refactor: streamline onboarding modal by removing unnecessary slide preparation logic and enhancing loading state management for improved user experience

.cursor/rules/elk-stack.mdc

@@ -0,0 +1,168 @@
+---
+description: This rule outlines best practices for developing and maintaining applications within the ELK (Elasticsearch, Logstash, Kibana) stack, including coding standards, configuration, performance, security, and testing.
+globs: *.conf, *.yml, *.json, *.log
+---
+## ELK Stack Best Practices
+
+This document provides comprehensive best practices for working with the ELK (Elasticsearch, Logstash, Kibana) stack, covering various aspects from code organization to security and testing.
+
+### 1. Code Organization and Structure
+
+- **Directory Structure Best Practices:**
+    - Organize configurations (Logstash pipelines, Kibana dashboards) in separate directories, such as `logstash/pipelines`, `kibana/dashboards`, and `elasticsearch/config`.
+    - Maintain a dedicated directory for custom scripts or plugins used within the ELK stack.
+    - Use a version control-friendly structure, where each configuration file is self-contained and easily traceable.
+
+- **File Naming Conventions:**
+    - Use descriptive and consistent naming conventions for configuration files, such as `application-name-pipeline.conf` for Logstash pipelines or `dashboard-name.json` for Kibana dashboards.
+    - Employ prefixes or suffixes to denote the purpose or environment of a configuration (e.g., `dev-`, `prod-`).
+
+- **Module Organization:**
+    - If using custom scripts or plugins, organize them into modules with clear interfaces and documentation.
+    - For Logstash pipelines, consider breaking down complex configurations into smaller, reusable modules using the `include` directive.
+
+- **Component Architecture:**
+    - Design a modular architecture where each component (e.g., data ingestion, processing, visualization) is loosely coupled and can be independently scaled or updated.
+    - Follow a layered approach, separating concerns such as data collection, transformation, storage, and presentation.
+
+- **Code Splitting Strategies:**
+    - For Logstash pipelines, split configurations into smaller files based on functionality (e.g., input, filter, output) to improve readability and maintainability.
+    - Use conditional logic (e.g., `if/else` statements) within Logstash pipelines to handle different data sources or processing requirements.
+
+### 2. Common Patterns and Anti-patterns
+
+- **Design Patterns:**
+    - **Pipeline Pattern:** Design data ingestion pipelines using Logstash or Beats to process and transform data before indexing it in Elasticsearch.
+    - **Dashboard Pattern:** Create reusable Kibana dashboards and visualizations to monitor key metrics and identify trends.
+    - **Alerting Pattern:** Implement alerting rules in Kibana or use external tools to trigger notifications based on specific events or thresholds.
+
+- **Recommended Approaches:**
+    - Use structured logging formats (e.g., JSON) to simplify data parsing and analysis.
+    - Implement data enrichment techniques (e.g., GeoIP lookup, user agent parsing) to add context to log data.
+    - Use Index Lifecycle Management (ILM) to automate index rotation, deletion, and optimization.
+
+- **Anti-patterns:**
+    - Avoid using overly complex Logstash pipelines that are difficult to understand and maintain.
+    - Do not store sensitive data (e.g., passwords, API keys) in plain text within configuration files.
+    - Avoid excessive use of wildcard queries in Elasticsearch, which can negatively impact performance.
+
+- **State Management:**
+    - For Logstash pipelines, use persistent queues to ensure data durability and prevent data loss during outages.
+    - Utilize Elasticsearch's snapshot and restore API to back up and restore data in case of disaster.
+
+- **Error Handling:**
+    - Implement error handling in Logstash pipelines to gracefully handle unexpected data formats or processing errors.
+    - Use dead-letter queues to capture failed events and allow for later reprocessing.
+    - Monitor the health of ELK stack components and set up alerts for critical errors.
+
+### 3. Performance Considerations
+
+- **Optimization Techniques:**
+    - Tune Elasticsearch settings such as shard allocation, refresh interval, and bulk indexing size to optimize indexing and search performance.
+    - Use appropriate data types and mappings in Elasticsearch to minimize storage space and improve query performance.
+    - Optimize Logstash pipelines by using efficient filters, reducing the number of operations, and avoiding unnecessary data transformations.
+
+- **Memory Management:**
+    - Configure JVM heap size for Elasticsearch and Logstash based on available memory and data volume.
+    - Monitor memory usage of ELK stack components and adjust settings as needed to prevent memory leaks or out-of-memory errors.
+
+- **Bundle Size Optimization:**
+    - N/A - ELK stack does not use bundles in the traditional web development sense, but optimize configurations and data structures.
+
+- **Lazy Loading Strategies:**
+    - N/A - ELK stack components load configurations on startup, but optimize data ingestion and indexing to improve overall performance.
+
+### 4. Security Best Practices
+
+- **Common Vulnerabilities:**
+    - Protect against Elasticsearch injection attacks by validating user inputs and sanitizing data before indexing it.
+    - Prevent unauthorized access to ELK stack components by enabling authentication and authorization.
+    - Secure communication between ELK stack components by enabling TLS encryption.
+
+- **Input Validation:**
+    - Validate data ingested into Logstash pipelines to prevent malicious or malformed data from being indexed in Elasticsearch.
+    - Sanitize user inputs in Kibana dashboards to prevent cross-site scripting (XSS) attacks.
+
+- **Authentication and Authorization:**
+    - Enable authentication in Elasticsearch and Kibana to restrict access to authorized users only.
+    - Use role-based access control (RBAC) to grant different levels of permissions to different users or groups.
+
+- **Data Protection:**
+    - Encrypt sensitive data at rest in Elasticsearch using encryption at rest features or external encryption tools.
+    - Implement data masking or anonymization techniques to protect sensitive data in Kibana dashboards.
+
+- **Secure API Communication:**
+    - Use HTTPS for all API communication with ELK stack components.
+    - Implement API rate limiting to prevent denial-of-service (DoS) attacks.
+
+### 5. Testing Approaches
+
+- **Unit Testing:**
+    - Write unit tests for custom scripts or plugins used within the ELK stack.
+    - Mock external dependencies to isolate components during testing.
+
+- **Integration Testing:**
+    - Test the integration between ELK stack components by simulating data ingestion, processing, and visualization scenarios.
+    - Verify that data is correctly indexed in Elasticsearch and displayed in Kibana dashboards.
+
+- **End-to-End Testing:**
+    - Perform end-to-end tests to ensure that the entire ELK stack is functioning correctly and meeting performance requirements.
+    - Simulate real-world scenarios to validate data flow and system behavior.
+
+- **Test Organization:**
+    - Organize tests into separate directories based on component or functionality.
+    - Use a consistent naming convention for test files and test cases.
+
+- **Mocking and Stubbing:**
+    - Use mocking frameworks to create mock objects for external dependencies during unit testing.
+    - Stub out API calls or database queries to isolate components during integration testing.
+
+### 6. Common Pitfalls and Gotchas
+
+- **Frequent Mistakes:**
+    - Not configuring sufficient resources (CPU, memory, disk space) for ELK stack components.
+    - Over-indexing data and creating unnecessarily large indices.
+    - Using inefficient queries or aggregations in Elasticsearch.
+
+- **Edge Cases:**
+    - Handling large volumes of data or high query loads.
+    - Dealing with complex data formats or unstructured log data.
+    - Recovering from node failures or data corruption.
+
+- **Version-Specific Issues:**
+    - Incompatibilities between different versions of ELK stack components.
+    - Deprecated features or APIs in newer versions of Elasticsearch, Logstash, and Kibana.
+
+- **Compatibility Concerns:**
+    - Ensuring compatibility between ELK stack components and other technologies in your environment.
+    - Addressing dependencies and conflicts between different plugins or libraries.
+
+- **Debugging Strategies:**
+    - Use Elasticsearch's explain API to analyze query performance and identify bottlenecks.
+    - Enable debug logging in Logstash pipelines to troubleshoot data processing issues.
+    - Monitor system logs and metrics to identify resource constraints or errors.
+
+### 7. Tooling and Environment
+
+- **Recommended Development Tools:**
+    - IDEs with support for YAML, JSON, and Groovy.
+    - Command-line tools for interacting with Elasticsearch, Logstash, and Kibana.
+    - Monitoring tools for tracking the health and performance of ELK stack components.
+
+- **Build Configuration:**
+    - Use configuration management tools (e.g., Ansible, Chef) to automate the deployment and configuration of ELK stack components.
+    - Store configuration files in version control to track changes and facilitate collaboration.
+
+- **Linting and Formatting:**
+    - Use linters and formatters to enforce coding standards and improve code quality.
+    - Configure IDEs to automatically format code on save.
+
+- **Deployment Best Practices:**
+    - Deploy ELK stack components on separate servers or virtual machines to isolate workloads and improve scalability.
+    - Use a load balancer to distribute traffic across multiple Elasticsearch nodes.
+
+- **CI/CD Integration:**
+    - Integrate ELK stack deployments into your CI/CD pipeline to automate testing and deployment processes.
+    - Use infrastructure-as-code (IaC) tools to provision and manage ELK stack environments.
+
+By following these best practices, you can build and maintain a robust, scalable, and secure ELK stack environment that meets your logging and analytics needs.