Joseph.Rawlings/Virtnet.bond
Private
Public Access
1
0
Fork 0
Code Activity

refactor: streamline onboarding modal by removing unnecessary slide preparation logic and enhancing loading state management for improved user experience

Browse Source
This commit is contained in:
Joseph.Rawlings
2025-12-04 00:11:26 -05:00
parent a6c8e14db0
commit 5cd9a52668
242 changed files with 59643 additions and 61 deletions
Download Patch File Download Diff File Expand all files Collapse all files

287
.cursor/rules/xgboost.mdc Normal file
View File

@@ -0,0 +1,287 @@
---
description: This rule provides best practices for developing with XGBoost, covering code organization, performance, security, testing, and common pitfalls.
globs: *.py,*.ipynb
---
# XGBoost Best Practices
This document outlines best practices for developing with XGBoost, focusing on code organization, common patterns, performance, security, testing, common pitfalls, and tooling.
## Library Information:
- Name: xgboost
- Tags: ai, ml, machine-learning, python, gradient-boosting
## 1. Code Organization and Structure
### 1.1 Directory Structure Best Practices
Adopt a clear and maintainable directory structure. Here's a recommended structure:
project_root/
├── data/
│ ├── raw/
│ ├── processed/
│ └── external/
├── models/
│ ├── trained_models/
│ └── evaluation_metrics/
├── src/
│ ├── features/
│ │ └── build_features.py # Data transformation and feature engineering
│ ├── models/
│ │ ├── train_model.py # Model training script
│ │ ├── predict_model.py # Model prediction script
│ │ └── xgboost_model.py # Definition of XGBoost model and related functions
│ ├── visualization/
│ │ └── visualize.py # Visualization tools
│ ├── utils/
│ │ └── helpers.py # Helper functions
│ └── __init__.py
├── notebooks/
│ └── exploratory_data_analysis.ipynb # EDA notebooks
├── tests/
│ ├── features/
│ ├── models/
│ └── utils/
├── .gitignore
├── README.md
├── requirements.txt
└── config.yaml # Configuration file
### 1.2 File Naming Conventions
- Use descriptive and consistent file names.
- Python scripts: `snake_case.py` (e.g., `train_model.py`, `build_features.py`).
- Configuration files: `config.yaml` or `config.json`.
- Data files: `descriptive_name.csv` or `descriptive_name.parquet`.
- Model files: `model_name.pkl` or `model_name.joblib`.
### 1.3 Module Organization
- Group related functions and classes into modules.
- Use `__init__.py` files to make directories importable as packages.
- Example `src/models/xgboost_model.py`:
python
import xgboost as xgb
class XGBoostModel:
def __init__(self, params=None):
self.params = params or {}
self.model = None
def train(self, X, y):
self.model = xgb.XGBRegressor(**self.params)
self.model.fit(X, y)
def predict(self, X):
return self.model.predict(X)
def evaluate(self, X, y):
# Evaluation metrics
pass
### 1.4 Component Architecture
- **Data Ingestion Layer**: Responsible for reading and validating input data.
- **Feature Engineering Layer**: Transforms raw data into features suitable for XGBoost.
- **Model Training Layer**: Trains the XGBoost model using the prepared features.
- **Prediction Layer**: Loads the trained model and makes predictions on new data.
- **Evaluation Layer**: Evaluates the model's performance using appropriate metrics.
### 1.5 Code Splitting
- Break down large scripts into smaller, reusable functions and classes.
- Use separate modules for data loading, preprocessing, model training, and evaluation.
- Leverage configuration files for managing parameters and settings.
## 2. Common Patterns and Anti-patterns
### 2.1 Design Patterns
- **Factory Pattern**: To create different types of XGBoost models based on configuration.
- **Strategy Pattern**: To implement different feature engineering techniques or evaluation metrics.
- **Observer Pattern**: To log training progress or monitor performance during training.
### 2.2 Recommended Approaches for Common Tasks
- **Hyperparameter Tuning**: Use `GridSearchCV`, `RandomizedSearchCV`, or Bayesian optimization techniques like `Hyperopt` or `Optuna` to find optimal hyperparameters.
- **Cross-Validation**: Use `KFold` or `StratifiedKFold` to evaluate model performance robustly.
- **Feature Importance**: Use `model.feature_importances_` or `xgb.plot_importance` to understand feature importance and perform feature selection.
- **Early Stopping**: Monitor validation performance during training and stop when performance plateaus to prevent overfitting. Use `early_stopping_rounds` parameter in `xgb.train` or `model.fit`.
### 2.3 Anti-patterns and Code Smells
- **Hardcoding parameters**: Avoid hardcoding hyperparameters or file paths directly in the code. Use configuration files or command-line arguments instead.
- **Ignoring feature scaling**: XGBoost is sensitive to feature scaling, so always normalize or standardize your features.
- **Overfitting**: Be cautious of overfitting, especially with complex models. Use regularization, early stopping, and cross-validation to mitigate this.
- **Ignoring data leakage**: Ensure that your training data is not contaminated with information from the test data.
- **Not tracking experiments**: Use experiment tracking tools like Neptune.ai, MLflow, or Weights & Biases to manage model versions, hyperparameters, and results effectively.
### 2.4 State Management
- Use classes to encapsulate model state and provide methods for training, prediction, and evaluation.
- Store trained models in a serialized format (e.g., `pickle`, `joblib`) for later use.
- Manage configuration parameters using a dedicated configuration file or object.
### 2.5 Error Handling
- Use `try-except` blocks to handle potential exceptions, such as file not found or invalid data.
- Log errors and warnings using the `logging` module.
- Provide informative error messages to help with debugging.
## 3. Performance Considerations
### 3.1 Optimization Techniques
- **GPU Acceleration**: Use GPU training to significantly speed up model training.
- **Parallel Processing**: XGBoost supports parallel processing, so utilize multiple CPU cores to speed up training.
- **Column Subsampling**: Reduce the number of features used in each tree to improve performance.
- **Row Subsampling**: Reduce the number of samples used in each boosting round to improve performance.
- **Data Types**: Use appropriate data types for your data (e.g., `float32` instead of `float64`) to reduce memory usage and improve performance.
### 3.2 Memory Management
- **Large Datasets**: For large datasets, consider using out-of-core training or distributed computing frameworks like Dask or Spark.
- **Feature Reduction**: Reduce the number of features used in the model to reduce memory usage.
- **Data Sampling**: Use sampling techniques to reduce the size of the training dataset.
### 3.3 Lazy Loading
- Implement lazy loading for large datasets or models to avoid loading them into memory until they are needed.
- Use generators or iterators to process data in chunks.
## 4. Security Best Practices
### 4.1 Common Vulnerabilities and Prevention
- **Model Poisoning**: Ensure your training data is from trusted sources to prevent malicious data from influencing the model.
- **Input Injection**: Sanitize user inputs before feeding them to the model to prevent injection attacks.
- **Model Extraction**: Protect your trained models from unauthorized access to prevent model extraction attacks.
### 4.2 Input Validation
- Validate input data to ensure it conforms to expected formats and ranges.
- Use schema validation libraries to enforce data quality.
- Sanitize user inputs to prevent injection attacks.
### 4.3 Authentication and Authorization
- Implement authentication and authorization mechanisms to restrict access to sensitive data and models.
- Use role-based access control (RBAC) to manage user permissions.
### 4.4 Data Protection
- Encrypt sensitive data at rest and in transit.
- Use data masking or anonymization techniques to protect personally identifiable information (PII).
- Comply with relevant data privacy regulations (e.g., GDPR, CCPA).
### 4.5 Secure API Communication
- Use HTTPS for secure communication between clients and servers.
- Implement API rate limiting to prevent denial-of-service attacks.
- Use API keys or tokens for authentication.
## 5. Testing Approaches
### 5.1 Unit Testing
- Test individual functions and classes in isolation.
- Use mocking and stubbing to isolate components and simulate dependencies.
- Write test cases for various scenarios, including edge cases and error conditions.
### 5.2 Integration Testing
- Test the interaction between different components or modules.
- Verify that data flows correctly between modules.
- Test the integration of XGBoost with other libraries or frameworks.
### 5.3 End-to-End Testing
- Test the entire application from end to end.
- Simulate user interactions and verify that the application behaves as expected.
- Test the deployment pipeline to ensure that the application can be deployed successfully.
### 5.4 Test Organization
- Organize tests into separate directories based on the module or component being tested.
- Use a consistent naming convention for test files and test functions.
- Keep tests concise and easy to understand.
### 5.5 Mocking and Stubbing
- Use mocking and stubbing to isolate components and simulate dependencies.
- Use mocking libraries like `unittest.mock` or `pytest-mock`.
- Create mock objects that mimic the behavior of real dependencies.
## 6. Common Pitfalls and Gotchas
### 6.1 Frequent Mistakes
- **Incorrect data types**: Ensure data types are appropriate (e.g. numerical features are not strings).
- **Not handling missing data**: XGBoost can handle missing values, but ensure you understand the implications and consider imputation.
- **Ignoring class imbalance**: Use `scale_pos_weight` or other techniques to address class imbalance in classification problems.
- **Not setting `eval_metric`**: Always define an evaluation metric to monitor training progress, especially when using early stopping.
### 6.2 Edge Cases
- **Rare categories**: Handle rare categories in categorical features appropriately, possibly by grouping them into a single category.
- **Outliers**: Consider the impact of outliers on the model and apply appropriate outlier detection and removal techniques.
- **Data drift**: Monitor the model's performance over time and retrain the model if data drift occurs.
### 6.3 Version-Specific Issues
- **API changes**: Be aware of API changes between XGBoost versions and update your code accordingly.
- **Compatibility issues**: Check for compatibility issues with other libraries or frameworks when upgrading XGBoost.
### 6.4 Compatibility Concerns
- Ensure compatibility between XGBoost and other libraries, such as scikit-learn, pandas, and NumPy.
- Use compatible versions of these libraries to avoid conflicts.
### 6.5 Debugging Strategies
- Use logging to track the flow of execution and identify errors.
- Use a debugger to step through the code and inspect variables.
- Use profiling tools to identify performance bottlenecks.
## 7. Tooling and Environment
### 7.1 Recommended Development Tools
- **IDE**: VS Code, PyCharm, Jupyter Notebooks.
- **Experiment Tracking**: MLflow, Neptune.ai, Weights & Biases.
- **Data Visualization**: Matplotlib, Seaborn, Plotly.
- **Profiling**: cProfile, memory_profiler.
- **Debugging**: pdb (Python Debugger).
### 7.2 Build Configuration
- Use a `requirements.txt` file to manage dependencies.
- Use a virtual environment to isolate project dependencies.
- Use a `setup.py` or `pyproject.toml` file for packaging and distribution.
- Specify versions for all dependencies to ensure reproducibility
### 7.3 Linting and Formatting
- Use a linter like `flake8` or `pylint` to enforce code style and detect errors.
- Use a formatter like `black` or `autopep8` to automatically format your code.
### 7.4 Deployment
- **Model Serialization**: Serialize the model using `pickle` or `joblib` for deployment.
- **Containerization**: Containerize the application using Docker for easy deployment and scalability.
- **Serving**: Serve the model using a web framework like Flask or FastAPI.
- **Cloud Platforms**: Deploy the application to a cloud platform like AWS, Azure, or Google Cloud.
### 7.5 CI/CD Integration
- Use a CI/CD pipeline to automate the build, test, and deployment process.
- Integrate with version control systems like Git.
- Use testing frameworks like pytest or unittest to run tests automatically.
- Automate deployment to staging and production environments.
By following these best practices, you can develop robust, maintainable, and high-performing XGBoost applications.