261 lines
12 KiB
Plaintext
261 lines
12 KiB
Plaintext
---
|
|
description: Enforces best practices for SQLAlchemy, covering code organization, performance, security, testing, and common pitfalls to ensure maintainable, efficient, and secure database interactions.
|
|
globs: *.py
|
|
---
|
|
# SQLAlchemy Best Practices and Coding Standards
|
|
|
|
This document outlines the best practices and coding standards for using SQLAlchemy in Python projects. Following these guidelines will help you write maintainable, efficient, and secure code.
|
|
|
|
## 1. Code Organization and Structure
|
|
|
|
### 1.1 Directory Structure
|
|
|
|
A well-organized directory structure improves code readability and maintainability. Here's a recommended structure for SQLAlchemy-based projects:
|
|
|
|
|
|
project_name/
|
|
├── app/
|
|
│ ├── __init__.py
|
|
│ ├── models/
|
|
│ │ ├── __init__.py
|
|
│ │ ├── user.py
|
|
│ │ ├── product.py
|
|
│ │ └── ...
|
|
│ ├── database.py # SQLAlchemy engine and session setup
|
|
│ ├── routes/
|
|
│ │ ├── __init__.py
|
|
│ │ ├── user_routes.py
|
|
│ │ ├── product_routes.py
|
|
│ │ └── ...
|
|
│ ├── schemas/
|
|
│ │ ├── __init__.py
|
|
│ │ ├── user_schema.py
|
|
│ │ ├── product_schema.py
|
|
│ │ └── ...
|
|
│ ├── utils.py
|
|
│ └── main.py # Entry point for the application
|
|
├── tests/
|
|
│ ├── __init__.py
|
|
│ ├── conftest.py # Fixtures for testing
|
|
│ ├── test_models.py
|
|
│ ├── test_routes.py
|
|
│ └── ...
|
|
├── migrations/
|
|
│ ├── versions/
|
|
│ │ ├── ... (Alembic migration scripts)
|
|
│ ├── alembic.ini
|
|
│ └── env.py
|
|
├── .env # Environment variables
|
|
├── requirements.txt
|
|
├── pyproject.toml # Define project dependencies
|
|
└── README.md
|
|
|
|
|
|
### 1.2 File Naming Conventions
|
|
|
|
* **Models:** Use descriptive names for model files (e.g., `user.py`, `product.py`).
|
|
* **Schemas:** Use `_schema.py` suffix for schema files (e.g., `user_schema.py`).
|
|
* **Routes/Controllers:** Use `_routes.py` or `_controllers.py` suffix (e.g., `user_routes.py`).
|
|
* **Database:** A central `database.py` or `db.py` file is standard.
|
|
* **Migrations:** Alembic manages migration script names automatically.
|
|
|
|
### 1.3 Module Organization
|
|
|
|
* **Models:** Group related models into separate modules for clarity.
|
|
* **Schemas:** Define schemas in separate modules for serialization/deserialization.
|
|
* **Routes/Controllers:** Organize API endpoints into logical modules.
|
|
|
|
### 1.4 Component Architecture
|
|
|
|
* **Data Access Layer (DAL):** Abstract database interactions into a separate layer using the Repository Pattern to decouple the application logic from the database implementation.
|
|
* **Service Layer:** Implement business logic in a service layer that utilizes the DAL.
|
|
* **Presentation Layer:** (Routes/Controllers) Handle request processing and response generation.
|
|
|
|
### 1.5 Code Splitting
|
|
|
|
* **Model Definition:** Split large models into smaller, manageable classes.
|
|
* **Query Logic:** Move complex query logic into reusable functions or methods.
|
|
* **Configuration:** Externalize configuration settings using environment variables.
|
|
|
|
## 2. Common Patterns and Anti-patterns
|
|
|
|
### 2.1 Design Patterns
|
|
|
|
* **Repository Pattern:** Centralizes data access logic, improving testability and maintainability. Example:
|
|
|
|
python
|
|
class UserRepository:
|
|
def __init__(self, session: Session):
|
|
self.session = session
|
|
|
|
def get_user_by_id(self, user_id: int) -> User | None:
|
|
return self.session.get(User, user_id)
|
|
|
|
* **Unit of Work Pattern:** Tracks changes to multiple entities and commits them as a single transaction, ensuring data consistency.
|
|
* **Data Mapper Pattern:** Provides a layer of indirection between the database and domain objects, allowing for independent evolution.
|
|
|
|
### 2.2 Recommended Approaches
|
|
|
|
* **Declarative Base:** Use `declarative_base()` to define models.
|
|
* **Context Managers:** Use context managers for session management to ensure sessions are properly closed.
|
|
* **Parameterized Queries:** Always use parameterized queries to prevent SQL injection.
|
|
* **Eager Loading:** Use `joinedload()`, `subqueryload()`, or `selectinload()` to optimize query performance and avoid the N+1 problem.
|
|
* **Alembic:** Use Alembic for database migrations.
|
|
|
|
### 2.3 Anti-patterns and Code Smells
|
|
|
|
* **Raw SQL:** Avoid writing raw SQL queries whenever possible; leverage SQLAlchemy's ORM or Core features.
|
|
* **Global Sessions:** Avoid using global session objects; create sessions within request/transaction scopes.
|
|
* **Long-Lived Sessions:** Keep sessions short-lived to prevent stale data and concurrency issues.
|
|
* **Over-Fetching:** Avoid retrieving more data than necessary; use targeted queries.
|
|
* **N+1 Query Problem:** Identify and address the N+1 query problem using eager loading.
|
|
|
|
### 2.4 State Management
|
|
|
|
* **Session Scope:** Manage the SQLAlchemy session within the scope of a request or transaction.
|
|
* **Thread Safety:** Ensure thread safety when using SQLAlchemy in multi-threaded environments.
|
|
* **Asynchronous Sessions:** Use asynchronous sessions for non-blocking database operations in asynchronous applications.
|
|
|
|
### 2.5 Error Handling
|
|
|
|
* **Exception Handling:** Implement robust exception handling to catch database errors and prevent application crashes.
|
|
* **Rollbacks:** Use `session.rollback()` to revert changes in case of errors.
|
|
* **Logging:** Log database errors and queries for debugging and monitoring purposes.
|
|
|
|
## 3. Performance Considerations
|
|
|
|
### 3.1 Optimization Techniques
|
|
|
|
* **Indexing:** Add indexes to frequently queried columns to improve query performance.
|
|
* **Query Optimization:** Analyze query execution plans and optimize queries accordingly.
|
|
* **Connection Pooling:** Configure connection pooling to reuse database connections and reduce overhead.
|
|
* **Caching:** Implement caching strategies to reduce database load.
|
|
* **Batch Operations:** Use batch operations for bulk inserts, updates, and deletes.
|
|
|
|
### 3.2 Memory Management
|
|
|
|
* **Session Management:** Close sessions promptly to release resources.
|
|
* **Result Set Size:** Limit the size of result sets to prevent memory exhaustion.
|
|
* **Streaming Results:** Use streaming results for large datasets to reduce memory usage.
|
|
|
|
### 3.3 Lazy Loading Strategies
|
|
|
|
* **Joined Loading**: Load related entities in a single query using a JOIN.
|
|
* **Subquery Loading**: Load related entities using a subquery, suitable for complex relationships.
|
|
* **Selectin Loading**: Load related entities using a separate SELECT IN query, efficient for collections.
|
|
|
|
## 4. Security Best Practices
|
|
|
|
### 4.1 Common Vulnerabilities
|
|
|
|
* **SQL Injection:** Prevent SQL injection by using parameterized queries and avoiding string concatenation.
|
|
* **Data Exposure:** Protect sensitive data by encrypting it at rest and in transit.
|
|
* **Authentication Bypass:** Implement robust authentication and authorization mechanisms to prevent unauthorized access.
|
|
|
|
### 4.2 Input Validation
|
|
|
|
* **Schema Validation:** Use schemas to validate input data and ensure it conforms to the expected format.
|
|
* **Sanitization:** Sanitize input data to remove malicious characters and prevent cross-site scripting (XSS) attacks.
|
|
|
|
### 4.3 Authentication and Authorization
|
|
|
|
* **Authentication:** Use secure authentication protocols such as OAuth 2.0 or JWT (JSON Web Tokens).
|
|
* **Authorization:** Implement role-based access control (RBAC) or attribute-based access control (ABAC) to restrict access to resources.
|
|
|
|
### 4.4 Data Protection
|
|
|
|
* **Encryption:** Encrypt sensitive data at rest and in transit using strong encryption algorithms.
|
|
* **Hashing:** Hash passwords and other sensitive data using strong hashing algorithms.
|
|
* **Data Masking:** Mask sensitive data in non-production environments to prevent data breaches.
|
|
|
|
### 4.5 Secure API Communication
|
|
|
|
* **HTTPS:** Use HTTPS to encrypt communication between the client and the server.
|
|
* **API Keys:** Use API keys to authenticate API requests.
|
|
* **Rate Limiting:** Implement rate limiting to prevent denial-of-service (DoS) attacks.
|
|
|
|
## 5. Testing Approaches
|
|
|
|
### 5.1 Unit Testing
|
|
|
|
* **Model Testing:** Test model methods and properties.
|
|
* **Repository Testing:** Test repository methods in isolation.
|
|
* **Service Testing:** Test service layer logic.
|
|
|
|
### 5.2 Integration Testing
|
|
|
|
* **Database Integration:** Test database interactions and ensure data integrity.
|
|
* **API Integration:** Test API endpoints and ensure they function correctly.
|
|
|
|
### 5.3 End-to-End Testing
|
|
|
|
* **Full Application Testing:** Test the entire application workflow to ensure all components work together seamlessly.
|
|
|
|
### 5.4 Test Organization
|
|
|
|
* **Test Directory:** Organize tests into a separate `tests` directory.
|
|
* **Test Modules:** Create separate test modules for each component.
|
|
* **Test Fixtures:** Use test fixtures to set up test data and dependencies.
|
|
|
|
### 5.5 Mocking and Stubbing
|
|
|
|
* **Mocking Databases**: Use `unittest.mock` or `pytest-mock` to mock the SQLAlchemy engine and session during testing.
|
|
* **Patching External Dependencies**: Patch external dependencies to isolate the component under test.
|
|
|
|
## 6. Common Pitfalls and Gotchas
|
|
|
|
### 6.1 Frequent Mistakes
|
|
|
|
* **Forgetting to Commit:** Always commit changes to the database after making modifications.
|
|
* **Incorrect Relationship Configuration:** Ensure relationships are configured correctly to avoid data integrity issues.
|
|
* **Not Handling Exceptions:** Always handle exceptions to prevent application crashes.
|
|
* **Lack of Query Optimization:** Neglecting to optimize queries can lead to performance bottlenecks.
|
|
|
|
### 6.2 Edge Cases
|
|
|
|
* **Concurrency Issues:** Be aware of concurrency issues when multiple users access the database simultaneously.
|
|
* **Data Type Mismatches:** Ensure data types in the application and the database are compatible.
|
|
* **Large Result Sets:** Handle large result sets efficiently to avoid memory issues.
|
|
|
|
### 6.3 Version-Specific Issues
|
|
|
|
* **API Changes:** Be aware of API changes between different SQLAlchemy versions.
|
|
* **Compatibility Issues:** Ensure compatibility between SQLAlchemy and other libraries.
|
|
|
|
### 6.4 Debugging Strategies
|
|
|
|
* **Logging:** Use logging to track database queries and errors.
|
|
* **Debugging Tools:** Use debugging tools to step through code and inspect variables.
|
|
* **Query Analysis:** Analyze query execution plans to identify performance bottlenecks.
|
|
|
|
## 7. Tooling and Environment
|
|
|
|
### 7.1 Recommended Development Tools
|
|
|
|
* **IDE:** Use a good IDE such as VS Code, PyCharm, or Spyder.
|
|
* **Database Client:** Use a database client such as pgAdmin, Dbeaver, or MySQL Workbench.
|
|
* **SQLAlchemy Profiler:** Use an SQLAlchemy profiler to analyze query performance.
|
|
|
|
### 7.2 Build Configuration
|
|
|
|
* **Dependencies:** Use `requirements.txt` or `pyproject.toml` to manage dependencies.
|
|
* **Environment Variables:** Use environment variables to configure the application.
|
|
|
|
### 7.3 Linting and Formatting
|
|
|
|
* **Linting:** Use linters such as pylint or flake8 to enforce code style.
|
|
* **Formatting:** Use formatters such as black or autopep8 to automatically format code.
|
|
|
|
### 7.4 Deployment Best Practices
|
|
|
|
* **Database Configuration:** Configure the database connection settings correctly.
|
|
* **Security Hardening:** Harden the server and database to prevent security breaches.
|
|
* **Monitoring:** Implement monitoring to track application performance and errors.
|
|
|
|
### 7.5 CI/CD Integration
|
|
|
|
* **Automated Testing:** Run automated tests during the CI/CD pipeline.
|
|
* **Database Migrations:** Apply database migrations during deployment.
|
|
* **Rollbacks:** Implement rollbacks in case of deployment failures.
|
|
|
|
By adhering to these best practices, you can build robust, scalable, and maintainable applications with SQLAlchemy. Remember to adapt these guidelines to your specific project requirements and context. |