185 lines
10 KiB
Plaintext
185 lines
10 KiB
Plaintext
---
|
|
description: Comprehensive guide to TensorFlow best practices, covering code organization, performance, testing, and security for robust and maintainable machine learning projects.
|
|
globs: *.py,*.tf,*.keras
|
|
---
|
|
- **Code Organization and Structure:**
|
|
- **Directory Structure:**
|
|
- Structure your project into logical directories. For example:
|
|
|
|
project_root/
|
|
├── data/
|
|
│ ├── raw/
|
|
│ └── processed/
|
|
├── models/
|
|
│ ├── training/
|
|
│ └── saved_models/
|
|
├── src/
|
|
│ ├── utils/
|
|
│ ├── layers/
|
|
│ ├── models/
|
|
│ ├── training/
|
|
│ └── evaluation/
|
|
├── notebooks/ #Jupyter notebooks for experimentation
|
|
├── tests/
|
|
├── configs/
|
|
└── README.md
|
|
|
|
|
|
- **File Naming Conventions:**
|
|
- Use descriptive and consistent names. For example:
|
|
- `model_name.py`
|
|
- `data_processing.py`
|
|
- `train.py`
|
|
- `evaluate.py`
|
|
- `layer_name.py`
|
|
|
|
- **Module Organization:**
|
|
- Break down code into reusable modules and functions.
|
|
- Use `tf.Module` and Keras layers to manage variables. This enables encapsulation and avoids global variable pollution.
|
|
- Import modules using explicit relative or absolute paths, such as `from src.models import MyModel`.
|
|
- Group related functionality into modules/packages.
|
|
|
|
- **Component Architecture:**
|
|
- Employ modular design principles.
|
|
- Keras `Layers` and `Models` promote a component-based architecture. Custom layers should inherit from `tf.keras.layers.Layer`. Custom models inherit from `tf.keras.Model`.
|
|
- Use dependency injection to decouple components and facilitate testing.
|
|
|
|
- **Code Splitting Strategies:**
|
|
- Refactor code into smaller, manageable modules.
|
|
- Separate data loading, preprocessing, model definition, training, and evaluation into distinct modules.
|
|
- Implement generator functions or `tf.data.Dataset` pipelines for large datasets to avoid loading all data into memory at once.
|
|
|
|
- **Common Patterns and Anti-patterns:**
|
|
- **Design Patterns:**
|
|
- **Strategy Pattern:** Use different strategies for optimization or regularization.
|
|
- **Factory Pattern:** Create model architectures dynamically based on configuration.
|
|
- **Observer Pattern:** Monitor training progress and trigger actions based on metrics.
|
|
|
|
- **Recommended Approaches:**
|
|
- Use Keras layers and models to manage variables. Keras handles the underlying TensorFlow operations.
|
|
- Leverage `tf.data.Dataset` for efficient data loading and preprocessing.
|
|
- Use `tf.function` to compile Python functions into TensorFlow graphs for improved performance.
|
|
|
|
- **Anti-patterns and Code Smells:**
|
|
- **God Classes:** Avoid monolithic classes that perform too many tasks. Break them into smaller, more focused classes or functions.
|
|
- **Copy-Pasted Code:** Refactor duplicated code into reusable functions or modules.
|
|
- **Magic Numbers:** Use named constants instead of hardcoded values.
|
|
- **Global Variables:** Minimize the use of global variables, especially for model parameters.
|
|
|
|
- **State Management:**
|
|
- Use Keras layers and models for managing model state (weights, biases).
|
|
- Use `tf.Variable` objects for persistent state that needs to be tracked during training.
|
|
- When creating a model subclass, define trainable weights as tf.Variable objects within the `build()` method.
|
|
- Consider using `tf.saved_model` to save and load the entire model state, including the computation graph and variable values.
|
|
|
|
- **Error Handling:**
|
|
- Use `tf.debugging.assert_*` functions to check tensor values during development and debugging.
|
|
- Implement try-except blocks to handle potential exceptions, such as `tf.errors.InvalidArgumentError` or `tf.errors.OutOfRangeError`.
|
|
- Log errors and warnings using `tf.compat.v1.logging` or the standard `logging` module.
|
|
- Ensure error messages are informative and actionable.
|
|
|
|
- **Performance Considerations:**
|
|
- **Optimization Techniques:**
|
|
- Use `tf.function` to compile Python functions into TensorFlow graphs for improved performance. Use autograph (automatic graph construction).
|
|
- Optimize data input pipelines using `tf.data.Dataset.prefetch` and `tf.data.Dataset.cache`.
|
|
- Experiment with different optimizers (e.g., Adam, SGD) and learning rates.
|
|
- Adjust the default learning rate for some `tf.keras.*` optimizers.
|
|
- Use mixed precision training with `tf.keras.mixed_precision.Policy` to reduce memory usage and improve performance on GPUs.
|
|
|
|
- **Memory Management:**
|
|
- Use `tf.data.Dataset` to stream data from disk instead of loading it all into memory.
|
|
- Release unnecessary tensors using `del` to free up memory.
|
|
- Use `tf.GradientTape` to compute gradients efficiently, and avoid keeping unnecessary tensors alive within the tape.
|
|
|
|
- **GPU Utilization:**
|
|
- Ensure that TensorFlow is using the GPU by checking `tf.config.list_physical_devices('GPU')`.
|
|
- Use larger batch sizes to maximize GPU utilization.
|
|
- Profile your code using TensorFlow Profiler to identify bottlenecks and optimize GPU usage.
|
|
|
|
- **Security Best Practices:**
|
|
- **Common Vulnerabilities:**
|
|
- **Untrusted Input:** Validate all user-provided input to prevent malicious code injection or data poisoning attacks.
|
|
- **Model Poisoning:** Protect against adversarial attacks that can manipulate the training data and degrade model performance.
|
|
- **Model Inversion:** Implement techniques to protect sensitive data from being extracted from the model.
|
|
|
|
- **Input Validation:**
|
|
- Sanitize and validate all input data to prevent SQL injection, cross-site scripting (XSS), and other security vulnerabilities.
|
|
- Use `tf.io.decode_image` to decode images safely and prevent potential vulnerabilities related to malformed image files.
|
|
- Input validation for image and text data is critical.
|
|
|
|
- **Data Protection:**
|
|
- Encrypt sensitive data at rest and in transit.
|
|
- Use differential privacy techniques to protect the privacy of training data.
|
|
- Regularly audit your code and infrastructure for security vulnerabilities.
|
|
|
|
- **Secure API Communication:**
|
|
- Use HTTPS to encrypt communication between the client and the server.
|
|
- Implement authentication and authorization mechanisms to restrict access to sensitive data and functionality.
|
|
|
|
- **Testing Approaches:**
|
|
- **Unit Testing:**
|
|
- Write unit tests for individual functions and classes using `unittest` or `pytest`.
|
|
- Use `tf.test.TestCase` for testing TensorFlow-specific code.
|
|
- Mock external dependencies to isolate the code being tested.
|
|
|
|
- **Integration Testing:**
|
|
- Test the integration of different modules and components.
|
|
- Verify that the data pipeline is working correctly.
|
|
- Ensure that the model is producing accurate predictions on real-world data.
|
|
|
|
- **End-to-End Testing:**
|
|
- Test the entire workflow from data loading to model deployment.
|
|
- Use tools like Selenium or Cypress to automate end-to-end tests.
|
|
- Test for performance and scalability.
|
|
|
|
- **Test Organization:**
|
|
- Organize tests into logical directories and modules.
|
|
- Use clear and descriptive test names.
|
|
- Follow the Arrange-Act-Assert pattern for writing tests.
|
|
|
|
- **Mocking and Stubbing:**
|
|
- Use mocking frameworks like `unittest.mock` or `pytest-mock` to replace external dependencies with mock objects.
|
|
- Use stubs to provide controlled responses from external dependencies.
|
|
|
|
- **Common Pitfalls and Gotchas:**
|
|
- **Version Compatibility:**
|
|
- Be aware of version-specific issues and compatibility concerns when upgrading TensorFlow versions.
|
|
- Use `tf.compat.v1` or `tf.compat.v2` to maintain compatibility with older versions of TensorFlow.
|
|
|
|
- **Eager Execution:**
|
|
- Understand the differences between eager execution and graph execution.
|
|
- Use `tf.function` to compile functions into graphs for improved performance in production.
|
|
|
|
- **Tensor Shapes and Data Types:**
|
|
- Pay attention to tensor shapes and data types to avoid errors.
|
|
- Use `tf.debugging.assert_shapes` and `tf.debugging.assert_type` to check tensor shapes and data types during development.
|
|
|
|
- **Variable Scope:**
|
|
- Be aware of variable scope when using `tf.Variable` objects.
|
|
- Use `tf.compat.v1.get_variable` to create or reuse variables within a specific scope.
|
|
|
|
- **Tooling and Environment:**
|
|
- **Recommended Development Tools:**
|
|
- Jupyter Notebooks or Google Colab for interactive development and experimentation.
|
|
- TensorBoard for visualizing training progress and model graphs.
|
|
- TensorFlow Profiler for identifying performance bottlenecks.
|
|
- Debuggers such as the Python Debugger (pdb) for stepping through code and inspecting variables.
|
|
|
|
- **Linting and Formatting:**
|
|
- Use linters like pylint or flake8 to enforce code style guidelines.
|
|
- Use formatters like black or autopep8 to automatically format your code.
|
|
|
|
- **Deployment Best Practices:**
|
|
- Use TensorFlow Serving to deploy models in production.
|
|
- Use Docker to containerize your application and ensure consistent deployments.
|
|
- Use a platform like Vertex AI for scalable model training and deployment.
|
|
|
|
- **CI/CD Integration:**
|
|
- Integrate your code with a continuous integration/continuous delivery (CI/CD) pipeline.
|
|
- Use tools like Jenkins, Travis CI, or CircleCI to automate testing and deployment.
|
|
|
|
- **References:**
|
|
- [TensorFlow Core](https://www.tensorflow.org/guide/effective_tf2)
|
|
- [TensorFlow testing best practices](https://www.tensorflow.org/community/contribute/tests)
|
|
- [Medium - 10 tips to improve your machine learning models with tensorflow](https://medium.com/decathlondigital/10-tips-to-improve-your-machine-learning-models-with-tensorflow-ba7c724761e2)
|
|
- [Quora - What are the best practices with TensorFlow](https://www.quora.com/What-are-the-best-practices-with-TensorFlow) |