first commit
This commit is contained in:
@@ -0,0 +1,117 @@
|
||||
# Auto-Import Pattern
|
||||
|
||||
## Overview
|
||||
|
||||
The Auto-Import pattern automatically discovers and imports all submodules in a directory, ensuring all components are registered without manual imports.
|
||||
|
||||
## Structure
|
||||
|
||||
```python
|
||||
# In module __init__.py (e.g., data_module/dataset/__init__.py)
|
||||
import os
|
||||
from src.utils.helpers import import_modules
|
||||
|
||||
models_dir = os.path.dirname(__file__)
|
||||
import_modules(models_dir, "src.data_module.dataset")
|
||||
```
|
||||
|
||||
## Helper Function
|
||||
|
||||
```python
|
||||
# In src/utils/helpers.py
|
||||
import os
|
||||
import importlib
|
||||
import pkgutil
|
||||
from typing import List
|
||||
|
||||
def import_modules(models_dir: str, package_name: str) -> List[str]:
|
||||
"""
|
||||
Import all Python modules in a directory.
|
||||
|
||||
Args:
|
||||
models_dir: Directory path to scan
|
||||
package_name: Full package name for imports
|
||||
|
||||
Returns:
|
||||
List of imported module names
|
||||
"""
|
||||
imported = []
|
||||
for module_loader, name, ispkg in pkgutil.iter_modules([models_dir]):
|
||||
if not name.startswith('_'):
|
||||
full_name = f"{package_name}.{name}"
|
||||
importlib.import_module(full_name)
|
||||
imported.append(name)
|
||||
return imported
|
||||
```
|
||||
|
||||
## Benefits
|
||||
|
||||
- **Zero maintenance**: Adding new file = auto-registration
|
||||
- **No遗漏**: Cannot forget to import new component
|
||||
- **Consistent**: All components follow same discovery path
|
||||
- **Scalable**: Works for any number of submodules
|
||||
|
||||
## Implementation Details
|
||||
|
||||
1. Scan directory for `.py` files
|
||||
2. Skip files starting with `_` (private)
|
||||
3. Import each module using full package path
|
||||
4. Import triggers decorator registration
|
||||
|
||||
## Directory Structure Example
|
||||
|
||||
```
|
||||
dataset/
|
||||
├── __init__.py # Contains import_modules() call
|
||||
├── simple_dataset.py # Auto-imported, registers "simple"
|
||||
├── custom_dataset.py # Auto-imported, registers "custom"
|
||||
└── _private.py # NOT imported (starts with _)
|
||||
```
|
||||
|
||||
## Best Practices
|
||||
|
||||
- **Skip private files**: Files starting with `_` are not imported
|
||||
- **Full package paths**: Use dot-notation for correct imports
|
||||
- **Idempotent**: Safe to call multiple times
|
||||
- **Error handling**: Import errors propagate for debugging
|
||||
|
||||
## Common Patterns
|
||||
|
||||
### Conditional Import
|
||||
|
||||
```python
|
||||
def import_modules(models_dir: str, package_name: str, skip: List[str] = None):
|
||||
skip = skip or []
|
||||
for module_loader, name, ispkg in pkgutil.iter_modules([models_dir]):
|
||||
if name not in skip and not name.startswith('_'):
|
||||
importlib.import_module(f"{package_name}.{name}")
|
||||
```
|
||||
|
||||
### Recursive Import
|
||||
|
||||
```python
|
||||
def import_modules_recursive(models_dir: str, package_name: str):
|
||||
"""Import modules and subpackages recursively."""
|
||||
for importer, name, ispkg in pkgutil.walk_packages([models_dir], prefix=f"{package_name}."):
|
||||
if not name.split('.')[-1].startswith('_'):
|
||||
importlib.import_module(name)
|
||||
```
|
||||
|
||||
### Dry-Run Mode
|
||||
|
||||
```python
|
||||
def import_modules(models_dir: str, package_name: str, dry_run: bool = False):
|
||||
if dry_run:
|
||||
return [name for _, name, _ in pkgutil.iter_modules([models_dir])
|
||||
if not name.startswith('_')]
|
||||
# ... actual import logic
|
||||
```
|
||||
|
||||
## Integration with Registry
|
||||
|
||||
The auto-import pattern is typically used WITH registry pattern:
|
||||
|
||||
1. **Import time**: `import_modules()` imports all files
|
||||
2. **Decorator execution**: `@register_dataset()` runs
|
||||
3. **Factory population**: `DATASET_FACTORY` dict populated
|
||||
4. **Runtime**: `DatasetFactory()` looks up registered classes
|
||||
@@ -0,0 +1,234 @@
|
||||
# Code Style Guidelines
|
||||
|
||||
## Type Annotations
|
||||
|
||||
Always use type hints for function signatures and class attributes:
|
||||
|
||||
```python
|
||||
from typing import Dict, List, Optional, Tuple
|
||||
import torch
|
||||
|
||||
def __getitem__(self, i: int) -> Dict[str, torch.Tensor]:
|
||||
"""Get item by index."""
|
||||
return self.data[i]
|
||||
|
||||
def compute_metrics(predictions: torch.Tensor, labels: torch.Tensor) -> Dict[str, float]:
|
||||
"""Compute evaluation metrics."""
|
||||
pass
|
||||
|
||||
class MyModel(nn.Module):
|
||||
hidden_dim: int # Class attribute type hints
|
||||
output_dim: int
|
||||
|
||||
def __init__(self, cfg):
|
||||
self.hidden_dim: int = cfg.model.hidden_dim
|
||||
self.output_dim: int = cfg.model.output_dim
|
||||
```
|
||||
|
||||
## Import Order
|
||||
|
||||
Organize imports in three sections with blank lines between:
|
||||
|
||||
```python
|
||||
# 1. Standard library imports
|
||||
import os
|
||||
from typing import Dict, List, Optional
|
||||
from pathlib import Path
|
||||
|
||||
# 2. Third-party imports
|
||||
import torch
|
||||
import torch.nn as nn
|
||||
from torch.utils.data import Dataset
|
||||
import numpy as np
|
||||
|
||||
# 3. Local imports
|
||||
from src.data_module.dataset import register_dataset
|
||||
from src.utils.helpers import import_modules
|
||||
from src.model_module.brain_decoder import register_model
|
||||
```
|
||||
|
||||
## __init__.py Files
|
||||
|
||||
### Module __init__.py (with factory)
|
||||
|
||||
Contains factory/registry logic and auto-import:
|
||||
|
||||
```python
|
||||
# src/data_module/dataset/__init__.py
|
||||
import os
|
||||
from typing import Dict, Callable, TypeVar
|
||||
from src.utils.helpers import import_modules
|
||||
|
||||
T = TypeVar('T')
|
||||
|
||||
DATASET_FACTORY: Dict[str, type] = {}
|
||||
|
||||
def register_dataset(name: str) -> Callable[[T], T]:
|
||||
"""Decorator to register dataset classes."""
|
||||
def decorator(cls: T) -> T:
|
||||
DATASET_FACTORY[name] = cls
|
||||
return cls
|
||||
return decorator
|
||||
|
||||
def DatasetFactory(data_name: str):
|
||||
"""Create dataset instance by name."""
|
||||
dataset = DATASET_FACTORY.get(data_name, None)
|
||||
if dataset is None:
|
||||
dataset = DATASET_FACTORY.get('simple')
|
||||
return dataset
|
||||
|
||||
# Auto-import all submodules
|
||||
models_dir = os.path.dirname(__file__)
|
||||
import_modules(models_dir, "src.data_module.dataset")
|
||||
```
|
||||
|
||||
### Subpackage __init__.py (can be empty)
|
||||
|
||||
```python
|
||||
# src/data_module/augmentation/__init__.py
|
||||
# Empty file - just marks as package
|
||||
```
|
||||
|
||||
Or with exports:
|
||||
|
||||
```python
|
||||
# src/data_module/__init__.py
|
||||
from .dataset import DatasetFactory, register_dataset
|
||||
from .augmentation import AugmentationFactory
|
||||
```
|
||||
|
||||
## Naming Conventions
|
||||
|
||||
### Files
|
||||
|
||||
- **Modules**: `simple_dataset.py`, `custom_model.py`
|
||||
- **Pipelines**: `training.sh`, `inference.sh`
|
||||
- **Configs**: `config.yaml`, `brain_decoder.yaml`
|
||||
- **Utilities**: `get_optimizer.py`, `helpers.py`, `compute_metrics.py`
|
||||
|
||||
### Classes and Functions
|
||||
|
||||
```python
|
||||
# Classes: PascalCase
|
||||
class SimpleDataset(Dataset):
|
||||
pass
|
||||
|
||||
class MyCustomModel(nn.Module):
|
||||
pass
|
||||
|
||||
# Functions and variables: snake_case
|
||||
def compute_accuracy(predictions, labels):
|
||||
pass
|
||||
|
||||
def get_optimizer(cfg):
|
||||
pass
|
||||
|
||||
learning_rate = 0.001
|
||||
batch_size = 32
|
||||
```
|
||||
|
||||
### Constants
|
||||
|
||||
```python
|
||||
# Constants: UPPER_SNAKE_CASE
|
||||
DEFAULT_HIDDEN_DIM = 256
|
||||
MAX_EPOCHS = 100
|
||||
LEARNING_RATE = 0.001
|
||||
```
|
||||
|
||||
## Docstrings
|
||||
|
||||
Use Google-style docstrings:
|
||||
|
||||
```python
|
||||
def DatasetFactory(data_name: str) -> type:
|
||||
"""Create dataset class by name.
|
||||
|
||||
Args:
|
||||
data_name: Name of the dataset to create.
|
||||
|
||||
Returns:
|
||||
Dataset class if found, otherwise simple dataset.
|
||||
|
||||
Raises:
|
||||
ValueError: If no dataset is found and no default exists.
|
||||
"""
|
||||
pass
|
||||
```
|
||||
|
||||
## Configuration-Driven Classes
|
||||
|
||||
Model classes must be config-driven:
|
||||
|
||||
```python
|
||||
@register_model('MyModel')
|
||||
class MyModel(nn.Module):
|
||||
def __init__(self, cfg):
|
||||
"""Initialize model from config.
|
||||
|
||||
Args:
|
||||
cfg: Hydra config object with model attributes.
|
||||
"""
|
||||
super().__init__()
|
||||
self.cfg = cfg
|
||||
|
||||
# ALL parameters from cfg
|
||||
self.hidden_dim = cfg.model.hidden_dim
|
||||
self.output_dim = cfg.dataset.target_size[cfg.dataset.task]
|
||||
self.dropout = cfg.model.dropout
|
||||
|
||||
def forward(self, x, labels=None, **kwargs):
|
||||
"""Forward pass.
|
||||
|
||||
Args:
|
||||
x: Input tensor.
|
||||
labels: Ground truth labels (training mode).
|
||||
**kwargs: Additional arguments.
|
||||
|
||||
Returns:
|
||||
Dict with loss, labels, and logits.
|
||||
"""
|
||||
# Implementation
|
||||
return {"loss": loss, "labels": labels, "logits": logits}
|
||||
```
|
||||
|
||||
## Error Handling
|
||||
|
||||
```python
|
||||
def DatasetFactory(data_name: str) -> type:
|
||||
"""Create dataset class by name."""
|
||||
dataset = DATASET_FACTORY.get(data_name)
|
||||
if dataset is None:
|
||||
available = ', '.join(DATASET_FACTORY.keys())
|
||||
raise ValueError(
|
||||
f"Dataset '{data_name}' not found. "
|
||||
f"Available: {available}"
|
||||
)
|
||||
return dataset
|
||||
```
|
||||
|
||||
## Logging
|
||||
|
||||
```python
|
||||
import logging
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
@register_dataset('custom')
|
||||
class CustomDataset(Dataset):
|
||||
def __init__(self, cfg):
|
||||
self.cfg = cfg
|
||||
logger.info(f"Initializing {self.__class__.__name__}")
|
||||
logger.debug(f"Config: {cfg.dataset}")
|
||||
```
|
||||
|
||||
## Code Review Checklist
|
||||
|
||||
- [ ] All functions have type hints
|
||||
- [ ] Imports are correctly ordered
|
||||
- [ ] Classes use PascalCase, functions use snake_case
|
||||
- [ ] Docstrings follow Google style
|
||||
- [ ] Model classes are config-driven
|
||||
- [ ] Registration decorators are used
|
||||
- [ ] Error messages are informative
|
||||
- [ ] Logging is added for key operations
|
||||
@@ -0,0 +1,53 @@
|
||||
# Factory Pattern
|
||||
|
||||
## Overview
|
||||
|
||||
The Factory pattern allows dynamic creation of instances without specifying the exact class. Each module uses a factory to decouple creation from usage.
|
||||
|
||||
## Structure
|
||||
|
||||
```python
|
||||
# In module __init__.py (e.g., data_module/dataset/__init__.py)
|
||||
DATASET_FACTORY: Dict[str, type] = {}
|
||||
|
||||
def DatasetFactory(data_name: str):
|
||||
"""Create dataset instance by name."""
|
||||
dataset = DATASET_FACTORY.get(data_name, None)
|
||||
if dataset is None:
|
||||
# Fallback to default
|
||||
dataset = DATASET_FACTORY.get('simple')
|
||||
return dataset
|
||||
```
|
||||
|
||||
## Usage
|
||||
|
||||
```python
|
||||
# Consumer code doesn't need to know concrete class
|
||||
dataset = DatasetFactory(cfg.dataset.name)
|
||||
```
|
||||
|
||||
## Benefits
|
||||
|
||||
- **Loose coupling**: Consumer doesn't import concrete classes
|
||||
- **Extensibility**: Add new types without changing consumer code
|
||||
- **Fallback handling**: Graceful degradation for unknown types
|
||||
- **Centralized registry**: Single source of truth for available types
|
||||
|
||||
## Implementation Details
|
||||
|
||||
1. Define factory dict at module level
|
||||
2. Factory function handles lookup and fallback
|
||||
3. Return class (not instance) for deferred initialization
|
||||
4. None result triggers fallback to default implementation
|
||||
|
||||
## Common Patterns
|
||||
|
||||
```python
|
||||
# With config integration
|
||||
def DatasetFactory(cfg):
|
||||
data_name = cfg.dataset.name
|
||||
dataset_cls = DATASET_FACTORY.get(data_name)
|
||||
if dataset_cls is None:
|
||||
raise ValueError(f"Unknown dataset: {data_name}")
|
||||
return dataset_cls(cfg)
|
||||
```
|
||||
@@ -0,0 +1,102 @@
|
||||
# Registry Pattern
|
||||
|
||||
## Overview
|
||||
|
||||
The Registry pattern allows components to register themselves via decorators, enabling automatic discovery and centralized management of available types.
|
||||
|
||||
## Structure
|
||||
|
||||
```python
|
||||
# In module __init__.py (e.g., data_module/dataset/__init__.py)
|
||||
from typing import Dict, Callable, TypeVar
|
||||
|
||||
T = TypeVar('T')
|
||||
|
||||
DATASET_FACTORY: Dict[str, type] = {}
|
||||
|
||||
def register_dataset(name: str) -> Callable[[T], T]:
|
||||
"""Decorator to register dataset classes."""
|
||||
def decorator(cls: T) -> T:
|
||||
DATASET_FACTORY[name] = cls
|
||||
return cls
|
||||
return decorator
|
||||
```
|
||||
|
||||
## Usage
|
||||
|
||||
```python
|
||||
# In implementation file (e.g., simple_dataset.py)
|
||||
from data_module.dataset import register_dataset
|
||||
|
||||
@register_dataset("simple")
|
||||
class SimpleDataset(Dataset):
|
||||
def __init__(self, cfg):
|
||||
# Implementation
|
||||
pass
|
||||
```
|
||||
|
||||
## Benefits
|
||||
|
||||
- **Automatic registration**: Components register themselves on import
|
||||
- **Declarative**: Single decorator line replaces manual registration code
|
||||
- **Import-time discovery**: Auto-import pattern finds all implementations
|
||||
- **Type-safe**: Preserves original class type
|
||||
|
||||
## Implementation Details
|
||||
|
||||
1. Decorator returns the class unchanged (for immediate use)
|
||||
2. Side effect: adds class to factory dict
|
||||
3. Name parameter must be unique per module
|
||||
4. Registration happens at module import time
|
||||
|
||||
## Advanced Patterns
|
||||
|
||||
### Registration with Config
|
||||
|
||||
```python
|
||||
def register_model(name: str):
|
||||
def decorator(cls):
|
||||
MODEL_FACTORY[name] = cls
|
||||
# Add config validation
|
||||
cls._config_schema = getattr(cls, '_config_schema', {})
|
||||
return cls
|
||||
return decorator
|
||||
```
|
||||
|
||||
### Conditional Registration
|
||||
|
||||
```python
|
||||
def register_dataset(name: str, experimental: bool = False):
|
||||
def decorator(cls):
|
||||
if not experimental or cfg.enable_experimental:
|
||||
DATASET_FACTORY[name] = cls
|
||||
return cls
|
||||
return decorator
|
||||
```
|
||||
|
||||
### Multi-Registry
|
||||
|
||||
```python
|
||||
# Multiple registries in one module
|
||||
DATASET_FACTORY = {}
|
||||
AUGMENTATION_FACTORY = {}
|
||||
|
||||
def register_dataset(name: str):
|
||||
def decorator(cls):
|
||||
DATASET_FACTORY[name] = cls
|
||||
return cls
|
||||
return decorator
|
||||
|
||||
def register_augmentation(name: str):
|
||||
def decorator(fn):
|
||||
AUGMENTATION_FACTORY[name] = fn
|
||||
return fn
|
||||
return decorator
|
||||
```
|
||||
|
||||
## Best Practices
|
||||
|
||||
- **Unique names**: Use descriptive, unique registration names
|
||||
- **Documentation**: Document required parameters in class docstring
|
||||
- **Validation**: Validate config in `__init__`, not in decorator
|
||||
- **Consistency**: Use same naming convention across modules
|
||||
@@ -0,0 +1,150 @@
|
||||
# Detailed Directory Structure
|
||||
|
||||
This document provides a comprehensive breakdown of the ML project template directory structure.
|
||||
|
||||
## Root Level Files
|
||||
|
||||
| File | Purpose |
|
||||
|------|---------|
|
||||
| `README.md` | Project documentation, installation guide, usage examples |
|
||||
| `TODO.md` | Task tracking with weekly focus and daily tasks |
|
||||
| `.gitignore` | Git ignore patterns for Python, Jupyter, IDEs, logs, cache |
|
||||
| `pyproject.toml` | Project configuration for build system and dependencies |
|
||||
| `uv.lock` | Locked dependency versions for reproducibility |
|
||||
|
||||
## run/ - Execution Layer
|
||||
|
||||
### pipeline/
|
||||
|
||||
Main workflow scripts organized by stage:
|
||||
|
||||
| Directory | Purpose |
|
||||
|-----------|---------|
|
||||
| `training/` | Training execution scripts (training.sh, inference.sh) |
|
||||
| `prepare_data/` | Data preparation and preprocessing pipelines |
|
||||
| `analysis/` | Evaluation and analysis workflows |
|
||||
|
||||
### conf/
|
||||
|
||||
Hydra configuration files organized by module:
|
||||
|
||||
| Directory | Purpose |
|
||||
|-----------|---------|
|
||||
| `training/` | Training hyperparameters, model configs, optimizer settings |
|
||||
| `dataset/` | Dataset configurations, data paths, preprocessing options |
|
||||
| `model/` | Model architecture configurations |
|
||||
| `prepare_data/` | Data preparation parameters |
|
||||
| `analysis/` | Analysis and evaluation configurations |
|
||||
| `dir/` | Directory path configurations |
|
||||
| `analysis/` | Analysis-specific settings |
|
||||
|
||||
## src/ - Source Code Layer
|
||||
|
||||
### data_module/ - Data Processing Module
|
||||
|
||||
```
|
||||
data_module/
|
||||
├── __init__.py # Module exports
|
||||
├── utils.py # Data-specific utility functions
|
||||
├── dataset/ # Dataset implementations
|
||||
│ ├── __init__.py # Dataset factory and registry
|
||||
│ └── simple_dataset.py # Simple dataset example
|
||||
├── augmentation/ # Data augmentation methods
|
||||
│ ├── __init__.py
|
||||
│ ├── mixup.py # Mixup augmentation
|
||||
│ ├── random_shift.py # Random shifting
|
||||
│ ├── channel_mask.py # Channel masking
|
||||
│ ├── time_masking.py # Time masking
|
||||
│ └── add_noise.py # Noise injection
|
||||
├── collate_fn/ # Batch collation functions
|
||||
│ ├── __init__.py
|
||||
│ └── simple_collate_fn.py
|
||||
├── compute_metrics/ # Metrics computation
|
||||
│ ├── __init__.py
|
||||
│ └── simple_compute_metrics.py
|
||||
├── prepare_data/ # Data preparation logic
|
||||
│ ├── __init__.py
|
||||
│ ├── prepare_data.py
|
||||
│ └── generate_yaml.py
|
||||
└── data_func/ # Data utility functions
|
||||
├── __init__.py
|
||||
└── simple_data_func.py
|
||||
```
|
||||
|
||||
### model_module/ - Model Module
|
||||
|
||||
```
|
||||
model_module/
|
||||
├── __init__.py # Module exports
|
||||
└── model/ # Model implementations
|
||||
└── [model files]
|
||||
```
|
||||
|
||||
### trainer_module/ - Training Module
|
||||
|
||||
Contains training loop logic, validation, and checkpoint management.
|
||||
|
||||
### analysis_module/ - Analysis Module
|
||||
|
||||
Contains evaluation, visualization, and result analysis code.
|
||||
|
||||
### llm/ - LLM Module
|
||||
|
||||
LLM-related code and integrations.
|
||||
|
||||
### utils/ - Shared Utilities
|
||||
|
||||
```
|
||||
utils/
|
||||
├── __init__.py
|
||||
├── helpers.py # Helper functions (import_modules, etc.)
|
||||
├── logging.py # Logging configuration
|
||||
├── get_optimizer.py # Optimizer factory
|
||||
├── get_scheduler.py # Learning rate scheduler factory
|
||||
├── get_callback.py # Training callbacks
|
||||
├── get_activation.py # Activation functions
|
||||
└── get_checkpoint_aggregation.py # Checkpoint handling
|
||||
```
|
||||
|
||||
## data/ - Data Layer
|
||||
|
||||
Following the Cookiecutter Data Science standard:
|
||||
|
||||
| Directory | Purpose |
|
||||
|-----------|---------|
|
||||
| `raw/` | Original, immutable data dump |
|
||||
| `processed/` | Cleaned, transformed data ready for use |
|
||||
| `external/` | Data from third-party sources |
|
||||
|
||||
## outputs/ - Output Layer
|
||||
|
||||
| Directory | Purpose |
|
||||
|-----------|---------|
|
||||
| `logs/` | Training logs, tensorboard logs |
|
||||
| `checkpoints/` | Model checkpoints for resuming training |
|
||||
| `tables/` | Result tables, CSV outputs |
|
||||
| `figures/` | Plots, visualizations, figures |
|
||||
|
||||
## Module Interaction Flow
|
||||
|
||||
```
|
||||
run/pipeline/ -> src/trainer_module/ -> src/model_module/
|
||||
src/data_module/ src/utils/
|
||||
src/utils/
|
||||
|
||||
run/conf/ -> Hydra config loader -> All modules
|
||||
```
|
||||
|
||||
## File Naming Conventions
|
||||
|
||||
- **Modules**: `simple_dataset.py`, `custom_model.py`
|
||||
- **Pipelines**: `training.sh`, `inference.sh`
|
||||
- **Configs**: `config.yaml`, dataset-specific names
|
||||
- **Utilities**: Descriptive names (`get_optimizer.py`, `helpers.py`)
|
||||
|
||||
## Python Package Structure
|
||||
|
||||
Each module is a proper Python package:
|
||||
- Has `__init__.py` with factory/registry logic
|
||||
- Can be imported as `from src.module import Component`
|
||||
- Subpackages are automatically discovered via `import_modules()`
|
||||
Reference in New Issue
Block a user