# Dynamic Hooks and Filters Extension

The Dynamic Hooks and Filters Extension enables YAML-based registration of hooks and filters, eliminating the need to hardcode them in plugin implementations.

## Overview

**Key Features:**
- **Configuration-driven**: Register hooks/filters via YAML instead of code
- **Universal workflow support**: Works with any workflow (inference, training, evaluation, etc.)
- **Registry-based security**: Validates all callbacks against `@discover_hook`/`@discover_filter` decorators
- **Parameter validation**: Schema-based validation with security pattern detection
- **Plugin dependency checks**: Verifies referenced plugins are loaded before callback registration

## Architecture

### Registration Flow

```
1. Application starts
2. Extension loads and registers post_argument_parsing hook
3. CLI parses arguments and loads YAML config
4. Extension reads vipr.{workflow}.hooks and vipr.{workflow}.filters
5. Validates each callback against discovery registry
6. Validates parameters against registry schema
7. Imports and wraps callbacks
8. Registers with Cement hook/filter system
```

### Security Model

**Three-layer validation:**
1. **Registry validation**: Callback must exist in `@discover_hook`/`@discover_filter` registry
2. **Parameter validation**: Parameters must match registry schema
3. **Content validation**: Blocks dangerous patterns (shell injection, path traversal, etc.)

## Configuration Structure

### Universal Workflow Support

Hooks and filters are configured under workflow sections:

```yaml
vipr:
  inference:  # or training, evaluation, custom_workflow, etc.
    hooks:
      HOOK_NAME:
      - class: package.module.ClassName
        method: method_name
        enabled: true
        weight: 0
        parameters:
          param1: value1
    
    filters:
      FILTER_NAME:
      - class: package.module.ClassName
        method: method_name
        enabled: true
        weight: 0
        parameters:
          param1: value1
```

**Note:** The extension automatically discovers hooks/filters from ALL workflow sections under `vipr`.

### Configuration Fields

| Field | Required | Description |
|-------|----------|-------------|
| `class` | ✅ | Full class path (e.g., `vipr.plugins.normalizers.log_normalizer.LogNormalizer`) |
| `method` | ✅ | Method name to call |
| `enabled` | ❌ | Enable/disable callback (default: `true`) |
| `weight` | ❌ | Execution priority, lower runs first (default: `0`) |
| `parameters` | ❌ | Method parameters (must match registry schema) |

## Usage Examples

### Registering a Filter

**Step 1: Declare filter with {py:func}`vipr.plugins.discovery.decorators.discover_filter` decorator**
```python
# vipr/plugins/normalizers/log_normalizer.py
from vipr.plugins.discovery.decorators import discover_filter

class LogNormalizer:
    @discover_filter('INFERENCE_NORMALIZE_PRE_FILTER', enabled_in_config=False)
    def normalize_filter(self, data, **kwargs):
        # Implementation
        return data
```

**Step 2: Configure in YAML**
```yaml
vipr:
  inference:
    filters:
      INFERENCE_NORMALIZE_PRE_FILTER:
      - class: vipr.plugins.normalizers.log_normalizer.LogNormalizer
        method: normalize_filter
        enabled: true
        weight: 0
```

### Registering a Hook with Parameters

**Step 1: Declare hook with {py:func}`vipr.plugins.discovery.decorators.discover_hook` decorator**
```python
# vipr_reflectometry/reflectorch/reflectorch_extension.py
from vipr.plugins.discovery.decorators import discover_hook

class Reflectorch:
    @discover_hook(
        'INFERENCE_LOAD_DATA_PRE_PRE_FILTER_HOOK',
        parameters={'test_params': {'type': 'boolean', 'default': True}},
        enabled_in_config=False
    )
    def _log(self, **kwargs):
        # Access parameter via kwargs
        test_val = kwargs.get('test_params')
        print(f"Test parameter: {test_val}")
```

**Step 2: Configure in YAML with parameter override**
```yaml
vipr:
  inference:
    hooks:
      INFERENCE_LOAD_DATA_PRE_PRE_FILTER_HOOK:
      - class: vipr_reflectometry.reflectorch.reflectorch_extension.Reflectorch
        method: _log
        enabled: true
        weight: 0
        parameters:
          test_params: false  # Override default value
```

### Multiple Callbacks with Weights

Execution order controlled by `weight` (lower = earlier):

```yaml
vipr:
  inference:
    filters:
      INFERENCE_NORMALIZE_PRE_FILTER:
      - class: vipr.plugins.normalizers.log_normalizer.LogNormalizer
        method: normalize_filter
        enabled: true
        weight: 0  # Runs first
      
      - class: vipr.plugins.normalizers.minmax_normalizer.MinMaxNormalizer
        method: normalize_filter
        enabled: false  # Disabled
        weight: 10
```

### Filter with Multiple Parameters

Parameters are declared in decorator and accessed via kwargs:

**Step 1: Declare filter with parameter schema**
```python
# vipr_reflectometry/shared/preprocessing/neutron_data_cleaner.py
from vipr.plugins.discovery.decorators import discover_filter

class NeutronDataCleaner:
    @discover_filter(
        'INFERENCE_PREPROCESS_PRE_FILTER',
        weight=-10,
        enabled_in_config=False,
        parameters={
            'error_threshold': {
                'type': 'float',
                'default': 0.5,
                'help': 'Relative error threshold (dR/R) for filtering'
            },
            'consecutive_errors': {
                'type': 'int',
                'default': 3,
                'help': 'Number of consecutive high-error points to trigger truncation'
            },
            'remove_single_errors': {
                'type': 'bool',
                'default': False,
                'help': 'Remove isolated high-error points before truncation'
            }
        }
    )
    def clean_experimental_data(self, data, **kwargs):
        # Access parameters via kwargs with defaults
        threshold = float(kwargs.get('error_threshold', 0.5))
        consecutive = int(kwargs.get('consecutive_errors', 3))
        remove_singles = bool(kwargs.get('remove_single_errors', False))
        
        # Use parameters in logic
        # ...
```

**Step 2: Configure in YAML with parameter overrides**
```yaml
vipr:
  inference:
    filters:
      INFERENCE_PREPROCESS_PRE_FILTER:
      - class: vipr_reflectometry.shared.preprocessing.neutron_data_cleaner.NeutronDataCleaner
        method: clean_experimental_data
        enabled: true
        weight: -10
        parameters:
          error_threshold: 0.5
          consecutive_errors: 3
          remove_single_errors: false
```

## Security Features

### Registry Validation

Only callbacks registered via `@discover_hook` or `@discover_filter` can be loaded:

```python
# ALLOWED - registered via decorator
@discover_filter('INFERENCE_NORMALIZE_PRE_FILTER')
def normalize_filter(self, data, **kwargs):
    pass

# BLOCKED - not in registry
def random_function():
    pass  # Cannot be loaded from YAML
```

### Parameter Validation

Parameters must match registry schema:

```python
@discover_filter('INFERENCE_NORMALIZE_PRE_FILTER', 
                 enabled_in_config=False,
                 scale_factor={'type': 'float', 'default': 1.0})
def normalize_filter(self, data, scale_factor=1.0, **kwargs):
    pass
```

```yaml
# ALLOWED - parameter declared in decorator
parameters:
  scale_factor: 2.0

# BLOCKED - parameter not in registry
parameters:
  unknown_param: value
```

### Dangerous Pattern Detection

Blocks potential security threats:

```yaml
# BLOCKED - shell injection
parameters:
  command: "rm -rf /"

# BLOCKED - path traversal  
parameters:
  file_path: "../../../etc/passwd"

# BLOCKED - Python injection
parameters:
  code: "__import__('os').system('ls')"
```

## Plugin Management

### Internal Plugins

Must be enabled in configuration:

```yaml
vipr:
  plugin:
    normalizers:
      enabled: true
```

### External Plugins

Must be installed via pip:

```bash
pip install vipr-reflectometry-plugin
```

The extension automatically detects installed external plugins.

## Legacy Support

For backward compatibility, the extension also reads from `vipr.hooks` and `vipr.filters` (non-workflow structure):

```yaml
vipr:
  hooks:
    HOOK_NAME:
    - class: package.ClassName
      method: method_name
  
  filters:
    FILTER_NAME:
    - class: package.ClassName
      method: method_name
```

**Recommendation:** Use workflow-based structure (`vipr.inference.hooks`) for better organization.

## Troubleshooting

### "Callback not in registry"

**Problem:** Trying to load a callback that isn't registered via decorator.

**Solution:** Add `@discover_hook` or `@discover_filter` decorator to the method.

### "Plugin not available"

**Problem:** Plugin containing the callback isn't loaded.

**Solution:** 
- Internal plugin: Enable in config with `plugin.{name}.enabled = true`
- External plugin: Install via `pip install vipr-plugin-{name}`

### "Parameter validation failed"

**Problem:** Using parameters not declared in decorator.

**Solution:** Add parameter to decorator's parameter schema or remove from YAML config.

### "Dangerous pattern in parameter"

**Problem:** Parameter contains potentially malicious content.

**Solution:** Review parameter value and ensure it's safe. Contact administrators if blocking legitimate use case.

## Best Practices

1. **Always use decorators**: Register all hooks/filters with `@discover_hook`/`@discover_filter`
2. **Declare parameters**: Define parameter schemas in decorators
3. **Use weights wisely**: Lower weights for earlier execution
4. **Enable selectively**: Use `enabled: false` to disable without removing config

## See Also

- [Inference Plugin](../plugins/built-in/inference.md) - Hook and filter usage in workflows
- [Normalizers Plugin](../plugins/built-in/normalizers.md) - Filter-based normalization example
- {py:mod}`vipr.ext.dynamic_hooks_filters.extension` - Extension implementation
- {py:mod}`vipr.plugins.discovery.decorators` - Discovery decorators API