Dynamic Hooks and Filters Extension

The Dynamic Hooks and Filters Extension enables YAML-based registration of hooks and filters, eliminating the need to hardcode them in plugin implementations.

Overview

Key Features:

  • Configuration-driven: Register hooks/filters via YAML instead of code

  • Universal workflow support: Works with any workflow (inference, training, evaluation, etc.)

  • Registry-based security: Validates all callbacks against @discover_hook/@discover_filter decorators

  • Parameter validation: Schema-based validation with security pattern detection

  • Plugin dependency checks: Verifies referenced plugins are loaded before callback registration

Architecture

Registration Flow

1. Application starts
2. Extension loads and registers post_argument_parsing hook
3. CLI parses arguments and loads YAML config
4. Extension reads vipr.{workflow}.hooks and vipr.{workflow}.filters
5. Validates each callback against discovery registry
6. Validates parameters against registry schema
7. Imports and wraps callbacks
8. Registers with Cement hook/filter system

Security Model

Three-layer validation:

  1. Registry validation: Callback must exist in @discover_hook/@discover_filter registry

  2. Parameter validation: Parameters must match registry schema

  3. Content validation: Blocks dangerous patterns (shell injection, path traversal, etc.)

Configuration Structure

Universal Workflow Support

Hooks and filters are configured under workflow sections:

vipr:
  inference:  # or training, evaluation, custom_workflow, etc.
    hooks:
      HOOK_NAME:
      - class: package.module.ClassName
        method: method_name
        enabled: true
        weight: 0
        parameters:
          param1: value1
    
    filters:
      FILTER_NAME:
      - class: package.module.ClassName
        method: method_name
        enabled: true
        weight: 0
        parameters:
          param1: value1

Note: The extension automatically discovers hooks/filters from ALL workflow sections under vipr.

Configuration Fields

Field

Required

Description

class

✅

Full class path (e.g., vipr.plugins.normalizers.log_normalizer.LogNormalizer)

method

✅

Method name to call

enabled

❌

Enable/disable callback (default: true)

weight

❌

Execution priority, lower runs first (default: 0)

parameters

❌

Method parameters (must match registry schema)

Usage Examples

Registering a Filter

Step 1: Declare filter with vipr.plugins.discovery.decorators.discover_filter() decorator

# vipr/plugins/normalizers/log_normalizer.py
from vipr.plugins.discovery.decorators import discover_filter

class LogNormalizer:
    @discover_filter('INFERENCE_NORMALIZE_PRE_FILTER', enabled_in_config=False)
    def normalize_filter(self, data, **kwargs):
        # Implementation
        return data

Step 2: Configure in YAML

vipr:
  inference:
    filters:
      INFERENCE_NORMALIZE_PRE_FILTER:
      - class: vipr.plugins.normalizers.log_normalizer.LogNormalizer
        method: normalize_filter
        enabled: true
        weight: 0

Registering a Hook with Parameters

Step 1: Declare hook with vipr.plugins.discovery.decorators.discover_hook() decorator

# vipr_reflectometry/reflectorch/reflectorch_extension.py
from vipr.plugins.discovery.decorators import discover_hook

class Reflectorch:
    @discover_hook(
        'INFERENCE_LOAD_DATA_PRE_PRE_FILTER_HOOK',
        parameters={'test_params': {'type': 'boolean', 'default': True}},
        enabled_in_config=False
    )
    def _log(self, **kwargs):
        # Access parameter via kwargs
        test_val = kwargs.get('test_params')
        print(f"Test parameter: {test_val}")

Step 2: Configure in YAML with parameter override

vipr:
  inference:
    hooks:
      INFERENCE_LOAD_DATA_PRE_PRE_FILTER_HOOK:
      - class: vipr_reflectometry.reflectorch.reflectorch_extension.Reflectorch
        method: _log
        enabled: true
        weight: 0
        parameters:
          test_params: false  # Override default value

Multiple Callbacks with Weights

Execution order controlled by weight (lower = earlier):

vipr:
  inference:
    filters:
      INFERENCE_NORMALIZE_PRE_FILTER:
      - class: vipr.plugins.normalizers.log_normalizer.LogNormalizer
        method: normalize_filter
        enabled: true
        weight: 0  # Runs first
      
      - class: vipr.plugins.normalizers.minmax_normalizer.MinMaxNormalizer
        method: normalize_filter
        enabled: false  # Disabled
        weight: 10

Filter with Multiple Parameters

Parameters are declared in decorator and accessed via kwargs:

Step 1: Declare filter with parameter schema

# vipr_reflectometry/shared/preprocessing/neutron_data_cleaner.py
from vipr.plugins.discovery.decorators import discover_filter

class NeutronDataCleaner:
    @discover_filter(
        'INFERENCE_PREPROCESS_PRE_FILTER',
        weight=-10,
        enabled_in_config=False,
        parameters={
            'error_threshold': {
                'type': 'float',
                'default': 0.5,
                'help': 'Relative error threshold (dR/R) for filtering'
            },
            'consecutive_errors': {
                'type': 'int',
                'default': 3,
                'help': 'Number of consecutive high-error points to trigger truncation'
            },
            'remove_single_errors': {
                'type': 'bool',
                'default': False,
                'help': 'Remove isolated high-error points before truncation'
            }
        }
    )
    def clean_experimental_data(self, data, **kwargs):
        # Access parameters via kwargs with defaults
        threshold = float(kwargs.get('error_threshold', 0.5))
        consecutive = int(kwargs.get('consecutive_errors', 3))
        remove_singles = bool(kwargs.get('remove_single_errors', False))
        
        # Use parameters in logic
        # ...

Step 2: Configure in YAML with parameter overrides

vipr:
  inference:
    filters:
      INFERENCE_PREPROCESS_PRE_FILTER:
      - class: vipr_reflectometry.shared.preprocessing.neutron_data_cleaner.NeutronDataCleaner
        method: clean_experimental_data
        enabled: true
        weight: -10
        parameters:
          error_threshold: 0.5
          consecutive_errors: 3
          remove_single_errors: false

Security Features

Registry Validation

Only callbacks registered via @discover_hook or @discover_filter can be loaded:

# ALLOWED - registered via decorator
@discover_filter('INFERENCE_NORMALIZE_PRE_FILTER')
def normalize_filter(self, data, **kwargs):
    pass

# BLOCKED - not in registry
def random_function():
    pass  # Cannot be loaded from YAML

Parameter Validation

Parameters must match registry schema:

@discover_filter('INFERENCE_NORMALIZE_PRE_FILTER', 
                 enabled_in_config=False,
                 scale_factor={'type': 'float', 'default': 1.0})
def normalize_filter(self, data, scale_factor=1.0, **kwargs):
    pass

# ALLOWED - parameter declared in decorator
parameters:
  scale_factor: 2.0

# BLOCKED - parameter not in registry
parameters:
  unknown_param: value

Dangerous Pattern Detection

Blocks potential security threats:

# BLOCKED - shell injection
parameters:
  command: "rm -rf /"

# BLOCKED - path traversal  
parameters:
  file_path: "../../../etc/passwd"

# BLOCKED - Python injection
parameters:
  code: "__import__('os').system('ls')"

Plugin Management

Internal Plugins

Must be enabled in configuration:

vipr:
  plugin:
    normalizers:
      enabled: true

External Plugins

Must be installed via pip:

pip install vipr-reflectometry-plugin

The extension automatically detects installed external plugins.

Legacy Support

For backward compatibility, the extension also reads from vipr.hooks and vipr.filters (non-workflow structure):

vipr:
  hooks:
    HOOK_NAME:
    - class: package.ClassName
      method: method_name
  
  filters:
    FILTER_NAME:
    - class: package.ClassName
      method: method_name

Recommendation: Use workflow-based structure (vipr.inference.hooks) for better organization.

Troubleshooting

“Callback not in registry”

Problem: Trying to load a callback that isn’t registered via decorator.

Solution: Add @discover_hook or @discover_filter decorator to the method.

“Plugin not available”

Problem: Plugin containing the callback isn’t loaded.

Solution:

  • Internal plugin: Enable in config with plugin.{name}.enabled = true

  • External plugin: Install via pip install vipr-plugin-{name}

“Parameter validation failed”

Problem: Using parameters not declared in decorator.

Solution: Add parameter to decorator’s parameter schema or remove from YAML config.

“Dangerous pattern in parameter”

Problem: Parameter contains potentially malicious content.

Solution: Review parameter value and ensure it’s safe. Contact administrators if blocking legitimate use case.

Best Practices

  1. Always use decorators: Register all hooks/filters with @discover_hook/@discover_filter

  2. Declare parameters: Define parameter schemas in decorators

  3. Use weights wisely: Lower weights for earlier execution

  4. Enable selectively: Use enabled: false to disable without removing config

See Also