Discovery Plugin Documentation

The Discovery Plugin provides a decorator-based registry system for all VIPR components, enabling automatic discovery and introspection of available handlers, hooks, and filters.

Overview

The Discovery Plugin serves as the central registry for:

  • Hooks & Filters: Readonly (hooks) and transformative (filters) callbacks for workflow extensions

  • Data Loaders: Components for loading input data

  • Model Loaders: Components for loading ML models

  • Predictors: Components for executing predictions

  • Postprocessors: Components for post-processing results

Key Features:

  • Decorator-based registration: Simple @discover_* decorators

  • CLI introspection: Query available components via command line

  • API-ready commands: Controller methods exposable as FastAPI endpoints via @api decorator

  • Plugin filtering: Filter components by plugin name

  • Automatic discovery: Components self-register on import

Architecture

Registry System

The plugin maintains separate registries for each component type:

# Central registries (in decorators.py)
hooks_discovery: dict[str, list[dict]]
filters_discovery: dict[str, list[dict]]
data_loaders_discovery: dict[str, dict]
model_loaders_discovery: dict[str, dict]
predictors_discovery: dict[str, dict]
postprocessors_discovery: dict[str, dict]

Registration Flow

1. Plugin loads and imports modules
2. Decorators execute on import (@discover_filter, @discover_data_loader, etc.)
3. Components register in central registries (hooks_discovery, filters_discovery, etc.)
4. Registries ready for introspection via CLI/API and programmatic access

Available Decorators

Hook & Filter Registration

Decorator

Purpose

Registry

@discover_hook()

Register workflow hooks

hooks_discovery

@discover_filter()

Register workflow filters

filters_discovery

Parameters:

  • hook_type / filter_type: Hook/filter name (e.g., INFERENCE_NORMALIZE_PRE_FILTER)

  • weight: Execution priority (default: 0, lower runs first)

  • enabled_in_config: Enable in generated configs (default: True)

  • **parameters: Parameter schema (for validation)

Handler Registration

Decorator

Purpose

Registry

@discover_data_loader()

Register data loaders

data_loaders_discovery

@discover_model_loader()

Register model loaders

model_loaders_discovery

@discover_predictor()

Register predictors

predictors_discovery

@discover_postprocessor()

Register postprocessors

postprocessors_discovery

Parameters:

  • handler_name: Handler identifier (e.g., csv_spectrareader)

  • parameters: Parameter schema (optional)

Usage Examples

Registering Hooks/Filters

Hook and filter callbacks are attached to these extension points via YAML configuration, which is passed to the CLI workflow command (e.g., vipr --config ./config.yaml inference run). See Dynamic Hooks and Filters Extension for details on registering callbacks.

Example of decorator registration:

from vipr.plugins.discovery.decorators import discover_filter

class LogNormalizer:
    @discover_filter('INFERENCE_NORMALIZE_PRE_FILTER', enabled_in_config=False)
    def normalize_filter(self, data, **kwargs):
        return data

Registering Data Loader

Real example from Reflectometry Plugin:

from vipr.handlers.data_loader import DataLoaderHandler
from vipr.plugins.discovery.decorators import discover_data_loader
from vipr.plugins.inference.dataset import DataSet

@discover_data_loader('csv_spectrareader', {
    'data_path': {
        'type': 'str', 
        'required': True,
        'description': 'Path to single CSV/DAT/TXT file'
    },
    'column_mapping': {
        'type': 'dict',
        'required': False,
        'default': {'q': 0, 'I': 1},
        'description': 'Column mapping for CSV data'
    }
})
class CSVSpectraReaderDataLoader(DataLoaderHandler):
    """CSV/DAT/TXT data loader for reflectometry data."""
    
    class Meta:
        label = 'csv_spectrareader'
    
    def _load_data(self, data_path: str, column_mapping: dict = None, **kwargs) -> DataSet:
        # Load CSV using SpectraReader
        # Returns DataSet with batch_size=1
        ...

Registering Model Loader

Real example from Reflectometry Plugin:

from vipr.handlers.model_loader import ModelLoaderHandler
from vipr.plugins.discovery.decorators import discover_model_loader
from reflectorch import EasyInferenceModel

@discover_model_loader('reflectorch', {
    'config_name': {'type': 'str', 'required': True},
    'device': {'type': 'str', 'required': False, 'default': 'cpu'}
})
class ReflectorchModelLoader(ModelLoaderHandler):
    """Reflectorch model loader for reflectometry analysis."""
    
    class Meta:
        label = 'reflectorch'
        
    def _load_model(self, **kwargs):
        config_name = kwargs.get('config_name')
        device = kwargs.get('device', 'cpu')
        
        # Load Reflectorch model
        model = EasyInferenceModel(
            config_name=config_name,
            root_dir=os.getenv('REFLECTORCH_ROOT_DIR'),
            device=device
        )
        return model

CLI Commands

Query Components Summary

# All components
vipr discovery plugins

# Filter by plugin
vipr discovery plugins --plugin normalizers

Output:

{
  "model_loaders": 2,
  "data_loaders": 3,
  "hooks": 15,
  "filters": 12,
  "predictors": 1,
  "postprocessors": 0
}

Query Specific Component Types

# List all model loaders
vipr discovery model-loaders

# List data loaders for specific plugin
vipr discovery data-loaders --plugin reflectometry

# List all hooks
vipr discovery hooks

# List hooks of specific type
vipr discovery hooks --hook-type INFERENCE_POSTPROCESS_PRE_PRE_FILTER_HOOK

# List hooks for specific plugin
vipr discovery hooks --plugin reflectometry

# List all filters
vipr discovery filters

# List filters of specific type
vipr discovery filters --filter-type INFERENCE_NORMALIZE_PRE_FILTER

# List predictors
vipr discovery predictors

Query All Components

# Complete registry dump
vipr discovery components

# Filter by plugin
vipr discovery components --plugin reflectorch

Query Available Types

# List all hook types (inference workflow)
vipr discovery hook-types

# List all filter types (inference workflow)
vipr discovery filter-types

API Integration

All discovery commands are available as HTTP endpoints via the API plugin.

Endpoints

Endpoint

Method

Description

/discovery/plugins

GET

Components summary

/discovery/model-loaders

GET

List model loaders

/discovery/data-loaders

GET

List data loaders

/discovery/hooks

GET

List hooks

/discovery/filters

GET

List filters

/discovery/predictors

GET

List predictors

/discovery/postprocessors

GET

List postprocessors

/discovery/components

GET

All components

/discovery/hook-types

GET

Available hook types

/discovery/filter-types

GET

Available filter types

Example API Call

# Get all model loaders
curl http://localhost:8000/discovery/model-loaders

# Get hooks for specific plugin
curl "http://localhost:8000/discovery/hooks?plugin=normalizers"

Plugin Filtering

The Discovery system supports filtering by plugin name for both internal and external plugins.

Internal Plugins

Pattern: vipr.plugins.{plugin_name}.

vipr discovery hooks --plugin normalizers
# Finds: vipr.plugins.normalizers.log_normalizer.LogNormalizer

External Plugins

External plugins are discovered via Python entry points. See Plugin Development Guide for details on entry point configuration and naming conventions.

Plugin Filtering Example:

vipr discovery hooks --plugin reflectometry
# Example match: vipr_reflectometry.reflectorch.reflectorch_extension.Reflectorch

How it works:

  • Input: --plugin reflectometry (plugin name from entry point)

  • Pattern constructed: vipr_reflectometry.* (automatic vipr_ prefix added)

  • Matches: All components in the vipr_reflectometry package

This is why the vipr_{plugin_name} naming convention is recommended – it enables automatic filtering by plugin name.

Integration with Dynamic Hooks Extension

The Discovery Plugin provides the registry foundation for the Dynamic Hooks and Filters Extension:

  1. Decorator Registration: Components use @discover_hook/@discover_filter

  2. Registry Validation: Dynamic Hooks Extension validates YAML configs against registries

  3. Security: Only registered components can be loaded from configuration

  4. Parameter Validation: Parameter schemas from decorators enforce type safety

Flow:

@discover_filter decorator
    ↓
Register in filters_discovery
    ↓
Dynamic Hooks Extension reads registry
    ↓
Validates YAML config against registry
    ↓
Imports and registers callback

Registry Data Structure

Hook/Filter Entry

{
    'enabled_in_config': False,
    'weight': 0,
    'method': 'normalize_filter',
    'class': 'vipr.plugins.normalizers.log_normalizer.LogNormalizer',
    'component_id': 'vipr.plugins.normalizers.log_normalizer.LogNormalizer.normalize_filter',
    'parameters': {
        'scale_factor': {'type': 'float', 'default': 1.0}
    }
}

Handler Entry

{
    'handler': 'csv_loader',
    'class': 'vipr.plugins.data_loader.csv_loader.CSVDataLoader',
    'parameters': {
        'file_path': {'type': 'string', 'required': True},
        'delimiter': {'type': 'string', 'default': ','}
    }
}

Best Practices

  1. Always use decorators: Register all components via @discover_* decorators

  2. Unique handler names: Use descriptive, unique names for handlers

  3. Document parameters: Include parameter schemas in decorators

  4. Class-only callbacks: Hooks/filters must be class methods (enforced)

  5. Set enabled_in_config appropriately: Use False for optional components

Programmatic Access

Query Registries in Code

from vipr.plugins.discovery.decorators import (
    get_discovered_hooks,
    get_discovered_filters,
    get_discovered_data_loaders,
    get_discovered_model_loaders
)

# Get all hooks
all_hooks = get_discovered_hooks()

# Get hooks of specific type
start_hooks = get_discovered_hooks(hook_type='INFERENCE_START_HOOK')

# Get hooks for plugin
plugin_hooks = get_discovered_hooks(plugin_name='normalizers')

# Get data loaders
loaders = get_discovered_data_loaders()

# Filter by plugin
reflectorch_loaders = get_discovered_data_loaders(plugin_name='reflectorch')

Configuration Generation

The Discovery system can be used to automatically generate YAML/JSON configurations based on registered components. This is particularly useful for:

  • Default configurations: Generate ready-to-use starter configs for plugins

  • User templates: Provide users with pre-configured, customizable setups

Use Case: Plugin Configuration Templates

Plugins can provide commands to generate standard configurations by querying the Discovery registries for their components.

Example from Reflectometry Plugin:

from vipr.plugins.discovery.decorators import (
    get_discovered_filters, get_discovered_hooks,
    get_discovered_data_loaders, get_discovered_model_loaders,
    get_discovered_predictors
)

def generate_standard_config(self):
    """Builds a standard configuration based on registry components."""
    # Query registered components for this plugin
    all_model_loaders = get_discovered_model_loaders('reflectometry')
    all_data_loaders = get_discovered_data_loaders('reflectometry')
    all_predictors = get_discovered_predictors('reflectometry')
    
    # Find specific handlers by name
    model_loader = self._find_handler(all_model_loaders, 'reflectorch')
    data_loader = self._find_handler(all_data_loaders, 'csv_spectrareader')
    predictor = self._find_handler(all_predictors, 'reflectorch_predictor')
    
    # Get all hooks and filters
    hooks = get_discovered_hooks()
    filters = get_discovered_filters()
    
    # Build configuration with default parameter values
    config = {
        'inference': {
            'load_model': self._component_to_config(model_loader),
            'load_data': self._component_to_config(data_loader),
            'prediction': self._component_to_config(predictor),
            'hooks': self._generate_hooks_config(hooks),
            'filters': self._generate_filters_config(filters)
        }
    }
    
    return config

def _component_to_config(self, component):
    """Converts a component to configuration format."""
    config = {'handler': component.get('handler')}
    
    # Extract default parameter values from schema
    parameters = {}
    for param_name, param_info in component.get('parameters', {}).items():
        if 'default' in param_info:
            parameters[param_name] = param_info['default']
    
    if parameters:
        config['parameters'] = parameters
    
    return config

CLI Usage

# Generate standard config (returns JSON)
vipr reflectometry generate-standard-config

# Write config to YAML file
vipr reflectometry write-standard-config-to-yaml --output config/my_config.yaml

See Also