Filter Documentation

Filters transform data during the inference pipeline. They are configured in YAML and executed in order of weight (lower values run first).

Quick Start for Reflectometry

Typical Reflectorch configs use these preprocessing filters:

  1. ErrorBarFilter.preprocess_error_bar_filter (weight: -10)

  2. InterpolationFilter.preprocess_interpolate (weight: 0)

Run an example:

vipr --config @vipr_reflectometry/reflectorch/examples/configs/Ni500.yaml inference run

Discover available filters:

vipr discovery filters

Available Filters

Normalization Filters (INFERENCE_PREPROCESS_PRE_FILTER)

Normalization is part of the preprocessing filter chain in the current 5-step workflow (there is no separate normalize step anymore).

These filters are discovered with enabled_in_config=False and weight=-10.

MinMaxNormalizer

Scales y values to [0, 1].

Formula:

y_norm = (y - min(y)) / (max(y) - min(y))
dy_norm = dy / (max(y) - min(y))

YAML configuration:

vipr:
  inference:
    filters:
      INFERENCE_PREPROCESS_PRE_FILTER:
      - class: vipr.plugins.normalizers.minmax_normalizer.MinMaxNormalizer
        enabled: true
        method: normalize_filter
        weight: -10

ZScoreNormalizer

Standardizes y to mean 0 and standard deviation 1.

Formula:

y_norm = (y - mean(y)) / std(y)
dy_norm = dy / std(y)

YAML configuration:

vipr:
  inference:
    filters:
      INFERENCE_PREPROCESS_PRE_FILTER:
      - class: vipr.plugins.normalizers.zscore_normalizer.ZScoreNormalizer
        enabled: true
        method: normalize_filter
        weight: -10

LogNormalizer

Applies logarithmic transformation to y.

Formula:

y_norm = log(y + offset)   # offset is used if min(y) <= 0
dy_norm = dy / (y + offset)

YAML configuration:

vipr:
  inference:
    filters:
      INFERENCE_PREPROCESS_PRE_FILTER:
      - class: vipr.plugins.normalizers.log_normalizer.LogNormalizer
        enabled: true
        method: normalize_filter
        weight: -10

Reflectometry Preprocessing Filters (INFERENCE_PREPROCESS_PRE_FILTER)

These run in preprocessing (step 3), before prediction (step 4).

ErrorBarFilter

Filters experimental curves based on relative error bars and truncation rules.

Class/method:

  • vipr_reflectometry.reflectorch.preprocess.error_bar_filter.ErrorBarFilter

  • preprocess_error_bar_filter

Default: disabled in generated discovery config (enabled_in_config=False)

Parameters:

  • filter_threshold (float, default 0.3): Relative error threshold dR/R

  • filter_remove_singles (bool, default true): Remove isolated high-error points

  • filter_remove_consecutives (bool, default true): Truncate at consecutive high-error points

  • filter_consecutive (int, default 3): Consecutive point count for truncation trigger

  • filter_q_start_trunc (float, default 0.1): Start checking truncation only above this Q

YAML configuration:

vipr:
  inference:
    filters:
      INFERENCE_PREPROCESS_PRE_FILTER:
      - class: vipr_reflectometry.reflectorch.preprocess.error_bar_filter.ErrorBarFilter
        enabled: true
        method: preprocess_error_bar_filter
        weight: -10
        parameters:
          filter_threshold: 0.3
          filter_consecutive: 3
          filter_remove_singles: true
          filter_remove_consecutives: true
          filter_q_start_trunc: 0.1

InterpolationFilter (Reflectorch)

Interpolates experimental data to the model Q-grid.

Class/method:

  • vipr_reflectometry.reflectorch.preprocess.interpolation_filter.InterpolationFilter

  • preprocess_interpolate

Default: enabled in generated discovery config (enabled_in_config=True)

YAML configuration:

vipr:
  inference:
    filters:
      INFERENCE_PREPROCESS_PRE_FILTER:
      - class: vipr_reflectometry.reflectorch.preprocess.interpolation_filter.InterpolationFilter
        enabled: true
        method: preprocess_interpolate
        weight: 0

FlowPreprocessor

Preprocessing for NSF-based flow models.

Class/method:

  • vipr_reflectometry.flow_models.preprocess.flow_preprocessor.FlowPreprocessor

  • _preprocess_flow

Default: disabled in generated discovery config (enabled_in_config=False)

YAML configuration:

vipr:
  inference:
    filters:
      INFERENCE_PREPROCESS_PRE_FILTER:
      - class: vipr_reflectometry.flow_models.preprocess.flow_preprocessor.FlowPreprocessor
        enabled: true
        method: _preprocess_flow
        weight: 0

Filter Chaining

All preprocessing filters share the same namespace and are ordered by weight.

vipr:
  inference:
    filters:
      INFERENCE_PREPROCESS_PRE_FILTER:
      - class: vipr_reflectometry.reflectorch.preprocess.error_bar_filter.ErrorBarFilter
        enabled: true
        method: preprocess_error_bar_filter
        weight: -10
        parameters:
          filter_threshold: 0.3

      - class: vipr_reflectometry.reflectorch.preprocess.interpolation_filter.InterpolationFilter
        enabled: true
        method: preprocess_interpolate
        weight: 0

Practical Examples

Reflectorch default preprocessing

vipr --config @vipr_reflectometry/reflectorch/examples/configs/Ni500.yaml inference run

This config uses ErrorBarFilter + InterpolationFilter.

Discovery

vipr discovery filters

Best Practices

Reflectorch models

  • Usually do not add external normalization filters.

  • Reflectorch models already apply their own internal scaling.

ErrorBarFilter tuning

  • Start with filter_threshold=0.3

  • Keep filter_consecutive=3 as robust default

  • Use filter_remove_singles=true only if isolated outliers are common

Notes

  • enabled_in_config is discovery metadata (default generation), not a runtime override by itself.

  • Runtime activation is controlled via YAML enabled: true/false.

  • INFERENCE_NORMALIZE_PRE_FILTER is legacy naming; current normalization filters run under INFERENCE_PREPROCESS_PRE_FILTER.