HuggingFace Plugin¶

Model management and downloads from HuggingFace repositories.

Overview¶

Enabled by Default: Manages HuggingFace model downloads with caching. Supports multiple repositories and models with configurable paths for weights and config files.

Configuration¶

Configure models in huggingface.yaml:

plugin.huggingface:
  enabled: true

huggingface:
  repositories:
    reflectorch:
      root_dir_env_var: "REFLECTOMETRY_ROOT_DIR"
      repo_id: "valentinsingularity/reflectivity"
      files:
        - source: "configs/b_mc_point_xray_conv_standard_L2_InputQ.yaml"
          target: "reflectorch/configs/b_mc_point_xray_conv_standard_L2_InputQ.yaml"
        - source: "saved_models/model_b_mc_point_xray_conv_standard_L2_InputQ.safetensors"
          target: "reflectorch/saved_models/model_b_mc_point_xray_conv_standard_L2_InputQ.safetensors"

    panpe:
      root_dir_env_var: "REFLECTOMETRY_ROOT_DIR"
      repo_id: "HW-SC/panpe-xrr-reflectometry"
      files:
        - source: "configs/panpe-2layers-xrr.yaml"
          target: "panpe/configs/panpe-2layers-xrr.yaml"
        - source: "saved_models/model_panpe-2layers-xrr.pt"
          target: "panpe/saved_models/model_panpe-2layers-xrr.pt"

Configuration Fields¶

Field	Type	Description
`repositories`	`dict`	Collection of HuggingFace repositories
`root_dir_env_var`	`str`	Environment variable that points to the local model root
`repo_id`	`str`	HuggingFace repository identifier (org/repo)
`files`	`list`	List of files to download from the repository
`source`	`str`	Path to the file inside the HuggingFace repository
`target`	`str`	Path below the local model root

Model Storage¶

Environment Variables¶

Two environment variable mechanisms control where models are stored:

HF_HOME - General HuggingFace cache directory (if unset, plugin uses a temporary directory):

export HF_HOME=${PWD}/storage/huggingface_cache

root_dir_env_var - Environment variable that defines the persistent local model root for a configured repository.

In the current VIPR configuration, both reflectometry repositories use the shared reflectometry root:

reflectorch -> REFLECTOMETRY_ROOT_DIR
panpe -> REFLECTOMETRY_ROOT_DIR

Example:

export REFLECTOMETRY_ROOT_DIR=${PWD}/storage/reflectometry

Directory Structure¶

With REFLECTOMETRY_ROOT_DIR set, reflectometry models are organized as:

storage/reflectometry/
├── reflectorch/
│   ├── saved_models/     # Model weights (.safetensors, .pt)
│   └── configs/          # Model configuration files (.yaml)
└── panpe/
    ├── saved_models/
    └── configs/

Other models use the standard HuggingFace cache structure in HF_HOME.

CLI Commands¶

Download Models¶

Download configured HuggingFace model files:

vipr huggingface download

This downloads all configured models from HuggingFace repositories to the configured local model roots. For custom deployment additions, set VIPR_HUGGINGFACE_CONFIG to an additional HuggingFace YAML before running the command.

Use Cases:

Pre-deployment: Download models before production
Offline scenarios: Cache models for air-gapped environments
CI/CD: Prepare models during build process

Adding Custom Models¶

Step 1: Identify Repository¶

Find your model on HuggingFace: https://huggingface.co/organization/repository-name

Step 2: Add to Configuration¶

huggingface:
  repositories:
    my_project:  # Unique identifier for your use
      root_dir_env_var: "MY_PROJECT_MODEL_ROOT"
      repo_id: "organization/repository-name"  # HuggingFace repo
      files:
        - source: "path/in/repo/config.yaml"
          target: "my_project/configs/config.yaml"
        - source: "path/in/repo/model.safetensors"
          target: "my_project/saved_models/model.safetensors"

Step 3: Download¶

vipr huggingface download

Supported Weight Formats¶

.safetensors - Recommended format (faster, safer, no pickle)
.pt - Standard PyTorch format
.bin - Older PyTorch format (pickle-based)

Example: Multiple Models¶

huggingface:
  repositories:
    reflectorch:
      root_dir_env_var: "REFLECTOMETRY_ROOT_DIR"
      repo_id: "valentinsingularity/reflectivity"
      files:
        - source: "configs/b_mc_point_xray_conv_standard_L2_InputQ.yaml"
          target: "reflectorch/configs/b_mc_point_xray_conv_standard_L2_InputQ.yaml"
        - source: "saved_models/model_b_mc_point_xray_conv_standard_L2_InputQ.safetensors"
          target: "reflectorch/saved_models/model_b_mc_point_xray_conv_standard_L2_InputQ.safetensors"

    panpe:
      root_dir_env_var: "REFLECTOMETRY_ROOT_DIR"
      repo_id: "HW-SC/panpe-xrr-reflectometry"
      files:
        - source: "configs/panpe-2layers-xrr.yaml"
          target: "panpe/configs/panpe-2layers-xrr.yaml"
        - source: "saved_models/model_panpe-2layers-xrr.pt"
          target: "panpe/saved_models/model_panpe-2layers-xrr.pt"

Download Behavior¶

vipr huggingface download attempts to download all configured model files.
Existing files are skipped automatically (unless forced in code).
API startup downloads HuggingFace models by default. Set VIPR_NO_MODEL_DOWNLOAD=true to skip the startup download.