Experiment Grid

SRToolkit.experiments.experiment_grid

Job-based experiment runner for symbolic regression experiments.

Provides three public classes:

• ExperimentInfo — lightweight metadata (seed, paths) for a single run.
• ExperimentJob — one atomic experiment: a single dataset × approach × seed triple. Can be run in-process or dispatched to a CLI worker.
• ExperimentGrid — a full cross-product grid of datasets and approaches. Manages serialization, parallelism via HPC command files, progress tracking, and result loading.

ExperimentInfo dataclass

ExperimentInfo(seed: int, result_path: str, top_k: int = 20, adapted_state_path: Optional[str] = None)

Metadata for a single experiment run.

Holds all job-specific information not contained in the dataset or approach config: the random seed, the path where the result should be written, how many top expressions to keep, and — for approaches with adaptation_scope="once" — where the pre-adapted state is stored.

ExperimentGrid builds these in memory inside build_job — both create_jobs and the run_job CLI derive the seed and result path from the grid plus the (dataset, approach, seed) identifiers, so no per-job info files are written to disk.

Examples:

>>> info = ExperimentInfo(seed=42, result_path="/results/exp_42.json")
>>> info.seed
42
>>> info.top_k
20
>>> d = info.to_dict()
>>> ExperimentInfo.from_dict(d) == info
True

Attributes:

Name	Type	Description
seed	int	Random seed passed to the evaluator and the approach's search() method.
result_path	str	File path where the result JSON will be written. If a directory is passed to ExperimentJob, the filename exp_{seed}.json is appended automatically.
top_k	int	Number of top-ranked expressions to retain in the result. Default 20.
adapted_state_path	Optional[str]	Base path to the pre-adapted state for "once"-scope approaches. None means the approach will adapt from scratch on every run and the state will not be saved.

to_dict

to_dict() -> dict

Serialise to a JSON-safe dictionary.

Returns:

Type	Description
dict	A flat dictionary with keys seed, result_path, top_k, and adapted_state_path, suitable for passing to from_dict.

Source code in SRToolkit/experiments/experiment_grid.py

def to_dict(self) -> dict:
    """
    Serialise to a JSON-safe dictionary.

    Returns:
        A flat dictionary with keys ``seed``, ``result_path``, ``top_k``, and ``adapted_state_path``, suitable for passing to [from_dict][SRToolkit.experiments.ExperimentInfo.from_dict].
    """
    return dataclasses.asdict(self)

from_dict classmethod

from_dict(d: dict) -> ExperimentInfo

Restore an ExperimentInfo from a dictionary produced by to_dict.

Parameters:

Name	Type	Description	Default
d	dict	Dictionary with keys seed, result_path, top_k, and adapted_state_path.	required

Returns:

Type	Description
ExperimentInfo	The reconstructed ExperimentInfo.

Source code in SRToolkit/experiments/experiment_grid.py

@classmethod
def from_dict(cls, d: dict) -> "ExperimentInfo":
    """
    Restore an [ExperimentInfo][SRToolkit.experiments.ExperimentInfo] from a dictionary
    produced by [to_dict][SRToolkit.experiments.ExperimentInfo.to_dict].

    Args:
        d: Dictionary with keys ``seed``, ``result_path``, ``top_k``, and
            ``adapted_state_path``.

    Returns:
        The reconstructed [ExperimentInfo][SRToolkit.experiments.ExperimentInfo].
    """
    return cls(**d)

ExperimentJob

ExperimentJob(dataset: Union[SR_dataset, str, dict], approach: Union[SR_approach, str, dict], info: Union[ExperimentInfo, str, dict], callbacks: Optional[Union[SRCallbacks, List[SRCallbacks], dict, List[dict]]] = None)

A single atomic experiment: one dataset × one approach × one seed.

An ExperimentJob is built from three components:

• dataset: the dataset to evaluate on — an SR_dataset instance, a path to a SR_dataset.to_dict() JSON file, or the dict itself.
• approach: the SR approach — an SR_approach instance, a path to an ApproachConfig.to_dict() JSON file, or the dict itself.
• info: job metadata — an ExperimentInfo instance, a path to an ExperimentInfo.to_dict() JSON file, or the dict itself.

For Python use, you can pass instances directly::

job = ExperimentJob(my_dataset, my_approach,
                    ExperimentInfo(seed=0, result_path="/out/"))
job.run()

Within a grid you rarely build these by hand — ExperimentGrid.build_job assembles one from the grid's stored configs, which is also how the run_job CLI executes a single (dataset, approach, seed) triple from just the grid.json file.

Attributes:

Name	Type	Description
dataset_name	str	Name of the dataset, resolved at construction time.
approach_name	str	Name of the approach, resolved at construction time.
seed		Random seed (from info).
result_path		File path where the experiment result is saved (from info).
info		The ExperimentInfo for this job.
is_complete	bool	True if the result file already exists on disk.

Parameters:

Name	Type	Description	Default
dataset	Union[SR_dataset, str, dict]	The dataset. One of: SR_dataset instance — used directly in memory. str — path to a JSON produced by SR_dataset.to_dict(). dict — the SR_dataset.to_dict() output directly.	required
approach	Union[SR_approach, str, dict]	The SR approach. One of: SR_approach instance — used directly in memory. str — path to a JSON produced by ApproachConfig.to_dict(). dict — the ApproachConfig.to_dict() output directly.	required
info	Union[ExperimentInfo, str, dict]	Job metadata. One of: ExperimentInfo instance. str — path to a JSON produced by ExperimentInfo.to_dict(). dict — the ExperimentInfo.to_dict() output directly.	required
callbacks	Optional[Union[SRCallbacks, List[SRCallbacks], dict, List[dict]]]	Optional callbacks to attach during run. Accepts a single SRCallbacks instance, a list of instances, a single serialised callback dict, or a list of dicts. Instances are serialised to dicts immediately so that run always reconstructs fresh instances (no shared state between jobs). Defaults to None.	None

Raises:

Type	Description
ValueError	If info.result_path is not a directory and does not end with .json.

Source code in SRToolkit/experiments/experiment_grid.py

def __init__(
    self,
    dataset: Union[SR_dataset, str, dict],
    approach: Union[SR_approach, str, dict],
    info: Union[ExperimentInfo, str, dict],
    callbacks: Optional[Union[SRCallbacks, List[SRCallbacks], dict, List[dict]]] = None,
) -> None:
    """
    Args:
        dataset: The dataset.  One of:

            - ``SR_dataset`` instance — used directly in memory.
            - ``str`` — path to a JSON produced by ``SR_dataset.to_dict()``.
            - ``dict`` — the ``SR_dataset.to_dict()`` output directly.

        approach: The SR approach.  One of:

            - ``SR_approach`` instance — used directly in memory.
            - ``str`` — path to a JSON produced by ``ApproachConfig.to_dict()``.
            - ``dict`` — the ``ApproachConfig.to_dict()`` output directly.

        info: Job metadata.  One of:

            - [ExperimentInfo][SRToolkit.experiments.ExperimentInfo] instance.
            - ``str`` — path to a JSON produced by ``ExperimentInfo.to_dict()``.
            - ``dict`` — the ``ExperimentInfo.to_dict()`` output directly.

        callbacks: Optional callbacks to attach during
            [run][SRToolkit.experiments.ExperimentJob.run].  Accepts a single
            [SRCallbacks][SRToolkit.evaluation.callbacks.SRCallbacks] instance, a list
            of instances, a single serialised callback dict, or a list of dicts.
            Instances are serialised to dicts immediately so that
            [run][SRToolkit.experiments.ExperimentJob.run] always reconstructs fresh
            instances (no shared state between jobs).  Defaults to ``None``.

    Raises:
        ValueError: If ``info.result_path`` is not a directory and does not end
            with ``.json``.
    """
    if isinstance(info, ExperimentInfo):
        self.info = info
    elif isinstance(info, str):
        with open(info) as f:
            self.info = ExperimentInfo.from_dict(json.load(f))
    else:
        self.info = ExperimentInfo.from_dict(dict(info))

    if isinstance(dataset, SR_dataset):
        self._dataset_instance: Optional[SR_dataset] = dataset
        self._dataset_dict: Optional[dict] = None
        self.dataset_name: str = dataset.dataset_name
    elif isinstance(dataset, str):
        with open(dataset) as f:
            self._dataset_dict = json.load(f)
        self._dataset_instance = None
        self.dataset_name = self._dataset_dict.get("dataset_name", "unnamed")
    else:
        self._dataset_dict = dict(dataset)
        self._dataset_instance = None
        self.dataset_name = self._dataset_dict.get("dataset_name", "unnamed")

    if isinstance(approach, SR_approach):
        self._approach_instance: Optional[SR_approach] = approach
        self._approach_dict: Optional[dict] = None
        self.approach_name: str = approach.name
    elif isinstance(approach, str):
        with open(approach) as f:
            self._approach_dict = json.load(f)
        self._approach_instance = None
        self.approach_name = self._approach_dict.get("name", "unknown")
    else:
        self._approach_dict = dict(approach)
        self._approach_instance = None
        self.approach_name = self._approach_dict.get("name", "unknown")

    self.seed = self.info.seed

    if callbacks is None:
        self._callback_configs: Optional[List[dict]] = None
    elif isinstance(callbacks, list):
        self._callback_configs = [cb if isinstance(cb, dict) else cb.to_dict() for cb in callbacks]
    else:
        self._callback_configs = [callbacks if isinstance(callbacks, dict) else callbacks.to_dict()]

    if os.path.isdir(self.info.result_path):
        self.result_path = os.path.join(self.info.result_path, f"exp_{self.seed}.json")
    else:
        _, extension = os.path.splitext(self.info.result_path)
        if extension.lower() != ".json":
            raise ValueError(
                f"Invalid file extension '{extension}'. SR_results can only be saved as '.json' files."
            )
        self.result_path = self.info.result_path

is_complete property

is_complete: bool

True if the result file at result_path already exists on disk.

run

run() -> None

Execute this experiment and save the result to result_path.

Handles adaptation according to SR_approach.adaptation_scope:

• "never": no adaptation.
• "once": loads pre-adapted state from ExperimentInfo's adapted_state_path if a path is set and the file exists, otherwise adapts (and saves if a path is set).
• "experiment": adapts fresh every run.

The result is saved via SR_results.save to result_path.

Source code in SRToolkit/experiments/experiment_grid.py

def run(self) -> None:
    """
    Execute this experiment and save the result to ``result_path``.

    Handles adaptation according to
    [SR_approach.adaptation_scope][SRToolkit.approaches.sr_approach.SR_approach.adaptation_scope]:

    - ``"never"``: no adaptation.
    - ``"once"``: loads pre-adapted state from
      [ExperimentInfo][SRToolkit.experiments.ExperimentInfo]'s ``adapted_state_path``
      if a path is set and the file exists, otherwise adapts (and saves if a path is set).
    - ``"experiment"``: adapts fresh every run.

    The result is saved via
    [SR_results.save][SRToolkit.evaluation.sr_evaluator.SR_results.save] to ``result_path``.
    """
    if self._dataset_instance is not None:
        dataset = self._dataset_instance
    else:
        if self._dataset_dict is None:
            raise ValueError("No dataset provided: pass a dataset instance or a dataset dict.")
        dataset = SR_dataset.from_dict(self._dataset_dict)

    if self._approach_instance is not None:
        approach = self._approach_instance
    else:
        if self._approach_dict is None:
            raise ValueError("No approach provided: pass an approach instance or an approach dict.")
        approach = SR_approach.from_config_dict(self._approach_dict)

    # Honors the contract for callers passing a live instance; a no-op for the
    # grid's own from-config reconstruction (nothing has accumulated yet).
    approach.prepare()

    if approach.adaptation_scope == "once":
        state_path = self.info.adapted_state_path
        if state_path is None:
            approach.adapt(dataset.X, dataset.symbol_library)
        else:
            if os.path.exists(state_path):
                approach.load_adapted_state(state_path)
            else:
                approach.adapt(dataset.X, dataset.symbol_library)
                dir_name = os.path.dirname(state_path)
                if dir_name:
                    os.makedirs(dir_name, exist_ok=True)
                approach.save_adapted_state(state_path)
    elif approach.adaptation_scope == "experiment":
        approach.adapt(dataset.X, dataset.symbol_library)

    evaluator = dataset.create_evaluator(seed=self.info.seed)
    evaluator._experiment_id = f"{self.dataset_name}_{self.approach_name}_{self.info.seed}"
    dispatcher: Optional[CallbackDispatcher] = None
    if self._callback_configs:
        cbs = [SRCallbacks.from_config_dict(d) for d in self._callback_configs]
        dispatcher = CallbackDispatcher(callbacks=cbs)
        evaluator.register_callbacks(dispatcher)

    event = ExperimentEvent(
        dataset_name=self.dataset_name,
        approach_name=self.approach_name,
        max_evaluations=evaluator.max_evaluations,
        success_threshold=evaluator.success_threshold,
        seed=self.info.seed,
    )
    if dispatcher is not None:
        dispatcher.on_experiment_start(event)

    start_time = time.monotonic()
    approach.search(evaluator, self.info.seed)
    elapsed = time.monotonic() - start_time

    results = evaluator.get_results(self.approach_name, self.info.top_k)
    results.results[-1].wall_time = elapsed
    if dispatcher is not None:
        dispatcher.on_experiment_end(event, results.results[-1])
    results.save(self.result_path)

ExperimentGrid

ExperimentGrid(datasets: Union[SR_dataset, List[Union[SR_dataset, SR_benchmark]], SR_benchmark], approaches: Union[SR_approach, List[SR_approach]], num_experiments: int, results_dir: str, initial_seed: int = 0, top_k: int = 20, adapted_states: Optional[Dict[str, Dict[str, str]]] = None, callbacks: Optional[Union[SRCallbacks, List[SRCallbacks]]] = None)

Defines and manages a grid of symbolic regression experiments across multiple datasets and approaches.

Each experiment is an independent ExperimentJob that runs one approach on one dataset with one seed. Jobs can be executed locally (iterate and call .run()) or on HPC clusters (generate a commands file with save_commands).

The grid spec is persisted via save and reloaded via load. Results are saved per-experiment to results_dir/{dataset}/{approach}/exp_{seed}.json, so parallel workers never write to the same file.

Seed scheme: job i (0-indexed) receives seed = initial_seed + i.

Examples:

>>> from SRToolkit.dataset import Nguyen
>>> from SRToolkit.approaches import ProGED
>>> from SRToolkit.experiments import ExperimentGrid
>>> bench = Nguyen()
>>> approach = ProGED()
>>> grid = ExperimentGrid(bench, approach, num_experiments=3,
...                       results_dir="/tmp/sr_run")

Parameters:

Name	Type	Description	Default
datasets	Union[SR_dataset, List[Union[SR_dataset, SR_benchmark]], SR_benchmark]	One or more datasets to run experiments on. Accepts a single SR_dataset, a SR_benchmark (all datasets in the benchmark are included), or a list containing SR_dataset-s and SR_dataset-s.	required
approaches	Union[SR_approach, List[SR_approach]]	One or more SR approaches. Accepts a single SR_approach or a list of them.	required
num_experiments	int	Number of independent experiments per (dataset, approach) pair.	required
results_dir	str	Root directory where all results and grid metadata are stored.	required
initial_seed	int	Seed for the first experiment. Subsequent experiments use initial_seed + 1, initial_seed + 2, etc.	0
top_k	int	Number of top expressions to highlight per experiment.	20
adapted_states	Optional[Dict[str, Dict[str, str]]]	Optional mapping {approach_name: {dataset_name: path}} providing paths for pre-adapted state files. Jobs for listed (approach, dataset) pairs will load state from the given path if it exists, or adapt and save to it otherwise. Pairs not listed will adapt on every run without saving. Several datasets may deliberately share one path for approaches that adapt per symbol space rather than per dataset (e.g. EDHiE): the path is then adapted once on the first dataset listed for it and the others reuse that state. An informational warning lists any such shared paths so an accidental collision can be spotted; give each dataset a distinct path if adaptation must differ per dataset.	None
callbacks	Optional[Union[SRCallbacks, List[SRCallbacks]]]	Optional callback or list of callbacks forwarded to every job created by create_jobs. Callbacks are serialised to dicts immediately so that each job reconstructs fresh instances in run (no shared state between jobs). They are inlined into the grid recipe by to_dict, so they travel inside grid.json — both save and the run_job CLI reconstruct them from that single file, with no extra files or flags. Defaults to None.	None

Source code in SRToolkit/experiments/experiment_grid.py

def __init__(
    self,
    datasets: Union[SR_dataset, List[Union[SR_dataset, SR_benchmark]], SR_benchmark],
    approaches: Union[SR_approach, List[SR_approach]],
    num_experiments: int,
    results_dir: str,
    initial_seed: int = 0,
    top_k: int = 20,
    adapted_states: Optional[Dict[str, Dict[str, str]]] = None,
    callbacks: Optional[Union[SRCallbacks, List[SRCallbacks]]] = None,
) -> None:
    self.num_experiments = num_experiments
    self.results_dir = results_dir
    self.initial_seed = initial_seed
    self.top_k = top_k
    self._adapted_states: Dict[str, Dict[str, str]] = adapted_states or {}

    if callbacks is None:
        self.callback_configs: Optional[List[dict]] = None
    elif isinstance(callbacks, list):
        self.callback_configs = [cb.to_dict() for cb in callbacks]
    else:
        self.callback_configs = [callbacks.to_dict()]

    # Build approach configs (plain serialisable dicts, no instance caching)
    self.approach_configs: List[dict] = []
    if isinstance(approaches, SR_approach):
        approaches = [approaches]
    for approach in approaches:
        self.add_approach(approach)

    # Serialise all datasets eagerly into config dicts (no arrays embedded)
    self.datasets: Dict[str, dict] = dict()
    if isinstance(datasets, list):
        for ds in datasets:
            self.add_dataset(ds)
    else:
        self.add_dataset(datasets)

add_approach

add_approach(approach: SR_approach) -> None

Add an approach to the grid.

The approach's config is serialised immediately (its adaptation_scope is recorded alongside) so the grid never caches a live instance. Safe to call after from_dict to extend a loaded grid.

Parameters:

Name	Type	Description	Default
approach	SR_approach	The SR_approach to add.	required

Raises:

Type	Description
ValueError	If an approach with the same name is already in the grid (names map to result subdirectories, so duplicates would collide).

Source code in SRToolkit/experiments/experiment_grid.py

def add_approach(self, approach: SR_approach) -> None:
    """
    Add an approach to the grid.

    The approach's config is serialised immediately (its ``adaptation_scope`` is
    recorded alongside) so the grid never caches a live instance. Safe to call after
    [from_dict][SRToolkit.experiments.ExperimentGrid.from_dict] to extend a loaded grid.

    Args:
        approach: The [SR_approach][SRToolkit.approaches.sr_approach.SR_approach] to add.

    Raises:
        ValueError: If an approach with the same ``name`` is already in the grid
            (names map to result subdirectories, so duplicates would collide).
    """
    cfg = approach.config.to_dict()
    cfg["adaptation_scope"] = approach.adaptation_scope
    name = cfg["name"]
    if any(c["name"] == name for c in self.approach_configs):
        raise ValueError(
            f"[ExperimentGrid] Duplicate approach name '{name}'. "
            f"Rename one of the approaches before adding it (e.g. set its config name)."
        )
    self.approach_configs.append(cfg)

add_dataset

add_dataset(dataset: Union[SR_dataset, SR_benchmark]) -> None

Add a dataset, or every dataset in a benchmark, to the grid.

Each dataset is serialised immediately to its config dict (no data arrays embedded). Safe to call after from_dict to extend a loaded grid.

Parameters:

Name	Type	Description	Default
dataset	Union[SR_dataset, SR_benchmark]	An SR_dataset or an SR_benchmark (all of its datasets are added).	required

Raises:

Type	Description
ValueError	If a dataset name already exists in the grid, or if dataset is neither an SR_dataset nor an SR_benchmark.

Source code in SRToolkit/experiments/experiment_grid.py

def add_dataset(self, dataset: Union[SR_dataset, SR_benchmark]) -> None:
    """
    Add a dataset, or every dataset in a benchmark, to the grid.

    Each dataset is serialised immediately to its config dict (no data arrays
    embedded). Safe to call after
    [from_dict][SRToolkit.experiments.ExperimentGrid.from_dict] to extend a loaded grid.

    Args:
        dataset: An [SR_dataset][SRToolkit.dataset.sr_dataset.SR_dataset] or an
            [SR_benchmark][SRToolkit.dataset.sr_benchmark.SR_benchmark] (all of its
            datasets are added).

    Raises:
        ValueError: If a dataset name already exists in the grid, or if ``dataset``
            is neither an ``SR_dataset`` nor an ``SR_benchmark``.
    """
    if isinstance(dataset, SR_benchmark):
        for name in dataset.list_datasets(verbose=False):
            self._add_one_dataset(dataset.create_dataset(name), name)
    elif isinstance(dataset, SR_dataset):
        self._add_one_dataset(dataset, dataset.dataset_name)
    else:
        raise ValueError(f"[ExperimentGrid] datasets must be an SR_dataset or SR_benchmark, got {type(dataset)}")

materialize_data

materialize_data() -> None

Materialise every dataset's data into the local cache, once, in this process.

Reconstructing each dataset via SR_dataset.from_dict triggers its data_source to download or sample the arrays into the shared data cache (a no-op for datasets already cached). Because this runs single-process, it is race-free — the intended counterpart to running it before parallel workers, which then only read the cache rather than each materialising the same dataset.

save_commands calls this by default, so the data is obtained once on the machine that writes the commands (which, on a cluster, is typically the only node with network access for UrlSource datasets).

Raises:

Type	Description
FileNotFoundError	If a null-source dataset has no data in the local cache (there is nothing to download or sample, so it cannot be materialised here).

Source code in SRToolkit/experiments/experiment_grid.py

def materialize_data(self) -> None:
    """
    Materialise every dataset's data into the local cache, once, in this process.

    Reconstructing each dataset via
    [SR_dataset.from_dict][SRToolkit.dataset.sr_dataset.SR_dataset.from_dict] triggers
    its ``data_source`` to download or sample the arrays into the shared data cache (a
    no-op for datasets already cached). Because this runs single-process, it is
    race-free — the intended counterpart to running it before parallel workers, which
    then only *read* the cache rather than each materialising the same dataset.

    [save_commands][SRToolkit.experiments.ExperimentGrid.save_commands] calls this by
    default, so the data is obtained once on the machine that writes the commands
    (which, on a cluster, is typically the only node with network access for
    ``UrlSource`` datasets).

    Raises:
        FileNotFoundError: If a ``null``-source dataset has no data in the local cache
            (there is nothing to download or sample, so it cannot be materialised here).
    """
    for dataset_name in self.datasets:
        SR_dataset.from_dict(self.datasets[dataset_name])

adapt_one

adapt_one(approach_name: str, dataset_name: str, force: bool = False) -> None

Adapt a single (approach, dataset) pair and persist its state.

This is the unit the per-pair adaptation commands in prepare.txt invoke (via the adapt CLI with --dataset/--approach), so independent pairs can adapt in parallel across a cluster. It loads the dataset, calls adapt, and saves the state via save_adapted_state.

Silently returns (no-op) when there is nothing to do: the approach is not "once"-scope, or has no registered state path for this dataset. By default it also no-ops when the state file already exists; pass force=True to re-adapt and overwrite it (e.g. after the adaptation logic or the dataset changed, or to replace a corrupt state). Because the dataset is named explicitly here, force is unambiguous even when several datasets share one state path — it adapts on this dataset and overwrites.

Parameters:

Name	Type	Description	Default
approach_name	str	Name of an approach in this grid.	required
dataset_name	str	Name of a dataset in this grid.	required
force	bool	Re-adapt and overwrite an existing state file instead of skipping it. Defaults to False.	False

Raises:

Type	Description
ValueError	If approach_name is not in the grid.
KeyError	If dataset_name is not in the grid.

Source code in SRToolkit/experiments/experiment_grid.py

def adapt_one(self, approach_name: str, dataset_name: str, force: bool = False) -> None:
    """
    Adapt a single (approach, dataset) pair and persist its state.

    This is the unit the per-pair adaptation commands in ``prepare.txt`` invoke (via the
    ``adapt`` CLI with ``--dataset``/``--approach``), so independent pairs can adapt in
    parallel across a cluster. It loads the dataset, calls
    [adapt][SRToolkit.approaches.sr_approach.SR_approach.adapt], and saves the state via
    [save_adapted_state][SRToolkit.approaches.sr_approach.SR_approach.save_adapted_state].

    Silently returns (no-op) when there is nothing to do: the approach is not
    ``"once"``-scope, or has no registered state path for this dataset. By default it also
    no-ops when the state file already exists; pass ``force=True`` to re-adapt and overwrite
    it (e.g. after the adaptation logic or the dataset changed, or to replace a corrupt
    state). Because the dataset is named explicitly here, ``force`` is unambiguous even when
    several datasets share one state path — it adapts on *this* dataset and overwrites.

    Args:
        approach_name: Name of an approach in this grid.
        dataset_name: Name of a dataset in this grid.
        force: Re-adapt and overwrite an existing state file instead of skipping it.
            Defaults to ``False``.

    Raises:
        ValueError: If ``approach_name`` is not in the grid.
        KeyError: If ``dataset_name`` is not in the grid.
    """
    try:
        approach_config = next(c for c in self.approach_configs if c["name"] == approach_name)
    except StopIteration:
        raise ValueError(f"[ExperimentGrid] No approach named {approach_name!r} in this grid.") from None
    if approach_config.get("adaptation_scope", "never") != "once":
        return
    state_path = self._get_adapted_state_ref_path(approach_name, dataset_name)
    if state_path is None or (os.path.exists(state_path) and not force):
        return
    dataset = SR_dataset.from_dict(self.datasets[dataset_name])
    approach = SR_approach.from_config_dict(approach_config)
    approach.adapt(dataset.X, dataset.symbol_library)
    approach.save_adapted_state(state_path)

adapt_if_missing

adapt_if_missing()

Pre-adapt all adaptation_scope="once" approaches where the state file is absent.

For each (approach, dataset) pair whose state file does not yet exist on disk, this method loads the dataset, calls adapt once, then persists the state via save_adapted_state. Pairs whose state file already exists are skipped.

Approaches whose adaptation_scope is not "once", or that have no entry in the adapted_states mapping passed at construction, are skipped entirely.

This adapts every pair sequentially in this process. To distribute the work, use save_commands, which emits one adapt_one command per pair into prepare.txt.

Source code in SRToolkit/experiments/experiment_grid.py

def adapt_if_missing(self):
    """
    Pre-adapt all ``adaptation_scope="once"`` approaches where the state file is absent.

    For each (approach, dataset) pair whose state file does not yet exist on disk,
    this method loads the dataset, calls
    [adapt][SRToolkit.approaches.sr_approach.SR_approach.adapt] once, then persists the
    state via
    [save_adapted_state][SRToolkit.approaches.sr_approach.SR_approach.save_adapted_state].
    Pairs whose state file already exists are skipped.

    Approaches whose ``adaptation_scope`` is not ``"once"``, or that have no entry in
    the ``adapted_states`` mapping passed at construction, are skipped entirely.

    This adapts every pair sequentially in this process. To distribute the work, use
    [save_commands][SRToolkit.experiments.ExperimentGrid.save_commands], which emits one
    [adapt_one][SRToolkit.experiments.ExperimentGrid.adapt_one] command per pair into
    ``prepare.txt``.
    """
    for approach_name, dataset_name in self._pairs_needing_adaptation():
        self.adapt_one(approach_name, dataset_name)

build_job

build_job(dataset_name: str, approach_name: str, seed: int) -> ExperimentJob

Construct the single ExperimentJob for one (dataset, approach, seed) triple, using this grid's stored configs.

This is the unit of work both create_jobs and the run_job CLI build from, so a worker only needs the grid file plus the three identifiers — no per-job config files on disk.

Parameters:

Name	Type	Description	Default
dataset_name	str	Name of a dataset in this grid.	required
approach_name	str	Name of an approach in this grid.	required
seed	int	Random seed for the run.	required

Returns:

Type	Description
ExperimentJob	The corresponding ExperimentJob.

Raises:

Type	Description
KeyError	If dataset_name is not in the grid.
ValueError	If approach_name is not in the grid.

Source code in SRToolkit/experiments/experiment_grid.py

def build_job(self, dataset_name: str, approach_name: str, seed: int) -> ExperimentJob:
    """
    Construct the single [ExperimentJob][SRToolkit.experiments.ExperimentJob] for one
    (dataset, approach, seed) triple, using this grid's stored configs.

    This is the unit of work both
    [create_jobs][SRToolkit.experiments.ExperimentGrid.create_jobs] and the
    ``run_job`` CLI build from, so a worker only needs the grid file plus the three
    identifiers — no per-job config files on disk.

    Args:
        dataset_name: Name of a dataset in this grid.
        approach_name: Name of an approach in this grid.
        seed: Random seed for the run.

    Returns:
        The corresponding [ExperimentJob][SRToolkit.experiments.ExperimentJob].

    Raises:
        KeyError: If ``dataset_name`` is not in the grid.
        ValueError: If ``approach_name`` is not in the grid.
    """
    if dataset_name not in self.datasets:
        raise KeyError(f"[ExperimentGrid] No dataset named {dataset_name!r} in this grid.")
    dataset_dict = self.datasets[dataset_name]
    try:
        approach_config = next(c for c in self.approach_configs if c["name"] == approach_name)
    except StopIteration:
        raise ValueError(f"[ExperimentGrid] No approach named {approach_name!r} in this grid.") from None
    info = ExperimentInfo(
        seed=seed,
        result_path=os.path.join(self.results_dir, dataset_name, approach_name, f"exp_{seed}.json"),
        top_k=self.top_k,
        adapted_state_path=self._get_adapted_state_ref_path(approach_name, dataset_name),
    )
    return ExperimentJob(dataset=dataset_dict, approach=approach_config, info=info, callbacks=self.callback_configs)

create_jobs

create_jobs(skip_completed: bool = True) -> List[ExperimentJob]

Return the list of ExperimentJob instances for this grid.

Does not trigger adaptation — call adapt_if_missing first if any approach has adaptation_scope="once". It also does not materialise data: running the returned jobs sequentially warms the cache lazily on first use, but if you dispatch them in parallel yourself, call materialize_data first so workers don't race to materialise the same dataset. (save_commands does both for you.)

Parameters:

Name	Type	Description	Default
skip_completed	bool	If True (default), omit jobs whose result file (exp_{seed}.json) already exists on disk.	True

Returns:

Type	Description
List[ExperimentJob]	List of jobs, one per (dataset, approach, seed) triple that has not yet completed.

Source code in SRToolkit/experiments/experiment_grid.py

def create_jobs(self, skip_completed: bool = True) -> List[ExperimentJob]:
    """
    Return the list of [ExperimentJob][SRToolkit.experiments.ExperimentJob] instances for
    this grid.

    Does **not** trigger adaptation — call
    [adapt_if_missing][SRToolkit.experiments.ExperimentGrid.adapt_if_missing] first if any
    approach has ``adaptation_scope="once"``. It also does **not** materialise data:
    running the returned jobs sequentially warms the cache lazily on first use, but if you
    dispatch them in parallel yourself, call
    [materialize_data][SRToolkit.experiments.ExperimentGrid.materialize_data] first so
    workers don't race to materialise the same dataset.
    ([save_commands][SRToolkit.experiments.ExperimentGrid.save_commands] does both for you.)

    Args:
        skip_completed: If ``True`` (default), omit jobs whose result file
            (``exp_{seed}.json``) already exists on disk.

    Returns:
        List of jobs, one per (dataset, approach, seed) triple that has not yet completed.
    """
    jobs: List[ExperimentJob] = []
    for approach_config in self.approach_configs:
        for dataset_name in self.datasets:
            for i in range(self.num_experiments):
                job = self.build_job(dataset_name, approach_config["name"], self.initial_seed + i)
                if skip_completed and job.is_complete:
                    continue
                jobs.append(job)
    return jobs

save_commands

save_commands(path: str, python_executable: str = 'python', skip_completed: bool = True, materialize_data: bool = True) -> Optional[str]

Write the experiment commands file (and, if needed, a prepare commands file).

Calls save first to persist the grid to results_dir/grid.json. Every command line references that single file plus the identifiers needed to rebuild the work — no per-dataset, per-approach, or per-job files are written.

The output is split into (up to) two files so a run is correct on any platform without scheduler-specific dependency tricks — run the prepare file to completion, then the experiments file:

• path — one run_job line per pending (dataset, approach, seed) triple::

  python -m SRToolkit.experiments run_job \ --grid /path/grid.json --dataset NG-1 --approach ProGED --seed 0

• prepare_<name> (sibling of path) — written only when one or more adaptation_scope="once" approaches still need state, one parallel-safe line per (approach, dataset) pair::

  python -m SRToolkit.experiments adapt \ --grid /path/grid.json --dataset NG-1 --approach ProGED

Data is materialised, once, in this process (materialize_data=True): every dataset's data_source is resolved into the shared cache before any command runs. This is single-process and therefore race-free, fetches each UrlSource exactly once, and happens on the machine writing the commands — typically the only node with network access on a cluster. Workers (adaptation and experiments) then only read the warm cache.

Parameters:

Name	Type	Description	Default
path	str	File path to write the experiment commands to.	required
python_executable	str	Python executable to use in the commands.	'python'
skip_completed	bool	If True (default), omit already-completed jobs.	True
materialize_data	bool	If True (default), resolve every dataset's data into the local cache before writing commands. Set False to only (re)generate the command text without touching data (e.g. the cache is already warm, or this machine cannot reach the data).	True

Returns:

Type	Description
Optional[str]	The path to the prepare commands file if one was written, otherwise None.

Source code in SRToolkit/experiments/experiment_grid.py

def save_commands(
    self,
    path: str,
    python_executable: str = "python",
    skip_completed: bool = True,
    materialize_data: bool = True,
) -> Optional[str]:
    """
    Write the experiment commands file (and, if needed, a ``prepare`` commands file).

    Calls [save][SRToolkit.experiments.ExperimentGrid.save] first to persist the grid
    to ``results_dir/grid.json``. Every command line references that single file plus
    the identifiers needed to rebuild the work — no per-dataset, per-approach, or
    per-job files are written.

    The output is split into (up to) two files so a run is correct on any platform
    without scheduler-specific dependency tricks — **run the prepare file to completion,
    then the experiments file**:

    - ``path`` — one ``run_job`` line per pending (dataset, approach, seed) triple::

          python -m SRToolkit.experiments run_job \\
              --grid /path/grid.json --dataset NG-1 --approach ProGED --seed 0

    - ``prepare_<name>`` (sibling of ``path``) — written **only** when one or more
      ``adaptation_scope="once"`` approaches still need state, one parallel-safe line
      per (approach, dataset) pair::

          python -m SRToolkit.experiments adapt \\
              --grid /path/grid.json --dataset NG-1 --approach ProGED

    Data is materialised, once, in this process (``materialize_data=True``):
    every dataset's ``data_source`` is resolved into the shared cache before any command
    runs. This is single-process and therefore race-free, fetches each ``UrlSource``
    exactly once, and happens on the machine writing the commands — typically the only
    node with network access on a cluster. Workers (adaptation and experiments) then
    only *read* the warm cache.

    Args:
        path: File path to write the experiment commands to.
        python_executable: Python executable to use in the commands.
        skip_completed: If ``True`` (default), omit already-completed jobs.
        materialize_data: If ``True`` (default), resolve every dataset's data into the
            local cache before writing commands. Set ``False`` to only (re)generate the
            command text without touching data (e.g. the cache is already warm, or this
            machine cannot reach the data).

    Returns:
        The path to the prepare commands file if one was written, otherwise ``None``.
    """
    self.save()
    if materialize_data:
        self.materialize_data()
    grid_path = shlex.quote(os.path.join(self.results_dir, "grid.json"))

    prepare_path: Optional[str] = None
    prepare_pairs = self._pairs_needing_adaptation()
    if prepare_pairs:
        abs_path = os.path.abspath(path)
        prepare_path = os.path.join(os.path.dirname(abs_path), "prepare_" + os.path.basename(abs_path))
        prepare_lines = [
            "# Adaptation prerequisites — run ALL of these to completion before the experiment commands.",
            "# Lines are independent and may run in parallel.",
        ]
        prepare_lines += [
            f"{python_executable} -m SRToolkit.experiments adapt "
            f"--grid {grid_path} "
            f"--dataset {shlex.quote(dataset_name)} "
            f"--approach {shlex.quote(approach_name)}"
            for approach_name, dataset_name in prepare_pairs
        ]

    header = [f"# results_dir: {self.results_dir}"]
    if prepare_path is not None:
        header.append(f"# Run '{os.path.basename(prepare_path)}' to completion FIRST, then these commands.")
    lines = list(header)
    for approach_config in self.approach_configs:
        for dataset_name in self.datasets:
            approach_name = approach_config["name"]
            for i in range(self.num_experiments):
                seed = self.initial_seed + i
                if skip_completed and self.build_job(dataset_name, approach_name, seed).is_complete:
                    continue
                lines.append(
                    f"{python_executable} -m SRToolkit.experiments run_job "
                    f"--grid {grid_path} "
                    f"--dataset {shlex.quote(dataset_name)} "
                    f"--approach {shlex.quote(approach_name)} "
                    f"--seed {seed}"
                )

    out_dir = os.path.dirname(os.path.abspath(path))
    if out_dir:
        os.makedirs(out_dir, exist_ok=True)
    with open(path, "w") as f:
        f.write("\n".join(lines) + "\n")
    if prepare_path is not None:
        with open(prepare_path, "w") as f:
            f.write("\n".join(prepare_lines) + "\n")
    return prepare_path

progress

progress() -> None

Print a dataset × approach progress table to stdout.

Each cell shows done/total experiments completed for that pair, based on the presence of the per-experiment exp_{seed}.json files on disk.

Example output::

Dataset        ProGED    EDHiE
-----------   -------   ------
NG-1             5/5      3/5
NG-2             2/5      0/5

Source code in SRToolkit/experiments/experiment_grid.py

def progress(self) -> None:
    """
    Print a dataset × approach progress table to stdout.

    Each cell shows ``done/total`` experiments completed for that pair, based on
    the presence of the per-experiment ``exp_{seed}.json`` files on disk.

    Example output::

        Dataset        ProGED    EDHiE
        -----------   -------   ------
        NG-1             5/5      3/5
        NG-2             2/5      0/5
    """
    dataset_names = list(self.datasets.keys())
    approach_names = [cfg["name"] for cfg in self.approach_configs]

    total_str = str(self.num_experiments)
    ds_w = max(len(n) for n in dataset_names + ["Dataset"]) + 2
    col_w = max(len(n) for n in approach_names + [f"{total_str}/{total_str}"]) + 2

    header = f"{'Dataset':<{ds_w}}" + "".join(f"{ap:>{col_w}}" for ap in approach_names)
    separator = "-" * ds_w + "".join("-" * col_w for _ in approach_names)
    print(header)
    print(separator)

    for dataset_name in dataset_names:
        row = f"{dataset_name:<{ds_w}}"
        for approach_name in approach_names:
            done = sum(
                1
                for i in range(self.num_experiments)
                if os.path.exists(
                    os.path.join(
                        self.results_dir,
                        dataset_name,
                        approach_name,
                        f"exp_{self.initial_seed + i}.json",
                    )
                )
            )
            row += f"{done}/{self.num_experiments}".rjust(col_w)
        print(row)

load_results

load_results(dataset_name: str, approach_name: str) -> SR_results

Load and merge all completed per-experiment results for a (dataset, approach) pair.

Examples:

>>> results = grid.load_results("Nguyen-1", "ProGED")
>>> len(results)  # number of completed experiments
5

Parameters:

Name	Type	Description	Default
dataset_name	str	Name of the dataset.	required
approach_name	str	Name of the approach.	required

Returns:

Type	Description
SR_results	An SR_results object containing one EvalResult per completed experiment. Returns an empty SR_results if no experiments have completed yet.

Source code in SRToolkit/experiments/experiment_grid.py

def load_results(self, dataset_name: str, approach_name: str) -> SR_results:
    """
    Load and merge all completed per-experiment results for a (dataset, approach) pair.

    Examples:
        >>> results = grid.load_results("Nguyen-1", "ProGED")  # doctest: +SKIP
        >>> len(results)  # number of completed experiments  # doctest: +SKIP
        5

    Args:
        dataset_name: Name of the dataset.
        approach_name: Name of the approach.

    Returns:
        An [SR_results][SRToolkit.evaluation.sr_evaluator.SR_results] object containing one [EvalResult][SRToolkit.evaluation.result_augmentation.EvalResult] per completed experiment.  Returns an empty ``SR_results`` if no experiments have completed yet.
    """
    merged = SR_results()
    for i in range(self.num_experiments):
        seed = self.initial_seed + i
        result_path = os.path.join(self.results_dir, dataset_name, approach_name, f"exp_{seed}.json")
        if os.path.exists(result_path):
            merged += SR_results.load(result_path)
    return merged

to_dict

to_dict() -> dict

Serialize the grid to a single self-contained, JSON-safe dict — the recipe.

The returned dict inlines every dataset config, approach config, and callback config alongside the run metadata (num_experiments, initial_seed, top_k). It is the portable, one-file form of the grid: pass it (or a JSON file holding it) straight to from_dict.

Unlike save, this carries no results and no machine-local execution state: results_dir is supplied at load time (anchored to the file's location) and the absolute adapted_states paths are dropped — the per-approach adaptation_scope already travels inside each approach config, so a recipient re-adapts to their own paths.

Dataset configs are the full per-dataset recipe (data_source + samplers + ground truth), not bare references, and contain no data arrays — those are regenerated or downloaded from each dataset's data_source on first use. A null-source dataset's data must already be present in the recipient's cache; ship it via export otherwise.

Returns:

Type	Description
dict	A JSON-safe dict with format_version 2, suitable for json.dump.

Source code in SRToolkit/experiments/experiment_grid.py

def to_dict(self) -> dict:
    """
    Serialize the grid to a single self-contained, JSON-safe dict — the *recipe*.

    The returned dict inlines every dataset config, approach config, and callback
    config alongside the run metadata (``num_experiments``, ``initial_seed``,
    ``top_k``). It is the portable, one-file form of the grid: pass it (or a JSON
    file holding it) straight to [from_dict][SRToolkit.experiments.ExperimentGrid.from_dict].

    Unlike [save][SRToolkit.experiments.ExperimentGrid.save], this carries **no
    results** and **no machine-local execution state**: ``results_dir`` is supplied
    at load time (anchored to the file's location) and the absolute
    ``adapted_states`` paths are dropped — the per-approach ``adaptation_scope``
    already travels inside each approach config, so a recipient re-adapts to their
    own paths.

    Dataset configs are the full per-dataset recipe (``data_source`` + samplers +
    ground truth), not bare references, and contain no data arrays — those are
    regenerated or downloaded from each dataset's ``data_source`` on first use. A
    ``null``-source dataset's data must already be present in the recipient's cache;
    ship it via [export][SRToolkit.experiments.ExperimentGrid.export] otherwise.

    Returns:
        A JSON-safe dict with ``format_version`` ``2``, suitable for ``json.dump``.
    """
    from SRToolkit import __version__

    return {
        "format_version": 2,
        "type": "ExperimentGrid",
        "srtk_version": __version__,
        "num_experiments": self.num_experiments,
        "initial_seed": self.initial_seed,
        "top_k": self.top_k,
        "datasets": {name: dict(cfg) for name, cfg in self.datasets.items()},
        "approaches": [dict(cfg) for cfg in self.approach_configs],
        "callbacks": ([dict(cfg) for cfg in self.callback_configs] if self.callback_configs is not None else None),
    }

from_dict classmethod

from_dict(config: Union[dict, str, Path], results_dir: Optional[str] = None, adapted_states: Optional[Dict[str, Dict[str, str]]] = None) -> ExperimentGrid

Reconstruct an ExperimentGrid from a recipe produced by to_dict.

Every embedded config (approaches, callbacks, and the samplers / sources / constraints nested inside each dataset config) is run through _auto_bind so that *_class paths pointing at an installed .srtk bundle resolve automatically. Dataset and approach instances are not created here — they are reconstructed lazily when jobs run. As a courtesy, any *_class path that cannot be imported on this machine triggers a warning naming the missing class; the hard failure is deferred to job-run time.

Parameters:

Name	Type	Description	Default
config	Union[dict, str, Path]	The recipe. One of: dict — the to_dict output directly. results_dir is then required. str / Path — path to a JSON file holding that dict. The directory containing the file becomes the grid's results_dir unless results_dir is given explicitly.	required
results_dir	Optional[str]	Where results are read from and written to. Overrides the file-anchored default; required when config is a dict.	None
adapted_states	Optional[Dict[str, Dict[str, str]]]	Optional mapping {approach_name: {dataset_name: path}} providing paths for pre-adapted state files. Jobs for listed (approach, dataset) pairs will load state from the given path if it exists, or adapt and save to it otherwise. Pairs not listed will adapt on every run without saving. Several datasets may deliberately share one path for approaches that adapt per symbol space rather than per dataset (e.g. EDHiE): the path is then adapted once on the first dataset listed for it and the others reuse that state. An informational warning lists any such shared paths so an accidental collision can be spotted; give each dataset a distinct path if adaptation must differ per dataset.	None

Returns:

Type	Description
ExperimentGrid	A fully configured ExperimentGrid.

Raises:

Type	Description
ValueError	If the config is not an ExperimentGrid recipe, its format_version is unsupported, or results_dir is omitted when loading from a dict.

Source code in SRToolkit/experiments/experiment_grid.py

@classmethod
def from_dict(
    cls,
    config: Union[dict, str, Path],
    results_dir: Optional[str] = None,
    adapted_states: Optional[Dict[str, Dict[str, str]]] = None,
) -> "ExperimentGrid":
    """
    Reconstruct an [ExperimentGrid][SRToolkit.experiments.ExperimentGrid] from a
    recipe produced by [to_dict][SRToolkit.experiments.ExperimentGrid.to_dict].

    Every embedded config (approaches, callbacks, and the samplers / sources /
    constraints nested inside each dataset config) is run through ``_auto_bind`` so
    that ``*_class`` paths pointing at an installed ``.srtk`` bundle resolve
    automatically. Dataset and approach **instances are not created here** — they
    are reconstructed lazily when jobs run. As a courtesy, any ``*_class`` path that
    cannot be imported on this machine triggers a warning naming the missing class;
    the hard failure is deferred to job-run time.

    Args:
        config: The recipe. One of:

            - ``dict`` — the [to_dict][SRToolkit.experiments.ExperimentGrid.to_dict]
              output directly. ``results_dir`` is then **required**.
            - ``str`` / ``Path`` — path to a JSON file holding that dict. The
              directory containing the file becomes the grid's ``results_dir``
              unless ``results_dir`` is given explicitly.

        results_dir: Where results are read from and written to. Overrides the
            file-anchored default; required when ``config`` is a dict.
        adapted_states: Optional mapping ``{approach_name: {dataset_name: path}}``
            providing paths for pre-adapted state files. Jobs for listed
            (approach, dataset) pairs will load state from the given path if it
            exists, or adapt and save to it otherwise. Pairs not listed will adapt
            on every run without saving.  Several datasets may deliberately share one
            ``path`` for approaches that adapt per symbol space rather than per dataset
            (e.g. EDHiE): the path is then adapted once on the first dataset listed for
            it and the others reuse that state. An informational warning lists any such
            shared paths so an accidental collision can be spotted; give each dataset a
            distinct ``path`` if adaptation must differ per dataset.

    Returns:
        A fully configured ``ExperimentGrid``.

    Raises:
        ValueError: If the config is not an ``ExperimentGrid`` recipe, its
            ``format_version`` is unsupported, or ``results_dir`` is omitted when
            loading from a dict.
    """
    default_results_dir: Optional[str] = None
    if isinstance(config, (str, Path)):
        file_path = os.path.abspath(str(config))
        with open(file_path) as f:
            d = json.load(f)
        default_results_dir = os.path.dirname(file_path)
    else:
        d = dict(config)

    if d.get("type") != "ExperimentGrid":
        raise ValueError(
            f"[ExperimentGrid.from_dict] Config is not an ExperimentGrid recipe (type={d.get('type')!r})."
        )
    fmt = d.get("format_version")
    if fmt != 2:
        raise ValueError(f"[ExperimentGrid.from_dict] Unsupported format_version: {fmt!r}. Expected 2.")
    _warn_if_outdated(d.get("srtk_version"))

    resolved_results_dir = results_dir if results_dir is not None else default_results_dir
    if resolved_results_dir is None:
        raise ValueError(
            "[ExperimentGrid.from_dict] results_dir is required when loading from a dict "
            "(there is no file location to anchor it to)."
        )

    grid = cls.__new__(cls)
    grid.num_experiments = d["num_experiments"]
    grid.results_dir = os.path.abspath(resolved_results_dir)
    grid.initial_seed = d["initial_seed"]
    grid.top_k = d["top_k"]
    grid._adapted_states = adapted_states or {}
    grid.datasets = {name: _safe_auto_bind(cfg) for name, cfg in d["datasets"].items()}
    grid.approach_configs = [_safe_auto_bind(cfg) for cfg in d["approaches"]]
    callbacks = d.get("callbacks")
    grid.callback_configs = [_safe_auto_bind(cfg) for cfg in callbacks] if callbacks is not None else None

    _warn_missing_class_deps(list(grid.datasets.values()) + grid.approach_configs + (grid.callback_configs or []))
    return grid

export

export(path: Union[str, Path], include_results: bool = False, additional_bundles: Optional[List[Union[str, Path]]] = None, strict: bool = False) -> None

Write a self-contained, shareable .zip gathering everything a recipient needs.

Unlike to_dict (a recipe that assumes the recipient can reach the data and code), export bundles the dependencies that wouldn't otherwise travel into one archive whose entries are:

grid.json            # the recipe (to_dict output), annotated with _bundle refs
data/<name>.zip      # only for datasets whose data isn't otherwise reachable
bundles/<name>.srtk  # only when custom (non-SRToolkit) classes are referenced
results/...          # only when include_results=True
MANIFEST.md          # inventory + recipient instructions

A dataset's data is archived when its source is null (data lives only in the cache) or a seedless sample source (regeneration wouldn't reproduce the exact numbers). url and seeded sample datasets are left referenced — the recipient downloads or regenerates them.

Custom code. Any non-SRToolkit class referenced anywhere in the grid must be covered by a .srtk bundle so the recipient can import it. Two sources are used, in order:

• already-installed bundles — a referenced class whose import path falls under an installed bundle is re-packed automatically from the installed source files (carrying its declared python_deps / srtk_min_version);
• additional_bundles — .srtk files you built yourself with pack for code that isn't installed as a bundle. Because you build them, multi-file implementations, dependencies, and the real version all travel correctly. export never tries to guess a bundle from a loose source file.

The recipe is annotated with _bundle / _version so the recipient's bind machinery relocates each class on load. Any custom class covered by neither source is not packed: it is listed in MANIFEST.md and a warning names each class and its source file. Pass strict=True to raise instead of warning.

The bundles are not installed automatically. from_export extracts them next to the archive so the recipient can install each before running jobs (installing executes third-party code and mutates the global bundle store, so it stays an explicit step); class binding is deferred to job-run time, so install order doesn't matter as long as it happens before the jobs run.

Load the result with from_export.

Parameters:

Name	Type	Description	Default
path	Union[str, Path]	Destination .zip path (parent directories created if absent). A non-.zip suffix is allowed but warns.	required
include_results	bool	If True, copy completed exp_*.json result files into results/ so the export carries the run's outputs too.	False
additional_bundles	Optional[List[Union[str, Path]]]	Optional list of .srtk paths (built with pack) providing custom classes that aren't installed as bundles. Each is copied verbatim into bundles/ and matched to the classes it provides; an explicitly supplied bundle takes precedence over a coincidentally-installed one. An entry that matches no referenced class warns.	None
strict	bool	If True, raise ValueError when any referenced custom class is covered by neither an installed bundle nor additional_bundles. Defaults to False (warn and flag it in MANIFEST.md instead).	False

Raises:

Type	Description
ValueError	If strict and a referenced custom class is uncovered, or if an additional_bundles path is not a readable .srtk.

Source code in SRToolkit/experiments/experiment_grid.py

def export(
    self,
    path: Union[str, Path],
    include_results: bool = False,
    additional_bundles: Optional[List[Union[str, Path]]] = None,
    strict: bool = False,
) -> None:
    """
    Write a self-contained, shareable ``.zip`` gathering everything a recipient needs.

    Unlike [to_dict][SRToolkit.experiments.ExperimentGrid.to_dict] (a recipe that
    *assumes* the recipient can reach the data and code), ``export`` bundles the
    dependencies that wouldn't otherwise travel into one archive whose entries are:

    ```
    grid.json            # the recipe (to_dict output), annotated with _bundle refs
    data/<name>.zip      # only for datasets whose data isn't otherwise reachable
    bundles/<name>.srtk  # only when custom (non-SRToolkit) classes are referenced
    results/...          # only when include_results=True
    MANIFEST.md          # inventory + recipient instructions
    ```

    A dataset's data is archived when its source is ``null`` (data lives only in the
    cache) or a seedless ``sample`` source (regeneration wouldn't reproduce the exact
    numbers). ``url`` and seeded ``sample`` datasets are left referenced — the
    recipient downloads or regenerates them.

    **Custom code.** Any non-SRToolkit class referenced anywhere in the grid must be
    covered by a ``.srtk`` bundle so the recipient can import it. Two sources are
    used, in order:

    - **already-installed bundles** — a referenced class whose import path falls under
      an installed bundle is re-packed automatically from the installed source files
      (carrying its declared ``python_deps`` / ``srtk_min_version``);
    - **``additional_bundles``** — ``.srtk`` files you built yourself with
      [pack][SRToolkit.bundle.pack] for code that isn't installed as a bundle. Because
      you build them, multi-file implementations, dependencies, and the real version
      all travel correctly. ``export`` never tries to guess a bundle from a loose
      source file.

    The recipe is annotated with ``_bundle`` / ``_version`` so the recipient's bind
    machinery relocates each class on load. Any custom class covered by neither source
    is **not** packed: it is listed in ``MANIFEST.md`` and a warning names each class
    and its source file. Pass ``strict=True`` to raise instead of warning.

    The bundles are **not** installed automatically.
    [from_export][SRToolkit.experiments.ExperimentGrid.from_export] extracts them next to
    the archive so the recipient can ``install`` each before running jobs (installing
    executes third-party code and mutates the global bundle store, so it stays an
    explicit step); class binding is deferred to job-run time, so install order doesn't
    matter as long as it happens before the jobs run.

    Load the result with
    [from_export][SRToolkit.experiments.ExperimentGrid.from_export].

    Args:
        path: Destination ``.zip`` path (parent directories created if absent). A
            non-``.zip`` suffix is allowed but warns.
        include_results: If ``True``, copy completed ``exp_*.json`` result files into
            ``results/`` so the export carries the run's outputs too.
        additional_bundles: Optional list of ``.srtk`` paths (built with
            [pack][SRToolkit.bundle.pack]) providing custom classes that aren't
            installed as bundles. Each is copied verbatim into ``bundles/`` and matched
            to the classes it provides; an explicitly supplied bundle takes precedence
            over a coincidentally-installed one. An entry that matches no referenced
            class warns.
        strict: If ``True``, raise ``ValueError`` when any referenced custom class is
            covered by neither an installed bundle nor ``additional_bundles``. Defaults
            to ``False`` (warn and flag it in ``MANIFEST.md`` instead).

    Raises:
        ValueError: If ``strict`` and a referenced custom class is uncovered, or if an
            ``additional_bundles`` path is not a readable ``.srtk``.
    """
    path = Path(path)
    if path.suffix.lower() != ".zip":
        warnings.warn(
            f"[ExperimentGrid.export] Export path {str(path)!r} does not end in '.zip'; "
            "a zip archive is written regardless.",
            stacklevel=2,
        )
    path.parent.mkdir(parents=True, exist_ok=True)
    # Build the export contents in a temp dir, then zip them at the archive root.
    with tempfile.TemporaryDirectory() as tmp:
        staging = Path(tmp)
        self._write_export_dir(staging, include_results, additional_bundles, strict)
        with zipfile.ZipFile(path, "w", zipfile.ZIP_DEFLATED) as zf:
            for file in sorted(staging.rglob("*")):
                if file.is_file():
                    zf.write(file, file.relative_to(staging))

from_export classmethod

from_export(path: Union[str, Path], results_dir: Optional[str] = None) -> ExperimentGrid

Load a grid from a .zip produced by export.

The archive is unpacked and:

• any data/*.zip archives are imported into the local data cache (so the bundled datasets resolve without a network or samplers);
• any bundles/ and MANIFEST.md are written into results_dir so the recipient has the loose .srtk files to install (from_export never auto-installs — that runs third-party code and mutates the global bundle store);
• the grid is rebuilt from grid.json and any shipped results/ are copied into results_dir so progress and load_results see them.

Custom classes still need their .srtk bundle(s) installed before jobs run — the returned grid is lazy and a warning names any that aren't importable yet. Class binding happens at job-run time, so installing after this call (before running) is fine. Install from <results_dir>/bundles/ per the extracted MANIFEST.md.

Parameters:

Name	Type	Description	Default
path	Union[str, Path]	A .zip written by export.	required
results_dir	Optional[str]	Where this machine's runs read/write results, and where bundles/ / MANIFEST.md are extracted. Defaults to the archive path with its suffix removed (e.g. run.zip → run/), created on load.	None

Returns:

Type	Description
ExperimentGrid	A fully configured ExperimentGrid.

Raises:

Type	Description
FileNotFoundError	If path does not exist or the archive has no grid.json.

Source code in SRToolkit/experiments/experiment_grid.py

@classmethod
def from_export(
    cls,
    path: Union[str, Path],
    results_dir: Optional[str] = None,
) -> "ExperimentGrid":
    """
    Load a grid from a ``.zip`` produced by
    [export][SRToolkit.experiments.ExperimentGrid.export].

    The archive is unpacked and:

    - any ``data/*.zip`` archives are imported into the local data cache (so the bundled
      datasets resolve without a network or samplers);
    - any ``bundles/`` and ``MANIFEST.md`` are written into ``results_dir`` so the
      recipient has the loose ``.srtk`` files to ``install`` (``from_export`` never
      auto-installs — that runs third-party code and mutates the global bundle store);
    - the grid is rebuilt from ``grid.json`` and any shipped ``results/`` are copied into
      ``results_dir`` so [progress][SRToolkit.experiments.ExperimentGrid.progress] and
      [load_results][SRToolkit.experiments.ExperimentGrid.load_results] see them.

    Custom classes still need their ``.srtk`` bundle(s) installed before jobs run — the
    returned grid is lazy and a warning names any that aren't importable yet. Class
    binding happens at job-run time, so installing after this call (before running) is
    fine. Install from ``<results_dir>/bundles/`` per the extracted ``MANIFEST.md``.

    Args:
        path: A ``.zip`` written by ``export``.
        results_dir: Where this machine's runs read/write results, and where ``bundles/``
            / ``MANIFEST.md`` are extracted. Defaults to the archive path with its suffix
            removed (e.g. ``run.zip`` → ``run/``), created on load.

    Returns:
        A fully configured ``ExperimentGrid``.

    Raises:
        FileNotFoundError: If ``path`` does not exist or the archive has no ``grid.json``.
    """
    path = Path(path)
    if not path.is_file():
        raise FileNotFoundError(f"[ExperimentGrid.from_export] No export archive at {str(path)!r}.")

    if results_dir is not None:
        resolved_results_dir = os.path.abspath(results_dir)
    else:
        resolved_results_dir = os.path.abspath(str(path.with_suffix("")))
    os.makedirs(resolved_results_dir, exist_ok=True)

    with tempfile.TemporaryDirectory() as tmp:
        staging = Path(tmp)
        with zipfile.ZipFile(path) as zf:
            zf.extractall(staging)

        grid_json = staging / "grid.json"
        if not grid_json.is_file():
            raise FileNotFoundError(
                f"[ExperimentGrid.from_export] {str(path)!r} is not an export archive (no grid.json)."
            )

        # Import bundled data into the local cache.
        data_dir = staging / "data"
        if data_dir.is_dir():
            for zip_path in sorted(data_dir.glob("*.zip")):
                # Side effect: extracts the archive's arrays into the data cache.
                SR_dataset.from_archive(zip_path)

        # Persist bundles + manifest so the recipient can install the loose .srtk files.
        bundles_src = staging / "bundles"
        if bundles_src.is_dir():
            shutil.copytree(bundles_src, Path(resolved_results_dir) / "bundles", dirs_exist_ok=True)
        manifest_src = staging / "MANIFEST.md"
        if manifest_src.is_file():
            shutil.copy2(manifest_src, Path(resolved_results_dir) / "MANIFEST.md")

        grid = cls.from_dict(str(grid_json), results_dir=resolved_results_dir)

        # Copy any shipped results into results_dir (staging is a temp dir, so always copy).
        shipped_results = staging / "results"
        if shipped_results.is_dir():
            for src in shipped_results.rglob("exp_*.json"):
                target = Path(resolved_results_dir) / src.relative_to(shipped_results)
                target.parent.mkdir(parents=True, exist_ok=True)
                shutil.copy2(src, target)

    return grid

save

save() -> None

Persist the grid to a single results_dir/grid.json file.

The file is the to_dict recipe plus the machine-local adapted_states paths (which to_dict omits, since they don't travel between machines). This is the local-persistence counterpart to the shareable export: it keeps your run resumable in place, but is not meant for handing to someone else.

save_commands calls this automatically, so a separate save() call is only needed when checkpointing the grid without generating a commands file.

Source code in SRToolkit/experiments/experiment_grid.py

def save(self) -> None:
    """
    Persist the grid to a single ``results_dir/grid.json`` file.

    The file is the [to_dict][SRToolkit.experiments.ExperimentGrid.to_dict] recipe
    plus the machine-local ``adapted_states`` paths (which ``to_dict`` omits, since
    they don't travel between machines). This is the local-persistence counterpart to
    the shareable [export][SRToolkit.experiments.ExperimentGrid.export]: it keeps your
    run resumable in place, but is not meant for handing to someone else.

    [save_commands][SRToolkit.experiments.ExperimentGrid.save_commands] calls this
    automatically, so a separate ``save()`` call is only needed when checkpointing the
    grid without generating a commands file.
    """
    os.makedirs(self.results_dir, exist_ok=True)
    payload = self.to_dict()
    if self._adapted_states:
        payload["adapted_states"] = self._adapted_states
    with open(os.path.join(self.results_dir, "grid.json"), "w") as f:
        json.dump(payload, f, indent=2)

load staticmethod

load(path: str) -> ExperimentGrid

Load a grid from a grid.json written by save.

Thin wrapper over from_dict that also restores the local adapted_states paths. Dataset and approach instances are not created at load time — they are reconstructed lazily when jobs run.

results_dir is taken from the directory containing path, not from any value stored in the file, so a grid directory can be moved, mounted under a different path, or copied elsewhere and still load (and write its results) correctly.

Parameters:

Name	Type	Description	Default
path	str	Path to the grid.json file. The directory holding it becomes the loaded grid's results_dir.	required

Returns:

Type	Description
ExperimentGrid	A fully configured ExperimentGrid.

Raises:

Type	Description
ValueError	If the file is not a supported ExperimentGrid recipe.

Source code in SRToolkit/experiments/experiment_grid.py

@staticmethod
def load(path: str) -> "ExperimentGrid":
    """
    Load a grid from a ``grid.json`` written by
    [save][SRToolkit.experiments.ExperimentGrid.save].

    Thin wrapper over [from_dict][SRToolkit.experiments.ExperimentGrid.from_dict] that
    also restores the local ``adapted_states`` paths. Dataset and approach instances
    are **not** created at load time — they are reconstructed lazily when jobs run.

    ``results_dir`` is taken from the directory containing ``path``, not from any value
    stored in the file, so a grid directory can be moved, mounted under a different
    path, or copied elsewhere and still load (and write its results) correctly.

    Args:
        path: Path to the ``grid.json`` file. The directory holding it becomes the
            loaded grid's ``results_dir``.

    Returns:
        A fully configured ``ExperimentGrid``.

    Raises:
        ValueError: If the file is not a supported ``ExperimentGrid`` recipe.
    """
    path = os.path.abspath(path)
    with open(path) as f:
        d = json.load(f)
    grid = ExperimentGrid.from_dict(d, results_dir=os.path.dirname(path))
    grid._adapted_states = d.get("adapted_states", {})
    return grid