Caching and Hydration¶

Consist's execution loop is:

declare inputs -> compute signature -> check cache -> hydrate or execute

This page focuses on the run signature, cache hit/miss behavior, hydration versus filesystem materialization, and the policy knobs you are likely to use. For runnable workflow patterns, see the Usage Guide.

Which Layer Am I In?¶

Consist uses a few related mechanisms to make prior work reusable. Keep the layer straight before choosing an option:

This page uses materialization to mean filesystem byte recovery or staging. DuckDB ingestion is a storage/querying choice covered in Data Storage and Ingestion.

Core Terms¶

Hydration and filesystem materialization are deliberately separate. A cache hit can return artifact metadata without copying bytes.

Run Signature¶

For Tracker.run(...), Tracker.trace(...), ScenarioContext.run(...), and ScenarioContext.trace(...), the practical signature model is:

signature = code identity + config identity + input identity

Code Identity¶

By default, Consist uses repository Git state. Consist requires a truthful, non-empty code identity before it performs reusable cache lookup. For function-shaped steps outside a Git repository, the default repo_git request falls back to hashing the callable's defining module. The resolved mode is persisted as callable_module; it is never mislabeled as repo_git.

For function-shaped steps, you can also select a code identity explicitly with CacheOptions:

from consist import CacheOptions

cache_options = CacheOptions(
    code_identity="callable_module",
    code_identity_extra_deps=["shared/helpers.py"],
)

Supported modes are:

Use code_identity_extra_deps when a callable-scoped mode also depends on helper files. callable_module is the safer non-Git fallback because it also captures same-file helpers and constants.

If repository identity is unavailable and no callable is present, Consist raises CodeIdentityUnavailableError before creating a reusable cache candidate. The same fail-closed behavior applies when an explicitly selected callable mode cannot inspect its callable; Consist never retries that request as repo_git and never uses a constant unknown_code_version or a timestamp nonce.

Consequently, unchanged callable-module code outside Git can reuse a completed run, while editing that module produces a new code digest and a cache miss.

Config Identity¶

config={...} contributes to the signature. Adapters can also contribute canonical config identity through adapter=.... Query-oriented facets are for filtering and grouping; do not rely on them as a replacement for identity inputs or config.

Identity-bearing configurations are canonicalized deterministically across Python processes. Set members are recursively normalized. Members that remain mutually comparable retain Consist's legacy natural ordering; heterogeneous JSON-compatible sets such as {"alpha", "beta", 1} use compact, sorted JSON tokens instead, giving them one stable identity even when their members are not mutually comparable in Python. Values that cannot be represented in JSON fail during identity construction; they do not receive a warning and a best-effort hash. This contract does not perform Unicode normalization or coerce numerically distinct values.

See Config Management for config/facet guidance.

Input Identity¶

Declared inputs={...} contribute to lineage and input hashing. Produced Consist artifacts link through their producing run signature. Raw files are hashed according to the active hashing strategy.

Use identity_inputs=[...] for additional hash-only files or directories that should affect the signature without being passed as callable inputs.

Hit and Miss Behavior¶

On a cache hit, Consist:

• Finds a matching completed prior run.
• Hydrates the prior output artifacts into the current result.
• Preserves lineage to the producing run.
• Skips callable execution for run(...)-style function execution.

On a cache hit, Consist does not automatically copy every output file into the new run directory. That only happens when a filesystem materialization policy asks for it.

On a cache miss, Consist executes the step, records a new run, logs declared outputs, and stores enough identity metadata to explain future hits and misses.

trace(...) is different from function-shaped run(...): the Python block is entered so you can perform explicit lifecycle work. Use run(...) when skip-on-hit callable execution is the important behavior.

Diagnosing Cache Misses¶

Start with the persisted identity summary:

run = tracker.get_run("run_id")
summary = run.identity_summary

print(summary["signature"])
print(summary["code_version"])
print(summary["config_hash"])
print(summary["input_hash"])
print(summary.get("identity_inputs"))

If a run re-executed, inspect the cache miss explanation:

explanation = run.meta.get("cache_miss_explanation")

if explanation:
    print(explanation["reason"])
    print(explanation.get("candidate_run_id"))
    print(explanation.get("matched_components"))
    print(explanation.get("mismatched_components"))
    print(explanation.get("details", {}))

Common reasons include config drift, input drift, code drift, invalid cached outputs, or no similar prior run.

Hydration vs Filesystem Materialization¶

Hydration recovers information about an artifact:

• artifact key
• producing run
• logical URI and resolved path
• metadata and format
• canonical fingerprint on artifact.hash

Filesystem materialization moves or recreates bytes:

• copy a cached output to a requested path
• stage an input file for a path-bound tool
• export an ingested CSV/Parquet artifact from DuckDB fallback
• restore historical outputs into a restart workspace

Default cache-hit behavior is metadata hydration. This keeps parameter sweeps and large simulations from duplicating files unnecessarily.

Use consist.load_df(...), consist.load(...), or consist.load_relation(...) when you need to read artifact bytes. See Data Storage and Ingestion for ingestion and database fallback behavior.

Cache Hydration Policies¶

Set byte-availability behavior with CacheOptions(cache_hydration=...) on run(...), or the equivalent lifecycle argument on lower-level APIs. Some policies apply only after a cache hit; inputs-missing applies before executing a cache miss.

For outputs-requested, provide explicit output paths. For outputs-all, provide a target output directory. run(...) accepts the high-level output_paths={...} pattern for requested outputs.

For cache reuse, outputs-requested is an all-or-miss contract. Consist stages and validates every requested output before it reports a cache hit. A candidate with a missing key, unavailable source, partial materialization, destination failure, or artifact identity mismatch is ineligible; Consist tries another same-identity candidate when available, otherwise it executes through the ordinary cache-miss path. Requested destination paths do not enter cache identity.

cache_hydration_failure="miss" remains available for lower-level direct hydration calls; cache reuse already uses strict candidate admission for this policy. It is not a replacement for eager cached-artifact validation, source-root selection, or restart recovery.

Input Binding and Hydration¶

input_binding controls what the callable receives. cache_hydration controls which cached bytes are made available. They often work together, but they are not the same knob.

Path-bound workflows often use cache_hydration="inputs-missing" when work moves across run directories. Loaded workflows can often keep cache_hydration="metadata" and load data on demand.

By default, inputs-missing restores only destinations that are absent. If a path-bound legacy workflow reuses the same filename for different logical artifacts, opt into validation:

from consist import CacheOptions

cache_options = CacheOptions(
    cache_hydration="inputs-missing",
    validate_materialized_inputs=True,
)

With validation enabled, Consist preserves an existing destination unless it can prove from a portable artifact hash that the bytes are stale. Proven stale inputs are restored from the historical root, artifact recovery_roots, or the ingested DB fallback when available.

Use this for in-place legacy workflows where one live filename can represent different logical artifacts at different points in the workflow, as long as the earlier artifact has recoverable bytes and a portable full-content hash.

Requested Input Staging¶

Requested input staging is input-side materialization attached to normal run execution. Use it when a declared input has a canonical provenance identity but the callable needs it at an exact workspace-local path.

from pathlib import Path

from consist import ExecutionOptions

result = tracker.run(
    fn=run_tool,
    inputs={"config_path": config_artifact},
    outputs=["report"],
    execution_options=ExecutionOptions(
        input_binding="paths",
        input_materialization="requested",
        input_materialization_mode="copy",
        input_paths={
            "config_path": tracker.run_dir / "workspace" / "tool-config.yaml"
        },
    ),
)

This staging runs before execution on cache misses and before returning cached results on cache hits. It keeps inputs={...} as the source of identity and lineage while making fixed-path tools practical.

Use low-level stage_artifact(...) or stage_inputs(...) only when staging is needed outside a run lifecycle. See Materialization.

Output Recovery and Historical Hydration¶

For restart or archive workflows, prefer hydrate_run_outputs(...) over manually copying old output paths. It returns keyed hydration records that say which requested outputs were restored, where they landed, and whether the resulting detached artifact is resolvable.

When restart code needs to choose the historical run first, use find_matching_run(...) with semantic fields such as model, stage, phase, year, iteration, cache_epoch, and run_scope, then pass the matched run ID to hydrate_run_outputs(...).

Recovery sources are checked in this order:

1. Per-call source_root=...
2. Historical source path or recorded mount root
3. artifact.meta["recovery_roots"]
4. DuckDB export fallback for ingested CSV/Parquet artifacts

See Materialization for the detailed status meanings and archive helpers.

Policy Summary¶

Intentional Invalidation¶

Use explicit version knobs when a schema or semantic change should invalidate cache even if file hashes did not change:

from consist import CacheOptions, Tracker, define_step

tracker = Tracker(run_dir="runs", cache_epoch=2)


@define_step(cache_version=3)
def simulate(...):
    ...


result = tracker.run(
    fn=simulate,
    cache_options=CacheOptions(cache_version=4),
)

Use the narrowest invalidation boundary that matches the change: step version, scenario epoch, or tracker epoch.

Related Pages¶

• Usage Guide: choosing run, trace, scenario, input binding, and step references.
• Data Storage and Ingestion: ingestion, hot/cold data, hybrid views, and loader fallback.
• Materialization: input staging, hydrate_run_outputs(...), and recovery status objects.
• Run API: accepted options and generated reference.
• Core Concepts Overview: one-line definitions and the mental model for runs, artifacts, scenarios, and couplers.