Skip to content

Reproducible case studies

CDS2 already ships runnable examples such as examples/experiment_fitting.py and examples/epidemic_simulation.py. This page defines the evidence standard for turning examples into research-grade case studies.

Required structure

Every substantial case study should contain:

  1. Question — the scientific or numerical question being answered.
  2. Data provenance — where the data came from, its license, and an immutable checksum or generated-data recipe.
  3. Assumptions — model family, noise model, preprocessing, boundary conditions, and any exclusions.
  4. Analysis — executable CDS2 code with a fixed seed where randomness is involved.
  5. Validation — an analytical result or independent implementation used as an oracle when possible.
  6. Uncertainty — parameter uncertainty, confidence/credible intervals, sensitivity, or an explicit explanation when uncertainty is not applicable.
  7. Diagnostics — residuals, convergence checks, conditioning, stability checks, or domain-specific diagnostics.
  8. Reproduction — exact command plus package/dependency versions and expected output artifacts.

Existing examples to promote

Example Scientific purpose Current strengths Next evidence step
experiment_fitting.py Michaelis–Menten parameter estimation seeded synthetic data, nonlinear fit, covariance-derived parameter errors, Welch and permutation tests cross-check fitted parameters against SciPy directly and publish tolerance/expected output
epidemic_simulation.py compartmental epidemic dynamics domain-specific simulation add conservation and sensitivity checks plus independent ODE integration
chaos_diagnostics.py nonlinear/chaotic dynamics domain diagnostics validate known regimes and statistical invariants rather than long-horizon pointwise trajectories
metaheuristic_race.py global optimization reproducible algorithm comparison compare against known benchmark minima over repeated seeds
ab_test_bayes.py Bayesian inference probabilistic workflow add analytical/conjugate oracle checks and calibration notes

Acceptance criteria

A case study is publication-quality for this repository when:

  • it runs from a clean environment,
  • all external inputs are pinned or checksummed,
  • stochastic results are seed-controlled,
  • expected numerical tolerances are stated,
  • plots and reports are generated by code rather than edited by hand,
  • the validation target is independent of the implementation being tested,
  • limitations and failure modes are described alongside successful results.

Prefer a small number of deep, auditable case studies over a large collection of shallow API demonstrations.

Guided-fit numerical-stack upgrade replay — 2026-10-06

Question

Does a saved guided-fit analysis expose dependency drift when it is replayed under a newer numerical stack, and does it distinguish that environment change from a material change in the fitted result?

Data provenance

The case uses a deterministic synthetic exponential_weak_offset dataset generated by scripts/reproduce_guided_fit_dependency_drift.py. No external data are downloaded. The saved manifest records the raw scientific input hash:

ad0ed8c6ea4dce837528607c52c323c918594409d741a078170dcab05ebcc478

The model is fixed to exponential, the guided-fit seed is 20261006, and missing/outlier policies are fixed.

Environments

Component Baseline Upgraded rerun
Python 3.12.14 3.12.14
NumPy 1.26.4 2.3.5
SciPy 1.11.4 1.18.1
pandas 2.2.3 2.2.3
Matplotlib 3.10.8 3.10.8

NumPy and SciPy are upgraded together because SciPy 1.18.1 requires NumPy >=2.0, so there is no honest common NumPy pin spanning this baseline and target pair. This is therefore a numerical-stack upgrade case, not a claim that SciPy alone caused any observed difference.

Actual rerun warning

The baseline manifest was created under the baseline environment, downloaded into the upgraded job, and replayed with the real guided-fit-rerun path. The rerun returned stability_warning=true with:

runtime version changed: numpy 1.26.4 -> 2.3.5
runtime version changed: scipy 1.11.4 -> 1.18.1

No material confidence-interval warning fired, because the measured numerical drift stayed below the configured 5% materiality threshold.

Measured numerical change

The largest absolute movement of any saved 95% confidence-interval bound was:

1.1747583668952899e-4

The L2 change across all confidence-interval bounds was 0.003364917% relative to the baseline intervals. Parameter-vector L2 movement was 3.4724873872523174e-5; relative RMSE movement was 6.350280320340965e-11.

The reliability label stayed:

reliable -> reliable

This is the intended distinction: the tool reports the environment change even when the scientific result remains numerically stable. It does not promote a small numerical difference into a material-drift claim.

Reproduction and archived evidence

Run the dedicated workflow manually:

.github/workflows/guided-fit-drift-case.yml

The workflow creates the old-stack manifest first, then downloads and replays that exact manifest under the upgraded stack. The executable recipe is:

scripts/reproduce_guided_fit_dependency_drift.py

The permanent result record, including exact confidence intervals, environment versions, input hash, workflow run ID, artifact digests and derived drift metrics, is:

benchmarks/history/guided_fit_stack_upgrade_20261006.json

The originating GitHub Actions run was 37465811285. Its probe commit was 0a17182587cd6cf5e118853923862fb45b1372ac.