Design: models

Design: models#

Note

A design document. It says how a trained model becomes an asset a pipeline can declare, document, version and hand to other code. Read Design: references first, since a model is addressed in that grammar. New to the project? Start with Design: start here.

Status: direction accepted 2026-10-12; the first slice, declared models, is built. A pipeline can declare a model, ask for its path, check it, and get a page and an index record for it. Versions, labels, parts and other pipelines’ models are not built. What is built says exactly what is there. Each section below says whether it is decided or proposed. Proposed parts are the author’s best design and still need the maintainer’s review, including the ones that are now implemented.

The problem#

A pipeline that trains a model leaves a file behind: a pickle, a .pt, a directory of weights. Six months later, nobody can answer the questions that matter:

Where is the model file?
What data was it trained on, and which version of that data?
What code trained it?
How good is it?
Which version is the one in use?

ChartBook already answers the first three for dataframes. The maintainer’s direction is that machine-learning operations should be a central use case, and models are the asset that makes it one.

What a model is, to ChartBook#

Decided. A model is a file or a directory that a pipeline produces, declared in chartbook.toml like any other asset. ChartBook records where it is, what it was trained on, its scores, and (later) its versions. It shows all of that on a page.

ChartBook does not know what is inside the file. It never imports PyTorch, scikit-learn or any other framework, and never loads, runs or serves a model.

A registry, not an experiment tracker#

Decided. Training a model means trying many things: hundreds of runs, most thrown away. Tools like MLflow and Weights & Biases record every attempt. ChartBook does not. It records the models a pipeline declares: the ones worth keeping.

ChartBook does	ChartBook does not
Record each declared model: its file, the data it was trained on, the code commit, its scores	Log every training run
Keep versions of a declared model, and labels like `champion`	Compare hyperparameter sweeps
Give any program the path to a model, at any version	Load a model, or run predictions
Show a page per model: description, scores, lineage	Serve a model behind an API
Say which models are out of date when their data changes	Decide when to retrain

These follow from the not-doing list: no required service, no orchestration, no run history. A team that wants an experiment tracker uses one, and declares in ChartBook the model it finally chose.

In the manifest#

Decided: the table and its four core fields. Proposed: the rest.

[models.rate_forecaster]
name = "Rate Forecaster"
description = "Forecasts the 10-year Treasury yield one month ahead."
path = "./_models/rate_forecaster.joblib"
trained_on = ["treasury_rates"]
metrics_path = "./_output/rate_forecaster_metrics.json"
docs_path = "./docs_src/models/rate_forecaster.md"
tags = ["Forecast", "Treasury"]

Field	Required	Meaning
`path`	yes	The model file, or a directory holding the model’s files. Relative to the pipeline root. Decided
`trained_on`	no	A list of references to the data it was trained on. The default kind is dataframe, so `"treasury_rates"` is a local dataframe and `"ftsfr/fed_yield_curve:treasury_rates"` is another pipeline’s. Both work today; see “What is built” for what the second one does and does not do yet. Decided
`metrics_path`	no	A JSON file of scores, written by the training code. See below. Decided
`docs_path` or `docs`	no	The model’s write-up (a “model card”): what it is for, how it was built, where it should not be used. Same convention as every other table. Decided
`name`, `description`, `tags`	no	As for dataframes and charts. Proposed
`[models.x.aliases]`	no	Labels that point at a version, such as `champion = "v3"`. See “Versions”. Proposed

trained_on becomes the model’s inputs in the index, the same field every other asset uses for “what was this made from”. The manifest keeps the friendlier word.

The metrics file#

Proposed, and implemented. One flat JSON object, names to numbers. A value may also be a string or true/false:

{ "rmse": 0.214, "mae": 0.171, "n_train": 5120, "n_test": 1280 }

Scores live in a file, not in chartbook.toml, because they are an output. They change every time the model is retrained, and the manifest is source that a person edits. ChartBook reads the file when it builds, copies the numbers into the index, and shows them on the model’s page. Anything that is not such a flat object is reported by chartbook check, and a build leaves the scores off the page and out of the index (with a warning) instead of failing.

Addressing a model#

Decided. A model is addressed like every other asset, with model/ in front:

model/rate_forecaster                          the model, in this pipeline
model/rate_forecaster#weights/best.pt          one file inside it, if it is a directory
model/rate_forecaster@v3                       version 3
model/rate_forecaster@champion                 the version labelled champion
ftsfr/fed_yield_curve:model/rate_forecaster    the model, in another pipeline

from chartbook import assets, models

assets.path("model/rate_forecaster")       # works for every kind
models.path("rate_forecaster")             # models only: the typed front door

$ chartbook path model/rate_forecaster
/work/fed_yield_curve/_models/rate_forecaster.joblib

chartbook.models is to models what chartbook.data is to dataframes: the same call with the kind fixed, which refuses anything else.

There is deliberately no models.load(). Loading is one line in the user’s own code, and keeping it there keeps every ML framework out of ChartBook:

import joblib
from chartbook import models

model = joblib.load(models.path("rate_forecaster"))

Code in any other language gets the path from chartbook path, as for dataframes.

The model’s page#

Proposed, and implemented. One page per model in the generated site, beside the dataframe and chart pages, at cb/models/<pipeline>/<model>.html:

the freshness line (“Model updated 2 days ago”), as dataframe pages have;
the write-up from docs_path or docs, if there is one;
Scores: the metrics file as a table;
Trained On: links to each dataframe’s page;
Files: for a directory, each file and its size;
Model Manifest: name, address, description, tags, the declared path, and the command that prints its path. The page shows the path as declared, never the absolute path on the machine that built the site;
once versions exist: a table of versions with their scores, and which labels point where.

A pipeline’s front page lists its models. A catalog gets a “Models” page and a card on its front page, but only when some pipeline declares a model, so a catalog without models is unchanged.

In the index#

Proposed, and implemented. A Model record, shaped like the other asset records in Design: the index (it has no built_by yet, since build records do not exist):

{
  "kind": "Model", "version": "chartbook/v1",
  "id": "ftsfr/fed_yield_curve:model/rate_forecaster",
  "ref": { "scope": "ftsfr", "pipeline": "fed_yield_curve",
           "kind": "model", "name": "rate_forecaster" },
  "pipeline": "ftsfr/fed_yield_curve",
  "name": "Rate Forecaster",
  "description": "Forecasts the 10-year Treasury yield one month ahead.",
  "location": { "kind": "file", "path": "./_models/rate_forecaster.joblib",
                "resolved_path": "/work/fed_yield_curve/_models/rate_forecaster.joblib" },
  "content_hash": "sha256:…",
  "size_bytes": 18234,
  "inputs": ["ftsfr/fed_yield_curve:treasury_rates"],
  "metrics": { "rmse": 0.214, "mae": 0.171 },
  "tags": ["Forecast", "Treasury"],
  "docs": { "path": "./docs_src/models/rate_forecaster.md", "rendered": "…" },
  "built_by": "sha256:…"
}

A model that is a directory also carries files, a list of { "path", "size_bytes", "content_hash" }, and its own content_hash covers all of them (the rule for hashing a directory is in Design: the index).

Versions#

Proposed

Decided: a model has versions; an exact version (@v3) and a label that moves (@champion) are both needed; each version records the data, the code and the scores. Proposed: everything else in this section. It depends on the store and on build records, which do not exist yet, so it is settled when they are built.

(Decided by the maintainer, 2026-10-13, and built the same day; Design: start here, “4d, as built”. Two things in this section went another way, and are marked where they are said. A version is made by a snapshot, not by every build that changes the model. And a version is named by the start of its content hash, with labels as the names people use, not v1, v2, v3. Where this section says “build record”, read “snapshot”: it is what the build record became.)

What a version is. One content hash of the model, together with the build record that produced it. The build record (Design: the asset graph) already holds what was asked for: the hash of every input (which data), the pipeline’s git commit (which code), the tool versions, and when. For a model it also keeps a copy of the metrics.

When a version is made. When a build finishes and a declared model’s content hash is one that has not been seen before. Rebuilding and getting the same bytes makes no new version. (As decided: when a snapshot is made and the model’s content hash is one that no earlier snapshot of the pipeline had. A build alone makes none.)

Where the bytes go. Into the store, by content hash, like every other pinned object. The file at path stays where it is; the store links to it or copies it (Design: the asset graph, “Links, not copies”). A team that wants to share model versions points at a shared store. That is the same mechanism as a shared build cache, and it needs no server.

How versions are named. v1, v2, v3, in the order they first appear, recorded in the pipeline’s committed chartbook.lock so the numbers are the same on every machine and are never reused. (As decided: by the start of the content hash, 7f3a9c2e41b0: twelve digits, and more when two versions would share them. Nothing is recorded to make the names agree between machines, because they are worked out from the bytes. See open question 1.)

Labels. A label is a name that points at a version, written in the manifest:

[models.rate_forecaster.aliases]
champion = "v3"
challenger = "v5"

(As built: champion = "7f3a9c2e41b0". A label may not itself look like a version.)

Promoting a model is then a one-line change, reviewed in a pull request like any other. There is no “promote” button and no service behind one.

How a reference resolves.

Reference	Resolves to
`model/rate_forecaster`	The file at `path`: what you are working on. Local references are live
`model/rate_forecaster@v3`	Version 3, in the store
`model/rate_forecaster@champion`	The label is looked up, then as above
`other:model/rate_forecaster`	The version this project’s lockfile pins. Cross-pipeline references are pinned
`other:model/rate_forecaster@champion`	The label is looked up when pinning and frozen in the lockfile, so a build is repeatable. `chartbook update` moves it

(Built: the first three rows, with a version written as @7f3a9c2e41b0. The fourth was built in step 3. The fifth on 2026-10-14: the label is read from the snapshot being pinned, which records its manifest’s labels, and written to the lockfile as a record of its own; Design: start here, “4h, as built”.)

Garbage collection. A recorded version is reachable, so chartbook store gc never collects it. (As built, 2026-10-13: a version a label points at is never collected. One no label names is kept for 30 days and while it is the newest of its pipeline, then goes with its snapshot. Design: start here, “Clearing old snapshots”. That is the answer to open question 3.)

What this gives an ML team#

In order, the loop this is meant to support:

Train. The pipeline’s own code writes the model file and the metrics file.
Record. The build notices a new model, makes version v4, and records the data hashes, the code commit and the scores.
Compare. The model’s page lists every version with its scores.
Promote. Someone changes champion = "v3" to champion = "v4" in a pull request.
Use. Serving code, in any language, asks for model/rate_forecaster@champion and gets a path. It never hardcodes a file name.
Audit. For the model in use, the page shows exactly which data and which commit made it.
Retrain. When upstream data changes, the model’s inputs no longer match, so it shows as out of date. chartbook update and a rebuild make v5. (Built 2026-10-14: the model’s page says “Out of date” and which inputs moved, the index record has stale_inputs, and chartbook check warns.)

Everything outside that loop belongs to another tool: experiment tracking (MLflow, Weights & Biases), serving (whatever the team runs), scheduling (cron, CI, an orchestrator), feature stores.

A worked example#

Built, in examples/yield_curve. It adds one small model to the pipeline and needs no new dependency: the model is a two-number regression fitted with NumPy and saved as JSON.

chartbook.toml:

[models.term_premium_ar1]
name = "Term Premium AR(1)"
description = "Predicts tomorrow's 10-year minus 2-year spread from today's."
path = "./_models/term_premium_ar1.json"
trained_on = ["term_premium"]
metrics_path = "./_output/term_premium_ar1_metrics.json"
docs_path = "./docs_src/models/term_premium_ar1.md"
tags = ["Yield Curve", "Forecast"]

src/train_term_premium_ar1.py, in outline:

from chartbook import data, models

spread = pd.read_parquet(data.path("term_premium"))["spread_10y_2y"].dropna()
train, test = ...                                       # first 80%, last 20%
slope, intercept = np.polyfit(train.values[:-1], train.values[1:], 1)

model_path = models.path("term_premium_ar1")            # ask the manifest where it goes
model_path.write_text(json.dumps({"intercept": ..., "slope": ...}))
metrics_path.write_text(json.dumps({"rmse": ..., "mae": ..., "n_train": ..., "n_test": ...}))

dodo.py has a task whose file_dep is data.path("term_premium") and whose targets are assets.path("model/term_premium_ar1") and the metrics file.

docs_src/models/term_premium_ar1.md is four short paragraphs: what the model is, what it is for (an example, not a forecast to act on), how it was trained, and its limits.

What you get:

$ cd examples/yield_curve && doit
$ chartbook path model/term_premium_ar1
/…/examples/yield_curve/_models/term_premium_ar1.json
$ chartbook check
chartbook.toml: OK (1 pipeline(s), 3 dataframes, 5 charts, 4 notebooks, 1 models)

On live data the fitted slope is 0.9988: a daily yield spread is so persistent that the model learns “tomorrow equals today”, which is what its write-up says it will. The built site has a page for the model with its write-up, a “Trained On” link to the Term Premium dataframe’s page, and a table with rmse, mae, n_train and n_test. The catalog built over the example has a Models page listing it, and index.json has its Model record.

A second example, examples/rate_forecast, is a separate pipeline whose model is trained_on = ["yield_curve:fed_yield_curve"]. It is the first real cross-pipeline example, for models and dataframes alike. (Built 2026-10-12, with step 3d.) It declares yield_curve in [dependencies], pins it, trains on data.load("yield_curve:fed_yield_curve"), and its Model record names the bytes it was pinned at.

Phasing#

Each slice is useful by itself. The step numbers are the Design: start here’s build order.

Slice	Needs	What gets built
M1. Declared models (step 2) — built	The index	The `[models]` table in the loader, with validation. `model` added to `RESOLVABLE_KINDS`, so `chartbook path model/x` works. `chartbook.models.path()`. `chartbook check` covers `path`, `metrics_path` and `docs_path` (the first two are build outputs, so `--skip-outputs` skips them). The model page. The `Model` record. Policy support: `[policy.required] models = […]`. The worked example above. No versions
M2. Parts (with M1 or soon after) — proposed 2026-10-13: not to be built	M1	`model/x#file` resolves to a file inside a directory model. Refuse a part that escapes the directory (`..`). (The maintainer decided that a reference does not select data (Design: start here, step 4, “4e”). By the same reasoning a file inside a model is `models.path("x") / "file"`, and the message for `model/x#file` says so. That extension is the builder’s, and waits on his yes.)
M3. Cross-pipeline (step 3) — built	Dependencies, lockfile, store	`other:model/x` resolves through the lockfile. Training data in another pipeline is a real dependency: declared, pinned, read from the pinned copy, and recorded by hash (`pinned_inputs`). The `examples/rate_forecast` pipeline. A dependency fetched with `git` has only what its repository commits
M4. Versions and labels (step 4) — built	Snapshots	A version on each new content hash a snapshot holds, `@7f3a9c2e41b0`, `[models.x.aliases]`, `@champion`, the versions table on the page. Staleness: a snapshot records what each model was trained on, and the page, the index and `check` say when those inputs have moved (2026-10-14; Design: start here, “4i, as built”)

What is built#

Slice M1, in these places:

What	Where
The `[models]` table, validated when the manifest loads	`chartbook/manifest.py`, `validate_models()`
`model/name` resolves; `chartbook path model/x`	`chartbook/resolve.py` (`RESOLVABLE_KINDS`)
`chartbook.models.path()`, `chartbook.models.read_metrics()`	`chartbook/models.py`
The model’s page, and the lists that link to it	`chartbook/writers/sphinx.py` (`model_page()`), `model_manifest.md` and `models.md` in the two theme directories
The `Model` record and its JSON Schema	`chartbook/index.py`, `chartbook/schemas/chartbook/v1/Model.json`
`check`: missing model, metrics or write-up; a malformed metrics file; required fields	`chartbook/check.py`, `chartbook/conf_validation.py`, `chartbook/diagnostics.py`
Incremental builds: a model’s files are inputs, its page belongs to its pipeline	`chartbook/incremental.py`
The example	`examples/yield_curve`
The tests	`tests/test_models.py`
Training data in a declared dependency: checked, and named by the dependency’s name (step 3a)	`chartbook/check.py` (`check_model_lineage`), `chartbook/index.py`, `tests/test_dependencies.py`

Not built: versions and labels (@v3, @champion, [models.x.aliases]) and parts (model/x#file). Each is refused with “reserved syntax and not yet supported”.

Slice M3 is built too, with step 3 of Design: start here, for a dependency that is a directory on disk: models.path("other:rate_forecaster") is the pinned copy of another pipeline’s model, and a model’s training data in another pipeline is declared, pinned, read from the pinned copy and named by hash. A dependency fetched with git works the same way, for what its repository commits: a model kept in git can be pinned and read; one that is built and git-ignored cannot, until ChartBook can run that pipeline.

Training data in another pipeline is the point, not an edge case

Decided by the maintainer, 2026-10-12: a model’s training data can be another pipeline’s dataframe, and it is allowed now, not held back until dependencies exist.

trained_on = ["ftsfr/fed_yield_curve:treasury_rates"] loads today. It is stated lineage:

the index records it as an input, by the ID that pipeline has in the build if it is part of it, and as written otherwise;
the model’s page lists it, and links to the dataframe’s page when the other pipeline is in the same catalog;
chartbook check, run on a catalog, reports a dataframe that the other pipeline does not declare. Where the other pipeline cannot be seen, nothing is checked.

What it is not yet: a dependency. Nothing fetches the data, pins a version of it, or records which bytes the model was trained on. Making it one is the reason step 3 exists, and whoever builds step 3 should treat “a model trained on another pipeline’s data, reproducibly” as its headline case. When that lands, a trained_on that names another pipeline must be backed by an entry in [dependencies]; until then it is taken on trust.

Since 2026-10-12, the first part of step 3: the other pipeline can be declared.

With "ftsfr/fed_yield_curve" = { path = "../fed_yield_curve" } in [dependencies], chartbook check reads that pipeline’s manifest and reports a dataframe it does not declare. No catalog is needed.
The index writes the input by the dependency’s name: fed_yield_curve:treasury_rates in the manifest is ftsfr/fed_yield_curve:treasury_rates in the index.
A trained_on that names a pipeline [dependencies] does not declare still loads, and now warns, saying what to add.

Later the same day, the rest of step 3 for a dependency on disk:

chartbook pin freezes the other pipeline’s data, and the training code reads the frozen copy with data.load("fed_yield_curve:treasury_rates").
The model’s record gains pinned_inputs: that input’s ID, and the content hash it is pinned at. (Decided by the maintainer: a field beside inputs, which stays a list of IDs.)
A trained_on that names an undeclared pipeline is now an error, as this box always said it would become.

pinned_inputs says what is pinned, which is what training code is handed. It does not prove what one training run read; a build record will. See “3d, as built” in Design: start here.

Decided while building it#

These were open or unstated when the slice was built.

Decided by the maintainer, 2026-10-12:

By default chartbook check reports a model that does not say what it was trained on. The default policy asks a model for name, description, trained_on, a write-up and path. A catalog can set its own list with [policy.required] models = […].
trained_on may name another pipeline’s dataframe, now. See the box above. (The first version of this slice refused it until dependencies existed. That was reversed.)
A model’s write-up is strongly recommended, not required. A model without one loads and gets a page; check reports it under the default policy. In a policy list the field is written docs, and is satisfied by either docs or docs_path.

Proposed: implemented, and waiting on the maintainer’s yes:

A metrics value may be a number, a string or true/false. Nothing nested.
A declared model or metrics file that is missing stops a build, as a missing data file does. chartbook check --skip-outputs does not report them, since training produces them.
A model’s key must be a legal name ([A-Za-z0-9._-]+), or the manifest does not load. The older tables are not held to that; a new one can be from its first day.
A catalog has a Models page only when a pipeline declares a model.
The page shows the declared path, not the absolute one. (The dataframe page shows the absolute path; that is older behaviour and was left alone.)
A series or a version in trained_on is still refused (treasury_rates#dgs10, treasury_rates@v2). Each needs the store to mean anything.

Asked, not answered: should a dataframe or a chart still be required to have a write-up? Today the manifest does not load without one, while a model’s is only recommended. The maintainer raised making all three the same. The recommendation is to do so: never refuse to load a manifest over a missing write-up; put docs in the default policy list for dataframes and charts as it now is for models; and let a catalog, or a pipeline, drop it. Not done, because it loosens something existing projects rely on being checked, and it was a question, not an instruction.

One thing is open and should be settled before models are used in earnest: hashing a directory counts every file, hidden ones included. A .DS_Store appearing in a model’s directory changes the model’s hash. That is the rule decided for the index, and the honest one (“the directory is the asset”), but for a model with versions it would make a new version out of nothing.

Decided by the maintainer, 2026-10-12: a short, fixed list of files that operating systems create by themselves is skipped (.DS_Store, Thumbs.db, desktop.ini, ._*), and every other file still counts. It is built with the store (Design: start here, “Answers to the open questions”).

Not doing#

No experiment tracking. Declared models only. Not every run.
No loading or running models. models.path() returns a path. No models.load(), no predict(), no framework imports.
No serving. No endpoints, no containers, no deployment.
No training orchestration. The pipeline’s own build trains. ChartBook never decides when.
No model-format handling. No conversion, no ONNX export, no signature inference. The file is opaque bytes with a hash.
No promotion workflow. No stages, approvals or buttons. A label is a line in a file under version control; the review process is the repository’s.
No automatic metric comparison or “best model” selection. The page shows scores; people choose.

Prior art#

MLflow Model Registry: registered models, numbered versions, and aliases such as @champion. The model for versions and labels. Its tracking server and run logging are the part deliberately not copied.
Hugging Face Hub: a model is a repository of files with a model card, addressed as org/name@revision. The model for “a model is a directory with a write-up”, and for @.
DVC: large files versioned by content hash beside git, with a cache and link strategies. The model for where the bytes live.
Model Cards (Mitchell et al., 2019): what a model’s write-up should say. The suggested outline for docs_path.
Weights & Biases Artifacts: versioned artifacts with aliases and lineage between them. Confirms that versions plus moving labels plus lineage is the useful core.

Open questions#

Version numbers and branches. Two branches that each retrain will both create v4 in their lockfiles, and the merge conflicts. Options: accept the conflict and renumber on merge; or name versions by a short content hash and keep sequential numbers out of it, leaving labels as the only human names. Needs a decision before M4. (Decided 2026-10-13: by the content hash, with labels.)
Which builds make a version. Every new content hash, or only builds on the default branch, or only when asked (chartbook pin)? Every-hash is simplest and fills the store fastest. (Decided 2026-10-13: only when asked, and the asking is a snapshot.)
Retention. Versions are never collected. A model retrained nightly makes 365 a year. Probably a per-model keep = N that spares labelled versions, but nothing is designed. (Answered 2026-10-13 without a new field: labelled versions are kept; the rest go with their snapshots after 30 days. Design: start here, “Clearing old snapshots”.)
Do other kinds get versions? A dataframe could keep named versions the same way. Nothing stops it; nothing has asked for it. Cross-pipeline pins already keep the versions that are depended on.
Non-numeric metrics. A confusion matrix or a curve is not a flat number. For now such things are charts or dataframes the pipeline declares separately.
Models that are not files. A model held in a registry elsewhere (an MLflow URI, a Hugging Face repo). The dataframe design’s typed location (Design: data access) is the seam; nothing is designed.