Design: the versioned catalog

Note

A design document, written 2026-10-16 after the maintainer and the last person to work on this talked through what an index is for and what a snapshot is for. The conclusion is here, with the plan it leads to. Status: the self-sufficient snapshot (step 1) is built; the rest is proposed. What the maintainer decided is marked decided.

The use cases this serves

1. A project grows and is split into repositories. One depends on the outputs of another, and the dependencies diverge: B reads A at one version while C reads A at a later one.

2. Depending upstream must be reproducible and show its lineage: which version of what, by commit and by hash.

3. One place to browse everything a person is working on, across every pipeline and every version of each, with links that follow the real dependencies.

4. An assistant browses the same thing from the command line.

5. While working on one repository, its own site is rebuilt now and then, for the author and for collaborators, from whatever is in the folder.

The first two exist: declared dependencies, pins by commit and hash, snapshots in the store, the provenance panel. The fifth is today’s pipeline site. The third and fourth are what this page plans.

Index and snapshot: the conclusion

Decided 2026-10-16. The two documents stay two documents, and the record format they share is the only merge. They are different acts:

	A snapshot	An index
Is	A preservation act: the bytes one pipeline built at one commit, copied into the store, declared ready	A description of what a site shows, taken from whatever is in the folders now
Needs	Everything committed, the bytes present	Nothing committed; a pipeline with missing outputs can be skipped
Covers	One pipeline, once	Many pipelines at once, the catalog’s own facts, and comparisons no one pipeline can make: a model out of date against its inputs now, versions read across commits
Lives	In the store	Beside the site, and a copy without machine paths travels with it

Neither is derivable from the other. A snapshot cannot be made from an index: the index has hashes and not bytes, no commit, and may describe a half-built folder. An index can almost be assembled from snapshots, but only from every pipeline’s snapshots plus the files at those commits plus the store, and the result describes what was declared ready rather than what is in front of you. That is a different site, and a useful one; it is the catalog below.

What was wrong before this conversation: a snapshot could not render a site. The index records facts and pointers, not the write-ups, notebooks, or chart files, and those came only from a checkout. So a version’s site, and a version’s data dictionary as it stood when the data was made, lived in git and nowhere else. The maintainer’s point: the snapshot of the data should carry the description of the data.

So a snapshot holds the index of its version. Not a merge; nesting. A snapshot contains the pipeline’s own index at that commit and the small documentation files a writer needs; a site’s index is assembled from version indexes plus the comparisons only a site can make. The vocabulary page says this now, where it used to say the two “converge”.

The catalog shows every version of every pipeline

Decided 2026-10-16. A catalog shows every snapshot in the store of every pipeline it knows, found by pipeline ID, plus the working version of each pipeline whose folder is on this machine: the folder as it is, committed or not, built or not, which is what the pipeline’s own site shows too. The store is the source of versions; the catalog’s manifest is the source of working copies, as it is today. A catalog with no snapshots is today’s catalog, so nothing breaks. “Latest” is not a word here: the working version is the working version, and the newest snapshot is the newest snapshot.

What that needs:

1. Which versions. Every snapshot of every pipeline the catalog lists, and the working version of each whose folder is present. A snapshot that store gc removes leaves the catalog with it: gc is the lifecycle.

2. Pages per version, each set under a path that carries the pipeline ID and the version: the commit’s start and the snapshot’s time, since one commit may have several snapshots when a pull script brought other data. Static pages are cheap.

3. Links follow pins. C’s pages link to A at the snapshot C pinned; B’s to the other. The link scheme is a function of the writer for this reason.

4. A version list on every pipeline page: the working version first, then snapshots newest first, with the commit and the date, each a link to the same page in that version or to the version’s front page where the page does not exist there. Both writers render it as a list; a drop-down is theme work.

5. Comparisons stay within a version. An out-of-date notice compares what a version recorded with what it pinned, never with a version it did not pin.

6. Rendering a snapshot costs once. A snapshot never changes, so its pages are rendered the first time a catalog build sees it and kept; only working versions are rendered every build. The incremental build is the home for that rule.

7. Search is scoped to working versions and newest snapshots by default, or every hit is a near-duplicate.

8. Diagnostics and policy apply to working versions; a snapshot’s metadata is history.

A shared store makes a shared catalog: with the S3 store, a team’s catalog is built from the bucket, and that is the use case for several machines (Design: stores and backends).

For the assistant

The catalog’s index.json is the assistant’s entry point already. Generated sites should also ship an assistant file written from the index, one line per pipeline, dataframe, chart and model with where its page is, so that a command-line agent finds its way without scraping.

The order of work

1. The self-sufficient snapshot. The snapshot carries the pipeline’s own index at that commit, in the published form, as an object in the store, and the documentation files a writer needs that are not assets: the manifest, the lockfile, the README, every write-up, the pipeline’s own site pages, the logo and favicon. Notebooks, notes, charts, dataframes and models are assets already. A snapshot can then be laid out as a folder from the store alone and a site built from that folder, which is the test. Built 2026-10-16: Snapshot.index and Snapshot.documents in the record, written by chartbook snapshot, kept alive by store gc like the assets; chartbook.snapshot.materialize() lays a snapshot out as a folder, and a test builds a site from one with the repository gone. As built, after a review and a live run on the examples:

   • What is taken, and how. Documents are named by the manifest’s text, so a manifest a docs build refuses still keeps them; the index is best-effort for the same reason, and the command says in one line why it was left out (it is not tried again at that commit unless the bytes change). Only files git tracks are taken, asked of git with literal pathspecs; a path must be plain (safe_relative: nothing absolute on any system, no .., no part named .git, no :), and no file that is a link or under one is taken; each one not kept is said. The review found that without these, glob characters, git’s pathspec magic and tracked links each reached files outside the folder.

   • A record is untrusted. A store may be shared, so materialize judges every path in a record by the same rule, follows no link inside an object, writes no .git, and writes nothing until every object the record names is there.

   • A laid-out version keeps its identity. materialize writes .chartbook-version.json (chartbook.identity.VERSION_FILE): the pipeline, the commit, when it was made and when its code last changed, from its index. A pipeline whose ID came from its folder’s name and its git remote keeps that ID in a folder of any name, and its code time is not asked of a git it does not have. The live run found both missing.

   • What a build writes and git does not keep is written back where the snapshot recorded it: a model’s scores to its metrics_path; a site_dir the manifest names is made if none of its files were kept. A table or query dataframe is in the store as parquet while the manifest names its database, so its page in a laid-out version says it is not built; the catalog’s renderer (step 2) will read the parquet instead.

   • Cost. Making the index hashes every file again, as a docs build would; for large data that doubles the hashing a snapshot does. Accepted for an index that says exactly what a build of that commit would say.

   • A second review found more, all fixed with tests and every exploit script rerun: a tracked decoy whose name differs from an ignored file’s only by a trailing space or a \ (git’s names are now taken exactly, never put in order, and decoded losslessly); .git aliases that HFS+ and NTFS make of other names (the rule is now git’s own: case, the invisible code points HFS+ drops, trailing dots and spaces, git~1), with a last check after layout that removes any .git and refuses the folder; a store object folder that is itself a link (the folder store now refuses one); a file swapped for a link between being looked at and being read (a document is opened without following a link and must be the very file looked at); and the version file honoured where it does not belong (it is read only in a folder that is not a repository’s own and where git does not track it, never when too large, and a laid-out version never has git asked about its code time). A named file git does not track is now said too.

2. The catalog renders snapshots. Working copies as today, plus the newest snapshot of each pipeline, each version at its own path. This proves rendering from the store. Built 2026-10-16, with every snapshot rather than the newest: a version’s site is built once on a machine and kept, so all of them cost what one did. Each Pipeline record of the index lists its snapshots in the store, newest first. A catalog build lays each one out from the store (materialize), builds it as the pipeline’s own site in a process of its own, with the site tool its manifest names, keeps it under versions/ in the machine’s cache, keyed by the ChartBook version, and copies it into the catalog’s site at versions/<pipeline>/<commit>-<made_at>/. A version that cannot be built gets a page that says why, and a warning: line, so a link to it never leads nowhere. A laid-out folder’s Pipeline record says from_snapshot, and its front page says which snapshot it is. Built otherwise than item 4 of the section above: the version list is one catalog page, cb/versions.md, every pipeline with its working version and its snapshots, the commit, the date, whether chartbook run built it and which pipeline of the catalog pins it; not a list on each pipeline page, because the catalog’s own pages are written on every build and a pipeline’s page is not when an incremental build reuses it. Search is scoped by construction: each version is its own site, so the catalog’s search finds the working versions only.

   After the review, 2026-10-16. Building versions automatically means running a docs build on files anyone who can write the store may have written, so a version’s build runs as a build of files that are not the user’s own (chartbook.config.untrusted_source): every template, a project’s own included, renders in Jinja’s sandbox, and a version takes no template from its folder at all; its site folder gives only pages and pictures; no page takes in a file (file_insertion_enabled off) and no picture or download that names a file outside the site is copied; notebooks are never run, as in any build; and each version’s build has a time limit. Two holes the review found are closed for every build, not only versions’: conf.py is rendered from the package’s own template, never from the build tree where a site page named conf.py.j2 could stand in for it, and a site page may not be conf.py*, _templates/ or the tool’s configuration. A snapshot record is checked when read (a full commit hash, a time as a snapshot writes one, a pipeline ID, a plain subdirectory), and . and .. are no longer pipeline ID parts, so that no value in a record becomes a path outside the site or the cache; the review showed a forged made_at could otherwise delete and write folders anywhere. One record that cannot be read no longer hides every version; only a catalog’s index lists snapshots, so a pipeline’s published index carries no store history; the store is read twice per catalog build rather than once per pipeline; the kept sites are keyed by ChartBook’s version and a fingerprint of its code, so an edit to ChartBook builds them again; a version whose snapshot store gc removed is removed from the site, the incremental build’s tree included; and a staging folder a stopped build left is cleared.

   After a second review, 2026-10-16. The restrictions above held against forged snapshots, and the data path did not: a forged manifest could name a DuckDB query that reads any file of the machine, or a dataframe path outside the folder, and the version’s pages showed the rows. In a build of files from a store, ChartBook now runs no table and no query at all (DuckDB refuses to open; the snapshot holds those rows as parquet, and the page says so), and the manifest loader refuses any path in [project] or an asset table (path and every *_path) that is not a plain path inside the folder, so that version’s build stops and its page says why. Both forgeries were rerun against the fix: neither secret reaches the site.

3. All versions. Every snapshot, links following pins, the version list, snapshot pages kept across builds, search scoped. What remains after step 2: links following pins, so that a page of C links to A at the version C pinned, inside a version’s site as in the catalog; a catalog entry by pipeline ID alone, for a pipeline whose folder is not on this machine; and clearing the kept sites, in the cache, of snapshots store gc removed. Built 2026-10-16, in part, after a review: the only input of another pipeline a page has is a model’s training data. In a catalog, a model’s “Trained On” links each such input to its page in the snapshot that held the bytes it was trained on (the index records it as trained_from, from trained_with and the store), and says when the pin has since moved; failing that, to the snapshot the lockfile pins for that input; failing that, to the working version, as before. The provenance panel links an input the lockfile pins to its page in the pinned snapshot; it linked nothing before. A link leads into a version only when the catalog shows that snapshot, and to the page that says why when its site could not be built, so the catalog builds the versions’ sites before its own pages. An incremental build regenerates a pipeline’s pages when any of this changes. store gc clears the kept site of every snapshot it removes. Not built: the same links inside a version’s own site, which is built once and kept, and so cannot know which other versions a catalog will show; its pages say the pinned commit and date in words. And a catalog entry by ID alone, which is the next step. Built 2026-10-16: a [pipelines] entry with no path names a pipeline whose folder is not on this machine; the catalog’s index has a Pipeline record for it with location: {kind: store}, named and described from its newest snapshot’s own index, and its snapshots; its page links its newest version and the Versions page; the manifest keeps it apart from the folder pipelines (store_pipelines), so check, run, the incremental plan and the policy pass it by. That completes step 3. Built 2026-10-16, after review: what a snapshot says of its pipeline is someone else’s text reaching the catalog’s own, trusted build, so the writer shows its name, description and maintainer as plain text (each ASCII punctuation character a character reference) and its links only when they are plain web addresses; the description is the newest snapshot’s, the one the page links. A [pipelines] entry takes only path and disabled, and a folder of the entry’s ID fills it in. A second review found that a page title (MkDocs) and Sphinx’s search results decode character references and set the text into HTML again, so HTML’s own characters (< > & " ', backtick) are replaced by look-alikes first; the stored index is read only as a regular file, not through a link, and up to a limit. Not changed: a version’s own site is its owner’s pages, raw HTML included, served with the catalog, as a listed repository’s pages are; the guide says so. A third review (2026-10-16): a character whose normalized form (NFKD) holds one of HTML’s own is replaced too, with look-alikes that no normalization turns back, and & becomes “and”; other text stays as written; a name of blank fillers is no name; the stored index is read up to 8 MiB. It also found that the S3 store wrote a file for any key under an object or under snapshots/, .. included, outside its cache: a key that is not a plain path is now no file. A fourth review found that the fix hid legitimate files named with : or \ on systems where those are ordinary (now refused on Windows only), that gc stopped on an object of stray keys, and that gc read whatever file a store’s root note named (now only a local chartbook.lock). A fifth found that a lockfile on a Windows network drive is noted by a network path, which gc then took as gone; a note gc does not read now keeps its pins and is never forgotten.

4. An assistant file for generated sites, written from the index. Built 2026-10-16: it is called llms.txt, after the llmstxt.org convention that agents already look for, and sits beside index.json at the root of every site. The site’s writer writes it from the index: a title and a summary, then a list each of pipelines, dataframes, charts and models, each line the name linking its page, the ID, and the description cut to one line. A version’s file says which snapshot the site is; a catalog’s lists each pipeline’s versions, each linking that version’s own llms.txt, or the page that says why its site could not be built. Text from a record is one line, with what would start a link escaped, so that the only links in it are the writer’s (in CommonMark; a renderer that links bare addresses may link one in a description). After review, the same day: the summary is the file’s own words only, and a description is in quotation marks after them, since a description at the start of a line made headings and lists (a version site’s summary was its owner’s description); a version’s file says its names and descriptions are its owner’s; a header says how to use an ID; each version is a line of its own, the newest ten, then the versions page.

5. A development build, later if wanted: a folder with gaps rendered with its assets marked not built and its notebooks unexecuted.

What this does not decide

How a version is named on a page (the commit’s start and the date is the proposal); whether a snapshot should also carry template overrides from docs_src/_templates; whether the version list should be a drop-down in each theme; what the assistant file is called (decided 2026-10-16 when it was built: llms.txt).