# Design: stores and backends

```{note}
A design document, written 2026-10-16 after cross-pipeline dependencies were made to work on one
machine ({doc}`overview`, step 4, and PR 65). It plans how ChartBook keeps the same experience for
the person using it, from a laptop with nothing installed to an organization's own storage and
warehouses. Status: **the settings layers, the store interface and the S3 store are built (2026-10-16); the backend interface and reading from Databricks, the first warehouse, are built (2026-10-17); pins in a warehouse are proposed.**
The decisions the maintainer has taken are marked *decided*; the rest is *proposed*.
```

## What this is for

ChartBook should scale to zero and to the enterprise with the same manifest, the same commands
and the same `data.load()` calls. On a laptop, `pip install chartbook` and nothing else: the store
is a folder under the home directory, and the data engine is DuckDB, which the package already
brings. In an organization, the store may be an object store or a database, the tables may live
in a warehouse, and the credentials may come from a secret manager, and the pipeline's repository
must not know any of that.

One rule makes it possible, and the sources follow it already: **the manifest names the kind of
thing; the machine says where it is.** A pipeline's `chartbook.toml` says `backend = "duckdb"` and, for a
file engine, the manifest's public path, and never a credential. Where that engine is on this machine comes from the environment or
the settings. This page extends the rule to the store, and says what it takes to add a store or a
backend without changing anything a pipeline's author sees.

## Two extension points, because data moves differently

ChartBook holds data in two ways, and they generalize separately.

| | A **store** | A **backend** |
|---|---|---|
| Holds | Bytes ChartBook *copied*: snapshots, and the pinned files pipelines read | Tables ChartBook *never copies*: a dataframe declared as a `table` or `query` in a `[sources]` entry |
| Is a fact about | The machine, or the organization | The pipeline |
| Named in | The settings (never the manifest) | The manifest, as the engine's name only |
| Today | A folder, content-addressed | DuckDB |
| Later | An object store (S3, GCS, Azure Blob), a shared drive, a database | Snowflake, Databricks, Postgres, … |

### The store is a setting of the machine, never of the pipeline

*Decided 2026-10-16.* A store is where snapshots of all of a person's, or an organization's,
pipelines are kept. Which store a machine uses is not a property of any one pipeline, so it never
appears in `chartbook.toml`, and no record ever names it. It is read from settings, in layers;
each layer is consulted only where the one above it is silent, key by key:

1. the environment, `CHARTBOOK_STORE`, for CI and a one-off run;
2. the user's own settings, `~/.chartbook/settings.toml`;
3. the organization's settings, system-wide: `/etc/chartbook/settings.toml` on Unix and macOS,
   `%ProgramData%\chartbook\settings.toml` on Windows, or the file `CHARTBOOK_SYSTEM_SETTINGS`
   names;
4. the default, `~/.chartbook/store`, so a machine with nothing configured works.

So an organization sets the store once, for every machine, and a person keeps their own choices
for everything else. A person who sets their own store overrides the organization's for
themselves, which is the right way round for a laptop that is sometimes offline.

The setting is a URL, not a path. A bare path is a folder, as today. `s3://bucket/prefix` is an
object store. A database URL is a database used as a store, which is what an organization that
runs everything in one warehouse wants. The key is `[store] url`; `[store] path` is read as a
folder URL for one release of notice, then goes.

### A backend is named by the pipeline, located by the machine

*As built.* Which engine a pipeline reads from is a fact about the pipeline, so `[sources.x]
backend = "duckdb"` stays in the manifest, with the manifest's public `url` for a file engine.
Where that engine is on this machine, and the credentials, come from `CHARTBOOK_SOURCE_<NAME>_URL`
or `~/.chartbook/sources.toml`; reading `sources.toml` in the same layers as the store is
proposed, not built. One pipeline
can read from Postgres while the organization's store is in Snowflake, and nothing in the
repository knows either address.

## Scales to zero: plugins behind extras

*Proposed; built for stores 2026-10-16.* `pip install chartbook` installs no cloud library and no warehouse driver. Each store
kind and each backend is a plugin found by an entry point, as plotting backends already are
({doc}`../guide/plotting-backends`): the groups `chartbook.stores` and `chartbook.backends`. An
extra installs one: `pip install chartbook[s3]`, `chartbook[snowflake]`. The local folder store
and DuckDB are in the package itself and are what you get with no extra. A setting or a manifest
that names a plugin that is not installed gets one message, naming the extra to install. *As
built:* the S3 store's module ships in the package and is registered in `chartbook.stores`; only
boto3 is behind the extra, and a store URL that needs it says `pip install chartbook[s3]`.

Credentials stay where they are now: never in a manifest, never in a record, never in a
snapshot. They come from the environment, the settings files, or whatever the organization's
secret manager hands to the process.

## The store interface

*Proposed.* Everything that stores or finds bytes already goes through one class,
`chartbook.store.Store`, and every record refers to bytes by content hash, so the records do not
change when the store does. The class becomes an interface with the local folder as its first
implementation:

| Method | What it does |
|---|---|
| `put(path) -> StoredObject` | Store what a declared path names; the object's name is the hash of what the store holds |
| `has(hash)`, `get(hash)` | Whether an object is there; where it is, as something a reader can open |
| `hashes()`, `size_of(hash)`, `age_of(hash)`, `delete(hash)` | What `verify` and `gc` need |
| `add_root`, `remove_root`, `roots()` | The lockfiles that keep objects alive |
| `write_snapshot`, `read_snapshot`, `snapshots(pipeline, commit)` | The snapshot records, which today are JSON files under `store/snapshots/`; four places in `chartbook.snapshot` reach into the folder for them and move behind this |

Three things tie the folder store to a local disk, and each is a seam, not a wall. `get()` hands
back a filesystem path and the readers open the parquet there; a remote store hands back a URL,
and the readers scan it through Polars, which reads `s3://` paths already, so `data.path()` may
then answer with a URL. `put()` makes objects by reflink, hard link or copy; those are local
speed tricks, and an upload replaces them. And the snapshot folder is listed directly; the
interface's snapshot methods replace that.

*Built otherwise, 2026-10-16:* the S3 store's `get()` hands back a path after all: it downloads
the object's files once into a cache on this machine (`~/.chartbook/cache/`, laid out as a folder
store), checks that they hash to the object's name, and returns that folder. So every reader,
every model loader and `chartbook path` work unchanged, a file is read from the bucket once and
then from disk, and a corrupt download is refused before anything reads it. `data.path()` is the
path of the cached copy. A URL-returning `get()` remains possible for a reader that wants to scan
a bucket directly, and nothing built here is in its way.

The interface was extracted before the second implementation, against the maintainer's
earlier preference, because every caller already went through one class and the snapshot
module's four folder accesses were the only thing left to move; a plugin will reshape it if it
must. The first remote store is S3 through one object-store library, behind
`chartbook[s3]`; the headline use is a build server running `chartbook run --frozen` against a
read-only shared store, which already needs no write to it.

## Pins in a warehouse

*Proposed.* A table in a DuckDB file is pinned by writing it out as parquet into the store (PR
65). A table in a warehouse should not be copied out: a warehouse with time travel, Delta
versions, Iceberg snapshots, Snowflake's `AT`, can pin a table as a *version or a timestamp*, and
`data.load("rates:curve")` then runs the query at that version. The lockfile record keeps the
source's `location` and gains a `pinned_at` (the version or timestamp) where it has a
`content_hash` today; the content hash of the result is recorded too when the engine can give
one cheaply, so staleness is still a comparison of hashes. An engine without time travel pins by
copy, as DuckDB does. What a backend must provide, then, is: scan a table or query to Arrow,
describe it, say whether it can pin by version, and run a pinned query. The interface is
formalized from what DuckDB does today, and the first warehouse adapter shapes the rest.

## The order of work

1. This page, and the settings layers with `[store] url`. *Built 2026-10-16:* the organization's file is read where the user's is silent, key by key; each layer is put in order before the merge, so a user's `[store] path` (read as `url` with a warning) still wins over the organization's `url`; a fault in the organization's file is a warning and the file is ignored, a fault in the user's names the file; a relative folder is relative to its settings file.
2. The store interface extracted, the folder store as its implementation, a test double, and the
   snapshot folder access moved behind it. No behaviour changes; the matrix and the live examples
   prove it. *Built 2026-10-16:* `chartbook.store.Store` is the interface (objects, roots, snapshot
   records), `FolderStore` the implementation; `store_for_url()` makes a store from the settings'
   URL and finds another kind by its scheme in the entry-point group `chartbook.stores`;
   `tests/test_store_contract.py` is what any kind must pass. In place of a test double, the
   contract test runs against each implementation.
3. The S3 store behind `chartbook[s3]`; readers accept URLs. *Built 2026-10-16, with readers taking
   paths through a cache rather than URLs (above):*
   `chartbook.stores.s3.S3Store`, found by the scheme `s3` in the entry-point group; the bucket
   is laid out exactly as a folder store, so a folder store synced into a bucket is this store;
   objects and records are read through a cache on this machine, so the readers take paths as
   before (see above); credentials are the SDK's own and never in the URL, which may carry
   `?endpoint=` and `?region=`; a read-only bucket serves `pin`, `run --frozen` and `data.load()`,
   and a write to it raises `ReadOnlyStoreError`, which `pin` treats as it treats a read-only
   folder; offline, an object read and checked before is served from the cache; the contract
   tests run against it in a mocked bucket. A table in a DuckDB file is pinned as parquet into
   the bucket as into a folder. One thing a bucket cannot do that a folder does: make an object
   of several files appear whole or not at all; a snapshot interrupted mid-upload leaves a
   partial object that readers refuse by name, and making the snapshot again completes it.
   *Amended 2026-10-18, after a review found `pin`, `update` and `--frozen` taking such an
   object as whole:* the S3 store tells a partial object as the folder store tells a half-synced
   one, by the listing against the record's files and sizes (`S3Store.incomplete`, one listing
   request), so the three say it is not fully uploaded yet and `update` does not move to it; a
   read through a lockfile asks `get()` before it asks whether the object is whole, so it meets
   the store's refusal by name, which says the same.
4. The backend interface formalized from DuckDB; the first warehouse adapter with version pins.
   *Built 2026-10-17, all but the pins:* `chartbook.backends` is a package; `Backend` is the
   interface (`ref`, `sql`, `scan`, `describe`, `span`, `connect`, `check_url`, and `reads_files`
   for whether a query there may name a file); a backend other than DuckDB is found by its name in
   the entry-point group `chartbook.backends`; `chartbook.backends.databricks.DatabricksBackend`
   reads a Databricks SQL warehouse through the Databricks SQL connector, behind
   `chartbook[databricks]`, with a lazy frame that asks for the columns at once and the rows when
   collected. One rule was added that this page did not foresee: a warehouse's credentials are sent
   only to the host the machine names (its URL for the source, or `DATABRICKS_HOST`), and a host a
   manifest or a record names is a check against it, never used in its place, since anyone who
   can write a manifest or a shared store could otherwise send this machine's token to their own
   server. A snapshot records a warehouse table without a copy until the pins are built.

## What this does not decide

Which warehouse comes first, Snowflake or Databricks; the first one shapes the backend
interface. *Decided by the maintainer 2026-10-17: Databricks, which has a free edition that does
not expire to test with.* Whether a warehouse pin should be a version where the engine offers one and a copy
only where it does not, which is the recommendation here. How a database used as a *store*
lays out objects and snapshots inside it. Each is marked where it arises, and each waits for the
maintainer.