chartbook.stores.s3

A store in an S3 bucket: [store] url = "s3://bucket/prefix".

Needs boto3, which pip install chartbook[s3] installs. The bucket is laid out exactly as a folder store is, under the prefix: store.json, objects/sha256/<2>/<62>/<file>, roots/<note>.json and snapshots/<commit>/<pipeline>/<stamp>.json. So aws s3 sync of a folder store into a bucket is this store, and back.

Readers open files, so what they read is a local copy: :meth:S3Store.get downloads an object’s files once into a cache folder on this machine, laid out as a folder store too, checks that they hash to the object’s name, and returns that folder. Each read while online is one listing request, which is how a name another pipeline added for the same bytes is seen; offline, an object that was read and checked before is served from the cache. Nothing else in ChartBook changes for a store that is remote; data.path() gives the path of the cached copy.

Credentials are the AWS SDK’s own: the environment, ~/.aws, an instance role. The URL never carries them. It may say where the service is, for a bucket that is not at AWS::

s3://bucket/prefix?endpoint=https://minio.example.org&region=us-east-1

A store that is read-only for this machine (s3:ListBucket and s3:GetObject) serves chartbook pin, chartbook run --frozen and data.load(); what would write raises :class:~chartbook.store.ReadOnlyStoreError and says what was refused. Writing needs s3:PutObject, and store gc s3:DeleteObject.

One thing the folder store does that this one cannot: make an object appear whole or not at all. Files are uploaded one by one, so a snapshot interrupted while uploading an object of several files leaves a partial one, which has() reports (a listing cannot tell). What tells is the record: :meth:S3Store.incomplete compares the keys under the object and their sizes with the files and sizes the record gives, as the folder store compares a synced folder’s files, so pin, update, pin --frozen and a read all say the object is not fully uploaded yet; get() refuses one whose files do not hash to its name. Making the snapshot again uploads what is missing and completes it.

Module Contents

Classes

S3Store

A store in an S3 bucket, with a cache of what was read on this machine.

Data

URL_OPTIONS
VERIFIED_DIR

API

class chartbook.stores.s3.S3Store(url: str, cache: Union[str, pathlib.Path, None] = None, client=None)

Bases: chartbook.store.Store

A store in an S3 bucket, with a cache of what was read on this machine.

Parameters:

• url – s3://bucket/prefix, with ?endpoint= and ?region= allowed; see the module.

• cache –

  The folder the downloaded objects and records are kept in. By default a folder for this bucket and prefix under

  func:

  chartbook.config.get_cache_path.

• client – A boto3 S3 client to use instead of making one; for tests, and for a program that has one already. It must be a boto3 client or behave as one, exceptions.NoSuchKey included.

Raises:

ValueError – If boto3 is not installed, or the URL is not a store URL.

Initialization

add_root(lockfile: Union[str, pathlib.Path], hashes: Iterable[str], snapshots: Iterable[tuple[str, str]] = (), machine: Optional[str] = None, labelled: Iterable[tuple[str, str, str, str]] = ()) → None

age_of(content_hash: str, now: Optional[float] = None) → float

Seconds since anything under the object’s prefix was written to the bucket.

all_snapshot_files() → list[pathlib.Path]

property cache: pathlib.Path

The folder on this machine that holds what was downloaded.

delete(content_hash: str) → bool

get(content_hash: str) → pathlib.Path

The folder on this machine with the object’s files, downloaded if they were not yet.

has(content_hash: str) → bool

hashes() → Iterator[str]

incomplete(content_hash: str, size_bytes: Optional[int], files: Optional[dict] = None, every: bool = False) → Optional[str]

Why an object is not whole as its record has it, by the keys under it and their sizes; None if it is.

Files are uploaded one at a time, so a snapshot interrupted while uploading an object of several files leaves some of its keys missing, and one being made now has them arriving. The sizes the listing gives are compared with the record’s, as the folder store compares a synced folder’s files (:func:chartbook.store.unsynced_part); one listing request, and nothing is downloaded. Offline, an object this machine downloaded and checked before is whole.

lacks(content_hash: str, names: Iterable[str], single: bool = True) → bool

Whether the bucket’s object lacks a key under one of these names: a put uploads the keys an object lacks.

note_origin(commit: str, name: str, top: str) → None

origins(commit: str, name: str) → list[str]

put(path: Union[str, pathlib.Path], links: Iterable[str] = LINKS) → chartbook.store.StoredObject

Copy what a declared path names into the cache, hash it, and upload it.

links is read for its errors only: a remote store holds copies, whatever was asked for, so the object never changes when the original does. An object the bucket already has is written again in metadata only, so that it counts as stored now and gc does not take it from under a pin in progress.

remove_root(lockfile: Union[str, pathlib.Path], machine: Optional[str] = None) → bool

remove_snapshot(path: pathlib.Path) → bool

root_of(lockfile: Union[str, pathlib.Path], machine: Optional[str] = None) → Optional[chartbook.store.Root]

roots() → list[chartbook.store.Root]

size_of(content_hash: str) → int

snapshot_commits() → set[str]

snapshot_files(commit: str, name: str) → list[pathlib.Path]

verify(content_hashes: Optional[Iterable[str]] = None) → list[chartbook.store.StoreProblem]

Download every object again, into a temporary folder, and report each that does not hash to its name.

The cache is left as it is: what this machine read and checked stays readable, and the problems are the bucket’s. The folder each object is downloaded into is removed after it is checked, so the disk holds one object at a time.

write_snapshot(commit: str, name: str, stamp: str, document: dict) → pathlib.Path

chartbook.stores.s3.URL_OPTIONS

(‘endpoint’, ‘region’)

chartbook.stores.s3.VERIFIED_DIR

‘verified’