Module store

Module store 

Source
Expand description

Analytics store implementation with TransactionalStore support.

This store supports two modes:

§Live Mode

Derives watermarks from file names via bucket iteration at startup, rather than storing them separately. File uploads inherently update the watermark since file names encode checkpoint ranges.

§Migration Mode

When migration_id is set, the store operates in migration mode:

  • Existing file ranges are loaded at startup and updated in-place.
  • Watermark is stored in a separate file: _metadata/watermarks/{pipeline}@migration_{id}.json
  • Conditional PUT with etag is used to prevent concurrent modification of data files

Structs§

AnalyticsConnection
Connection to the analytics store.
AnalyticsStore
Analytics store wrapper that delegates to an inner store mode.
Batch
Rows accumulated across commits, waiting to be flushed to a file.
FileRangeEntry
Entry for a single file range in the index.
FileRangeIndex
Index of existing file ranges for a pipeline.
LiveStore
Live mode - derives watermarks from file names.
MigrationStore
Migration mode - uses explicit watermark files and conditional PUT.

Enums§

StoreMode
The operational mode of the analytics store.
WatermarkUpdateError
Error type for watermark updates.