Structs§
- Committer
Watermark - Represents the highest checkpoint for some pipeline that has been processed by the indexer
framework. When read from the
Store, this represents the inclusive upper bound checkpoint of data that has been written to the Store for a pipeline. - Init
Watermark - Ingested checkpoint range during pipeline initialization.
- Pruner
Watermark - A watermark that represents the bounds for the region that the pruner is allowed to prune, and the time in milliseconds the pruner must wait before it can begin pruning data.
- Reader
Watermark - Represents the inclusive lower bound of available data in the Store for some pipeline.
Traits§
- Concurrent
Connection - Extension of
Connectionfor concurrent pipeline watermark operations. - Concurrent
Store - Extension of
Storefor stores that support concurrent pipeline operations, including task-based pipeline watermark tracking. - Connection
- Represents a database connection that can be used by the indexer framework to manage watermark operations, agnostic of the underlying store implementation.
- Sequential
Connection - Extension of
Connectionfor sequential pipeline operations. - Sequential
Store - Extension of
Storefor stores that support sequential pipeline operations, including transactional capabilities used within the framework for atomic or transactional writes. - Store
- A storage-agnostic interface that provides database connections for both watermark management
and arbitrary writes. The indexer framework accepts this
Storeimplementation to manage watermarks operations through its associatedConnectiontype. This store is also passed to the pipeline handlers to perform arbitrary writes to the store.
Functions§
- pipeline_
task - Check that the pipeline name does not contain the store’s delimiter, and construct the string
used for tracking a pipeline’s watermarks in the store. This is either the pipeline name itself,
or
{pipeline}{ConcurrentStore::DELIMITER}{task}if a task name is provided.