Struct IndexerCluster

pub struct IndexerCluster { /* private fields */ }

An IndexerCluster combines an Indexer with a MetricsService and a tracing subscriber (outputting to stderr) to provide observability. It is a useful starting point for an indexer binary.

Implementations

impl IndexerCluster

pub async fn new( database_url: Url, args: Args, migrations: Option<&'static EmbeddedMigrations>, ) -> Result<Self>

Create a new cluster with most of the configuration set to its default value. Use Self::new_with_configs to construct a cluster with full customization.

pub async fn new_with_configs( database_url: Url, db_args: DbArgs, args: Args, ingestion_config: IngestionConfig, migrations: Option<&'static EmbeddedMigrations>, metric_label: Option<String>, ) -> Result<Self>

Create a new cluster.

• database_url and db_args configure its database connection.
• args configures where checkpoints are come from, what is indexed and metrics.
• ingestion_config controls how the ingestion service is set-up (its concurrency, polling intervals, etc).
• migrations are any database migrations the indexer needs to run before starting to ensure the database schema is ready for the data that is about to be committed.
• metric_label is an optional custom label to add to metrics reported by this service.

pub fn metrics(&self) -> &Arc<IndexerMetrics>

Access to the indexer’s metrics. This can be cloned before a call to Self::run, to retain shared access to the underlying metrics.

pub fn cancel(&self) -> &CancellationToken

This token controls stopping the indexer and metrics service. Clone it before calling Self::run to retain the ability to stop the service after it has started.

pub async fn run(self) -> Result<JoinHandle<()>>

Starts the indexer and metrics service, returning a handle to await the service’s exit. The service will exit when the indexer has finished processing all the checkpoints it was configured to process, or when it receives an interrupt signal.

Methods from Deref<Target = Indexer>

pub fn db(&self) -> &Db

The database connection pool used by the indexer.

pub fn ingestion_client(&self) -> &IngestionClient

The ingestion client used by the indexer to fetch checkpoints.

pub fn metrics(&self) -> &Arc<IndexerMetrics>

The indexer’s metrics.

pub fn pipelines(&self) -> impl Iterator<Item = &'static str> + '_

The pipelines that this indexer will run.

pub async fn concurrent_pipeline<H>( &mut self, handler: H, config: ConcurrentConfig, ) -> Result<()>

where H: Handler<Store = Db> + Send + Sync + 'static,

Adds a new pipeline to this indexer and starts it up. Although their tasks have started, they will be idle until the ingestion service starts, and serves it checkpoint data.

Concurrent pipelines commit checkpoint data out-of-order to maximise throughput, and they keep the watermark table up-to-date with the highest point they can guarantee all data exists for, for their pipeline.

pub async fn sequential_pipeline<H>( &mut self, handler: H, config: SequentialConfig, ) -> Result<()>

where H: Handler<Store = Db> + Send + Sync + 'static,

Adds a new pipeline to this indexer and starts it up. Although their tasks have started, they will be idle until the ingestion service starts, and serves it checkpoint data.

Sequential pipelines commit checkpoint data in-order which sacrifices throughput, but may be required to handle pipelines that modify data in-place (where each update is not an insert, but could be a modification of an existing row, where ordering between updates is important).

The pipeline can optionally be configured to lag behind the ingestion service by a fixed number of checkpoints (configured by checkpoint_lag).

Trait Implementations

impl Deref for IndexerCluster

type Target = Indexer

The resulting type after dereferencing.

fn deref(&self) -> &Self::Target

Dereferences the value.

impl DerefMut for IndexerCluster

fn deref_mut(&mut self) -> &mut Self::Target

Mutably dereferences the value.