Struct IngestionService

pub struct IngestionService { /* private fields */ }

Implementations

impl IngestionService

pub fn new( args: ClientArgs, config: IngestionConfig, metrics: Arc<IndexerMetrics>, cancel: CancellationToken, ) -> Result<Self>

TODO: If we want to expose this as part of the framework, so people can run just an ingestion service, we will need to split IngestionMetrics out from IndexerMetrics.

pub fn subscribe( &mut self, ) -> (Receiver<Arc<CheckpointData>>, UnboundedSender<(&'static str, u64)>)

Add a new subscription to the ingestion service. Note that the service is susceptible to the “slow receiver” problem: If one receiver is slower to process checkpoints than the checkpoint ingestion rate, it will eventually hold up all receivers.

The ingestion service can optionally receive checkpoint high watermarks from its subscribers. If a subscriber provides a watermark, the ingestion service will commit to not run ahead of the watermark by more than the config’s buffer_size.

Returns the channel to receive checkpoints from and the channel to accept watermarks from.

pub async fn run<I>( self, checkpoints: I, ) -> Result<(JoinHandle<()>, JoinHandle<()>)>

where I: IntoIterator<Item = u64> + Send + Sync + 'static, I::IntoIter: Send + Sync + 'static,

Start the ingestion service as a background task, consuming it in the process.

Checkpoints are fetched concurrently from the checkpoints iterator, and pushed to subscribers’ channels (potentially out-of-order). Subscribers can communicate with the ingestion service via their channels in the following ways:

• If a subscriber is lagging (not receiving checkpoints fast enough), it will eventually provide back-pressure to the ingestion service, which will stop fetching new checkpoints.
• If a subscriber closes its channel, the ingestion service will interpret that as a signal to shutdown as well.

If ingestion reaches the leading edge of the network, it will encounter checkpoints that do not exist yet. These will be retried repeatedly on a fixed retry_interval until they become available.