Operating an Aggregator or Publisher

Run a Walrus aggregator or publisher to expose the HTTP API.

Start a local daemon

Run a local Walrus daemon through the walrus binary using one of three commands:

• walrus aggregator: Starts an aggregator that offers an HTTP interface to read blobs from Walrus.
• walrus publisher: Starts a publisher that offers an HTTP interface to store blobs in Walrus.
• walrus daemon: Offers the combined functionality of an aggregator and publisher on the same address and port.

Run an aggregator

The aggregator does not perform any on-chain actions, and only requires specifying the address on which it listens:

$ walrus aggregator --bind-address "127.0.0.1:31415"

Sample systemd configuration

The following example shows an aggregator node that hosts an HTTP endpoint you can use to fetch data from Walrus over the web.

Run the aggregator process through the walrusclient binary using a systemd service:

[Unit]
Description=Walrus Aggregator

[Service]
User=walrus
Environment=RUST_BACKTRACE=1
Environment=RUST_LOG=info
ExecStart=/opt/walrus/bin/walrus --config /opt/walrus/config/client_config.yaml aggregator --bind-address 0.0.0.0:9000
Restart=always

LimitNOFILE=65536

Support large files

As of Walrus v1.38.0 the aggregator can perform concatenation of multiple blobs through the /v1alpha/blobs/concat endpoint. This endpoint enables delivery of very large files, which would otherwise be unsupported due to individual blob size restrictions. The walrus-store-sliced.sh script that follows shows an example of how a very large file can be sliced and uploaded to Walrus. Once the slices are uploaded, the very large file can be read directly by your downstream users by GET URL construction (listing the blob slices in the query params), or through a POST request with a JSON body listing the ids. Details of this api are available in the online aggregator docs. These can be found at <your-aggregator-url>/v1/api. This endpoint is still under development and its specifications or behavior might change prior to becoming stable.

walrus-store-sliced.sh

#!/bin/bash
# Copyright (c) Walrus Foundation
# SPDX-License-Identifier: Apache-2.0

set -euo pipefail

error() {
  echo "$0: error: $1" >&2
}

note() {
  echo "$0: note: $1" >&2
}

die() {
  echo "$0: error: $1" >&2
  exit 1
}

usage() {
  echo "Usage: $0 -f <file> -s <size> [-- <walrus store args>...]"
  echo ""
  echo "Split a file into chunks and store them using walrus store."
  echo ""
  echo "OPTIONS:"
  echo "  -f <file>             Input file to split (required)"
  echo "  -s <size>             Chunk size (e.g., 10M, 100K, 1G) (required)"
  echo "  -h                    Print this usage message"
  echo "  --                    Delimiter for walrus store arguments"
  echo ""
  echo "EXAMPLES:"
  echo "  $0 -f large_file.txt -s 10M -- --epochs 5"
  echo "  $0 -f video.mp4 -s 100M -- --epochs max --force"
  echo ""
  echo "The chunks will be named: basename_0.ext, basename_1.ext, etc."
  echo "Chunks are automatically deleted when the script exits."
}

file=""
chunk_size=""
walrus_args=()

# Parse arguments
while [[ $# -gt 0 ]]; do
  case "$1" in
    -f)
    file="$2"
    shift 2
    ;;
    -s)
    chunk_size="$2"
    shift 2
    ;;
    -h)
    usage
    exit 0
    ;;
    --)
    shift
    walrus_args=("$@")
    break
    ;;
    *)
    error "Unknown option: $1"
    usage
    exit 1
    ;;
  esac
done

# Validate required arguments
if [[ -z "$file" ]]; then
  error "input file (-f) is required"
  usage
  exit 1
fi

if [[ -z "$chunk_size" ]]; then
  error "chunk size (-s) is required"
  usage
  exit 1
fi

if [[ ! -f "$file" ]]; then
  die "file not found: $file"
fi

# Extract basename and extension
file_basename=$(basename "$file")
file_name="${file_basename%.*}"
file_ext="${file_basename##*.}"

# Handle case where file has no extension
if [[ "$file_name" == "$file_ext" ]]; then
  file_ext=""
else
  file_ext=".$file_ext"
fi

# Create temp directory for chunks
temp_dir=$(mktemp -d -t walrus-chunks-XXXXXX)
trap 'rm -rf "'"$temp_dir" EXIT
note "splitting $file into chunks of size $chunk_size in $temp_dir..." >&2

# Split the file into chunks with numeric suffixes
split -b "$chunk_size" "$file" "$temp_dir/chunk_"

# Rename chunks to the desired format: basename_i.ext
chunk_files=()
i=0
for chunk in "$temp_dir"/chunk_*; do
  if [[ "$file_ext" == "" ]]; then
    new_name="$temp_dir/${file_name}_${i}"
  else
    new_name="$temp_dir/${file_name}_${i}${file_ext}"
  fi
  mv "$chunk" "$new_name"
  chunk_files+=("$new_name")
  ((i++))
done

note "created ${#chunk_files[@]} chunks"

# Display the chunks
for chunk in "${chunk_files[@]}"; do
  note "  - $(basename "$chunk")"
done

# Call walrus store for each chunk individually
note "storing ${#chunk_files[@]} chunks..."

for chunk_file in "${chunk_files[@]}"; do
  note "running: walrus store ${walrus_args[*]} $chunk_file"

  if ! walrus store "${walrus_args[@]}" "$chunk_file"; then
    exit_code=$?
    error "✗ walrus store failed with exit code: $exit_code"
    note "failed to store entire file. please address issue above and try again."
    exit $exit_code
  fi
done

note "✓ all chunks stored successfully"

For more information about maximum blob sizes, run walrus info.

Run a publisher

The publisher and daemon perform on-chain actions and thus require a Sui Testnet wallet with sufficient SUI and WAL balances. To enable handling many parallel requests without object conflicts, they create internal sub-wallets since version 1.4.0, which are funded from the main wallet. These sub-wallets are persisted in a directory specified with the --sub-wallets-dir argument. Any existing directory can be used. If it already contains sub-wallets, they are reused.

By default, 8 sub-wallets are created and funded. You can change this with the --n-clients argument. For simple local testing, 1 or 2 sub-wallets are usually sufficient.

Run a publisher with a single sub-wallet stored in the Walrus configuration directory:

PUBLISHER_WALLETS_DIR=~/.config/walrus/publisher-wallets
mkdir -p "$PUBLISHER_WALLETS_DIR"
walrus publisher \
  --bind-address "127.0.0.1:31416" \
  --sub-wallets-dir "$PUBLISHER_WALLETS_DIR" \
  --n-clients 1

Replace publisher by daemon to run both an aggregator and publisher on the same address and port.

warning

While the aggregator does not perform Sui on-chain actions, and therefore consumes no gas, the publisher does perform actions on-chain and consumes both SUI and WAL tokens. Ensure only authorized parties can access it, or use other measures to manage gas costs, especially in a future Mainnet deployment.

Set sub-wallets and upload concurrency

The publisher uses sub-wallets to allow storing blobs in parallel. By default, the publisher uses 8 sub-wallets, meaning it can handle 8 blob store HTTP requests concurrently.

To operate a high-performance and concurrency publisher, use the following options:

• The --n-clients <NUM> option creates a number of separate wallets used to perform concurrent Sui chain operations. Increase this to allow more parallel uploads. A higher number requires more SUI and WAL coins initially to be distributed to more wallets.
• The --max-concurrent-requests <NUM> determines how many concurrent requests can be handled, including Sui operations (limited by number of clients) but also uploads. After this is exceeded, more requests are queued up to the --max-buffer-size <NUM>, after which requests are rejected with an HTTP 429 code.

Manage SUI coins in sub-wallets

Each of the sub-wallets requires funds to interact with the chain and purchase storage. A background process checks periodically if the sub-wallets have enough funds. In steady state, each of the sub-wallets has a balance of 0.5-1.0 SUI and WAL. Configure the amount and triggers for coin refills through CLI arguments.

To tweak how refills are handled, use the --refill-interval <REFILL_INTERVAL>, --gas-refill-amount <GAS_REFILL_AMOUNT>, --wal-refill-amount <WAL_REFILL_AMOUNT>, and --sub-wallets-min-balance <SUB_WALLETS_MIN_BALANCE> arguments.

Understand Blob object lifecycle

Each store operation in Walrus creates a Blob object on Sui. This blob object represents the (partial) ownership over the associated data, and allows certain data management operations (for example, in the case of deletable blobs).

When the publisher stores a blob on behalf of a client, the Blob object is initially owned by the sub-wallet that stored the blob. Then, the following cases are possible, depending on the configuration:

• If the client requests to store a blob and specifies the send_object_to query parameter (see the relevant section for examples), then the Blob object is transferred to the specified address. This is a way for clients to get back the created object for their data.
• If the send_object_to query parameter is not specified, two cases are possible:
  • By default the sub-wallet transfers the newly-created blob object to the main wallet, such that all these objects are kept there. Change this behavior by setting the --burn-after-store flag, and the blob object is then immediately deleted.
  • However, this flag does not affect the use of the send_object_to query parameter. Regardless of this flag's status, the publisher sends created objects to the address in the send_object_to query parameter, if it is specified in the PUT request.

Use authenticated publishers

The setup and use of an authenticated publisher is covered in a separate section.

Limitations

By default, store blob requests are limited to 10 MiB. Increase this limit through the --max-body-size option. Store quilt requests are limited to 100 MiB by default, and you can increase them using the --max-quilt-body-size option.

If the aggregator or publisher is hosted on a third-party platform, ensure that any additional platform-imposed limits or constraints are compatible with your intended use case.

View daemon metrics

Services by default export a metrics endpoint accessible through curl http://127.0.0.1:27182/metrics. Change it using the --metrics-address <METRICS_ADDRESS> CLI option.