Skip to main content

Hashi Node Backups

Hashi automatically backs up the data a node needs to rejoin the committee after data loss. Every epoch, Hashi writes the node config, referenced key files, and database snapshot to a tar archive, then compresses and encrypts it as an ASCII-armored PGP message. The backup files use the suffix .tar.asc. The key files and database snapshot can contain the node's operator, TLS, and MPC private keys (see Suspected node compromise), so treat a decrypted backup as sensitive key material.

Encryption only needs the public PGP certificate, so the backup key does not need to be present on the server during normal operation. Restore needs access to the matching private key, either as a local PGP secret-key file or through gpg-agent for YubiKey-backed keys. Because you can forward gpg-agent over SSH, you can restore on a remote server using a YubiKey plugged into your laptop instead of plugging the YubiKey into the server itself.

Prerequisites​

The section that uses each item describes it in detail:

  • For a YubiKey backup key: a new YubiKey 5 Series key with firmware 5.7 or newer, a USB-capable machine, oct, GnuPG, YubiKey Manager, jq, and a checkout of the Hashi repository.
  • For a local backup key file: sequoia-sq.
  • For remote backups: a private S3 or GCS bucket, write-only node credentials, and separate recovery credentials kept off-node. See Remote backups for provider-specific authentication and safeguards.

Create a backup key​

We recommend generating a PGP private key entirely on one YubiKey and using its public key to encrypt the node backup. This makes it practically impossible for anyone to decrypt the backup without physical access to the YubiKey.

warning

Do not lose the YubiKey. Without it, you cannot decrypt backups encrypted to its public key.

Use a USB-capable machine and a new YubiKey 5 Series with firmware 5.7 or newer. Connect only this YubiKey; disconnect other smartcards. Normal backups need only its public certificate, not a connected YubiKey.

  1. Install oct, GnuPG (gpg and gpgconf), YubiKey Manager (ykman), and jq. Ensure your system's smartcard service is available.
  2. From a checkout of the Hashi repository, run the setup script in an interactive terminal:
bash scripts/node-operator/provision-yubikey-backup.sh --output-dir ./backup-key
  1. Follow the prompts for an optional User ID (Enter to omit), PIN changes, and physical touches. The script refuses cards that already contain keys, generates the private keys on-device, enables touch for decryption, exports public artifacts, and checks encryption and decryption. Keep the new User and Admin PINs in a secure password manager. Do not replace the factory attestation (ATT) key or certificate.
  2. Save these 4 public files. <fingerprint> is the full encryption subkey fingerprint:
    • yubikey-backup-<fingerprint>.asc
    • yubikey-backup-<fingerprint>.attestation-device.pem
    • yubikey-backup-<fingerprint>.attestation-dec.pem
    • yubikey-backup-<fingerprint>.fingerprints.txt

Use the .asc file directly as backup-pgp-cert in your node configuration. The script also creates YubiKey attestation evidence. If desired, share all 4 public files to assemble a multi-recipient backup certificate, which can verify the attestations.

For YubiKey restores with --use-gpg-agent, import the public certificate and link the card in your normal GnuPG home (the script's test keyring is temporary):

gpg --import "./backup-key/yubikey-backup-<fingerprint>.asc"
gpg --card-status

Replace <fingerprint> with the generated filename's fingerprint and keep the YubiKey connected; if using a separate recovery GnuPG home, use it for both commands.

Preserve the filenames. Store the YubiKey securely; do not share it or the PINs. If setup fails, retain the public files and inspect the card before retrying. You can restart provisioning if no data depends on any newly generated keys. Otherwise, preserve the keys and follow the script's public-artifact recovery guidance: wiping or regenerating them would destroy access to existing backups.

Option 2: Create a local private key file​

If you do not want to use a YubiKey, you can generate a PGP private key locally and encrypt with it. Because a normal file holds the private key instead of a YubiKey, there is a greater risk of unauthorized use of the private key. Store the file carefully.

  1. Install tools:

  2. Generate a new key pair. Be sure to update the name and email to real values.

sq key generate \
  --own-key \
  --name "Your Name" \
  --email "you@example.com" \
  --expiration never \
  --without-password \
  --output private-key.asc \
  --rev-cert revocation-cert.asc
  1. Export the public key to a file. Despite the name, this doesn't actually delete the private key, it just extracts the public key.
sq key delete \
  --cert-file private-key.asc \
  --output public-key.asc
  1. Test encrypting a file to the public key:
echo "test message" > test.txt
sq encrypt \
  --for-file public-key.asc \
  --without-signature \
  --output test.txt.asc \
  test.txt
  1. Test decrypting the file with the private key:
sq decrypt --recipient-file private-key.asc test.txt.asc

Confirm the output matches the original file contents, then delete the test files.

  1. Save the revocation, public key, and private key files in a secure location.

Assemble a multi-recipient backup certificate​

You can combine multiple YubiKey recipients into one OpenPGP certificate. Any selected recipient can independently decrypt the backup; this is 1-of-n, not threshold recovery. Participants exchange only public certificates, so the YubiKeys never need to be in the same location. Hashi also supports software-key recipients.

This command requires the optional backup-assemble-cert feature. Build it with cargo build -p hashi --release --features backup-assemble-cert; the binary is target/release/hashi. Normal node builds omit this command and its GPG-agent dependencies.

The certifier's primary key must support certification and signing and be available through gpg-agent. Pass its public certificate to --primary-cert:

hashi backup assemble-cert \
  --primary-cert certifier-public.asc \
  --recipient alice-public.asc \
  --recipient bob-public.asc \
  --output hashi-backup-cert.asc

Repeat --recipient for each public recipient certificate. If a certificate has multiple usable encryption subkeys, select one with --recipient "alice-public.asc=FULL_ENCRYPTION_SUBKEY_FINGERPRINT". The certifier is not automatically a recipient; list it explicitly if needed. Use --gpg-homedir PATH to select a different GnuPG home. The output must not already exist; configure it as backup-pgp-cert below.

The command reports each key's expiry. A finite certifier expiry limits the whole certificate: replace it before then to avoid backup and node-startup failures. If the certifier advertises SEIPDv2, every recipient must support it; a v1-only certifier can include recipients that also support v2.

To verify a recipient's YubiKey provenance, add --recipient-attestation SUBKEY_FINGERPRINT DEVICE_PEM DEC_PEM, supplying the device attestation certificate and DEC statement. Repeat for each attested recipient. Supplied evidence is always verified; add --require-recipient-attestation to require it for every recipient.

For example, using the files exported during YubiKey setup (replace FULL_ENCRYPTION_SUBKEY_FINGERPRINT with the [E] subkey fingerprint from the recipient's .fingerprints.txt file, without spaces). Replace the recipient paths with the matching files produced by the script:

hashi backup assemble-cert \
  --primary-cert certifier-public.asc \
  --recipient yubikey-backup-FULL_ENCRYPTION_SUBKEY_FINGERPRINT.asc \
  --recipient-attestation FULL_ENCRYPTION_SUBKEY_FINGERPRINT \
    yubikey-backup-FULL_ENCRYPTION_SUBKEY_FINGERPRINT.attestation-device.pem \
    yubikey-backup-FULL_ENCRYPTION_SUBKEY_FINGERPRINT.attestation-dec.pem \
  --require-recipient-attestation \
  --output hashi-backup-cert.asc

For multiple YubiKeys, collect each recipient's public certificate, device PEM, DEC PEM, and .fingerprints.txt file under distinct filenames. Repeat both --recipient and --recipient-attestation for each recipient. With --require-recipient-attestation, software-generated keys cannot be recipients.

Verification checks the device certificate against pinned Yubico issuers and checks that the DEC statement attests on-device generation of the selected key. It does not establish current possession of the YubiKey or enforce a touch policy, and it does not check X.509 expiry or revocation.

Before relying on the certificate, make a backup and confirm that each recipient can restore that same archive independently.

Run hashi backup test --backup-pgp-cert hashi-backup-cert.asc --use-gpg-agent for a temporary-message round trip with a YubiKey or GnuPG key, or replace --use-gpg-agent with --backup-pgp-secret-key private.asc for a local private key. Test files are automatically cleaned up on success or error.

Configure Hashi backups​

Every node config must provide backup-pgp-cert (a certificate file path or inline armored text) and backup-dir:

backup-pgp-cert = "/path/to/hashi-backup-cert.asc"
backup-dir = "/var/lib/hashi/backups"

The backup-pgp-cert file is the public key you created in Create a backup key: the yubikey-backup-<fingerprint>.asc file from Option 1, public-key.asc from Option 2, or a certificate from Assemble a multi-recipient backup certificate. It is the public half of the backup key pair. Restore uses the matching private half, passed as --backup-pgp-secret-key (for example, private-key.asc from Option 2) or reached through --use-gpg-agent for YubiKey-backed keys.

Use a persistent, node-specific backup directory outside the database directory, with write access restricted to the node's user.

Automatic cleanup​

On epoch housekeeping, before the save, Hashi applies these rules in backup-dir:

  1. Hashi deletes archives older than 14 days by filename timestamp, but keeps the newest current-epoch MPC recovery archive until it saves a newer one or that epoch ends. Ordinary backups cannot replace that recovery copy. Hashi keeps the newest archive of any kind while the node is still attempting saves, or up to 30 days otherwise. Archives that manual backups place in backup-dir follow the same policy.
  2. Hashi removes abandoned .hashi-backup-* files and .hashi-restore-* directories older than 14 days by modification time, without retaining the newest staging entry.
  3. Hashi leaves recent staging entries, symlinks, and unrelated files alone.
  4. Entry errors do not stop the sweep, and a cleanup failure does not prevent the backup attempt.

Each sweep logs its epoch, directory, whether anyone requested a backup, and removed/failed counts; expiring the newest archive also warns.

The node stages archives in backup-dir, then finalizes and syncs them to disk before publication; it syncs the directory afterward on a best-effort basis. It logs a directory sync failure at error level without failing the already-published backup. Failed saves remove their temporary files and never overwrite existing archives.

What operators must clean up​

Automatic cleanup does not cover the following; handle them yourself:

  • A powered-off node sweeps nothing, so delete a decommissioned host's archives yourself.
  • Manual backups do not trigger cleanup. Apply backup expiry to all copies except the monthly off-machine copies.
  • Restore staging and completed restore output contain decrypted keys and database data; use a private restore output directory. Automatic backup cleanup does not sweep completed restores, .hashi-db-restore-* staging directories beside the destination database, or restore output elsewhere. Remove these plaintext copies when you no longer need them.

Backup expiry​

Use a 14-day retention target, measured from the archive's original creation, for every backup copy except the monthly off-machine copies: local archives (including the newest), other off-machine copies, and all versions or generations of remote uploads. Copying or re-uploading must not reset this target; Remote backups describes provider-specific versioning and deletion timing.

Active nodes check daily and refresh their current-epoch MPC recovery backup once it is 3 days old, even during stalled reconfiguration. This intentionally preserves recovery of the active shares; it does not extend retention for historical epochs. If refresh is failing, preserve the last usable recovery copy and follow the incident guidance below rather than deleting it to meet the target.

Old shares can still reconstruct the active MPC key after resharing. Keeping backups indefinitely gives an attacker more time to collect enough decryptable backups from the same epoch to recover that key. Expiry limits this opportunity while preserving a recent disaster-recovery window. You keep the monthly off-machine copies longer, for three months, so that you can recover the key even if an attacker wipes every node.

Operators must configure deletion in each storage location, leaving out only the monthly copies, which you destroy by hand under their own rule; Hashi's newest-backup exception alone does not enforce this policy. Account for retention locks and holds, and monitor deletion failures or delays. Remove temporary decrypted copies immediately after use.

During an incident, pause expiry for backups needed for recovery: disable the applicable cloud lifecycle rules and other off-machine deletion jobs, and stop destroying monthly copies. Hashi's local cleanup still runs, so preserve encrypted copies outside backup-dir in a restricted location exempt from deletion. Confirm the copies are accessible; changing lifecycle rules is not instantaneous. Resume expiry and remove incident copies when recovery no longer needs them.

Monthly off-machine copy​

Once a month, copy the backup from the first epoch that starts in that month (UTC) to storage that none of your nodes can reach, since anyone who takes over a node can delete what its credentials permit. Do not put it in the remote bucket that receives automatic backups. Use that epoch's -mpc- archive, which holds the node's recovery record. Recovery only combines records from the same epoch, so every operator should copy the same epoch. Copy it within 14 days, before local cleanup and cloud lifecycle rules remove it.

Test each copy when you make it, on a machine outside the validator fleet, by exporting its recovery record with hashi backup export-mpc-recovery-record to your backup-pgp-cert, then delete the export. If that epoch has no -mpc- archive or the export fails, keep your older copies and find out why. Do not copy a different epoch instead.

Keep each copy for three months after its epoch, then destroy it everywhere it is stored, including old versions, replicas and snapshots. Destroy expired copies only after this month's copy passes its test, unless your node has left the committee. Keep a way to decrypt every copy you still hold, even after you replace backup-pgp-cert.

Restore from a backup​

hashi backup restore decrypts an archive and extracts it into a directory you choose. It installs nothing. The full procedure, from choosing a backup to restarting the node, is in Backup and Restore. For remote restore commands and an on-page recovery checklist, see Restore and infrastructure limits and Machine-loss recovery below.

Remote backups​

Configure one remote destination using a [backup-remote] table. For S3, region is required:

[backup-remote]
destination = "s3://my-hashi-backups/"
region = "us-east-1"

For GCS, omit region:

[backup-remote]
destination = "gs://my-hashi-backups/"

Use a bucket-only URI; Hashi supplies the object prefix. Configure the bucket permissions, retention, and audit logging described below. Hashi does not set up or verify these controls.

S3 safeguards​

Uploads and restores use the AWS default credential chain (environment, profiles, or IAM roles), not credentials in the node config. The secure Mainnet S3 baseline requires operators to configure:

  • A private bucket with S3 Versioning and Object Lock with bucket-default retention; Hashi does not set retention per object.
  • A bucket policy explicitly denying object-creation s3:PutObject requests without If-None-Match: * (s3:if-none-match), rather than relying on the client's header alone.
  • A node role restricted to s3:PutObject within its prefix, without object read or delete, bucket or retention administration, or governance-retention bypass. Restrict reads to approved recovery roles and keep their credentials off-node.
  • CloudTrail read and write object data events scoped to the backup bucket, retained in an independent audit store outside backup writers' permissions. Use per-node credentials for attribution: events identify the credential principal, not a trusted archive creator.
  • A 14-day remote retention target, using lifecycle expiration:
    • Expire current objects after 14 days (Expiration.Days: 14).
    • Delete noncurrent versions after 1 day (NoncurrentVersionExpiration.NoncurrentDays: 1), without retaining a minimum number of newer versions.
    • Remove expired delete markers with a separate rule (Expiration.ExpiredObjectDeleteMarker: true). Apply these rules to the backup prefix (or the whole dedicated backup bucket). Current expiry creates a delete marker; the data becomes noncurrent and is eligible for deletion 1 day later, not another 14 days later. Lifecycle uses S3 upload time, so late uploads need separate expiry tracking. Align bucket-default Object Lock retention with operator policy and verify it before locking. Object Lock can delay deletion, and lifecycle processing is asynchronous: this is not a hard deadline or a guaranteed recovery window.

GCS safeguards​

GCS uses Application Default Credentials (ADC). Prefer an attached service account or workload identity. Keep recovery credentials on a separate trusted host, and never put credentials in node configuration.

  • Bucket: enable uniform bucket-level access and public access prevention.
  • Permissions: grant each node roles/storage.objectCreator within its backup prefix, without read, delete, or administrative access. Recovery identities need storage.objects.get, plus storage.objects.list to inspect history.
  • Retention lock: enable Object Versioning and Bucket Lock. Check the retention period first: locking is irreversible and the period cannot be reduced.
  • Audit: enable Data Access logs for DATA_READ and DATA_WRITE (disabled by default). Use per-node identities and retain logs outside their control. Lifecycle deletions are not logged.
  • Expiry: apply these lifecycle rules to the backup prefix or dedicated bucket:
    • Delete with age: 14 and isLive: true.
    • Delete with daysSinceNoncurrentTime: 1 and isLive: false.
  • Soft delete: account for its default 7-day retention, which extends storage beyond lifecycle deletion. To recover a soft-deleted object, restore it through GCS first, then use its new generation with Hashi.

Cloud age starts at upload, not archive creation; preserve the original expiry target when copying or recovering objects. Locks, holds, and asynchronous lifecycle processing can delay deletion.

Remote archive locations​

Both providers use the same identity-derived namespace, with no custom prefix:

s3://<bucket>/<sui-chain-id>/<hashi-object-id>/<validator-address>/hashi-backup-epoch-<epoch>-<timestamp>.tar.asc
gs://<bucket>/<sui-chain-id>/<hashi-object-id>/<validator-address>/hashi-backup-epoch-<epoch>-<timestamp>.tar.asc

If the -mpc- marker is present after the epoch number, it identifies backups containing MPC key-recovery information.

Failed uploads leave the local backup intact but are not retried automatically; check logs for failures.

Manual backups​

Stop the node before opening its database for a manual backup:

hashi backup save /path/to/validator.toml --output-dir /var/lib/hashi/backups

Replace /path/to/validator.toml with the path to your node config, and /var/lib/hashi/backups with the directory to write the archive to (for example, your configured backup-dir).

This saves locally, then uploads to the configured remote destination. Upload failure returns a nonzero exit status while preserving the local archive. With a remote destination configured, a --backup-pgp-cert override with different usable encryption recipients requires --local-only or updating the configured certificate.

Restore and infrastructure limits​

Restore from a full remote URI without the original node configuration. Use independently retained receipts to pin an S3 version or GCS generation:

hashi backup restore 's3://my-bucket/<namespace>/<archive>.tar.asc' \
  --region us-east-1 --version-id '<version-id>' \
  --backup-pgp-secret-key /secure/secret-key.asc \
  --output-dir /private/restore
hashi backup restore 'gs://my-bucket/<namespace>/<archive>.tar.asc' \
  --generation '<generation>' \
  --backup-pgp-secret-key /secure/secret-key.asc \
  --output-dir /private/restore

Replace the quoted placeholders with values from your backup receipts: <namespace> is <sui-chain-id>/<hashi-object-id>/<validator-address>, and <archive> is the full archive basename before .tar.asc. Keys are literal object keys, not URL-decoded paths. S3 recovery credentials need s3:GetObject for the current object or s3:GetObjectVersion for a pinned version; inspecting history also requires s3:ListBucketVersions. GCS recovery requires storage.objects.get, plus storage.objects.list to inspect history.

Remote restore accepts only encrypted .tar.asc archives. Choose --backup-pgp-secret-key or --use-gpg-agent (with optional --gpg-homedir). Local restores also accept plaintext .tar archives, which need neither decryptor. --region is required for S3, and --version-id is S3-only. GCS rejects both options and accepts --generation, a positive signed 64-bit integer. S3 rejects --generation; all three remote options are invalid for local restores. Downloads have no failed-request retry or resume.

Downloads use private temporary storage under --output-dir, and the command removes them afterward. Restore publishes a complete extraction, refuses an existing destination, and does not install files at their original paths.

Keep backup URI and S3 version ID or GCS generation receipts, the S3 region, and decryption-key access outside the node. Use independently retained receipts and audit history to select an exact version or generation, then pin it with --version-id or --generation. Neither the newest object nor successful decryption proves its origin: encryption does not authenticate the creator. Unexpected versions, generations, or delete markers warrant investigation, but lifecycle rules and authorized administrators can also change object history.

Machine-loss recovery​

Use this path for accidental machine or disk loss without suspected compromise. For suspected compromise, use the separate procedure below, not a snapshot restore.

  1. Stop or fence the old node. Select a backup using independently kept receipts and restore its pinned version or generation on a trusted replacement host into a private directory (umask 077); you need no original config or database.
  2. Verify the restored chain, Hashi object, and validator identity against trusted records. Install the config and key files at trusted paths, not paths taken blindly from the manifest. Put hashi-db-snapshot at the configured db path; never merge it into an existing database.
  3. Update config paths and endpoints, restrict permissions to the node user, and start the node. Check chain catch-up and committee participation.
  4. Remove plaintext recovery output and temporary key copies when finished. Keep decryption-key access independent of the replacement host.

Suspected node compromise​

Assume the host-held operator, TLS, and MPC private keys have leaked. Even a known pre-compromise snapshot can contain those same keys; its age does not make them safe to restore.

For a single compromised node with sufficient healthy peers:

  1. Fence the old host. From trusted infrastructure, use the validator account authority to revoke the old operator delegation and register fresh operator and TLS identities in onchain metadata. Follow the key generation and registration and operator revocation procedures with the replacement config; do not authorize recovery with the compromised operator key.
  2. Provision a fresh machine with an empty database, not restored key material. Startup generates fresh node encryption and signing keys and registers them; see server startup.
  3. Rejoin through reconfiguration with the healthy peers, and confirm the replacement identities and participation.

Suspected quorum-wide or global MPC key compromise requires coordinated incident response, not this single-node procedure. Resharing preserves the global Bitcoin key; it does not repair a stolen global private key.

Catastrophic MPC key recovery​

A zero-day compromise could steal and delete MPC key shares across the validator fleet. Catastrophic data loss could also leave every node without its shares. Backups allow you to recover the existing MPC key when no functioning committee remains.

Scheduled backups with the -mpc- marker include hashi-mpc-recovery.bcs. This file contains the node's shares and the information needed to combine them with other nodes' backups to recover the MPC key. Recovery needs a sufficient set of backups from the same epoch and works even after old onchain certificates are gone. Manual backups do not include this file.

Operators should keep their monthly off-machine copies and keep access to their backup decryption keys separate from the validator fleet. MPC key recovery is an incident-response operation, not part of ordinary node restoration.

Recovery assembles the full MPC private key and writes it PGP-encrypted. You must still handle it as a highly sensitive secret. Recovering it does not invalidate a copy already stolen by an attacker.

Sharing recovery records​

For a coordinated recovery, use hashi backup export-mpc-recovery-record to decrypt your backup locally and export only its recovery information, encrypted to the coordinator's public PGP certificate. The command prints the recipient certificate's fingerprint; verify it through a trusted channel before sharing the export.

hashi backup export-mpc-recovery-record backup.tar.asc \
  --backup-pgp-secret-key /secure/operator-secret.asc \
  --output-pgp-cert coordinator.asc --output recovery-record.bcs.asc

Use --use-gpg-agent instead of --backup-pgp-secret-key for agent-backed keys. Send only the encrypted .bcs.asc file; you need to share neither your backup decryption key nor the full backup. The command never writes a plaintext record. The coordinator can pass these files to hashi backup recover-mpc-key and decrypt them with their own key.