Skip to main content

Hashi Node Backups

Hashi automatically backs up the data a node needs to rejoin the committee after data loss. Every epoch, Hashi writes the node config, referenced key files, and database snapshot to a tar archive, then compresses and encrypts it as an ASCII-armored PGP message. The backup files use the suffix .tar.asc.

Encryption only needs the public PGP certificate, so the backup key does not need to be present on the server during normal operation. Restore needs access to the matching private key, either as a local PGP secret-key file or through gpg-agent for YubiKey-backed keys. Because gpg-agent can be forwarded over SSH, you can restore on a remote server using a YubiKey plugged into your laptop instead of plugging the YubiKey into the server itself.

Create a backup Key​

We recommend generating a PGP private key entirely on one YubiKey and using its public key to encrypt the node backup. This makes it practically impossible for anyone to decrypt the backup without physical access to the YubiKey.

warning

Do not lose the YubiKey. Without it, you cannot decrypt backups encrypted to its public key.

Use a USB-capable machine and a new YubiKey 5 Series with firmware 5.7 or newer. Connect only this YubiKey; disconnect other smartcards. Normal backups need only its public certificate, not a connected YubiKey.

  1. Install oct, GnuPG (gpg and gpgconf), YubiKey Manager (ykman), and jq. Ensure your system's smartcard service is available.
  2. From a checkout of the Hashi repository, run the setup script in an interactive terminal:
bash scripts/node-operator/provision-yubikey-backup.sh --output-dir ./backup-key
  1. Follow the prompts for an optional User ID (Enter to omit), PIN changes, and physical touches. The script refuses cards that already contain keys, generates the private keys on-device, enables touch for decryption, exports public artifacts, and checks encryption/decryption. Keep the new User and Admin PINs in a secure password manager. Do not replace the factory attestation (ATT) key or certificate.
  2. Save these 4 public files. <fingerprint> is the full encryption subkey fingerprint:
    • yubikey-backup-<fingerprint>.asc
    • yubikey-backup-<fingerprint>.attestation-device.pem
    • yubikey-backup-<fingerprint>.attestation-dec.pem
    • yubikey-backup-<fingerprint>.fingerprints.txt

Use the .asc file directly as backup-pgp-cert in your node configuration. The script also creates YubiKey attestation evidence. If desired, share all 4 public files to assemble a multi-recipient backup certificate, which can verify the attestations.

For YubiKey restores with --use-gpg-agent, import the public certificate and link the card in your normal GnuPG home (the script's test keyring is temporary):

gpg --import "./backup-key/yubikey-backup-<fingerprint>.asc"
gpg --card-status

Replace <fingerprint> with the generated filename's fingerprint and keep the YubiKey connected; if using a separate recovery GnuPG home, use it for both commands.

Preserve the filenames. Store the YubiKey securely; do not share it or the PINs. If setup fails, retain the public files and inspect the card before retrying. You can restart provisioning if no data depends on any newly generated keys. Otherwise, preserve the keys and follow the script's public-artifact recovery guidance: wiping or regenerating them would destroy access to existing backups.

Option 2: Create a local private key file​

If you do not want to use a YubiKey, you can generate a PGP private key locally and encrypt with it. Because the private key is stored in a normal file instead of on a YubiKey, there is a greater risk of unauthorized use of the private key. Store the file carefully.

  1. Install tools:

  2. Generate a new key pair. Be sure to update the name and email to real values.

sq key generate \
  --own-key \
  --name "Your Name" \
  --email "you@example.com" \
  --expiration never \
  --without-password \
  --output private-key.asc \
  --rev-cert revocation-cert.asc
  1. Export the public key to a file. Note that despite the name, this doesn't actually delete the private key, it just extracts the public key.
sq key delete \
  --cert-file private-key.asc \
  --output public-key.asc
  1. Test encrypting a file to the public key:
echo "test message" > test.txt
sq encrypt \
  --for-file public-key.asc \
  --without-signature \
  --output test.txt.asc \
  test.txt
  1. Test decrypting the file with the private key:
sq decrypt --recipient-file private-key.asc test.txt.asc

Confirm the output matches the original file contents, then delete the test files.

  1. Save the revocation, public key, and private key files in a secure location.

Assemble a multi-recipient backup certificate​

You can combine multiple YubiKey recipients into one OpenPGP certificate. Any selected recipient can independently decrypt the backup; this is 1-of-n, not threshold recovery. Only public certificates are exchanged, so the YubiKeys never need to be in the same location. Software-key recipients are also supported.

This command requires the optional backup-assemble-cert feature. Build it with cargo build -p hashi --release --features backup-assemble-cert; the binary is target/release/hashi. Normal node builds omit this command and its GPG-agent dependencies.

The certifier's primary key must support certification and signing and be available through gpg-agent. Pass its public certificate to --primary-cert:

hashi backup assemble-cert \
  --primary-cert certifier-public.asc \
  --recipient alice-public.asc \
  --recipient bob-public.asc \
  --output hashi-backup-cert.asc

Repeat --recipient for each public recipient certificate. If a certificate has multiple usable encryption subkeys, select one with --recipient "alice-public.asc=FULL_ENCRYPTION_SUBKEY_FINGERPRINT". The certifier is not automatically a recipient; list it explicitly if needed. Use --gpg-homedir PATH to select a different GnuPG home. The output must not already exist; configure it as backup-pgp-cert below.

The command reports each key's expiry. A finite certifier expiry limits the whole certificate: replace it before then to avoid backup and node-startup failures. If the certifier advertises SEIPDv2, every recipient must support it; a v1-only certifier can include recipients that also support v2.

To verify a recipient's YubiKey provenance, add --recipient-attestation SUBKEY_FINGERPRINT DEVICE_PEM DEC_PEM, supplying the device attestation certificate and DEC statement. Repeat for each attested recipient. Supplied evidence is always verified; add --require-recipient-attestation to require it for every recipient.

For example, using the files exported during YubiKey setup (replace FULL_ENCRYPTION_SUBKEY_FINGERPRINT with the [E] subkey fingerprint from the recipient's .fingerprints.txt file, without spaces). Replace the recipient paths with the matching files produced by the script:

hashi backup assemble-cert \
  --primary-cert certifier-public.asc \
  --recipient yubikey-backup-FULL_ENCRYPTION_SUBKEY_FINGERPRINT.asc \
  --recipient-attestation FULL_ENCRYPTION_SUBKEY_FINGERPRINT \
    yubikey-backup-FULL_ENCRYPTION_SUBKEY_FINGERPRINT.attestation-device.pem \
    yubikey-backup-FULL_ENCRYPTION_SUBKEY_FINGERPRINT.attestation-dec.pem \
  --require-recipient-attestation \
  --output hashi-backup-cert.asc

For multiple YubiKeys, collect each recipient's public certificate, device PEM, DEC PEM, and .fingerprints.txt file under distinct filenames. Repeat both --recipient and --recipient-attestation for each recipient. With --require-recipient-attestation, software-generated keys cannot be recipients.

Verification checks the device certificate against pinned Yubico issuers and checks that the DEC statement attests on-device generation of the selected key. It does not establish current possession of the YubiKey or enforce a touch policy, and it does not check X.509 expiry or revocation.

Before relying on the certificate, make a backup and confirm that each recipient can restore that same archive independently.

Run hashi backup test --backup-pgp-cert hashi-backup-cert.asc --use-gpg-agent for a temporary-message round trip with a YubiKey or GnuPG key, or replace --use-gpg-agent with --backup-pgp-secret-key private.asc for a local private key. Test files are automatically cleaned up on success or error.

Configuring hashi​

Every node config must provide backup-pgp-cert (a certificate file path or inline armored text) and backup-dir:

backup-pgp-cert = "/path/to/hashi-backup-cert.asc"
backup-dir = "/var/lib/hashi/backups"

Use a persistent, node-specific backup directory outside the database directory, with write access restricted to the node's user.

Automatic backup maintenance deletes archives older than 14 days by filename timestamp, but preserves the newest current-epoch MPC recovery archive until a newer one is successfully saved or its epoch ends. Ordinary backups cannot replace that recovery copy. The newest archive of any kind is kept while saves are attempted, or up to 30 days otherwise; older unprotected archives expire after 14 days. A powered-off node sweeps nothing; remove decommissioned hosts' archives yourself.

Each sweep logs its epoch, directory, whether a backup was requested, and removed/failed counts; expiring the newest archive also warns. Unrelated files are preserved, entry errors do not stop the sweep, and a cleanup failure does not prevent the backup attempt.

Manual backups do not trigger cleanup, but archives placed in backup-dir follow the same automatic policy. Apply backup expiry to all copies.

Archives are staged in backup-dir, finalized and synced to disk before publication; the directory is synced afterward on a best-effort basis. A directory sync failure is logged at error level without failing the already-published backup. Failed saves remove their temporary files and never overwrite existing archives. Within backup-dir, cleanup removes abandoned .hashi-backup-* files and .hashi-restore-* directories older than 14 days by modification time, without retaining the newest staging entry. Recent staging entries and symlinks are left alone. Restore staging and completed restore output contain decrypted keys and database data; use a private restore output directory. Completed restores, .hashi-db-restore-* staging directories beside the destination database, and restore output elsewhere are not swept by automatic backup cleanup. Operators must remove these plaintext copies when no longer needed.

Backup expiry​

Use a 14-day retention target, measured from the archive's original creation, for every backup copy: local archives (including the newest), off-machine copies, and all S3 object versions. Copying or re-uploading must not reset this target; S3's versioning and deletion timing are described below.

Active nodes check daily and refresh their current-epoch MPC recovery backup once it is 3 days old, even during stalled reconfiguration. This intentionally preserves recovery of the active shares; it does not extend retention for historical epochs. If refresh is failing, preserve the last usable recovery copy and follow the incident guidance below rather than deleting it to meet the target.

Old shares can still reconstruct the active MPC key after resharing. Keeping backups indefinitely gives an attacker more time to collect enough decryptable backups from the same epoch to recover that key. Expiry limits this opportunity while preserving a recent disaster-recovery window.

Operators must configure deletion for each storage location; Hashi's newest-backup exception alone does not enforce this policy. Account for Object Lock and monitor deletion failures or delays. Remove temporary decrypted copies immediately after use.

During an incident, pause expiry for backups needed for recovery: disable the applicable S3 lifecycle rules and other off-machine deletion jobs. Hashi's local cleanup still runs, so preserve encrypted copies outside backup-dir in a restricted location exempt from deletion. Confirm the copies are accessible; changing lifecycle rules is not instantaneous. Resume expiry and remove incident copies when recovery no longer needs them.

Restoring from a backup​

hashi backup restore decrypts an archive and extracts it into a directory you choose. It installs nothing. The full procedure, from choosing a backup to restarting the node, is in the operator runbook.

Cloud backup​

Enable S3 uploads with this optional node-config table; both values are required. Use a bucket name, not an S3 URI. Omit the table for local-only backups.

[backup-s3]
bucket = "my-hashi-backups"
region = "us-east-1"

Uploads and restores use the AWS default credential chain (environment, profiles, or IAM roles), not credentials in the node config. The secure Mainnet baseline requires operators to configure:

  • A private bucket with S3 Versioning and Object Lock with bucket-default retention; Hashi does not set retention per object.
  • A bucket policy explicitly denying object-creation s3:PutObject requests without If-None-Match: * (s3:if-none-match), rather than relying on the client's header alone.
  • A node role restricted to s3:PutObject within its prefix, without object read/delete, bucket/retention administration, or governance-retention bypass. Restrict reads to approved recovery roles and keep their credentials off-node.
  • CloudTrail read and write object data events scoped to the backup bucket, retained in an independent audit store outside backup writers' permissions. Use per-node credentials for attribution: events identify the credential principal, not a trusted archive creator.
  • A 14-day remote retention target, using lifecycle expiration:
    • Expire current objects after 14 days (Expiration.Days: 14).
    • Delete noncurrent versions after 1 day (NoncurrentVersionExpiration.NoncurrentDays: 1), without retaining a minimum number of newer versions.
    • Remove expired delete markers with a separate rule (Expiration.ExpiredObjectDeleteMarker: true). Apply these rules to the backup prefix (or the whole dedicated backup bucket). Current expiry creates a delete marker; the data becomes noncurrent and is eligible for deletion 1 day later, not another 14 days later. Lifecycle uses S3 upload time, so late uploads need separate expiry tracking. Object Lock can delay deletion, and lifecycle processing is asynchronous: this is not a hard deadline or a guaranteed recovery window.

Hashi neither provisions nor preflights these controls. Mysten Testnet's Pulumi configuration defines the bucket and audit safeguards, but leaves read access unchanged. Recovery-only reads remain Mainnet rollout work; node credential provisioning is separate.

The destination is derived from the node identity, with no custom prefix:

s3://<bucket>/<sui-chain-id>/<hashi-object-id>/<validator-address>/hashi-backup-epoch-<epoch>-<timestamp>.tar.asc

If the -mpc- marker is present after the epoch number, it identifies backups containing MPC key-recovery information.

Scheduled backups upload the completed encrypted local archive once, without retries or a queue. Failures preserve local backups and retention, and log the local path and S3 URI.

Manual backups​

Stop the node before opening its database for a manual backup:

hashi backup save /path/to/validator.toml --output-dir /var/lib/hashi/backups

This saves locally, then uploads if configured. Add --local-only to skip S3. Upload failure returns a nonzero exit status while preserving the local archive. With S3 configured, a --backup-pgp-cert override with different usable encryption recipients requires --local-only or updating the configured certificate.

Restore and infrastructure limits​

Restore from a full S3 URI without the original node configuration:

hashi backup restore s3://my-bucket/<namespace>/hashi-backup-<timestamp>.tar.asc \
  --region us-east-1 --backup-pgp-secret-key /secure/secret-key.asc \
  --output-dir /private/restore

Use --version-id '<version-id>' to select an exact object version. Keys are literal object keys, not URL-decoded paths. Recovery credentials need s3:GetObject for the current object or s3:GetObjectVersion for a pinned version; inspecting version history also requires s3:ListBucketVersions.

S3 restore accepts only encrypted .tar.asc archives. Choose --backup-pgp-secret-key or --use-gpg-agent (with optional --gpg-homedir). Local restores also accept plaintext .tar archives, which need neither decryptor. --region is required for S3; it and --version-id are invalid for local restores.

Downloads use private temporary storage under --output-dir and are removed afterward. Restore publishes a complete extraction, refuses an existing destination, and does not install files at their original paths.

Keep backup URI/version receipts, the region, and decryption-key access outside the node. Use independently retained receipts and audit history to select an exact version, then pin it with --version-id. Neither the newest object nor successful decryption proves its origin: encryption does not authenticate the creator. Unexpected versions or delete markers warrant investigation, but lifecycle rules and authorized administrators can also create them.

Machine-loss recovery​

Use this path for accidental machine or disk loss without suspected compromise. For suspected compromise, use the separate procedure below, not a snapshot restore.

  1. Stop or fence the old node. Restore on a trusted replacement host into a private directory (umask 077); no original config or database is needed.
  2. Verify the restored chain, Hashi object, and validator identity against trusted records. Install the config and key files at trusted paths, not paths taken blindly from the manifest. Put hashi-db-snapshot at the configured db path; never merge it into an existing database.
  3. Update config paths and endpoints, restrict permissions to the node user, and start the node. Check chain catch-up and committee participation.
  4. Remove plaintext recovery output and temporary key copies when finished. Keep decryption-key access independent of the replacement host.

Suspected node compromise​

Assume host-held operator, TLS, and MPC private keys are exposed. Even a known pre-compromise snapshot can contain those same keys; its age does not make them safe to restore.

For a single compromised node with sufficient healthy peers:

  1. Fence the old host. From trusted infrastructure, use the validator account authority to revoke the old operator delegation and register fresh operator and TLS identities in onchain metadata. Follow the runbook's key generation and registration and operator revocation procedures with the replacement config; do not authorize recovery with the compromised operator key.
  2. Provision a fresh machine with an empty database, not restored key material. Startup generates fresh node encryption/signing keys and registers them; see server startup.
  3. Rejoin through reconfiguration with the healthy peers, and confirm the replacement identities and participation.

Suspected quorum-wide or global MPC key compromise requires coordinated incident response, not this single-node procedure. Resharing preserves the global Bitcoin key; it does not repair a stolen global private key.

Catastrophic MPC key recovery​

A zero-day compromise could steal and delete MPC key shares across the validator fleet. Catastrophic data loss could also leave every node without its shares. Backups allow the existing MPC key to be recovered when no functioning committee remains.

Scheduled backups include hashi-mpc-recovery.bcs when the node holds MPC shares. This file contains the node's shares and the information needed to combine them with other nodes' backups to recover the MPC key. Recovery needs a sufficient set of backups from the same epoch and works even after old onchain certificates have been deleted. Manual backups do not include this file.

Operators should preserve off-machine encrypted backups and keep access to their backup decryption keys separate from the validator fleet. MPC key recovery is an incident-response operation, not part of ordinary node restoration.

Recovery assembles the full MPC private key and writes it PGP-encrypted. It must still be handled as a highly sensitive secret. Recovering it does not invalidate a copy already stolen by an attacker.

Sharing recovery records​

For a coordinated recovery, use hashi backup export-mpc-recovery-record to decrypt your backup locally and export only its recovery information, encrypted to the coordinator's public PGP certificate. The command prints the recipient certificate's fingerprint; verify it through a trusted channel before sharing the export.

hashi backup export-mpc-recovery-record backup.tar.asc \
  --backup-pgp-secret-key /secure/operator-secret.asc \
  --output-pgp-cert coordinator.asc --output recovery-record.bcs.asc

Use --use-gpg-agent instead of --backup-pgp-secret-key for agent-backed keys. Send only the encrypted .bcs.asc file; neither your backup decryption key nor the full backup needs to be shared. The command never writes a plaintext record. The coordinator can pass these files to hashi backup recover-mpc-key and decrypt them with their own key.