Walrus is open source. The Walrus protocol health is overseen by an independent foundation that is now well resourced to support future development and growth. And, the community is collecting tools, resources and dapps for Walrus under Awesome Walrus!

This is a significant milestone a little over 12 months after an initial small team started designing the Walrus protocol. And a special thanks is due to the community of storage operators that supported the development of Walrus through consecutive Testnet(s), as well as all community members that integrated the protocol early and provided feedback to improve it.

New features​

Besides the promise of stability and security, the Mainnet release of Walrus comes with a few notable new features and changes:

• Blob attributes. Each Sui blob objects can have multiple attributes and values attached to it, to encode application meta-data. The aggregator uses this facility for returning common HTTP headers.

• Burn blob objects on Sui. The command line walrus tool is extended with commands to "burn" Sui blob objects to reclaim the associated storage fee, making it cheaper to store blobs.

• CLI expiry time improvements. Now blob expiry can be expressed more flexibly when storing blobs, through --epochs max, an RFC3339 date with --earliest-expiry-time, or the concrete end epoch with --end-epoch.

• RedStuff with Reed-Solomon codes. The erasure code underlying RedStuff was changed from RaptorQ to Reed-Solomon (RS) codes. Intense benchmarking suggests their performance, for our parameter set, is similar, and the use of RS codes provides perfect robustness, in that blobs can always be reconstructed given a threshold of slivers. We thank Joachim Neu for extensive feedback on this topic.

• TLS handing for storage node. Storage nodes can now be configured to serve TLS certificates that are publicly trusted, such as those issues by cloud providers and public authorities such as Let's Encrypt. This allows JavaScript clients to directly store and retrieve blobs from Walrus.

• JWT authentication for publisher. Now the publisher can be configured to only provide services to authenticated users via consuming JWT tokens that can be distributed through any authentication mechanism.

• Extensive logging and metrics. All services, from the storage node to aggregator and publisher export metrics that can be used to build dashboards, and logs at multiple levels to troubleshoot their operation.

• Health endpoint. Storage nodes and the CLI include a health command to check the status and basic information of storage nodes. This is useful when allocating stake as well as monitoring the network.

• Walrus Sites updates. The Mainnet Walrus Sites public portal will be hosted on the wal.app domain name. Now Walrus Sites support deletable blobs to make their updates more capital efficient. Those operating their own portals may also use their own domain names to serve a Walrus Site. The Staking, Docs, Snowreads, and Flatland Walrus Sites are now on Mainnet.

• A subsidies contract. The Mainnet release of Walrus requires WAL for storing blobs, however to enable early adopters to try the system, transition to it, the Walrus foundation operates a smart contract to acquire subsidized storage. The CLI client uses it automatically when storing blobs.

Testnet future plans​

The current Walrus Testnet will soon be wiped and restarted to align the codebase to the Mainnet release. Going forward we will regularly wipe the Testnet, every few months. Developers should use the Walrus Mainnet to get any level of stability. The Walrus Testnet is only there to test new features before they are deployed in production.

We will not be operating a public portal to serve Walrus Sites on Testnet, to reduce costs and incident response associated with free web hosting. Developers can still run a local portal pointed to Testnet to test Walrus Sites.

Open source Walrus codebase​

The Walrus codebase, including all smart contracts in Move, services in Rust, and documentation, is now open sourced under an Apache 2.0 license, and hosted on GitHub. Since the main Walrus repository is now open to all, and contains both documentation and smart contracts, we are retiring the walrus-docs repository.

Developers may find the Rust CLI client, and associated aggregator and publisher services of most interest. These can be extended to specialize services to specific operational needs. They are also a good illustration of how to interface with Walrus using Rust clients. A cleaner and more stable Rust SDK is in the works as well.

Publisher improvements and plans​

Publishing blobs on Mainnet consumes real WAL and SUI. To avoid uncontrolled costs to operations the publisher service was augmented with facilities to authenticate requests and account for costs per user.

Note that all blob data on the previous Testnet instance has been wiped. All blobs need to be re-uploaded to the new Testnet instance, including Walrus Sites. In addition, there is a new version of the WAL token, so your previous WAL tokens will not work anymore. To use the Testnet v2, you need to obtain new WAL tokens.

In the following sections, we describe the notable changes and the actions required for existing Walrus Sites.

Epoch duration​

The epoch duration has been increased from one day to two days to emphasize that this duration is different from Sui epochs (at Mainnet, epochs will likely be multiple weeks long). In addition, the maximum number of epochs a blob can be stored for has been reduced from 200 to 183 (corresponding to one year).

The walrus store command now also supports the --epochs max flag, which will store the blob for the maximum number of epochs. Note that the --epochs flag is now mandatory.

New features​

Besides many improvements to the contracts and the storage-node service, the latest Walrus release also brings several user-facing improvements.

• The walrus store command now supports storing multiple files at once. This is faster and more cost-effective compared to storing each file separately as transactions can be batched through PTBs. Notably, this is compatible with glob patterns offered by many shells, so you can for example run a command like walrus store *.png --epochs 100 to store all PNG files in the current directory.
• The walrus CLI now supports creating, funding, and extending shared blobs using the walrus share, walrus store --share, and walrus fund-shared-blob commands. Shared blobs are an example of collectively managed and funded blobs. See the shared blobs section for more details.

New WAL token​

Along with the redeployment of Walrus, we have also deployed a fresh WAL contract. This means that you cannot use any WAL token from the previous Testnet instance with the new Testnet instance. You need to request new WAL tokens through the Testnet WAL faucet.

Backwards-incompatible changes​

One reason for a full redeployment is to allow us to make some changes that are backwards-incompatible. Many of those are related to the contracts and thus less visible to users. There are, however, some changes that may affect you.

Configuration files​

The format of the configuration files for storage nodes and clients has been changed. Make sure to use the latest version of the configuration files, see the configuration section.

CLI options​

Several CLI options of the walrus CLI have been changed. Notably, all "short" variants of options (e.g., -e instead of --epochs) have been removed to prevent future confusion with new options. Additionally, the --epochs flag is now mandatory for the walrus store command (this also affects the JSON API).

Please refer to the CLI help (walrus --help, or walrus <command> --help) for further details.

HTTP APIs​

The paths, request, and response formats of the HTTP APIs have changed for the storage nodes, and also the aggregator and publisher. Please refer to the section on the HTTP API for further details.

Effects on and actions required for existing Walrus Sites​

The Walrus Sites contracts have not changed, which means that all corresponding objects on Sui are still valid. However, the resources now point to blob IDs that do not yet exist on the new Testnet. The easiest way to fix existing sites is to simply update them with the --force flag:

site-builder update --epochs <number of epochs> --force <path to site> <existing site object>

New Move contracts & documentation​

As part of the new Testnet release of Walrus, the Move smart contracts have been updated; the deployed version can be found in the walrus-docs repository.

Leveraging innovations in erasure coding, Walrus enables fast and robust encoding of unstructured data blobs into smaller slivers distributed and stored over a network of storage nodes. A subset of slivers can be used to rapidly reconstruct the original blob, even when up to two-thirds of the slivers are missing. This is possible while keeping the replication factor down to a minimal 4x-5x, similar to existing cloud-based services, but with the additional benefits of decentralization and resilience to more widespread faults.

The Replication Challenge​

Sui is the most advanced blockchain system in relation to storage on validators, with innovations such as a storage fund that future-proofs the cost of storing data onchain. Nevertheless, Sui still requires complete data replication among all validators, resulting in a replication factor of 100x or more in today’s Sui Mainnet. While this is necessary for replicated computing and smart contracts acting on the state of the blockchain, it is inefficient for simply storing unstructured data blobs, such as music, video, blockchain history, etc.

Introducing Walrus: Efficient and Robust Decentralized Storage​

To tackle the challenge of high replication costs, Mysten Labs has developed Walrus, a decentralized storage network offering exceptional data availability and robustness with a minimal replication factor of 4x-5x. Walrus provides two key benefits:

1. Cost-Effective Blob Storage: Walrus allows for the uploading of gigabytes of data at a time with minimal cost, making it an ideal solution for storing large volumes of data. Walrus can do this because the data blob is transmitted only once over the network, and storage nodes only spend a fraction of resources compared to the blob size. As a result, the more storage nodes the system has, the fewer resources each storage node uses per blob.

2. High Availability and Robustness: Data stored on Walrus enjoys enhanced reliability and availability under fault conditions. Data recovery is still possible even if two-thirds of the storage nodes crash or come under adversarial control. Further, availability may be certified efficiently without downloading the full blob.

Decentralized storage can take multiple forms in modern ecosystems. For instance, it offers better guarantees for digital assets traded as NFTs. Unlike current designs that store data off-chain, decentralized storage ensures users own the actual resource, not just metadata, mitigating risks of data being taken down or misrepresented.

Additionally, decentralized storage is not only useful for storing data such as pictures or files with high availability; it can also double as a low-cost data availability layer for rollups. Here, sequencers can upload transactions on Walrus, and the rollup executor only needs to temporarily reconstruct them for execution.

We also believe Walrus will accompany existing disaster recovery strategies for millions of enterprise companies. Not only is Walrus low-cost, it also provides unmatched layers of data availability, integrity, transparency, and resilience that centralized solutions by design cannot offer.

Walrus is powered by the Sui Network and scales horizontally to hundreds or thousands of networked decentralized storage nodes. This should enable Walrus to offer Exabytes of storage at costs competitive with current centralized offerings, given the higher assurance and decentralization.

The Future of Walrus​

By releasing this developer preview we hope to share some of the design decisions with the decentralized app developer community and gather feedback on the approach and the APIs for storing, retrieving, and certifying blobs. In this developer preview, all storage nodes are operated by Mysten Labs to help us understand use cases, fix bugs, and improve the performance of the software.

Future updates to Walrus will allow for dynamically changing the set of decentralized storage nodes, as well as changing the mapping of what slivers are managed by each storage node. The available operations and tools will also be expanded to cover more storage-related use cases. Many of these functions will be designed with the feedback we gather in mind.

Stay tuned for more updates on how Walrus will revolutionize data storage in the web3 ecosystem.

What can developers build?​

As part of this developer preview, we provide a binary client (currently macOS, ubuntu) that can be operated from the command line interface, a JSON API, and an HTTP API. We also offer the community an aggregator and publisher service and a Devnet deployment of 10 storage nodes operated by Mysten Labs.

We hope developers will experiment with building applications that leverage the Walrus Decentralized Store in a variety of ways. As examples, we hope to see the community build:

• Storage of media for NFT or dapps: Walrus can directly store and serve media such as images, sounds, sprites, videos, other game assets, etc. This is publicly available media that can be accessed using HTTP requests at caches to create multimedia dapps.

• AI-related use cases: Walrus can store clean data sets of training data, datasets with a known and verified provenance, model weights, and proofs of correct training for AI models. Or it may be used to store and ensure the availability and authenticity of an AI model output.

• Storage of long term archival of blockchain history: Walrus can be used as a lower-cost decentralized store to store blockchain history. For Sui, this can include sequences of checkpoints with all associated transaction and effects content, as well as historic snapshots of the blockchain state, code, or binaries.

• Support availability for L2s: Walrus enables parties to certify the availability of blobs, as required by L2s that need data to be stored and attested as available to all. This may also include the availability of extra audit data such as validity proofs, zero-knowledge proofs of correct execution, or large fraud proofs.

• Support a full decentralized web experience: Walrus can host full decentralized web experiences including all resources (such as js, css, html, and media). These can provide content but also host the UX of dapps, enabling fully decentralized front- and back-ends on chain. It brings the full "web" back into "web3".

• Support subscription models for media: Creators can store encrypted media on Walrus and only provide access via decryption keys to parties that have paid a subscription fee or have paid for content. (Note that Walrus provides the storage; encryption and decryption must be done off Walrus).

We are excited to see what else the web3 developer community can imagine!

Getting Started​

For this developer preview the public Walrus Devnet is openly available to all developers. Developer documentation is available at <https://docs.wal.app>.

SUI Testnet token is the main currency for interacting with Walrus. Developers pay for Walrus Devnet storage using SUI Testnet tokens which can be acquired at the Sui Testnet Discord faucet.

One more thing …​

The Walrus Sites website, the Walrus docs, and this very blog are hosted on Walrus. To learn more about Walrus Sites and how you can deploy your own, click here.

Blob deletion​

The most important user-facing new feature is optional blob deletion. The uploader of a blob can optionally indicate a blob is "deletable". This information is stored in the Sui blob metadata object, and is also included in the event denoting when the blob is certified. Subsequently, the owner of the Sui blob metadata object can "delete" it. As a result storage for the remaining period is reclaimed and can be used by subsequent blob storage operations.

Blob deletion allows more fine-grained storage cost management: smart contracts that wrap blob metadata objects can define logic that stores blobs and delete them to minimize costs, and reclaim storage space before Walrus epochs end.

However, blob deletion is not an effective privacy mechanism in itself: copies of the blob may exist outside Walrus storage nodes on caches and end-user stores or devices. Furthermore, if the identical blob is stored by multiple Walrus users, the blob will still be available on Walrus until no copy exists. Thus deleting your own copy of a blob cannot guarantee that it is deleted from Walrus as a whole.

• Find out how to upload and delete deletable blobs through the CLI.
• Find out more about how delete operations work.

Epochs​

Walrus Testnet enables multiple epochs. Initially, the epoch duration is set to a single day to ensure the logic of epoch change is thoroughly tested. At Mainnet, epochs will likely be multiple weeks long.

The progress of epochs makes the expiry epoch of blobs meaningful, and blobs will become unavailable after their expiry epoch. The store command may be used to extend the expiry epoch of a blob that is still available. This operation is efficient and only affects payments and metadata, and does not re-upload blob contents.

• Find out the current epoch through the CLI.
• Find out how to store a blob for multiple epochs.

The WAL token and the Testnet WAL faucet​

Payments for blob storage and extending blob expiry are denominated in Testnet WAL, a Walrus token issued on the Sui Testnet. Testnet WAL has no value, and an unlimited supply; so no need to covet or hoard it, it's just for testing purposes and only issued on Sui Testnet.

WAL also has a smaller unit called FROST, similar to MIST for SUI. 1 WAL is equal to 1 billion (1000000000) FROST.

To make Testnet WAL available to all who want to experiment with the Walrus Testnet we provide a utility and smart contract to convert Testnet SUI (which also has no value) into Testnet WAL using a one-to-one exchange rate. This is chosen arbitrarily, and generally one should not read too much into the actual WAL denominated costs of storage on Testnet. They have been chosen arbitrarily.

Find out how to request Testnet WAL tokens through the CLI.

Decentralization through staking & unstaking​

The WAL token may also be used to stake with storage operators. Staked WAL can be unstaked and re-staked with other operators or used to purchase storage.

Each epoch storage nodes are selected and allocated storage shards according to their delegated stake. At the end of each epoch payments for storing blobs for the epoch are distributed to storage nodes and those that delegate stake to them. Furthermore, important network parameters (such as total available storage and storage price) are set by the selected storage operators each epoch according to their stake weight.

A staking web dApp is provided to experiment with this functionality. Community members have also created explorers that can be used to view storage nodes when considering who to stake with. Staking ensures that the ultimate governance of Walrus, directly in terms of storage nodes, and indirectly in terms of parameters and software they chose, rests with WAL Token holders.

Under the hood and over the next months we will be testing many aspects of epoch changes and storage node committee changes: better shard allocation mechanisms upon changes or storage node stake; efficient ways to sync state between storage nodes; as well as better ways for storage nodes to follow Sui event streams.

• Explore the Walrus staking dApp.
• Look at recent activity on the Walrus Explorer.

New Move contracts & documentation​

As part of the Testnet release of Walrus, the documentation and Move Smart contracts have been updated, and can be found in the walrus-docs repository.

New Walrus Sites features​

With the move to Walrus Testnet, Walrus Sites have also been updated! The new features in this update greatly increase the flexibility, speed, and security of Walrus Sites. Developers can now specify client-side routing rules, and add custom HTTP headers to the portals' responses for their site, expanding the possibilities for what Walrus Sites can do.

Migrate now to take advantage of these new features! The old Walrus Sites, based on Walrus Devnet, will still be available for a short time. However, Devnet will be wiped soon (as described below), so it is recommended to migrate as soon as possible.

Discontinuation of Walrus Devnet​

The previous Walrus Devnet instance is now deprecated and will be shut down after 2024-10-31. All data stored on Walrus Devnet (including Walrus Sites) will no longer be accessible at that point. You need to re-upload all data to Walrus Testnet if you want it to remain accessible. Walrus Sites also need to be migrated.

It is time to unveil the next stage of the project: Walrus will become an independent decentralized network with its own utility token, WAL, that will play a key role in the operation and governance of the network. Walrus will be operated by storage nodes through a delegated proof-of-stake mechanism using the WAL token. An independent Walrus foundation will encourage the advancement and adoption of Walrus, and support its community of users and developers.

Today, we published the Walrus whitepaper (also on GitHub) that offers additional details, including:

• The encoding scheme and Read / Write operations Walrus uses to ensure both security and efficient scaling to 100s and 1000s of storage nodes, including interactions with the Sui blockchain which serves as a coordination layer for Walrus’ operations.
• The reconfiguration of storage nodes across epochs, and how the protocol ensures available blobs on Walrus remain available over long periods of time.
• The tokenomics of Walrus based on the WAL token, including how staking and staking rewards are structured, how pricing and payments for storage are handled and distributed in each epoch, and the governance of key system parameters.
• Forward-looking design options, such as a cheap mechanism to challenge and audit storage nodes, options for ensuring reads with a higher service quality, possibly against a payment, and designs that empower light nodes to meaningfully contribute to the protocol’s robustness, serve reads, and be rewarded.

The whitepaper focuses on the steady-state design aspects of Walrus. Further details about the project, such as timelines, opportunities for community participation, how to join the network as a storage node, and plans around light nodes, will be shared in subsequent posts.

To be part of this journey:

• Follow us on Twitter
• Join our Discord
• Build apps on Walrus
• Publish a Walrus Site and share it

Migration and Re-deployment of Walrus Sites​

You can obtain the latest version of the walrus binary and the new configuration as described in the setup chapter.

If you had deployed any Walrus Sites, the site object on Sui and any SuiNS name are still valid. However, you need to re-store all blobs on Walrus. You can achieve this by running the site-builder tool (from the walrus-sites directory) as follows:

./target/release/site-builder --config site-builder/assets/builder-example.yaml update --force \
    <path to the site> <site object ID>

Changes​

Besides many improvements to the storage nodes, the new version of Walrus includes the following user-facing changes:

• Improved coin management: The client now better selects coins for gas and storage fees. Users no longer require multiple coins in their wallet.
• Improved connection management: The client now limits the number of parallel connections to improve performance for users with low network bandwidth storing large blobs.
• OpenAPI specification: Walrus storage nodes, aggregators, and publishers expose their API specifications at the path /v1/api.
• System info in JSON: The info command is now also available in JSON mode.
• Client version: The walrus CLI now has a --version option.
• Support for the empty blob: The empty blob is now supported by Walrus.
• Default configuration-file paths: The client now looks for configuration files in ~/.config/walrus in addition to ~/.walrus and recognizes the extension .yml in addition to .yaml.
• Home directory in paths: Paths specified in configuration files now expand the ~ symbol at the beginning to the user's home directory.
• More robust store and status check: The store and blob-status commands are now more robust against Sui full nodes that aggressively prune past events and against load-balancers that send transactions to different full nodes.
• Fix CLI parsing: The walrus CLI now properly handles hyphens in blob IDs.

This update also increases the number of shards to 1000, which is more representative of the expected value in Testnet and Mainnet.