Klaytn State Migration: An Efficient Way to Reduce Blockchain Data

In this post we will introduce the concept of “State Trie Migration’’, which is a process for reducing chain data on Klaytn. This is a more detailed version of the If(Kakao) 2020 presentation (in Korean) given by our developer Ethan.

NOTE: If you are curious about how to run state migration, please refer to Klaytn v1.5.0 State Migration: Saving Node Storage.

What is Blockchain Storage?

Blockchain storage is where all the blockchain-related data gets saved. It looks something like this:

Blockchain data is basically a chain of blocks connected to one another, and each block includes a block header, the body and the receipt. The data of the current state forms the so-called trie structure, which can be accessed through the state root contained in the block header. As such, various data elements exist on the blockchain, which eventually get stored in the database (LevelDB).

Motivation for Development and Current State of Storage

On Klaytn’s mainnet Cypress, about 1.2TB of chaindata (the entire data of a given blockchain) have been accumulated since the genesis block. This chaindata is what is required if you want to operate a node on Cypress, but the costs ensuing from the storage have been pretty demanding.

The size of the chaindata on Cypress as of December 3rd 2020 is as follows:

Full node: 1.2TB in Full Node (Node which stores state in the storage for every 128 blocks)

942G ./statetrie
92G ./header
72G ./body
63G ./receipts
9.3G ./txlookup
4.1G ./misc
288K ./bridgeservice
1.2T .

A full node, which stores the state of the last 128 blocks, uses up about 1.2TB of data, 942GB of which are attributable to the state trie.

Archive node: 2.3TB (Node which stores state for every new block)

2.0T ./statetrie
92G ./header
72G ./body
63G ./receipts
9.3G ./txlookup
4.1G ./misc
48K ./bridgeservice
2.3T .

An archive node that stores the state for every block contains 2.3TB of chaindata, about 2TB are basically state tries.

Characteristics of the State Trie

Let’s take a closer look at the state trie, which takes up the most storage space in the chaindata. A state trie contains nonces, balances and account keys of all accounts (including contracts) that exist on Klaytn. In the case of a contract account, it has a separate storage trie, which also gets stored in the state trie.

The data is saved in the form of a trie and each trie node is stored in the form of a key/value. As can be seen below, each trie node is stored in the existing database in the form of a key/value. A key is the hash of the trie node’s value, so even if there are several identical trie nodes, they will be stored as one unique record in the database.

The Idea behind State Trie Migration

As we have seen so far, every block contains a state trie. But notice below the trie nodes A and B, which only exist in block #100 and are not necessary for processing block #102.

That is how unnecessary trie nodes start to pile up. And the idea was that removing these trie nodes would not affect the processing of new blocks, even though past state would not be retrievable.

On Ethereum there is the “fast sync” option; basically, when you start a new node, you sync the last state via p2p, and only the new blocks added from that point on will be processed. But Klaytn went further and thought of ways to reduce data size without having to compromise the newly added as well as the currently running nodes.

Process of State Migration

Here is a step-by-step process:

  1. Create a new DB (a new state trie database). First, you need to create a DB for a new state trie once you know which block you want to perform the migration on. This new DB will store the state trie of the newly added blocks including the block, which you chose for the migration.
  2. Traverse the state trie of the migrated block and copy it in the new state trie DB.
  3. After the migration is complete, delete the old DB.
  4. Now you have a new DB, which stores the state of the specified block and the new blocks.

State Trie Migration may be a simple process, but it’s not really a quick one. And new blocks will be added during the migration and they should also be processed, with their results written on the new DB as well. In the diagram below, the new block #102 is being processed while migration for block #101 is still in progress, which means both new and existing DBs have to be queried.

Result of the State Migration

Let’s take a look at the changes in the storage after the migration. The state trie, which used to take up 942GB of storage space in the full node, has been greatly reduced by 94% to 55GB.

Migrated Full node: 294GB

(node which saves state for block #45,131,309 and every 128 blocks)

55G ./statetrie_migrated_45131309
92G ./header
72G ./body
63G ./receipts
9.3G ./txlookup
4.1G ./misc
344K ./bridgeservice
294G .

Technical Issues and Resolution

In the process of developing State Migration, we achieved higher speed through performance optimization.

Performance/Memory Optimization for State Migration

  1. Parallel Processing

After the first implementation of State Migration, it was found that it took more than several weeks on AWS EC2 m5.4xlarge instance. It was then reduced to three days by parallel copying of the DB.

2. Optimized writing of duplicate trie nodes

As of block #45,131,309, there are about 800 million nodes in the state trie, but excluding the duplicates, about 400 million. In order to eliminate the duplicates, the hash of the state trie already written on the DB gets stored. Before requesting a store to the DB, it checks whether the state trie node is already stored in the DB. It cuts down the required time from three days to one.

3. LRU-based Key Caching

When processing duplicate trie nodes, all trie node keys have to be stored, which requires over 20GB memory space for Cypress. But there are about 100 million duplicate nodes, and, by using 2GB LRU cache, the achieved performance was just like how it would have been if you had stored and checked all the trie node keys.

Aborted Migration and Errors

Since State Migration takes more than a day, the possibility of a termination or an unexpected error must also be taken into account. For this reason, we made it so that the trie nodes from the latest block can be stored in both the existing as well as the new DBs even when the State Migration is in process. Even when the migration is aborted, it is still possible to keep the existing DB and delete the new. And when the migration is complete, the new DB can be kept in place of the old one.

We’ve taken a look at the technical aspects of the State Migration process. We hope this could make the operation of Klaytn Node easier and more affordable by reducing the size of the chaindata on Cypress. But we won’t stop exploring more options in order to promptly remove nonessential trie nodes while each new block is being processed, ultimately to enhance the sustainability of state trie operations.

State Trie Migration can be implemented by calling startStateMigration API, with Klaytn v1.5.0 or higher. We encourage you to give it a go. For more details on how to use State Migration, refer to this link: Klaytn v1.5.0 State Migration: Saving Node Storage.

We hope that this post is helpful in reducing your block data. Klaytn team is always committed to providing our users new and better features by putting ourselves in the users’ shoes, and we will continue to do so in the future!

Thank you and stay tuned for our next post!

MORE FROM Technology

Ecosystem

Bringing access to the Klaytn metaverse through CEXes

07/06/2022

Announcements

Monthly round-up: May 2022

31/05/2022

Ecosystem

Gaming: Our key to metaverse global adoption

27/05/2022

CATEGORIES

© Klaytn Foundation 2022. All rights reserved.
© Klaytn Foundation 2022. All rights reserved.