In this post we will introduce the concept of “State Trie Migration’’, which is a process for reducing chain data on Klaytn. This is a more detailed version of the If(Kakao) 2020 presentation (in Korean) given by our developer Ethan.

NOTE: If you are curious about how to run state migration, please refer to Klaytn v1.5.0 State Migration: Saving Node Storage.

What is Blockchain Storage?

Blockchain storage is where all the blockchain-related data gets saved. It looks something like this:

Blockchain data is basically a chain of blocks connected to one another, and each block includes a block header, the body and the receipt. The data of the current state forms the so-called trie structure, which can be accessed through the state root contained in the block header. As such, various data elements exist on the blockchain, which eventually get stored in the database (LevelDB).

Motivation for Development and Current State of Storage

On Klaytn’s mainnet Cypress, about 1.2TB of chaindata (the entire data of a given blockchain) have been accumulated since the genesis block. This chaindata is what is required if you want to operate a node on Cypress, but the costs ensuing from the storage have been pretty demanding.

The size of the chaindata on Cypress as of December 3rd 2020 is as follows:

Full node: 1.2TB in Full Node (Node which stores state in the storage for every 128 blocks)

942G ./statetrie
92G ./header
72G ./body
63G ./receipts
9.3G ./txlookup
4.1G ./misc
288K ./bridgeservice
1.2T .

A full node, which stores the state of the last 128 blocks, uses up about 1.2TB of data, 942GB of which are attributable to the state trie.

Archive node: 2.3TB (Node which stores state for every new block)

2.0T ./statetrie
92G ./header
72G ./body
63G ./receipts
9.3G ./txlookup
4.1G ./misc
48K ./bridgeservice
2.3T .

An archive node that stores the state for every block contains 2.3TB of chaindata, about 2TB are basically state tries.

Characteristics of the State Trie

Let’s take a closer look at the state trie, which takes up the most storage space in the chaindata. A state trie contains nonces, balances and account keys of all accounts (including contracts) that exist on Klaytn. In the case of a contract account, it has a separate storage trie, which also gets stored in the state trie.

The data is saved in the form of a trie and each trie node is stored in the form of a key/value. As can be seen below, each trie node is stored in the existing database in the form of a key/value. A key is the hash of the trie node’s value, so even if there are several identical trie nodes, they will be stored as one unique record in the database.

The Idea behind State Trie Migration

As we have seen so far, every block contains a state trie. But notice below the trie nodes A and B, which only exist in block #100 and are not necessary for processing block #102.

That is how unnecessary trie nodes start to pile up. And the idea was that removing these trie nodes would not affect the processing of new blocks, even though past state would not be retrievable.

On Ethereum there is the “fast sync” option; basically, when you start a new node, you sync the last state via p2p, and only the new blocks added from that point on will be processed. But Klaytn went further and thought of ways to reduce data size without having to compromise the newly added as well as the currently running nodes.

Process of State Migration

Here is a step-by-step process:

  1. Create a new DB (a new state trie database). First, you need to create a DB for a new state trie once you know which block you want to perform the migration on. This new DB will store the state trie of the newly added blocks including the block, which you chose for the migration.
  2. Traverse the state trie of the migrated block and copy it in the new state trie DB.
  3. After the migration is complete, delete the old DB.
  4. Now you have a new DB, which stores the state of the specified block and the new blocks.

State Trie Migration may be a simple process, but it’s not really a quick one. And new blocks will be added during the migration and they should also be processed, with their results written on the new DB as well. In the diagram below, the new block #102 is being processed while migration for block #101 is still in progress, which means both new and existing DBs have to be queried.

Result of the State Migration

Let’s take a look at the changes in the storage after the migration. The state trie, which used to take up 942GB of storage space in the full node, has been greatly reduced by 94% to 55GB.

Migrated Full node: 294GB

(node which saves state for block #45,131,309 and every 128 blocks)

55G ./statetrie_migrated_45131309
92G ./header
72G ./body
63G ./receipts
9.3G ./txlookup
4.1G ./misc
344K ./bridgeservice
294G .

Technical Issues and Resolution

In the process of developing State Migration, we achieved higher speed through performance optimization.

Performance/Memory Optimization for State Migration

  1. Parallel Processing

After the first implementation of State Migration, it was found that it took more than several weeks on AWS EC2 m5.4xlarge instance. It was then reduced to three days by parallel copying of the DB.

2. Optimized writing of duplicate trie nodes

As of block #45,131,309, there are about 800 million nodes in the state trie, but excluding the duplicates, about 400 million. In order to eliminate the duplicates, the hash of the state trie already written on the DB gets stored. Before requesting a store to the DB, it checks whether the state trie node is already stored in the DB. It cuts down the required time from three days to one.

3. LRU-based Key Caching

When processing duplicate trie nodes, all trie node keys have to be stored, which requires over 20GB memory space for Cypress. But there are about 100 million duplicate nodes, and, by using 2GB LRU cache, the achieved performance was just like how it would have been if you had stored and checked all the trie node keys.

Aborted Migration and Errors

Since State Migration takes more than a day, the possibility of a termination or an unexpected error must also be taken into account. For this reason, we made it so that the trie nodes from the latest block can be stored in both the existing as well as the new DBs even when the State Migration is in process. Even when the migration is aborted, it is still possible to keep the existing DB and delete the new. And when the migration is complete, the new DB can be kept in place of the old one.

We’ve taken a look at the technical aspects of the State Migration process. We hope this could make the operation of Klaytn Node easier and more affordable by reducing the size of the chaindata on Cypress. But we won’t stop exploring more options in order to promptly remove nonessential trie nodes while each new block is being processed, ultimately to enhance the sustainability of state trie operations.

State Trie Migration can be implemented by calling startStateMigration API, with Klaytn v1.5.0 or higher. We encourage you to give it a go. For more details on how to use State Migration, refer to this link: Klaytn v1.5.0 State Migration: Saving Node Storage.

We hope that this post is helpful in reducing your block data. Klaytn team is always committed to providing our users new and better features by putting ourselves in the users’ shoes, and we will continue to do so in the future!

Thank you and stay tuned for our next post!

In this post we introduce you to Klaytn v1.5.0’s new feature called the “State Migration”, which allows you to keep only the latest states on the chain starting from a certain block while dumping the unnecessary states and data. In the end you can save up to 75% of node storage!

Overview

What is the State Trie?

On Klaytn, all account information is stored in a data structure called the Merkle Patricia Trie, also called the state trie. This trie is stored in the Klaytn Node storage.

Every block in the blockchain has a state trie. In other words, one Klaytn block contains one state trie, and it stores all the account information.

What is State Migration?

State Migration is a process by which the state trie before a specific block is completely deleted from the node storage. By keeping only the state tries from that block along with the blocks that will be added afterwards, the node will eventually have only the fresh chaindata that you need for operating Endpoint Nodes (EN), which saves a lot of space.

State Migration consists of the following steps:

1. Choose the block for Migration

Choose which block’s state trie you want remained in the EN Storage. In other words, you are determining a certain point from which you want to keep the data.

2. Copy the block’s state trie

Once you have chosen the block for migration, a new database can be created. By traversing the state trie of the that block in the existing database, every node in the state trie is copied to the new database.

NOTE: As the state trie of the specified block is being copied, new blocks are continuously being added to the blockchain.

3. Validate the copied state trie

Once the state trie of the specified block has been copied into the new database, it should now be validated to see if all the data has been copied intactly.

4. Delete the old database, start storing on the new database

After validation, it deletes the old database and starts storing the state tries for newly added blocks in the new database.

Recommended System Requirements

In order to perform State Migration, a m5.4xlarge instance is required on AWS at minimum. To complete one session of State Migration, you would need about 50 hours (for Cypress, as of August 2020). This is the time that is required for synchronizing the latest blocks, traversing the block’s state trie, copying the trie nodes into the new database, and validating them. You will need less time with a larger instance. It takes about 30 hours with m5.8xlarge.

Prerequisites

To run State Migration, Klaytn v1.5.0 or later is required. To update, download Klaytn 1.5.0 binary (link), stop the existing node, replace the downloaded binary and then install Klaytn again. The update process may vary according to the system environment. For details, please refer to the EN installation guide.

Getting Started with State Migration

Run State Migration

To run State Migration, you must be able to use Admin API on your Klaytn node. Connect to the node via IPC or temporarily activate Admin API on RPC/WS. For detailed information about how to connect to the Klaytn node via IPC or using Admin API, please refer to JSON-RPC APIs and Admin APIs.

> ken attach ~/datadir/ken.ipcWelcome to the Klaytn JavaScript console!
instance: Klaytn/v1.5.0+94c52174fe/linux-amd64/go1.14.1
datadir: /data/kend/datamodules: admin:1.0 debug:1.0 governance:1.0 istanbul:1.0 klay:1.0 net:1.0 personal:1.0 rpc:1.0 txpool:1.0 web3:1.0> admin.startStateMigration()null

If your start was successful, admin.startStateMigration() will return null. Once State Migration has kicked off, the state trie of the specified block will be traversed and then copied into the new database while new blocks are continuously being processed.

Log at the Start of State Migration

After a successful start of State Migration, the following logs will be shown in the log file:

INFO[07/20,20:44:35 +09] [5] State migration is prepared expectedMigrationStartingBlockNumber=29911450…INFO[07/20,20:55:51 +09] [5] State migration is started block=29911450 root=06b082d~60b893INFO[07/20,20:56:06 +09] [44] Add a new stakingInfo to stakingInfoCache and stakingInfoDB blockNum=29894400INFO[07/20,20:56:06 +09] [46] Start setting a new database for state trie migration blockNum=29911450INFO[07/20,20:56:06 +09] [46] LevelDB configurations path=/data/kend/data/klay/chaindata/statetrie_migrated_29911450/0 levelDBCacheSize=36 openFilesLimit=97 useBufferPool=true compressionType=none compactionTableSize(MB)=2 compactionTableSizeMultiplier=1.000INFO[07/20,20:56:06 +09] [46] LevelDB configurations path=/data/kend/data/klay/chaindata/statetrie_migrated_29911450/1 levelDBCacheSize=36 openFilesLimit=97 useBufferPool=true compressionType=none compactionTableSize(MB)=2 compactionTableSizeMultiplier=1.000INFO[07/20,20:56:06 +09] [46] LevelDB configurations path=/data/kend/data/klay/chaindata/statetrie_migrated_29911450/2 levelDBCacheSize=36 openFilesLimit=97 useBufferPool=true compressionType=none compactionTableSize(MB)=2 compactionTableSizeMultiplier=1.000INFO[07/20,20:56:06 +09] [46] LevelDB configurations path=/data/kend/data/klay/chaindata/statetrie_migrated_29911450/3 levelDBCacheSize=36 openFilesLimit=97 useBufferPool=true compressionType=none compactionTableSize(MB)=2 compactionTableSizeMultiplier=1.000INFO[07/20,20:56:06 +09] [46] Created a new database for state trie migration newStateTrieDB=/data/kend/data/klay/chaindata/statetrie_migrated_29911450INFO[07/20,20:56:06 +09] [47] Persisted nodes from memory database by Cap nodes=0 size=0.00B time=6.031µs flushnodes=0 flushsize=0.00B flushtime=5.704µs livenodes=1 livesize=0..00B

Monitoring State Migration

Checking State Migration Progress with Logs

While State Migration is ongoing, you can regularly check the progress in logs as shown below:

Progress Logs Printed with a 10-second Interval

INFO[07/20,20:56:28 +09] [5] State migration progress progress=0% totalRead=2321 totalCommitted=0 pending=37134 read=2321 readElapsed=22.184223002s processElapsed=33.163185ms written=0 writeElapsed=17.257µs elapsed=22.217s totalElapsed=22.21783843sINFO[07/21,08:44:33 +09] [5] State migration progress progress=64.917% totalRead=106657041 totalCommitted=106654913 pending=13469 read=19456 readElapsed=11h36m47.910769711s processElapsed=5m7.371806884s written=19717 writeElapsed=5m25.733869118s elapsed=10.147s totalElapsed=11h48m26.757580868s

Checking State Migration Status via RPC API

You can also check the progress of State Migration via RPC API. The result below shows that the state trie copying is about 64% complete.

> admin.stateMigrationStatus{
committed: 106743172,
err: “null”,
isMigration: true,
migrationBlockNumber: 29911450,
pending: 12036,
progress: 64.9169921875,
read: 106745105
}

Validation of the Copied State Trie

Once copying the state trie is done, your logs will look like as shown below. Next, you have to validate whether the state trie has been intactly copied.

The status of the validation can be checked in the logs as shown below:

Validation Status Logs

INFO[07/23,11:11:22 +09] [5] State migration progress progress=100% totalRead=368572444 totalCommitted=368572444 pending=0 read=5374 readElapsed=61h36m58.773140567s processElapsed=17m45.115580174s written=6488 writeElapsed=16m36.198342833s elapsed=4.207s totalElapsed=62h15m15.550216445sINFO[07/23,11:11:22 +09] [5] State migration : Copy is done totalRead=368572444 totalCommitted=368572444 totalElapsed=62h15m15.550264864s committed per second=1644.564INFO[07/23,11:11:22 +09] [6] CheckStateConsistencyParallel is started root=0x06b082dd3a077dee8fa7d0016ea9e96f64d8d40f3332ecff44c779b35f60b893 len(children)=16INFO[07/23,11:11:32 +09] [6] CheckStateConsistencyParallel progress cnt=35793…

Completion of State Migration

Checking State Migration Completion Logs

Once the validation for the copied state tries is done, State Migration is officially complete. If you see logs printed as below, it means that the state migration has been successful.

State Migration Completion Log

INFO[07/25,09:20:12 +09] [6] CheckStateConsistencyParallel is done cnt=649698856 err=nilINFO[07/25,09:20:12 +09] [5] State migration is completed copyElapsed=62h15m15.550264864s checkElapsed=46h8m50.500767758s

Results: Checking Chaindata Data Size Reduction

The chaindata storage for Cypress, Klaytn’s mainnet, is as follows (as of August 2020):

> du -d1 -h651G ./statetrie
56G ./body
2.7G ./misc
240K ./bridgeservice
60G ./header
6.5G ./txlookup
40G ./receipts
815G .

After booting a new EN and performing State Migration for Cypress on this EN, the chaindata is reduced as shown below.

> du -d1 -h129G ./statetrie_migrated_29911450
61G ./body
3.0G ./misc
240K ./bridgeservice
69G ./header
7.3G ./txlookup
47G ./receipts
315G .

(The reason why the data for other directories has increased is because new blocks were added in the process of the Migration.)

Things to Keep in Mind During State Migration

Here are some things to keep in mind during State Migration:

Conclusion

As demonstrated above, State Migration greatly reduces data in the storage on a Klaytn node. But having to manually carry out State Migration may be a bit too demanding from the node operator’s perspective. That is why Klaytn Team is planning to keep on providing Cypress’ latest chaindata reduced by State Migration, which will be performed periodically to optimize storage usage. With this chaindata, a node operator won’t have to migrate state tries him or herself, and instead directly downloads and replaces the chaindata on his/her node. To download the latest snapshot of the chaindata, check out the following links: CypressBaobab.

We hope that this post is helpful in freeing up your storage size. Klaytn Team is always committed to making your Klaytn experience as convenient as possible. Thank you for reading and stay tuned for our next post!