Klaytn v1.5.0 State Migration: Saving Node Storage

In this post we introduce you to Klaytn v1.5.0’s new feature called the “State Migration”, which allows you to keep only the latest states on the chain starting from a certain block while dumping the unnecessary states and data. In the end you can save up to 75% of node storage!

Overview

What is the State Trie?

On Klaytn, all account information is stored in a data structure called the Merkle Patricia Trie, also called the state trie. This trie is stored in the Klaytn Node storage.

Every block in the blockchain has a state trie. In other words, one Klaytn block contains one state trie, and it stores all the account information.

What is State Migration?

State Migration is a process by which the state trie before a specific block is completely deleted from the node storage. By keeping only the state tries from that block along with the blocks that will be added afterwards, the node will eventually have only the fresh chaindata that you need for operating Endpoint Nodes (EN), which saves a lot of space.

State Migration consists of the following steps:

1. Choose the block for Migration

Choose which block’s state trie you want remained in the EN Storage. In other words, you are determining a certain point from which you want to keep the data.

2. Copy the block’s state trie

Once you have chosen the block for migration, a new database can be created. By traversing the state trie of the that block in the existing database, every node in the state trie is copied to the new database.

NOTE: As the state trie of the specified block is being copied, new blocks are continuously being added to the blockchain.

3. Validate the copied state trie

Once the state trie of the specified block has been copied into the new database, it should now be validated to see if all the data has been copied intactly.

4. Delete the old database, start storing on the new database

After validation, it deletes the old database and starts storing the state tries for newly added blocks in the new database.

Recommended System Requirements

In order to perform State Migration, a m5.4xlarge instance is required on AWS at minimum. To complete one session of State Migration, you would need about 50 hours (for Cypress, as of August 2020). This is the time that is required for synchronizing the latest blocks, traversing the block’s state trie, copying the trie nodes into the new database, and validating them. You will need less time with a larger instance. It takes about 30 hours with m5.8xlarge.

Prerequisites

To run State Migration, Klaytn v1.5.0 or later is required. To update, download Klaytn 1.5.0 binary (link), stop the existing node, replace the downloaded binary and then install Klaytn again. The update process may vary according to the system environment. For details, please refer to the EN installation guide.

Getting Started with State Migration

Run State Migration

To run State Migration, you must be able to use Admin API on your Klaytn node. Connect to the node via IPC or temporarily activate Admin API on RPC/WS. For detailed information about how to connect to the Klaytn node via IPC or using Admin API, please refer to JSON-RPC APIs and Admin APIs.

> ken attach ~/datadir/ken.ipcWelcome to the Klaytn JavaScript console!
instance: Klaytn/v1.5.0+94c52174fe/linux-amd64/go1.14.1
datadir: /data/kend/datamodules: admin:1.0 debug:1.0 governance:1.0 istanbul:1.0 klay:1.0 net:1.0 personal:1.0 rpc:1.0 txpool:1.0 web3:1.0> admin.startStateMigration()null

If your start was successful, admin.startStateMigration() will return null. Once State Migration has kicked off, the state trie of the specified block will be traversed and then copied into the new database while new blocks are continuously being processed.

Log at the Start of State Migration

After a successful start of State Migration, the following logs will be shown in the log file:

INFO[07/20,20:44:35 +09] [5] State migration is prepared expectedMigrationStartingBlockNumber=29911450…INFO[07/20,20:55:51 +09] [5] State migration is started block=29911450 root=06b082d~60b893INFO[07/20,20:56:06 +09] [44] Add a new stakingInfo to stakingInfoCache and stakingInfoDB blockNum=29894400INFO[07/20,20:56:06 +09] [46] Start setting a new database for state trie migration blockNum=29911450INFO[07/20,20:56:06 +09] [46] LevelDB configurations path=/data/kend/data/klay/chaindata/statetrie_migrated_29911450/0 levelDBCacheSize=36 openFilesLimit=97 useBufferPool=true compressionType=none compactionTableSize(MB)=2 compactionTableSizeMultiplier=1.000INFO[07/20,20:56:06 +09] [46] LevelDB configurations path=/data/kend/data/klay/chaindata/statetrie_migrated_29911450/1 levelDBCacheSize=36 openFilesLimit=97 useBufferPool=true compressionType=none compactionTableSize(MB)=2 compactionTableSizeMultiplier=1.000INFO[07/20,20:56:06 +09] [46] LevelDB configurations path=/data/kend/data/klay/chaindata/statetrie_migrated_29911450/2 levelDBCacheSize=36 openFilesLimit=97 useBufferPool=true compressionType=none compactionTableSize(MB)=2 compactionTableSizeMultiplier=1.000INFO[07/20,20:56:06 +09] [46] LevelDB configurations path=/data/kend/data/klay/chaindata/statetrie_migrated_29911450/3 levelDBCacheSize=36 openFilesLimit=97 useBufferPool=true compressionType=none compactionTableSize(MB)=2 compactionTableSizeMultiplier=1.000INFO[07/20,20:56:06 +09] [46] Created a new database for state trie migration newStateTrieDB=/data/kend/data/klay/chaindata/statetrie_migrated_29911450INFO[07/20,20:56:06 +09] [47] Persisted nodes from memory database by Cap nodes=0 size=0.00B time=6.031µs flushnodes=0 flushsize=0.00B flushtime=5.704µs livenodes=1 livesize=0..00B

Monitoring State Migration

Checking State Migration Progress with Logs

While State Migration is ongoing, you can regularly check the progress in logs as shown below:

Progress Logs Printed with a 10-second Interval

INFO[07/20,20:56:28 +09] [5] State migration progress progress=0% totalRead=2321 totalCommitted=0 pending=37134 read=2321 readElapsed=22.184223002s processElapsed=33.163185ms written=0 writeElapsed=17.257µs elapsed=22.217s totalElapsed=22.21783843sINFO[07/21,08:44:33 +09] [5] State migration progress progress=64.917% totalRead=106657041 totalCommitted=106654913 pending=13469 read=19456 readElapsed=11h36m47.910769711s processElapsed=5m7.371806884s written=19717 writeElapsed=5m25.733869118s elapsed=10.147s totalElapsed=11h48m26.757580868s

Checking State Migration Status via RPC API

You can also check the progress of State Migration via RPC API. The result below shows that the state trie copying is about 64% complete.

> admin.stateMigrationStatus{
committed: 106743172,
err: “null”,
isMigration: true,
migrationBlockNumber: 29911450,
pending: 12036,
progress: 64.9169921875,
read: 106745105
}

Validation of the Copied State Trie

Once copying the state trie is done, your logs will look like as shown below. Next, you have to validate whether the state trie has been intactly copied.

The status of the validation can be checked in the logs as shown below:

Validation Status Logs

INFO[07/23,11:11:22 +09] [5] State migration progress progress=100% totalRead=368572444 totalCommitted=368572444 pending=0 read=5374 readElapsed=61h36m58.773140567s processElapsed=17m45.115580174s written=6488 writeElapsed=16m36.198342833s elapsed=4.207s totalElapsed=62h15m15.550216445sINFO[07/23,11:11:22 +09] [5] State migration : Copy is done totalRead=368572444 totalCommitted=368572444 totalElapsed=62h15m15.550264864s committed per second=1644.564INFO[07/23,11:11:22 +09] [6] CheckStateConsistencyParallel is started root=0x06b082dd3a077dee8fa7d0016ea9e96f64d8d40f3332ecff44c779b35f60b893 len(children)=16INFO[07/23,11:11:32 +09] [6] CheckStateConsistencyParallel progress cnt=35793…

Completion of State Migration

Checking State Migration Completion Logs

Once the validation for the copied state tries is done, State Migration is officially complete. If you see logs printed as below, it means that the state migration has been successful.

State Migration Completion Log

INFO[07/25,09:20:12 +09] [6] CheckStateConsistencyParallel is done cnt=649698856 err=nilINFO[07/25,09:20:12 +09] [5] State migration is completed copyElapsed=62h15m15.550264864s checkElapsed=46h8m50.500767758s

Results: Checking Chaindata Data Size Reduction

The chaindata storage for Cypress, Klaytn’s mainnet, is as follows (as of August 2020):

> du -d1 -h651G ./statetrie
56G ./body
2.7G ./misc
240K ./bridgeservice
60G ./header
6.5G ./txlookup
40G ./receipts
815G .

After booting a new EN and performing State Migration for Cypress on this EN, the chaindata is reduced as shown below.

> du -d1 -h129G ./statetrie_migrated_29911450
61G ./body
3.0G ./misc
240K ./bridgeservice
69G ./header
7.3G ./txlookup
47G ./receipts
315G .

(The reason why the data for other directories has increased is because new blocks were added in the process of the Migration.)

Things to Keep in Mind During State Migration

Here are some things to keep in mind during State Migration:

  • After State Migration, accounts will not be retrievable on past blocks since the old state tries have been deleted.
  • Do not restart a node while Migration is still in progress. Restarting a node will make State Migration start all over again, taking up more time.
  • It can take an excessive amount of time if State Migration is taking place on a smaller instance.
  • A m5.4xlarge is recommended at minimum.
  • SSD is preferred, not HDD.
  • Due to Klaytn’s frequent I/O, meeting the IOPS requirement is crucial for better performance.

Conclusion

As demonstrated above, State Migration greatly reduces data in the storage on a Klaytn node. But having to manually carry out State Migration may be a bit too demanding from the node operator’s perspective. That is why Klaytn Team is planning to keep on providing Cypress’ latest chaindata reduced by State Migration, which will be performed periodically to optimize storage usage. With this chaindata, a node operator won’t have to migrate state tries him or herself, and instead directly downloads and replaces the chaindata on his/her node. To download the latest snapshot of the chaindata, check out the following links: CypressBaobab.

We hope that this post is helpful in freeing up your storage size. Klaytn Team is always committed to making your Klaytn experience as convenient as possible. Thank you for reading and stay tuned for our next post!

MORE FROM Tutorials

Ecosystem

Bringing access to the Klaytn metaverse through CEXes

07/06/2022

Announcements

Monthly round-up: May 2022

31/05/2022

Ecosystem

Gaming: Our key to metaverse global adoption

27/05/2022

CATEGORIES

© Klaytn Foundation 2022. All rights reserved.
© Klaytn Foundation 2022. All rights reserved.