Import Database

This page will guide you through the process of starting a node in import-db mode, allowing the reprocessing of older transactions.

Nodes can revalidate an existing database by importing it and initializing with designated flags.

Blockchain State Verification and Data Indexing

Checking the blockchain state at a specific point in time, ensuring software compatibility, validating blockchain integrity, and indexing data from genesis to the present are critical tasks for maintaining a secure and efficient blockchain system. This guide covers how to:

• Check Blockchain State at a Specific Time: This involves using APIs or direct database access to examine the state of the blockchain at a given block number (e.g., block 255255). For example, one could import the database module and configure the node to pause at the block of interest.

• Ensure Backwards Compatibility with New Software Versions: It's crucial to verify that updates or new versions of the blockchain software do not introduce compatibility issues with existing data and functionalities.

• Validate Blockchain State: Regularly validating the integrity and consistency of the blockchain state helps in identifying and rectifying discrepancies early.

• Index Data in Elasticsearch: For enhanced query capabilities and analytics, you can index all the blockchain data, from the genesis block to the latest, into a search engine like Elasticsearch.

How to start the process​

Let's suppose we have the following data structure:

  ~/of-chain-go/cmd/node

The node binary is in the specified path. We also have a database built from a continuously syncing observer with the chain from genesis without shard switching.. This database will be placed in a directory, let's presume we will place it near the node's binary, yielding a data structure as follows:

.
├── config
│    ├── api.toml
│    ├── config.toml
│    ...
├── import-db
│    └── db
│        └── 1
│            ├── Epoch_0
│            │     └── Shard_1
│            │         ├── BlockHeaders
│            │         │   ...
│            │         ├── BootstrapData
│            │         │   ...
│            │         ...
│            └── Static
│                  └── Shard_1
│                      ...
├── node

Ensure the db directory is a subdirectory of import-db. Verify that the config directory, especially the prefs.toml file, matches the original node's configuration.

WARNING

Before you begin the import-db process, ensure that the /of-chain-go/cmd/node/db directory is completely empty. This will allow the import to start from the genesis block and proceed up to the last available epoch.

Next, the node can be started by using:

 cd ~/of-chain-go/cmd/node
 ./node -use-log-view -log-level *:INFO -import-db ./import-db

NOTE

The -import-db flag designates the path to the source database directory. In the given example, it's assumed the directory is named import-db and is situated close to the node executable.

The node will start the reprocessing of the provided database. It will end with a message like:

import ended because data from epochs [x] or [y] does not exist

NOTE

To accelerate the import-db process, omit checking block header signatures when using data from a trusted source. Add the -import-db-no-sig-check flag upon starting the node, alongside previously mentioned flags.

Import-DB with populating an Elasticsearch cluster​

Utilizing the import-db mechanism allows for the efficient population of an Elasticsearch cluster by re-processing data through this process.

NOTE

Import-DB for populating an Elasticsearch cluster should be used only for a full setup (a node in each Shard + a Metachain node)

• To prepare, update the external.toml file on each node.

• Use Import-DB only for full setups.

• Nodes will push re-processed data to the Elasticsearch cluster if configured correctly.

More details will be released when the testnet phase starts.