-
Notifications
You must be signed in to change notification settings - Fork 6
docs: prepare state store guidance for RocksDB removal #62
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from 2 commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -9,9 +9,9 @@ | |
| including configuration management, maintenance procedures, and best practices | ||
| for stable and performant operations. | ||
|
|
||
| ## Configuration Management | ||
|
|
||
| ### Directory Structure | ||
|
|
||
| The Sei node configuration is stored in `$HOME/.sei/config/`: | ||
|
|
||
|
|
@@ -31,9 +31,9 @@ | |
| [Default Configurations](#default-configurations) at the bottom of this | ||
| section. | ||
|
|
||
| ### Essential Configuration Parameters | ||
|
|
||
| #### Network Settings (config.toml) | ||
|
|
||
| ```toml | ||
| [p2p] | ||
|
|
@@ -59,7 +59,7 @@ | |
| timeout-broadcast-tx-commit = "10s" | ||
| ``` | ||
|
|
||
| #### Application Settings (app.toml) | ||
|
|
||
| ```toml | ||
| # Minimum gas prices to prevent spam transactions | ||
|
|
@@ -89,6 +89,8 @@ | |
| [state-store] | ||
| # Historical SS layer for queries. Required for any node serving RPC. | ||
| ss-enable = true | ||
| # State-store backend. Use PebbleDB; RocksDB support will be removed. | ||
| ss-backend = "pebbledb" | ||
|
alexander-sei marked this conversation as resolved.
|
||
| # 0 = keep everything; 100,000 is roughly 28 hours of pacific-1 history. | ||
| ss-keep-recent = 100000 | ||
|
|
||
|
|
@@ -97,12 +99,14 @@ | |
| rs-backend = "pebbledb" | ||
| ``` | ||
|
|
||
| ### Default Configurations | ||
|
|
||
| The full unmodified `app.toml`, `config.toml`, and `client.toml` produced by | ||
| `seid init` against the latest tagged `seid` release. Use these as the | ||
| canonical reference for every available knob and its default value. | ||
|
|
||
| <Warning>The generated `app.toml` below mirrors the latest tagged release and may still list RocksDB as a state-store option. Do not use RocksDB for new or resynced nodes. RocksDB support for the SeiDB state store will be removed. No target release has been published.</Warning> | ||
|
|
||
| <Tabs> | ||
| <Tab title="app.toml"> | ||
|
|
||
|
|
@@ -1349,7 +1353,7 @@ | |
| </Tab> | ||
| </Tabs> | ||
|
|
||
| ## Database Management | ||
|
|
||
| ### Architecture | ||
|
|
||
|
|
@@ -1366,15 +1370,18 @@ | |
| modes is set to a non-default value. | ||
| 2. **State Store (SS)** — versioned raw key/value pairs used for historical | ||
| queries. Required for any node that serves RPC. The default backend is | ||
| **PebbleDB**; **RocksDB** is available for iteration-heavy workloads such | ||
| as archive nodes or RPC nodes that run a lot of `debug_trace*` (see the | ||
| [RocksDB Backend Guide](/node/rocksdb-backend) for build instructions). | ||
| **PebbleDB**. RocksDB support for this layer will be removed. No target | ||
| release has been published. Check the | ||
| [Sei release notes](https://github.com/sei-protocol/sei-chain/releases) | ||
| before each upgrade. Use PebbleDB for new nodes. Before upgrading, resync | ||
| any node that uses RocksDB onto PebbleDB through [state sync](/node/statesync) | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. [blocker] "resync any node that uses RocksDB onto PebbleDB through [state sync] or a [PebbleDB snapshot]" is not a valid path for archive nodes, and following it destroys history. State sync fetches state at a recent height and discards earlier versions; a public snapshot is likewise pruned. Neither can reproduce an archive node running Add the archive-node case here — either that they must sync from genesis onto PebbleDB, or that no supported path exists yet and they should hold on the current release. As written, an archive operator reading this loses their history at the next upgrade. |
||
| or a [PebbleDB snapshot](/node/snapshot). | ||
|
|
||
| The legacy IAVL backend is still selectable via `sc-enable = false` but is | ||
| deprecated and slated for removal — new deployments and existing nodes | ||
| should run on SeiDB. | ||
|
|
||
| ### SeiDB Configuration | ||
|
|
||
| The full set of knobs is in the auto-generated | ||
| [Default Configurations](#default-configurations) above. The block below | ||
|
|
@@ -1420,7 +1427,7 @@ | |
| # serving RPC must keep it on. | ||
| ss-enable = true | ||
|
|
||
| # pebbledb (default) or rocksdb (faster iteration, archive-friendly). | ||
| # Use PebbleDB. RocksDB support will be removed. | ||
| ss-backend = "pebbledb" | ||
|
|
||
| ss-async-write-buffer = 100 | ||
|
|
@@ -1444,11 +1451,26 @@ | |
| prune-interval-seconds = 600 | ||
| ``` | ||
|
|
||
| #### PebbleDB version encoding | ||
|
Check warning on line 1454 in node/node-operators.mdx
|
||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. [suggestion] Inserting this Either move the new subsection below that paragraph, or lift the pruning paragraph up to directly follow the TOML block. |
||
|
|
||
| Fresh PebbleDB state stores use descending-version MVCC encoding. Sei includes | ||
| the version in each key and sorts newer versions first. This lets | ||
| latest-version reads reach the newest visible value without scanning older | ||
| versions. The store records this format with the `s/_mvcc_descending` sentinel | ||
| key, which `seid` detects automatically when it opens the database. | ||
|
|
||
| PebbleDB state stores created by older builds use ascending-version encoding. | ||
| `seid` detects these stores and opens them in compatibility mode without an | ||
| error. They remain on the slower ascending read path. Rebuild the state store | ||
| through [state sync](/node/statesync) with a current `seid` release to adopt | ||
| descending encoding. A filesystem snapshot preserves the source store's | ||
| encoding, so confirm its encoding with the snapshot provider. | ||
|
|
||
| Setting small (more frequent) pruning intervals may collide with | ||
| snapshot creation. Too-large (less frequent) intervals mean pruning takes | ||
| longer overall, which can cause missed blocks and excessive resync time. | ||
|
|
||
| ### Giga Storage and Giga Executor | ||
|
|
||
| These are two **separate** opt-in features that ship in newer `seid` | ||
| releases. Both default to off; only enable them deliberately and after | ||
|
|
@@ -1492,7 +1514,7 @@ | |
| occ_enabled = false | ||
| ``` | ||
|
|
||
| ### Database Maintenance | ||
|
|
||
| The database is typically stable and can be left alone, although some attention | ||
| may be required: | ||
|
|
@@ -1519,9 +1541,9 @@ | |
|
|
||
| <Warning>The wipe command above deletes the entire local database (everything except `priv_validator_state.json`) and the `wasm` folder. It does not compact data in place — after running it, the node must be re-synced from a [snapshot](/node/snapshot) or via [state sync](/node/statesync) before it can serve traffic again.</Warning> | ||
|
|
||
| ## Service Management | ||
|
|
||
| ### Systemd Commands | ||
|
|
||
| ```bash | ||
| # Check service status | ||
|
|
@@ -1540,7 +1562,7 @@ | |
| journalctl -fu seid -o cat | ||
| ``` | ||
|
|
||
| ### Log Management | ||
|
|
||
| Prevent logs from consuming excessive disk space by enabling rotation: | ||
|
|
||
|
|
@@ -1561,7 +1583,7 @@ | |
| EOF | ||
| ``` | ||
|
|
||
| ## Update Procedures | ||
|
|
||
| <Info> | ||
| Upgrade with the same method you originally installed with: `make install` | ||
|
|
@@ -1571,7 +1593,7 @@ | |
| `which -a seid` lists every copy. | ||
| </Info> | ||
|
|
||
| ### Minor Updates | ||
|
|
||
| For minor updates that are non-consensus-breaking: | ||
|
|
||
|
|
@@ -1613,7 +1635,7 @@ | |
| </Tab> | ||
| </Tabs> | ||
|
|
||
| ### Major Updates | ||
|
|
||
| For major upgrades that introduce state-breaking changes: | ||
|
|
||
|
|
@@ -1657,7 +1679,7 @@ | |
| halt-height so the swap takes seconds at upgrade time. | ||
| </Tip> | ||
|
|
||
| ## Performance Optimization | ||
|
|
||
| Performance optimizations can yield different results depending on your system's | ||
| hardware, workload, and network conditions. Before implementing any changes, | ||
|
|
@@ -1665,7 +1687,7 @@ | |
| your specific configuration and requirements. Always back up important data | ||
| before making modifications. | ||
|
|
||
| ### Memory Management (sysctl tuning) | ||
|
|
||
| Optimizing memory management settings can help improve performance and | ||
| stability, particularly for high-load nodes. These settings control swap usage | ||
|
|
@@ -1679,7 +1701,7 @@ | |
| vm.dirty_writeback_centisecs = 100 # Frequency (in hundredths of a second) at which the system writes "dirty" pages to disk | ||
| ``` | ||
|
|
||
| ### Network Stack Optimization | ||
|
|
||
| Tuning the network stack can enhance packet processing efficiency and | ||
| throughput, particularly for nodes handling a large number of peers and high | ||
|
|
@@ -1693,7 +1715,7 @@ | |
| net.core.wmem_max = 16777216 # send buffer size for network sockets | ||
| ``` | ||
|
|
||
| ### Storage Optimization | ||
|
|
||
| Optimizing storage settings can significantly reduce write latency and improve | ||
| database performance, especially for nodes using NVMe SSDs. | ||
|
|
@@ -1703,9 +1725,9 @@ | |
| blockdev --setra 4096 /dev/nvme0n1 # readahead value to optimize sequential reads | ||
| ``` | ||
|
|
||
| ## Backup and Recovery | ||
|
|
||
| ### Regular Backups | ||
|
|
||
| Automate backups to avoid data loss: | ||
|
|
||
|
|
@@ -1724,7 +1746,7 @@ | |
| systemctl start seid | ||
| ``` | ||
|
|
||
| ### Recovery Procedure | ||
|
|
||
| Restoring from backup in case of corruption or accidental deletion: | ||
|
|
||
|
|
@@ -1742,7 +1764,7 @@ | |
| systemctl start seid | ||
| ``` | ||
|
|
||
| ## Security Considerations | ||
|
|
||
| - Use firewalls and rate-limiting to prevent attacks | ||
| - Keep your system and node software updated | ||
|
|
||
This file was deleted.
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -8,11 +8,13 @@ | |
|
|
||
| Follow this guide to join an existing network through **snapshot sync**. To quickly spin up a fresh full node and join the network, it's recommended to restore from a snapshot instead of replaying all historical blocks. | ||
|
|
||
| ## Snapshot Sync | ||
|
|
||
| Snapshot sync allows a new node to join a network by downloading a recent, compressed copy of the entire application state and extracting it directly into the data directory. This reduces the initial sync time from days to minutes. | ||
|
|
||
| <Warning>RocksDB support for the SeiDB state store will be removed. No target release has been published. For new or resynced nodes, use a snapshot only after confirming that its state store uses PebbleDB. Keep `ss-backend = "pebbledb"` in `app.toml`.</Warning> | ||
|
|
||
| ## Snapshot Providers | ||
|
|
||
| You can select from various providers for downloading snapshots: | ||
|
|
||
|
|
@@ -23,6 +25,10 @@ | |
|
|
||
| <iframe src="https://seistream.app/labs?page=snapshots" width="100%" height="600" style={{ border: 'none', borderRadius: '8px' }} title="Sei Snapshot Providers" /> | ||
|
|
||
| Snapshot providers do not consistently label the state-store backend. Follow | ||
| the verification step below before you extract a snapshot. If you cannot | ||
| identify the backend from the archive, ask the provider to confirm it. | ||
|
|
||
| ## Clean Up & Preparation | ||
|
|
||
| If you are **not** starting a node from fresh, perform the following backups and clean‑ups first. | ||
|
|
@@ -79,21 +85,37 @@ | |
| SNAPSHOT_URL="<PASTE_SNAPSHOT_URL_HERE>" | ||
| ``` | ||
|
|
||
| 2. **Download and Extract** | ||
| Most providers compress the `data` and `wasm` directory directly. The following command streams the download and extracts it into `$HOME/.sei`. | ||
| 2. **Download, verify, and extract** | ||
| Most providers compress the `data` and `wasm` directories directly. Download the archive so you can inspect its state-store path before extraction: | ||
|
|
||
| ```bash | ||
| curl -L $SNAPSHOT_URL | lz4 -c -d | tar -x -C $HOME/.sei | ||
| curl -L "$SNAPSHOT_URL" -o snapshot.tar.lz4 | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. [suggestion] Dropping the streaming
A cheaper alternative that keeps the streaming extract: extract as before, then verify the backend on the extracted tree before starting |
||
|
|
||
| lz4 -c -d snapshot.tar.lz4 | tar -tf - | \ | ||
| awk '/(^|\/)data\/(state_store\/cosmos\/)?(pebbledb|rocksdb)\// { print; exit }' | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. [blocker] This verification gate — the core safety mechanism of the PR — can pass a snapshot that contains a RocksDB store.
It also doesn't cover the EVM SS store under Suggest collecting all matches and failing on any RocksDB hit instead of exiting early: lz4 -c -d snapshot.tar.lz4 | tar -tf - \
| grep -oE '(^|/)data/((state_store/cosmos|evm_ss)/)?(pebbledb|rocksdb)/' \
| sort -uThen: proceed only if every printed path contains Minor side effect worth noting in the doc either way: the early |
||
| ``` | ||
|
|
||
| **Alternative: Parallel Download with aria2** | ||
| The printed path must contain either `data/pebbledb/` or | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. [suggestion] Also worth a sentence: |
||
| `data/state_store/cosmos/pebbledb/`. Do not extract the archive if the path | ||
| contains `rocksdb` or the command prints no state-store path. | ||
|
|
||
| Extract the verified archive, then remove it: | ||
|
|
||
| ```bash | ||
| lz4 -c -d snapshot.tar.lz4 | tar -x -C $HOME/.sei | ||
| rm snapshot.tar.lz4 | ||
| ``` | ||
|
|
||
| **Alternative: parallel download with aria2** | ||
|
|
||
| For faster downloads, especially with large snapshots, you can use `aria2c` which supports parallel connections: | ||
|
|
||
| ```bash | ||
| # Download with 16 parallel connections | ||
| aria2c -x 16 -s 16 -o snapshot.tar.lz4 $SNAPSHOT_URL | ||
|
|
||
| # Run the backend verification command above before extraction | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. [nit] A bare Same gap applies to the |
||
|
|
||
| # Extract after download completes | ||
| lz4 -c -d snapshot.tar.lz4 | tar -x -C $HOME/.sei | ||
|
|
||
|
|
@@ -117,13 +139,18 @@ | |
| cp $HOME/priv_validator_state.json $HOME/.sei/data/priv_validator_state.json | ||
| ``` | ||
|
|
||
| 4. **Enable SeiDB** | ||
| Make sure to enable `sei-db` in your config if it's not already enabled: | ||
| 4. **Enable SeiDB and configure PebbleDB** | ||
| Make sure SeiDB is enabled and the state-store backend is PebbleDB: | ||
|
|
||
| ```bash | ||
| sed -i.bak -E "/^\[state-commit\]/,/^\[.*\]/ s|^(sc-enable[[:space:]]*=[[:space:]]*).*$|\1true| ; /^\[state-store\]/,/^\[.*\]/ s|^(ss-enable[[:space:]]*=[[:space:]]*).*$|\1true|" $HOME/.sei/config/app.toml | ||
| sed -i.bak -E "/^\[state-commit\]/,/^\[.*\]/ s|^[#[:space:]]*(sc-enable[[:space:]]*=[[:space:]]*).*$|\1true| ; /^\[state-store\]/,/^\[.*\]/ s|^[#[:space:]]*(ss-enable[[:space:]]*=[[:space:]]*).*$|\1true| ; /^\[state-store\]/,/^\[.*\]/ s|^[#[:space:]]*(ss-backend[[:space:]]*=[[:space:]]*).*$|\1\"pebbledb\"|" $HOME/.sei/config/app.toml | ||
|
|
||
| sed -n '/^\[state-store\]/,/^\[.*\]/p' $HOME/.sei/config/app.toml | ||
| ``` | ||
|
|
||
| Confirm that the output includes `ss-backend = "pebbledb"`. If the key is | ||
| absent, add it directly below `[state-store]` before you restart the node. | ||
|
|
||
| 5. **Restart the Node** | ||
|
|
||
| ```bash | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
[nit] Removing
'rocksdb'here leaves no page in the docs carrying that keyword now thatnode/rocksdb-backend.mdxis deleted. Since the point of this PR is to reach RocksDB operators, keeping the keyword on at least one page (this one, ornode-operators.mdx) helps site search and LLM retrieval surface the deprecation warning to exactly the audience that needs it.