Run an indexer

ksana-indexer talks to no Ksana node. It needs only a bitcoind on the same Bitcoin network, from which it collects the block headers and bodies that settlements and republishes wrote into Bitcoin. From those it builds the same block DAG and main chain as a node, executes the main-chain blocks, and checks that every block’s state_root matches what it computed.

Why run one

• Check a node. When your indexer and a node report the same state root at a height, that node’s state there is exactly what the data on Bitcoin implies. This works for the hosted node too.
• Depend on Bitcoin alone. Everything the indexer uses is on Bitcoin, so it shows that the chain can be rebuilt even if every Ksana node disappeared.
• Check finality. It decides which blocks are final from Bitcoin with the same rules as a node, and raises an alarm when something does not add up.

The signet wallet’s Verify page shows the same comparison for the hosted indexer and node.

Build and run

The indexer builds with the node (see Run a node), or on its own:

cargo build --release --locked -p ksana-indexer

It needs bitcoind on signet with -txindex=1, synced past the signet’s genesis anchor; Run a node shows how to start one. Then, from the repository’s directory:

export KSANA_BTC_RPC_PASS='<your bitcoind RPC password>'
target/release/ksana-indexer --chain signet --data-dir ~/ksana-signet/indexer \
  --genesis signet/genesis.json \
  --btc-rpc-url http://127.0.0.1:38332 --btc-rpc-user ksana

--chain signet and the genesis config must be the same as the nodes’, and the data directory remembers the genesis config it was made with. The password comes from KSANA_BTC_RPC_PASS (or --btc-rpc-pass), and bitcoind is reached over plain http:// only.

The indexer has no CORS option: a browser page reaches it only through a proxy on the page’s own origin, as the hosted wallet does with https://signet.ksana.network/indexer.

At startup it logs [ksana-indexer] started: verified height …, state root … to standard error and ksana-indexer rpc=127.0.0.1:8546 to standard output. As it catches up it logs verified up to height … (block tree main-chain height …, BTC height …), and BTC-final: block … at height … each time finality moves forward. A first sync of the signet takes as long as reading its Bitcoin blocks since the genesis anchor.

How far it trails the node

The indexer sees a block only once a settlement or republish has carried the block’s data into a Bitcoin block. It then verifies a main-chain block once that block has 2 confirmations on its main chain (SETTLEMENT_DEPTH on the signet) or is final. So its verified height trails a node’s tip by the blocks whose data has not reached a Bitcoin block yet, usually the last several minutes of blocks on the signet.

The JSON-RPC

The indexer answers two methods, over HTTP and WebSocket. indexer_status reports how far it got:

curl -sS https://signet.ksana.network/indexer -H 'content-type: application/json' \
  --data '{"jsonrpc":"2.0","id":1,"method":"indexer_status","params":[]}'

{"jsonrpc":"2.0","id":1,"result":{"height":"282","hash":"76ed3966dec4ddcd40fe31a777add64381a75b0daef81be1780549cf86e5bf7a","state_root":"7f546297e3cdf56a8724539b4930d527550d5cdcda5eee9aa714fe61cf412630","btc_height":326325,"finalized":{"height":"246","hash":"c45e4b58173b529c164e66c61219befe4bf7b3e93c319f9c80dbd98982793071"},"halted":null}}

• height, hash and state_root: the highest block it has executed and verified.
• btc_height: the Bitcoin height it has read up to (a number, unlike the fast-lane heights).
• finalized: its finalized tip, decided from Bitcoin by the same rule as a node’s.
• halted: null while all is well; otherwise why it stopped (below).

indexer_stateRoot takes a height, "latest" or a block hash, and answers for verified blocks only; for any other it returns -32004, the block has not yet been executed and verified by the indexer:

curl -sS https://signet.ksana.network/indexer -H 'content-type: application/json' \
  --data '{"jsonrpc":"2.0","id":1,"method":"indexer_stateRoot","params":["262"]}'

{"jsonrpc":"2.0","id":1,"result":{"height":"262","hash":"2f22a435469f702c2d23b1142ac39034d019fd406331a094485a789a1cc7d7b3","state_root":"e49f287ddfd7c232e655e8b9ae42626e4c745b5fcb26315dfa8ab2844f6e5e3e"}}

ksana-cli cannot talk to an indexer, as it needs a node’s methods; use curl or the TypeScript SDK’s indexer client.

Compare state roots with a node

This script compares the last five heights the indexer verified with the node’s blocks at the same heights. Set NODE and INDEXER to yours, or keep the hosted ones; it needs curl and jq.

NODE=https://signet.ksana.network/rpc
INDEXER=https://signet.ksana.network/indexer
call() {  # url method params
  curl -sS "$1" -H 'content-type: application/json' \
    --data "{\"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"$2\",\"params\":$3}" | jq -c .result
}
top=$(call "$INDEXER" indexer_status '[]' | jq -r .height)
for h in $(seq $((top - 4)) "$top"); do
  a=$(call "$INDEXER" indexer_stateRoot "[\"$h\"]" | jq -c '[.hash, .state_root]')
  b=$(call "$NODE" ksana_getBlock "[\"$h\"]" | jq -c '[.hash, .header.state_root]')
  if [ "$a" = "$b" ]; then echo "$h same $a"; else echo "$h DIFFERENT indexer $a node $b"; fi
done

278 same ["d5e70875e331fb4093b9f8d5c1d460f1f6b557f063d85e93b5a0fa8fbf437d01","7f546297e3cdf56a8724539b4930d527550d5cdcda5eee9aa714fe61cf412630"]
279 same ["ca81971355fdbae2693c4fe0b8a31670c5c410314524fd9e01dbd97aa01b5fe2","7f546297e3cdf56a8724539b4930d527550d5cdcda5eee9aa714fe61cf412630"]
280 same ["792fd0dbb60d85f8195326e0ac83847c10d031ccb8b225a6fa64f3764289fee2","7f546297e3cdf56a8724539b4930d527550d5cdcda5eee9aa714fe61cf412630"]
281 same ["d18e91c9383400d27166695177decadb89399456e55d79eb6e6164d625edaf95","7f546297e3cdf56a8724539b4930d527550d5cdcda5eee9aa714fe61cf412630"]
282 same ["76ed3966dec4ddcd40fe31a777add64381a75b0daef81be1780549cf86e5bf7a","7f546297e3cdf56a8724539b4930d527550d5cdcda5eee9aa714fe61cf412630"]

The block hashes must match as well as the state roots. Here every state root is the same because these were empty blocks, which change no state.

A difference right after a Bitcoin reorganization or a switch of main chain can last until both sides have followed it. A difference that stays at heights both have long passed means that the node and the indexer executed the same block differently: keep both data directories and report it as the Security page describes.

Alarms

The indexer reports problems on standard error as lines that start with [ksana-indexer] alarm. Most describe something it handles by itself:

Three alarms of the highest level need you:

• alarm (highest level): … processing halted, also shown in indexer_status as halted, stops the indexer. A block that Bitcoin made final turned out invalid: either the fast lane finalized an invalid block, or Ksana’s software computes differently on different machines. Keep the data directory and report it.
• the payload contains a block header of protocol version …, but this indexer only supports up to … means the network has upgraded its rules. The indexer stops before the change; update it from the repository.
• cannot check block header … in the payload means this machine could not compute a header’s proof of work, a local fault such as memory it could not allocate. The indexer judges nothing, logs sync failed (will retry later) and tries again; if the alarm keeps coming back, look at the machine.

Restarts and pruning

The indexer keeps its work in the data directory (state/, index/, indexer.json and pruning.ckpt). After a restart it first checks its stored state against the data on Bitcoin, waiting while bitcoind is behind, and then logs verification against the BTC data complete: continuing from height ….

Pruning is on by default and works as on a node: memory keeps only what the main chains it can still follow need, the verified main chain below that is answered from the index on disk, and each pruning writes a checkpoint the next start continues from. In a 10-hour devnet soak test the indexer ended at 67 MiB of resident memory with a data directory of about 45 MB after 7,308 blocks.

The indexer starts over from the genesis anchor by itself when it cannot continue from what it kept: after a Bitcoin reorganization deeper than that, when the main chain leaves its pruning root, or when its index is lost. It logs an alarm and rebuilds everything from bitcoind, which takes as long as a first sync. After a signet restart from genesis, give it the new genesis config and a new data directory.