Skip to main content

One post tagged with "scale"

View All Tags

Querying Substrate Storage via RPC

· 10 min read
Shawn Tabrizi
Shawn Tabrizi
Software Engineer
In this post, we will investigate how you can interact with the Substrate RPC endpoint in order to read storage items from your Substrate runtime.​

This post was updated to the latest changes in Substrate as of January 2020.

Most of the posts I have written about Substrate so far have showed you how easy it is to build custom blockchains with this next generation framework. However, there is an entire set of parallel development and tools needed to enable users to easily interact with these new blockchain systems.

Our ultimate goal in this post is to query the balance of a Substrate user using the Substrate RPC. Along the way, we will paint a better picture of how Substrate interacts with the outside world by investigating storage structures, hashing algorithms, encoding schemes, public endpoints, metadata, and more!

Substrate RPC Methods​

Substrate provides a set of RPC methods by default which allow you to interact, query, and submit to the actual node. The available RPC methods that Substrate exposes are documented as part of the Polkadot-JS docs. Your node actually exposes this information behind another RPC endpoint: rpc_methods.

To query the balance of a Substrate user, we will need to read into the runtime storage of the Balances module. This is done by calling the getStorage method in state:

getStorage(key: StorageKey, block?: Hash): StorageData

summary: Retrieves the storage for a key

Note that specifying a block here is optional. By default, it will query the latest block.

The actual RPC method name is generated by combining the category with the documented function name, like so:

  • state_getStorage

However, to start simple, we will first query the Metadata endpoint for our Substrate node, which requires only knowledge of the method name: state_getMetadata.

Substrate RPC Endpoint​

To actually call these methods, you need access to a Substrate RPC endpoint. When you start a local Substrate node, two endpoints are made available to you:

Most of the Substrate front-end libraries and tools use the more powerful WebSocket endpoint to interact with the blockchain. Through WebSockets, you can subscribe to various items, like events, and receive push notifications whenever changes in your blockchain occur.

For the purposes of this post, we will continue to keep things simple and use the HTTP endpoint to make JSON-RPC queries to our blockchain.

Let's first use this endpoint to call the RPC methods endpoint:

$ curl -H "Content-Type: application/json" -d '{"id":1, "jsonrpc":"2.0", "method": "rpc_methods"}' http://localhost:9933/

> {"jsonrpc":"2.0","result":{"methods":["account_nextIndex","author_insertKey","author_pendingExtrinsics","author_removeExtrinsic","author_rotateKeys","author_submitAndWatchExtrinsic","author_submitExtrinsic","author_unwatchExtrinsic","chain_getBlock","chain_getBlockHash","chain_getFinalisedHead","chain_getFinalizedHead","chain_getHead","chain_getHeader","chain_getRuntimeVersion","chain_subscribeFinalisedHeads","chain_subscribeFinalizedHeads","chain_subscribeNewHead","chain_subscribeNewHeads","chain_subscribeRuntimeVersion","chain_unsubscribeFinalisedHeads","chain_unsubscribeFinalizedHeads","chain_unsubscribeNewHead","chain_unsubscribeNewHeads","chain_unsubscribeRuntimeVersion","contracts_call","state_call","state_callAt","state_getChildKeys","state_getChildStorage","state_getChildStorageHash","state_getChildStorageSize","state_getKeys","state_getMetadata","state_getRuntimeVersion","state_getStorage","state_getStorageAt","state_getStorageHash","state_getStorageHashAt","state_getStorageSize","state_getStorageSizeAt","state_queryStorage","state_subscribeRuntimeVersion","state_subscribeStorage","state_unsubscribeRuntimeVersion","state_unsubscribeStorage","subscribe_newHead","system_accountNextIndex","system_chain","system_health","system_name","system_networkState","system_nodeRoles","system_peers","system_properties","system_version","unsubscribe_newHead"],"version":1},"id":1}

Here you see a full list of available RPC apis. We can try calling the Metadata endpoint:

$ curl -H "Content-Type: application/json" -d '{"id":1, "jsonrpc":"2.0", "method": "state_getMetadata"}' http://localhost:9933/

> {"jsonrpc":"2.0","result":"0x6d65746107481853797374656d011853797374656d3c304163636f756e744e6f6e636501010130543a3a4163636f756e74496420543a3a496e64657800200000000000000000047c2045787472696e73696373206e6f6e636520666f72206163636f756e74732e3845787472696e736963436f756e...

Yay! A basic RPC call to get the metadata from Substrate is successful! However, you will notice the result is a large hex value, which really isn't that helpful...

There is more to the story.

Substrate Encoding​

What we haven't touched on yet are the various encoding mechanisms used by Substrate to both optimize serialization of data, but also provide safeties to the blockchain system.

SCALE Codec​

If we try to naively decode the hex returned from the metadata endpoint using JavaScript, we get something like:

// From StackOverflow question 3745666
function hex_to_string(metadata) {
  return metadata
    .match(/.{1,2}/g)
    .map(function (v) {
      return String.fromCharCode(parseInt(v, 16));
    })
    .join("");
}

hex_to_string(
  "0x6d65746107481853797374656d011853797374656d3c304163636f756e744e6f6e636501010130543a3a4163636f756e74496420543a3a496e64657800200000000000000000047c2045787472696e73696373206e6f6e636520666f72206163636f756e74732e3845787472696e736963436f756e..."
) >
  "\u0000meta\u0007H\u0018System\u0001\u0018System<0AccountNonce\u0001\u0001\u00010T::AccountId T::Index\u0000 \u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0004| Extrinsics nonce for accounts.8ExtrinsicCoun...";

There is real data in there! However, it is not well formed.

To correctly parse the metadata, you will need to become familiar with is Parity's SCALE codec:

SCALE is a light-weight format which allows encoding (and decoding) which makes it highly suitable for resource-constrained execution environments like blockchain runtimes and low-power, low-memory devices.

Parity uses SCALE for a number of reasons. Gav mentioned that:

  • It does not use Rust STD, and thus can compile to Wasm.
  • It is zero-copy and uses next to no memory on little-endian hardware for elementary numeric types.
  • It is built to have great support in Rust for deriving codec logic for new types: just add #[derive(Encode, Decode)].
  • It is about as thin and lightweight as can be.

Using the SCALE codec and parsing the Substrate metadata could be it's own blog post, so I will not go much deeper here; I just wanted to point out the main encoding scheme used by Substrate, and which shows up in the examples we have done so far.

Storage Keys​

For our goal, what we really want to learn is how to generate the storage keys for our various runtime storage items.

Substrate has a single key-value database for powering the entire blockchain framework. From this minimal data structure, additional abstractions can be constructed such as a Merkle Patricia tree ("trie") that is used throughout Substrate.

At a base level, to gain access to any runtime storage item, you simply need to know it's storage key for the core key-value database. To prevent key collisions, a special schema is used to generate keys for Runtime module storage items:

  • For storage values:

    xxhash128("ModuleName") + xxhash128("StorageName")

  • For storage maps:

    xxhash128("ModuleName") + xxhash128("StorageName") + blake256hash("StorageItemKey")

  • For storage double maps:

    xxhash128("ModuleName") + xxhash128("StorageName") + blake256hash("FirstKey") + blake256hash("SecondKey")

This constructs a "prefix trie", where all storage items for a module share a common prefix, where all storage keys under a storage item share a common prefix, and so on... This may not make a lot of sense right now, but we will do some practical examples below to hopefully clarify.

Learn More: Check out my deep-dive into Substrate storage here.

Querying Runtime Storage​

We are almost to the finish line. Now that you know the different storage key encoding patterns, we can try to construct and query the runtime storage for a Substrate chain. Since you will need to use some cryptographic hash functions to try this yourself, I have loaded them for you on this blog post.

Open your browser console, and you will find utility functions under util.*, util_crypto.*, and keyring.*. These come from the polkadot-js/common and will give you access to the hash functions like util_crypto.xxhashAsHex or util_crypto.blake2AsHex.

Storage Value Query​

Let's start with a simple storage value, for instance getting the Sudo user for a Substrate chain. The module name is Sudo and the storage item which holds the AccountId is named Key.

Thus we would do the following:

util_crypto.xxhashAsHex("Sudo", 128) > "0x5c0d1176a568c1f92944340dbfed9e9c";

util_crypto.xxhashAsHex("Key", 128) > "0x530ebca703c85910e7164cb7d1c9e47b";

So the combined storage key would be:

0x5c0d1176a568c1f92944340dbfed9e9c530ebca703c85910e7164cb7d1c9e47b

Note: Note that we use XXHash to output a 128 bit hash. However, XXHash only supports 32 bit and 64 bit outputs. To correctly generate the 128 bit hash, we need to hash the same phrase twice, with seed 0 and seed 1, and concatenate them.

Now we can form an RPC request using this value as the params when calling the state_getStorage endpoint:

$ curl -H "Content-Type: application/json" -d '{"id":1, "jsonrpc":"2.0", "method": "state_getStorage", "params": ["0x5c0d1176a568c1f92944340dbfed9e9c530ebca703c85910e7164cb7d1c9e47b"]}' http://localhost:9933/

> {"jsonrpc":"2.0","result":"0xd43593c715fdd31c61141abd04a99fd6822c8558854ccde39a5684e7a56da27d","id":1}

Success! The result here is the SCALE encoded AccountID of the Sudo user:

keyring.encodeAddress(
  "0xd43593c715fdd31c61141abd04a99fd6822c8558854ccde39a5684e7a56da27d"
) > "5GrwvaEF5zXb26Fz9rcQpDWS57CtERHpNehXCPcNoHGKutQY";

This is the familiar Alice account which we would expect on a --dev chain, and also matches what we get using the Polkadot-JS UI:

Sudo Key for the Substrate --dev node

Storage Map Query​

Note: The construction of storage keys for maps has slightly changed from when this was written. You can find an up to date version of that key construction in my post "Transparent Keys in Substrate".

As a final challenge, we will look to query a storage map like the balance of an account. The module name is Balances and the storage item we are interested in is named FreeBalance. They mapping for this storage item is from AccountId -> Balance, so the storage item key we want to use is an AccountId.

We need to follow the same pattern as before, but append to the end the hash of the AccountId:

util_crypto.xxhashAsHex("Balances", 128) > "0xc2261276cc9d1f8598ea4b6a74b15c2f";

util_crypto.xxhashAsHex("FreeBalance", 128) >
  "0x6482b9ade7bc6657aaca787ba1add3b4";

util_crypto.blake2AsHex(
  keyring.decodeAddress("5GrwvaEF5zXb26Fz9rcQpDWS57CtERHpNehXCPcNoHGKutQY"),
  256
) > "0x2e3fb4c297a84c5cebc0e78257d213d0927ccc7596044c6ba013dd05522aacba";

So the final storage key in this case is:

0xc2261276cc9d1f8598ea4b6a74b15c2f6482b9ade7bc6657aaca787ba1add3b42e3fb4c297a84c5cebc0e78257d213d0927ccc7596044c6ba013dd05522aacba

Just like before, we can form an RPC request using this value as the params:

$ curl -H "Content-Type: application/json" -d '{"id":1, "jsonrpc":"2.0", "method": "state_getStorage", "params": ["0xc2261276cc9d1f8598ea4b6a74b15c2f6482b9ade7bc6657aaca787ba1add3b42e3fb4c297a84c5cebc0e78257d213d0927ccc7596044c6ba013dd05522aacba"]}' http://localhost:9933

{"jsonrpc":"2.0","result":"0x0000a0dec5adc9353600000000000000","id":1}

The result here is now a SCALE encoded version of the Balance type, which is a u64 and thus trivially decodable (now that you know it is little endian):

util.hexToBn("0x0000a0dec5adc9353600000000000000", { isLe: true }).toString() >
  "1000000000000000000000";

Woohoo!

Prefix Tries​

Hopefully, you should be able to see they this storage key generation forms "prefix tries".

Let's say you wanted to query another user's balance. Well the construction of the key would make the first 256 bits exactly the same!

# All Balances -> FreeBalance storage keys start with
0xc2261276cc9d1f8598ea4b6a74b15c2f6482b9ade7bc6657aaca787ba1add3b4

This means you could actually use the state_getKeys API to get all the storage keys for all the free balances in your system!

curl -H "Content-Type: application/json" -d '{"id":1, "jsonrpc":"2.0", "method": "state_getKeys", "params": ["0xc2261276cc9d1f8598ea4b6a74b15c2f6482b9ade7bc6657aaca787ba1add3b4"]}' http://localhost:9933

> {"jsonrpc":"2.0","result":[
  "0xc2261276cc9d1f8598ea4b6a74b15c2f6482b9ade7bc6657aaca787ba1add3b4024cd62ab7726e039438193d4bbd915427f2d7de85afbcf00bd16fadbcad6aed",
  "0xc2261276cc9d1f8598ea4b6a74b15c2f6482b9ade7bc6657aaca787ba1add3b42e3fb4c297a84c5cebc0e78257d213d0927ccc7596044c6ba013dd05522aacba",
  "0xc2261276cc9d1f8598ea4b6a74b15c2f6482b9ade7bc6657aaca787ba1add3b44724e5390fcf0d08afc9608ff4c45df257266ae599ac7a32baba26155dcf4402",
  "0xc2261276cc9d1f8598ea4b6a74b15c2f6482b9ade7bc6657aaca787ba1add3b454b75224d766c852ac60eb44e1329aec5058574ae8daf703d43bc2fbd9f33d6c",
  "0xc2261276cc9d1f8598ea4b6a74b15c2f6482b9ade7bc6657aaca787ba1add3b465d0de2c1f75d898c078307a00486016783280c8f3407db41dc9547d3e3d651e",
  "0xc2261276cc9d1f8598ea4b6a74b15c2f6482b9ade7bc6657aaca787ba1add3b46b1ab1274bcbe3a4176e17eb2917654904f19b3261911ec3f7a30a473a04dcc8",
  "0xc2261276cc9d1f8598ea4b6a74b15c2f6482b9ade7bc6657aaca787ba1add3b477d14a2289dda9bbb32dd9313db096ef628101ac5bbb3b19301ede2c61915b89",
  "0xc2261276cc9d1f8598ea4b6a74b15c2f6482b9ade7bc6657aaca787ba1add3b4927407fbcfe5afa14bcfb44714a843c532f291a9c33612677cb9e0ae5e2bd5de",
  "0xc2261276cc9d1f8598ea4b6a74b15c2f6482b9ade7bc6657aaca787ba1add3b494772f97f5f6b539aac74e798bc395119f39603402d0c85bc9eda5dfc5ae2160",
  "0xc2261276cc9d1f8598ea4b6a74b15c2f6482b9ade7bc6657aaca787ba1add3b49a9304efeee429067b2e8dfbcfd8a22d96f9d996a5d6daa02899b96bd7a667b1",
  "0xc2261276cc9d1f8598ea4b6a74b15c2f6482b9ade7bc6657aaca787ba1add3b49ea52149af6b15f4d523ad4342f63089646e29232a1777737159c7bc84173597",
  "0xc2261276cc9d1f8598ea4b6a74b15c2f6482b9ade7bc6657aaca787ba1add3b4a315ee9e56d2f3bb24992a1cff6617b0f7510628a15722b680c42c2be8bb7452",
  "0xc2261276cc9d1f8598ea4b6a74b15c2f6482b9ade7bc6657aaca787ba1add3b4c4a80eb5e32005323fb878ca749473d7e5f40d60ed5e74e887bc125a3659f258"],"id":1}

This basically allows you to enumerate across all the balances in a Substrate blockchain! Although, you would not necessarily know the AccountId for these balances...

Note: Now you CAN figure out the AccountId for all these balances! Learn how in my post "Transparent Keys in Substrate".

Next Steps​

If you made it this far, you probably have come to the same conclusion as me, which is that interacting with the Substrate RPC is not trivial. Substrate is optimized for performance, bandwidth, and execution, which leaves tasks like encoding and decoding of transactions, storage, metadata, etc... to the outside world.

That being said, once you are able to walk through these examples step by step, I think it becomes easier to understand what is going on, and even reproduce this logic on other platforms and languages. Certainly this is needed for the future Substrate ecosystem.

I have started a project called Substrate RPC Examples:

https://github.com/shawntabrizi/substrate-rpc-examples

The idea of this project is to provide some easy to read, "minimal library magic" examples of interacting with the Substrate RPC. So far, I have only used the tools available in util, util_crypto, and keyring, and ideally this can be reduced by introducing a few hand written functions.

The two samples I have described in this blog post (getting metadata, querying storage) are implemented. I hope to also add to it an example of a balance transfer, which will show how to sign a message. If you have any good ideas or examples that you would want to share with the world, feel free to open a PR.

I think the next follow up from this post should be a deep dive into the SCALE codec and how you can turn the Metadata you receive from a Substrate node into valid JSON.

As always, if you are enjoying the content I have produced, take a look at my donations page to see how you can continue to support me.