The multikey feature allows a node instance to hold more than one key. These type of nodes used in multikey operations can be assimilated as a hybrid between an observer node and a validator. It behaves as an observer by holding in the validatorKey.pem file, a BLS key that will never be part of the consensus group. The node behaves also as a validator (or multiple validators) by monitoring and emitting consensus messages, whenever required on the behalf of the managed keys set.
Since an observer already performs block validation, it can be easily used to manage a group of validator keys and propose or validate blocks on behalf of the keys it possesses. To summarize, this type of node can use any provided keys, in any combination, to generate consensus messages provided that those used keys are part of the consensus group in the current round. With the multikey feature, the relationship now becomes n:2, providing that n is the number of keys managed by an entity.
INFO
This feature is purely optional. Normal 1:1 relationship between the keys and the nodes is still supported. The multikey mode should optimize the costs when running a set of keys (check Economics running multikey node section)
The following description outlines the relationship between keys and nodes in single operation mode compared to multikey operation mode.
Nodes employing the multikey feature, in addition to determining the consensus group (a task typically performed by each node individually), have the capability to access and utilize the provided set of keys. They can use one or multiple keys in any combination if the node detects that at least one managed key is a part of the consensus group. The introduction of code changes to accommodate multikey nodes primarily impacting the consensus, keyManagement, and heartbeat packages.
Enhanced Security Through Virtual Peer IDs
The managing group, referred to as multikey nodes or the multikey group, enhances security by assigning validators' BLS information to "virtual" peer IDs. These virtual peer IDs are unique p2p identities that, unlike regular IDs, do not have a real address linked to them, making it impossible for the p2p network to establish a direct connection. This innovative approach adds an extra layer of security by obscuring the relationship between the validator BLS keys and the actual hosts managing those keys.
Enhanced Redundancy Sub-system for Multikey Operations
The redundancy sub-system has been enhanced to support multikey operations effectively, ensuring robustness across multiple fallback redundancy groups. A unique multikey fallback group will now independently monitor each key within a managed node for any missed consensus activities. This allows for a more resilient approach, where issues such as misconfiguration or unavailability in the primary group's nodes activate fallback mechanisms automatically.
For instance, if the primary multikey group is configured to manage keys from [key_0, key_1 ... key_e-1, key_e+1 ... key_n] (excluding key_e), and the redundancy multikey fallback group is set to [key_0, key_1 ... key_e-1, key_e, key_e+1 ... key_n], the system will trigger the fallback for key_e after detecting k missed consensus activities (such as proposing or signing a block) within the primary group. The parameter k is configurable via the prefs.toml file under the RedundancyLevel setting, ensuring flexibility in threshold adjustments for activating fallback operations.
As for n managed keys we will need at least a group of nodes, there is a threshold that a staking operator will want to consider when deciding to switch the operation towards the multikey mode. The switch becomes attractive for the operator when he has more than one key. So, for the time being, when we have at least 2 keys that are either eligible or waiting, the switch to multikey mode becomes feasible.
Recommended Multikey Group Size for Node Operators
While there are no explicit constraints in the source code on the number of keys a multikey group can have, the OneFinity team advises node operators to limit themselves to 32 keys per group. Exceeding this recommended number could potentially harm the blockchain. Specifically, a node with an excessive number of keys might be able to propose multiple incorrect blocks in succession, thereby hindering block synchronization and cross-notarization processes.
Transitioning to multikey operation involves compiling all BLS keys into a single file named allValidatorsKeys.pem. This file should be located in the same directory as the validatorKey.pem file but can be specified elsewhere using the --all-validator-keys-pem-file binary flag. Below is an example of the contents of an allValidatorsKeys.pem file:
-----BEGIN PRIVATE KEY for e296e97524483e6b59bce00cb7a69ec8c0d1ac4227925f07fdd57b3ab4ec2f64b240728a0a3c5be2930aea570bf12c12314e25d942b106472800e51524add26ec9546475c1cfae91dd7e799f256d1b0758e17aaa3898c29d489bd87c86d04498-----
YzJlODM0NTdmOTVmYMDVjZGRiNzdiODc1N2YyZGEx
ZGRhYWY5MTI5Y2NlOWQyOQ==
-----END PRIVATE KEY for e296e97524483e6b59bce00cb7a69ec8c0d1ac4227925f07fdd57b3ab4ec2f64b240728a0a3c5be2930aea570bf12c12314e25d942b106472800e51524add26ec9546475c1cfae91dd7e799f256d1b0758e17aaa3898c29d489bd87c86d04498-----
-----BEGIN PRIVATE KEY for 5585ddceb6b7bf0d308162efd895d0717b22bab6b0412f09fb9cee234be73d197bfef8ae10064be5733472c573894015029672b70f63e0b58c7ab2e831ee0aff88b868e4d712bec0baf9a1cd1982e138af9b6cc55e4454b01cb8ad02a064f515-----
MzNlZjQyYTRhZDc3ZDBkZDk1M2JmNGIwNWE2MzczMmYxZWUy
ZWVkNzNiOGQ1ZDQ0NmEzMg==
-----END PRIVATE KEY for 5585ddceb6b7bf0d308162efd895d0717b22bab6b0412f09fb9cee234be73d197bfef8ae10064be5733472c573894015029672b70f63e0b58c7ab2e831ee0aff88b868e4d712bec0baf9a1cd1982e138af9b6cc55e4454b01cb8ad02a064f515-----
-----BEGIN PRIVATE KEY for 791c7e2bd6a5fb1371af18269267ad8ef9e56e264c4c95703c57526b16b84dd8df6347c0cc14f93d595a12316d38ae11264e05d2fa26d80387d12db52c1a98e93064d073d02549c71ec4e352d73724c21c02245b25d3643b532fac25d7580f0b-----
OTcxYjYyNWMzMzlkY2JhNTAyODMwNzZlYjMyY2MxMmYzNThiMjNiNzYz
NTA4YjFjMTVlYTIwNDYyMw==
-----END PRIVATE KEY for 791c7e2bd6a5fb1371af18269267ad8ef9e56e264c4c95703c57526b16b84dd8df6347c0cc14f93d595a12316d38ae11264e05d2fa26d80387d12db52c1a98e93064d073d02549c71ec4e352d73724c21c02245b25d3643b532fac25d7580f0b-----
In systems where both NodeDisplayName and Identity fields are implemented across all managed and loaded BLS keys, there's a specific naming convention to follow. Specifically, the NodeDisplayName will have an appended order index for every managed key. As an illustration, consider a scenario with NodeDisplayName configured to example. In such cases, the naming convention for the managed keys would adhere to the following pattern:
If a subset of BLS keys must operate under a distinct identity or employ a different naming convention, the NamedIdentity section is particularly useful. Continuing with our example, to assign a new identity or node name to the 791c7e2bd6a5fb1371... key, the section should be defined as follows:
# NamedIdentity represents an identity that runs nodes on the multikey# There can be multiple identities set on the same node, each one of them having different bls keys, just by duplicating the NamedIdentity[[NamedIdentity]]# Identity represents the keybase/GitHub identity for the current NamedIdentity Identity ="identity2"# NodeName represents the name that will be given to the names of the current identity NodeName ="random"# BLSKeys represents the BLS keys assigned to the current NamedIdentity BLSKeys = ["791c7e2bd6a5fb1371af18269267ad8ef9e56e264c4c95703c57526b16b84dd8df6347c0cc14f93d595a12316d38ae11264e05d2fa26d80387d12db52c1a98e93064d073d02549c71ec4e352d73724c21c02245b25d3643b532fac25d7580f0b" ]
The multi-key feature enables the use of multiple keys on a small group of nodes. At first glance, this appears to potentially weaken security by offering attackers more targets within a large staking provider. However, there are strategies to diminish these concerns, as outlined below:
Use the recommendation found in this page regarding the maximum number of keys per multikey group;
Ensure that each primary multi-key group has at least one backup multi-key group as a precautionary measure in case of any unforeseen issues.
Use the NamedIdentity configuration explained above to obfuscate the BLS keys and their declared identity from the actual nodes that manage the keys.
Regarding point 3, each managed BLS key will create a virtual P2P identity that no node from the network can connect to, as it does not advertise the connection info but is only used to sign P2P messages. Associated with a separate named identity, the system will render the BLS key virtually unreachable, and its origin hidden from the multikey nodes. Therefore, node operators will need to apply the following changes to the prefs.toml file:
In the [Preference] section, the two options called NodeDisplayName and Identity should be changed to terms different from those used in BLS definitions to avoid easy matching. Generic names like gateway or observer are suitable for this section. Additionally, completely random strings can be employed to facilitate the identification of nodes in the explorer. The Identity field can be left empty.
In the [[NamedIdentity]] section, the two options called NodeName and Identity will be changed to the actual identities of the BLS keys, such as the staking provider brand names. They must be different from those defined in the [Preference] section.
In this manner, the operation will bear resemblance to the sentinel nodes encountered in other contexts. The key distinction in our scenario lies in the significantly simplified setup, as there's no need to maintain a separate network for the protected nodes. Implementing points 1, 2, and 3 will ensure that the security of our arrangement is on par with a _sentinel setup.
Let's assume we possess 5 BLS keys belonging to a staking provider named testing-staking-provider, and we aim to implement the security measures discussed earlier. For illustrative purposes, we have generated 5 random BLS keys. The allValidatorsKeys.pem file should therefore contain entries similar to the following:
-----BEGIN PRIVATE KEY for 15eb03756fae81d2fbae392a4d7d82abdf7618ce3056b89376c2a46bc6e8403ed3cc84e12bc819c0b088ee46e7c28302d2b666b011714cc8ea2b75488907d07e194a6e83f0f3d15c7699de412de425314be5cc3ce6ab2c594690006f9915dd15-----
NDA5MWVjODMwZjU3MDhkYmQwNzk5ZWEwNjg2MDc0MzUzYmZjNThjM2ZhYzU2Y2I1
ZGRhMjY3YTY1NjhkZjI1YQ==
-----END PRIVATE KEY for 15eb03756fae81d2fbae392a4d7d82abdf7618ce3056b89376c2a46bc6e8403ed3cc84e12bc819c0b088ee46e7c28302d2b666b011714cc8ea2b75488907d07e194a6e83f0f3d15c7699de412de425314be5cc3ce6ab2c594690006f9915dd15-----
-----BEGIN PRIVATE KEY for ff12bc7f471e2e375c6e8b981f13ed823dcca857c41a2ffc3a0956283a8428a95754375dabc0b412df3ec41d2a51ef1490a8d23f4e4f9348787f9615093e0129969085488b59d2ab550467cd0d0fa33df22e2ed2d8c8c0c0f59042dafd0c1098-----
MTcwN2ZlMzFhMzk3Y2VjOWM4ZjdmMWU3Njg4MjY3YTAwOWU5ZjJmMWYxY2Y0ZjFl
MzI2Y2M5NGJiZGFjNGQwZA==
-----END PRIVATE KEY for ff12bc7f471e2e375c6e8b981f13ed823dcca857c41a2ffc3a0956283a8428a95754375dabc0b412df3ec41d2a51ef1490a8d23f4e4f9348787f9615093e0129969085488b59d2ab550467cd0d0fa33df22e2ed2d8c8c0c0f59042dafd0c1098-----
-----BEGIN PRIVATE KEY for 3dec570c02a4444197c1ed53fefd7e57acb9bc99ae47db7661cfbfb47170418702162a46ed40e113e3381d68b713e903e286ffaf9cac77fed8f9c79e83f2abb0ccd690ef4f689607b6414a6f893e0c0ced93d7456240bbccbf223f7603dd8e05-----
ZWMwYWRjYjNiYTQ0YmM4MGM5ZjhmNTlkNTU5YTRlMWJlMTI2ODFmMDlmM2JiNTM4
MmMyYzdlYmNhYjNkNTk2MA==
-----END PRIVATE KEY for 3dec570c02a4444197c1ed53fefd7e57acb9bc99ae47db7661cfbfb47170418702162a46ed40e113e3381d68b713e903e286ffaf9cac77fed8f9c79e83f2abb0ccd690ef4f689607b6414a6f893e0c0ced93d7456240bbccbf223f7603dd8e05-----
-----BEGIN PRIVATE KEY for 38a93e3c00128c31769823710aa7deb145591b99a78c87dbd74c894afd540ade6de3906b45001d3f5a5882db34eaf30e412bef77ed43cf5a394edd0aa70254a74db1c80eef5d41342cae76fbbae596bc811fa491e00f16a7e011a836f7ceaa15-----
YWMzMDk2ZjY3NmExNjhiNTQ5ODQzM2JiM2NiZWFmNzkyYjQyYWZhZjJlZmMwNjNl
YzdhMWI5OGM1ZDdjODg1MQ==
-----END PRIVATE KEY for 38a93e3c00128c31769823710aa7deb145591b99a78c87dbd74c894afd540ade6de3906b45001d3f5a5882db34eaf30e412bef77ed43cf5a394edd0aa70254a74db1c80eef5d41342cae76fbbae596bc811fa491e00f16a7e011a836f7ceaa15-----
-----BEGIN PRIVATE KEY for 1fce426b632e5a5941d9989e4f8bbb93a0a08a0e85dfe16d4d65c08b351dfbff1a1104d5e75e1be7565b4bbc6a583103bfc4b4075727133a54fa421983d894e549576364694b3e8910359b3de5260360bfe9f9bea2fec1cb50c2cf79a3fd590d-----
ZmYzMjM2ODljODQwMDRiMDI1MGU0NjcyMzhjYjJlMDNlNzg0OGI0YzQ1ZTM0ZjQz
YTZkZDVmNTBjYjAwMjAyNg==
-----END PRIVATE KEY for 1fce426b632e5a5941d9989e4f8bbb93a0a08a0e85dfe16d4d65c08b351dfbff1a1104d5e75e1be7565b4bbc6a583103bfc4b4075727133a54fa421983d894e549576364694b3e8910359b3de5260360bfe9f9bea2fec1cb50c2cf79a3fd590d-----
Staking operators responsible for creating the allValidatorsKeys.pem file used on the chain should merge all keys from their respective validatorKey.pem files using a text editor. The merged content should look similar to the example provided.
For the prefs.toml file, we can have definitions like:
[Preferences]# DestinationShardAsObserver represents the desired shard when running as observer# value will be given as string. For example: "0", "1", "15", "metachain"#if "disabled" is provided then the node will start in the corresponding shard for its public key or 0 otherwise DestinationShardAsObserver ="0"# NodeDisplayName represents the friendly name a user can pick for his node in the status monitor when the node does not run in multikey mode# In multikey mode, all bls keys not mentioned in NamedIdentity section will use this one as default NodeDisplayName ="s14"# Identity represents the GitHub identity when the node does not run in multikey mode# In multikey mode, all bls keys not mentioned in NamedIdentity section will use this one as default Identity =""# RedundancyLevel represents the level of redundancy used by the node (-1 = disabled, 0 = main instance (default), # 1 = first backup, 2 = second backup, etc.) RedundancyLevel =0# FullArchive, if enabled, will make the node able to respond to requests from past, old epochs.# It is highly recommended to enable this flag on an observer (not on a validator node) FullArchive =false# PreferredConnections holds an array containing valid ips or peer ids from nodes to connect with (in top of other connections)# Example:# PreferredConnections = [# "127.0.0.10",# "16Uiu2HAm6yvbp1oZ6zjnWsn9FdRqBSaQkbhELyaThuq48ybdorrr"# ] PreferredConnections = []# ConnectionWatcherType represents the type of the connection watcher needed.# possible options:# - "disabled" - no connection watching should be made# - "print" - new connection found will be printed in the log file ConnectionWatcherType ="disabled"# OverridableConfigTomlValues represents an array of items to be overloaded inside other configuration files, which can be helpful# so that certain config values need to remain the same during upgrades.# (for example, an Elasticsearch user wants external.toml->ElasticSearchConnector.Enabled to remain true all the time during upgrades, while the default # configuration of the node has the false value)# The Path indicates what value to change, while Value represents the new value in string format. The node operator must make sure# to follow the same type of the original value (ex: uint32: "37", float32: "37.0", bool: "true")# File represents the file name that holds the configuration. Currently, the supported files are: config.toml, external.toml, p2p.toml and enableEpochs.toml# -------------------------------# Un-comment and update the following section in order to enable config values overloading# -------------------------------# OverridableConfigTomlValues = [# { File = "config.toml", Path = "StoragePruning.NumEpochsToKeep", Value = "4" },# { File = "config.toml", Path = "MiniBlocksStorage.Cache.Name", Value = "MiniBlocksStorage" },# { File = "external.toml", Path = "ElasticSearchConnector.Enabled", Value = "true" }#]# BlockProcessingCutoff can be used to stop processing blocks at a certain round, nonce or epoch.# This can be useful for snapshotting different stuff and also for debugging purposes.[BlockProcessingCutoff]# If set to true, the node will stop at the given coordinate Enabled =false# Mode represents the cutoff mode. possible values: "pause" or "process-error".# "pause" mode will halt the processing at the block with the given coordinates. Useful for snapshots/analytics# "process-error" will return an error when processing the block with the given coordinates. Useful for debugging Mode ="pause"# CutoffTrigger represents the kind of coordinate to look after when cutting off the processing.# Possible values: "round", "nonce", or "epoch" CutoffTrigger ="round"# The minimum value of the cutoff. For example, if CutoffType is set to "round", and Value to 20, then the node will stop processing at round 20+ Value =0# NamedIdentity represents an identity that runs nodes on the multikey# There can be multiple identities set on the same node, each one of them having different bls keys, just by duplicating the NamedIdentity[[NamedIdentity]]# Identity represents the GitHub identity for the current NamedIdentity Identity ="testing-staking-provider"# NodeName represents the name that will be given to the names of the current identity NodeName ="tsp"# BLSKeys represents the BLS keys assigned to the current NamedIdentity BLSKeys = ["15eb03756fae81d2fbae392a4d7d82abdf7618ce3056b89376c2a46bc6e8403ed3cc84e12bc819c0b088ee46e7c28302d2b666b011714cc8ea2b75488907d07e194a6e83f0f3d15c7699de412de425314be5cc3ce6ab2c594690006f9915dd15","ff12bc7f471e2e375c6e8b981f13ed823dcca857c41a2ffc3a0956283a8428a95754375dabc0b412df3ec41d2a51ef1490a8d23f4e4f9348787f9615093e0129969085488b59d2ab550467cd0d0fa33df22e2ed2d8c8c0c0f59042dafd0c1098","3dec570c02a4444197c1ed53fefd7e57acb9bc99ae47db7661cfbfb47170418702162a46ed40e113e3381d68b713e903e286ffaf9cac77fed8f9c79e83f2abb0ccd690ef4f689607b6414a6f893e0c0ced93d7456240bbccbf223f7603dd8e05","38a93e3c00128c31769823710aa7deb145591b99a78c87dbd74c894afd540ade6de3906b45001d3f5a5882db34eaf30e412bef77ed43cf5a394edd0aa70254a74db1c80eef5d41342cae76fbbae596bc811fa491e00f16a7e011a836f7ceaa15","1fce426b632e5a5941d9989e4f8bbb93a0a08a0e85dfe16d4d65c08b351dfbff1a1104d5e75e1be7565b4bbc6a583103bfc4b4075727133a54fa421983d894e549576364694b3e8910359b3de5260360bfe9f9bea2fec1cb50c2cf79a3fd590d" ]
INFO
These 2 configuration files allValidatorsKeys.pem and prefs.toml should be copied on all n nodes that assemble the multikey group of nodes.
Do not forget to change the DestinationShardAsObserver accordingly for each node.
After starting the multikey nodes, within approximately 10 minutes, the explorer will reflect the changes. All nodes participating in the multikey group will broadcast their identity as an empty string, and their names will be depicted as s14. Conversely, the identities of the BLS keys will be named and identified as follows:
