[Feature Request] Modification of Mining Module

[gzelda]

Is your feature request related to a problem? Please describe.

Right now the official code for a Stacks 2.0 node does not readily expose information/data/API that is both necessary and desired by STX Miners. This extensively limits the operations an individual miner can easily execute, as well as the ability for third parties to provide STX Mining solutions that make it easy for anyone to participate. The two biggest issues are described in detail below:

The operating mechanism of existing nodes is to read configuration information from the configuration file of .toml, such as whether to mine, miner address, etc. This method of statically reading files limits the operations related to mining. For example, starting and stopping mining are currently controlled by opening and closing the node program, and the node controller cannot change the behavior during the node operation. (For example, only normal node status at the beginning of the node, and then perform mining operation later).

Specifically, the two missing features that are of highest priority are:

• 1. A Stacks node can start or stop mining at any time while it is running. When mining is stopped, the node is just running as a normal node, synchronizing network progress.
• 2. The node can change the mining-related parameters (e.g. amount of BTC) at any time while it is running.

Describe the solution you'd like

Daemon Technologies has, up until now, been modifying the official Stacks Blockchain code in order to facilitate the above. These changes, however, are unofficial/not part of master, and therefore need to be redone every time Blockstack PBC changes the official code. Given that the changes we propose will be necessary, and large value add for any and all STX Miners, in addition to data that will be important for other products such as the Explorer, Daemon Technologies would like to propose working with Blockstack PBC to merge a solution to the issues described above to the master/official Stacks Blockchain code.

The initial solution Daemon Technologies is proposing to the issues above is as follows:
Since the pre-run check and runtime state changes of existing nodes are both in run_loop, the following points need to be adjusted at this stage:

• 1. For the first issue, we are looking at Ethereum as a good example of how running a node and mining can interact. Refer to Geth, firstly initialize the miner-related component instance in run_loop module (for example, develop a package called miner responsible for mining-related processing), and then wait for the call of the miner instance related method (Here I want to design Miner-related RPC to achieve), then miner.start() miner.stop() and the method of configuring miner related parameters (miner btc private key) can be realized through RPC method.
• 2. Regarding the issue of dynamically modifying the mining-related parameters (the amount of BTC), we cannot refer to other frameworks, because this is a unique mechanism of Blockstack, the time interval for stacks-blockchain to select a new leader (the same amount as the Bitcoin network block generation Level) is long enough for users to operate, so in this process we want to modify some member variables in the miner class, mentioned above, through the RPC method (the number of bitcoins burned at this stage is a fixed value, written in a variable called burn_fee_cap , Please see link for details, so that you can modify the number of burned BTC to participate in the next round of elections.

Describe alternatives you've considered

If Blockstack PBC is not willing to work with Daemon Technologies to merge these features to master at this stage(maybe because of time/energy/priority), Daemon plans to continue it's development, but with a delayed timeline. If these features are not offered in the official Stacks chain code, Daemon will need to take more time to develop and release the STX Mining bot. If these changes are merged to master and officially offered, we believe we can release a fully usable version of the mining bot by mainnet launch. If not, we anticipate the official release of the mining bot occuring 1-2 months post mainnet launch.

Additional context

[xanbots]

Thanks Gavin! Just to echo the above, we think some of the changes we're making a lot of sense to be offered officially by Blockstack PBC. The first we're in the process of working on, and the second we've created one possible solution for.

Appreciate the consideration @muneeb-ali @diwakergupta @kantai @jcnelson

[diwakergupta]

Thanks @tyGavinZJU @xanbots -- please do send PRs! Some relevant issues that add context on prior discussions about what functionality belongs in stacks-blockchain proper vs. mining software: #1451 , #1441 , #1420

[jcnelson]

Hey @tyGavinZJU @xanbots, appreciate your input. This is something that @kantai, @lgalabru, and myself have discussed in the past (as early as April), and we have a pretty good idea about what mining is going to look like in the future. If you want to help us with this, there's a specific architecture we had in mind that we'd like you to use.

The biggest difference between how the node behaves right now and how the node will be have in the future is that most mining activity will occur in a separate program (#1441). The mining component was never intended to be part of the Stacks node. It's only there now because it was the fastest way to enable mining on the testnet.

What we intend to do -- and would accept PRs for -- is to split the mining component out of the Stacks node to create a separate mining program, and expand the RESTful API to make it possible for the mining program to generate and broadcast Stacks blocks and Bitcoin transactions.

A Stacks mining program would need to do the following:

• Manage the user's mining keychains -- one for sending Bitcoin transactions, one for generating VRF proofs, and one for signing Stacks microblocks.
• Generate and broadcast the LeaderKeyRegister and LeaderBlockCommit Bitcoin transactions to both register a VRF key for the miner and commit blocks to the chain. This would include fetching and signing UTXOs from a Bitcoin node in order to form these transactions.
• Fetch transactions from the Stacks node's mempool to package into an anchored Stacks block (whose hash will be written to the Bitcoin chain in a LeaderBlockCommit transaction).
• Fetch transactions from the Stacks node's mempool to package into a microblock stream.
• Upload Stacks blocks and microblocks to the Stacks node for broadcasting.

All of this currently happens within the Stacks testnet software, and could be split out into a separate Rust binary. However, the Stacks node still needs the following RESTful API endpoints at a minimum to make this happen:

• An API endpoint for generating a block template (Add mine_block RPC endpoint #1420). This currently happens in the mining component as part of building a Stacks block, but would need to be exposed as a (privileged) API endpoint. The API endpoint would return a Stacks block without the VRF proof or microblock public key hash set. The miner program would need to fill these fields in as part of generating a LeaderBlockCommit.
• An API endpoint for generating a microblock template. This is similar to the above -- it just returns an unsigned microblock.
• An API endpoint for uploading a signed Stacks block (Add RPC endpoint for uploading a mined block #1451). The API endpoint would need to ensure the block gets scheduled to be broadcasted and processed.
• An API endpoint for uploading a signed Stacks microblock. The API endpoint would need to ensure that the microblock gets scheduled to be broadcasted and processed.

[gzelda]

Hey @tyGavinZJU @xanbots, appreciate your input. This is something that @kantai, @lgalabru, and myself have discussed in the past (as early as April), and we have a pretty good idea about what mining is going to look like in the future. If you want to help us with this, there's a specific architecture we had in mind that we'd like you to use.

The biggest difference between how the node behaves right now and how the node will be have in the future is that most mining activity will occur in a separate program (#1441). The mining component was never intended to be part of the Stacks node. It's only there now because it was the fastest way to enable mining on the testnet.

What we intend to do -- and would accept PRs for -- is to split the mining component out of the Stacks node to create a separate mining program, and expand the RESTful API to make it possible for the mining program to generate and broadcast Stacks blocks and Bitcoin transactions.

A Stacks mining program would need to do the following:

• Manage the user's mining keychains -- one for sending Bitcoin transactions, one for generating VRF proofs, and one for signing Stacks microblocks.
• Generate and broadcast the LeaderKeyRegister and LeaderBlockCommit Bitcoin transactions to both register a VRF key for the miner and commit blocks to the chain. This would include fetching and signing UTXOs from a Bitcoin node in order to form these transactions.
• Fetch transactions from the Stacks node's mempool to package into an anchored Stacks block (whose hash will be written to the Bitcoin chain in a LeaderBlockCommit transaction).
• Fetch transactions from the Stacks node's mempool to package into a microblock stream.
• Upload Stacks blocks and microblocks to the Stacks node for broadcasting.

All of this currently happens within the Stacks testnet software, and could be split out into a separate Rust binary. However, the Stacks node still needs the following RESTful API endpoints at a minimum to make this happen:

• An API endpoint for generating a block template (Add mine_block RPC endpoint #1420). This currently happens in the mining component as part of building a Stacks block, but would need to be exposed as a (privileged) API endpoint. The API endpoint would return a Stacks block without the VRF proof or microblock public key hash set. The miner program would need to fill these fields in as part of generating a LeaderBlockCommit.
• An API endpoint for generating a microblock template. This is similar to the above -- it just returns an unsigned microblock.
• An API endpoint for uploading a signed Stacks block (Add RPC endpoint for uploading a mined block #1451). The API endpoint would need to ensure the block gets scheduled to be broadcasted and processed.
• An API endpoint for uploading a signed Stacks microblock. The API endpoint would need to ensure that the microblock gets scheduled to be broadcasted and processed.

A quick question, does the mining-program contain a stacks-node for sync blocks and broadcast transactions?

I guess the answer is NO, because from your description that the mining-program is just like a signing machine which process certain transaction of Stacks and Bitcoin. But in reality, each miner should has his own node to broadcast new transactions like LeaderKeyRegister and LeaderBlockCommit. Other nodes could refuse the outer transaction for some reason(maybe evil).

[psq]

Synchronizing your own regtest bitcoind locally with the one krypton uses is easy:

bitcoin.conf

server=1
regtest=1
rpcallowip=0.0.0.0/0
rpcallowip=::/0
rpcuser=blockstack
rpcpassword=blockstacksystem
txindex=1
listen=1
debug=1
rpcserialversion=0
datadir=/absolute/path/to/data/bitcoin-psq2

[regtest]
bind=0.0.0.0:19444
rpcbind=0.0.0.0:19443
addnode=bitcoind.krypton.blockstack.org:18444

Start bitcoind

bitcoind -conf=/absolute/path/to/bitcoin.conf &

and to query with bitcoin-cli:

bitcoin-cli -regtest -conf=/absolute/path/to/bitcoin.conf -rpcport=19443 getblockchaininfo

not sure why it can't pick up the rpcport from the conf file, but whatever network operations are whitelisted are sufficient for the node to synchronize, and now you have access to all rpc calls.

[psq]

Here are are some thoughts on how I would approach mining, and incrementally splitting some of the mining responsibilities.

First of all, the mining process needs to be driven by the node, which reacts to other nodes when it receives new btc blocks, new stacks blocks and new transactions. So when considering an RPC API, the miner is the server, and the node a client to the miner, in the same fashion the stacks-blockchain-api node is a server and the stacks-blockchain node a client.

Secondly, the more granular you make the api, the more serialization/deserialization cost you'll be incurring, slowing down the process, and making it more complex (and also more brittle), so you need to keep the exchanges to a minimum (when speed may matter).

I started writing this thinking you'd split the current mining code into smaller chunks, the node sending data needed for each basic operation (register key, generate proof, build block, sign and commit, new block received, new transaction received, ...), but it turns out building a block outside of the node is not really feasible without pulling most of the node code (Clarity VM, including its state for example), so here's a more nimble proposal.

The node keeps managing the Leader Key, and assembles the block, and once the block is assembled, calls the miner to let it decide whether it wants to mine the block (based on the current chain tip, the tx count, the tx fees, and a few other parameters) and how much it wants to transfer to stackers, and what fee to pay to the btc network, or the miner can decide to skip this block. The node will call the miner each time it detects a new chain tip, which can be more than once per epoch (btc block), but gives the miner the best chance of mining a block on the best known fork for the current epoch as the node learns about it.

This should be fairly simple and only a few modifications to the current node code, and a few extra configuration parameters (miner's address and port may be sufficient), and can be modeled on the event dispatcher (or even be an extension of it with a few more events).

A second version could let the miner pick which transactions to include, and in what order (a first call the node would make to the miner), then let the node assemble the block, and again, the node would call the miner to let it decide once the block has been assembled, whether to commit to it or not. The miner could already reuse the existing events used to get new transactions and blocks so it could manage its own mempool and does not need to get a copy of the mempool for each block).

This does not take microblocks into accounts, but once this has been enabled, the miner would also need to decide which transaction to mine in a microblock, and when to stop mining microblocks for the current epoch, some fairly simple addition to the protocol.

[diwakergupta]

This should be fairly simple and only a few modifications to the current node code, and a few extra configuration parameters (miner's address and port may be sufficient), and can be modeled on the event dispatcher (or even be an extension of it with a few more events).

Interesting idea @psq ! There's some elegance in re-using the event dispatcher pattern to "push" updates to the miner software. In this model, would the miner program would still make API calls into the node to upload the block etc (basically the set of endpoints in Jude's earlier comment)? I think a quick diagram (whiteboard picture would also work) might help clarify what you have in mind.

@blockstack/blockchain-team please chime in with your thoughts! 🙏🏽

[jcnelson]

All you need to do to get the node to produce a block template is add a RPC endpoint that takes a VRF proof, a microblock public key hash, and a list of txids to mine as input, and spits out an unsigned Stacks block that includes those transactions in the order given by the txids (or, returns an error of the node is missing one or more of those transactions or if the transactions cannot be applied in the given order). Internally, the node would use a StacksChainState instance to create an anchored block (i.e. by instantiating a StacksBlockBuilder, loading the referenced transactions out of the mempool, and calling make_anchored_block_from_txs()). This is what #1420 is.

There's absolutely no need at all to involve the event-dispatcher. It's the mining program's job to identify transactions for inclusion, not the node's. Moreover, the node shouldn't be producing blocks that the mining program didn't ask for, and it shouldn't be polluting the event stream (to which the side-car is attached) with data that it didn't want in the first place. Making a block template is expensive and time-consuming, and should only be done at the request of the miner (implying that this RPC endpoint should be privileged, and/or accessible only from localhost). The mining program, not the node, should be driving block template generation. This is how most mining software for existing blockchains work -- the node makes the block template at the miner's request, and the miner fills in the template with any missing information to finish mining the block (in our case, the VRF proof and the block-commit transaction).

Let's not over-complicate the design.