need two-phase commit of shard block

[qizhou]

We currently impose the following assumption:

• If the master has a validated shard block header, then the shard block is persisted in slave's db

This assumption holds when the cluster is running properly. However, it can be broken if

• the slave's machine/container is crashed so that even the block is put, it may not be persisted as rocksdb's fsync is disabled by default rocksdb may lose data after shutting down the machine/container directly #486
• the slave's may persist the block, but the cluster is shut down before the block header is added to master. Therefore, after restarting the cluster, the slave assumes the block is validated in the cluster, but actually is not

The second issue may be more common. A fix is to introduce two-phase commitment. In general, there are two fixes

• write phase 1 information that a block is not commit
• after the shard block is broadcasted to all slaves and master, the block is committed

Additionally, after restarting the cluster, the block that is not committed will be scanned and recovered before the cluster is active

[qizhou]

Another fix is when a shard block is added to local

• check if it is committed, and do nothing if committed
• broadcast to all slaves
• send to master
• mark the block is committed

This may broadcast the block to slaves and send to master multiple times, which is OK.

[ninjaahhh]

as discussed offline, another potentially easier fix would be

1. after cluster reboot, master server will scan a certain number of root blocks
2. delete root blocks which have inconsistent states (ie missing shard blocks), which will force the cluster to re-download from peers

[ninjaahhh]

however this only fixes the inconsistent states in the canonical root chain. if we have another inconsistent state in a fork, during recovery we may not find it. then the shit will hit the fan when a root block of another fork is added.

solution:

1. as mentioned in my comment here, use memory to keep the canonical chain and force the master to re-download all other forks
2. or add aforementioned additional commit stage

[qizhou]

fixed by #493