Intermittent Ouroboros/ChainDB error when syncing on Testnet

[oneEdoubleD]

Issue:

Testnet nodes are intermittently crashing while syncing. When testing four nodes they crashed a total of 7 times in under an hour. There were no indications of memory issues in the process explorer while running these nodes. The error was:

cardano-node: ExceptionInLinkedThread "ThreadId 80" (ClosedDBError [("getEnvSTM",SrcLoc {srcLocPackage = "ouroboros-consensus-0.1.0.0-R0AVfeY78d7fCNQvHwSpA", srcLocModule = "Ouroboros.Storage.ChainDB.Impl", srcLocFile = "src/Ouroboros/Storage/ChainDB/Impl.hs", srcLocStartLine = 169, srcLocStartCol = 34, srcLocEndLine = 169, srcLocEndCol = 64})])

While most of the testing was completed on jlotoski/cardano-node-issue-461 I have also confirmed it occurs on Master.

This is a fatal error that crashes the node. Notably, this does not occur on Staging.

Steps to recreate:

• Clone a node (or four, to speed up the process of having one crash as it's seemingly random)
• Build with nix-build -A scripts.testnet.node
• Run nodes

Edit:

To run nodes, run ./result in the nodes directory.

[nfrisby]

Thanks! I think I've been investigating similar exceptions in the ouroboros-network test suite and am optimistic that I'll be able to address this relatively soon and simply.

[mrBliss]

Linking to IntersectMBO/ouroboros-network#1470

[oneEdoubleD]

I have completely synced several nodes last Friday, and @ArturWieczorek ran 6 nodes for several hours without seeing this issue, so I think it's safe to say it is fixed.