Better parachain liveness and finality metrics

[sandreim]

To help us further make decision on what are the most impactful areas to invest in for improving parachain throughput and latency we need to make some measurements. Some of the metrics required should already exist so what is needed is just some dashboard, but other stuff needs to be implemented as well.

High level view of what is the current impact on parachains:

• Produced parachain blocks
• Finalized parachain blocks
• Finalized to produced ratio

To further drill down and understand why, we also need more parachain consensus observability.

Prospective parachains candidate drop reasons:

• Parachain produced forks
• Candidate’s relay parent went out of scope before it could get backed on chain ( Availability of a previously backed candidate took more than one block, Session change? Invalidate a complete prospective chain; Relay chain block was full ? Block producer did not put the candidate in for unknown reasons?)

Other metrics of interest:

• collator connectivity to backing group (how fast we connect to validators and to how many we connect/error)
• Statement distribution: candidate’s relay parent went out of scope before it was successfully backed by enough backers.
• Availability: availability timed out and thus a candidate was dropped.
• Relay chain reorgs
• Collator protocol: the candidate could not be transmitted to backers; throttling (fairness)
• Backing dropped invalid candidates

[sandreim]

On the collator side I've added a few new metrics and logs in #8230 to be able to identify relay chain blocks when collators miss slots:

• time till collation fetched
• backing latency (counting from RP)
• backing latency (counting from collation fetch)
• inclusion latency
• expired collations (not backed, not advertised, not fetched)

I've been running these metrics on Kusama People chain. So far I identified 2 situations when we see blocks being skipped that reflect in the newly added backing latency metrics.

High collation fetch latency

Collation fetch times of up more than 5s have been observed, see here: https://grafana.teleport.parity.io/goto/ak1bvx1HR?orgId=1 .

tokio-runtime-worker parachain::collator-protocol-stats: [Relaychain] Collation fetch latency is 5696ms para_head=0x7a7f5b0bbf6d18baad11fe7b953f7ca2f28f17a2c72085bffff32a64a8d8f08c our_para_id=1004

The issue seems to be that connect to validators took a long time. A collation is advertised to one of the after almost 6s.

Also saw some libp2p issues: https://grafana.teleport.parity.io/goto/BAb1vbJHR?orgId=1

Collation backed 2 blocks after it was seconded.

Logs: https://grafana.teleport.parity.io/goto/ta01WzxHR?orgId=1
Metrics: https://grafana.teleport.parity.io/goto/JexS7kbHR?orgId=1

Candidate 0xf8a788f4d902f705e7c5604b8b395e4df21d5931a1e85bda95fda53597085465 is backed 2 blocks after being seconded.

You can see in the chart below, there are 2 instances when this happens, fetched to backed latency goes > 10s, and also backing latency measured from RP to backed block goes to 2.

[AndreiEres]

We can observe collators connectivity (quantity) using desired_peer_count and peer_count for collator protocol.

Added it to the dashboard with Collator Performance Metrics