Skip to content

Conversation

leandregagnonlewis
Copy link
Contributor

Enable Prometheus Recording Rule Backfilling for Testing

This PR enhances the Docker Compose setup for development and testing by adding support for pre-seeding Prometheus with recording rule data. This change:

  1. Modifies the Prometheus service in our docker-compose.yaml to use a custom entrypoint script
  2. Adds a backfilling mechanism that generates 5 hours of historical data for a test recording rule
  3. Re-enables a previously skipped Prometheus range query test that now has sufficient historical data to work with

Details

  • Created a custom entrypoint script (testdata/prometheus-entrypoint.sh) that:

    • Starts Prometheus initially to make its API available
    • Uses promtool tsdb create-blocks-from rules to backfill 5 hours of data for a simple test recording rule
    • Restarts Prometheus to ensure the backfilled data is immediately available
  • Added a simple recording rule definition in testdata/prometheus-seed.yml that creates a metric called test with a constant value of 1

  • Updated our test to use the new test metric instead of up, removing the skip annotation since we now have predictable data to test against

This change will make our tests more reliable by ensuring consistent test data is available, regardless of how long the Prometheus instance has been running.

@leandregagnonlewis leandregagnonlewis requested a review from a team as a code owner May 23, 2025 18:22
Copy link
Contributor

@csmarchbanks csmarchbanks left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Love the solution, thanks!

@csmarchbanks csmarchbanks merged commit dd2811a into grafana:main May 28, 2025
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants