adding flushall after memtier benchmark failure. #320
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…se may be in a corrupted state for the next test
slice4e
changed the title
fcostaoliveira
approved these changes
Oct 15, 2025
slice4e
added a commit
to slice4e/redis-benchmarks-specification
that referenced
this pull request
Oct 16, 2025
Apply black formatting to runner.py for FLUSHALL changes from PR redis#320 to comply with CI code style checks.
fcostaoliveira
pushed a commit
that referenced
this pull request
Oct 27, 2025
* fix(tests): Use architecture-specific streams in coordinator tests Fix 7 failing tests in test_self_contained_coordinator_memtier.py by: - Adding messages to arch-specific streams instead of base stream - Fixing consumer group creation parameters (arch and id) - Updating assertions to check arch-specific streams This aligns tests with the arch-specific stream routing implemented in the coordinator, which reads from streams like: - oss:api:gh/redis/redis/builds:amd64 (for amd64) - oss:api:gh/redis/redis/builds:arm64 (for arm64) Fixes: - test_self_contained_coordinator_dockerhub_preload - test_self_contained_coordinator_dockerhub - test_self_contained_coordinator_dockerhub_iothreads - test_self_contained_coordinator_dockerhub_valkey - test_dockerhub_via_cli - test_dockerhub_via_cli_airgap - test_self_contained_coordinator_duplicated_ts * style: Format test file with black Apply black formatting to test_self_contained_coordinator_memtier.py to comply with CI code style checks. * style: Format runner.py with black Apply black formatting to runner.py for FLUSHALL changes from PR #320 to comply with CI code style checks. * fix(runner): Skip validation for untested operation types Improve metric validation to skip operation-specific metrics when those operations are not in the tested-commands list. For example, a SET-only test (--ratio 1:0) will now skip validation of Gets.Ops/sec metric, which would legitimately be 0. This fixes CI test failures where SET-only tests were failing validation because Gets.Ops/sec was 0 (below the 10 QPS threshold). The validation now: 1. Checks benchmark_config for 'tested-commands' list 2. Skips metrics for operation types (gets, sets, hgets, etc.) that are not in the tested-commands list 3. Still validates metrics for operations that are actually being tested * fix(tests): Add git_version to benchmark stream requests Add git_version parameter when calling generate_benchmark_stream_request in tests. This ensures version information is propagated through the stream to the coordinator, allowing by.version keys to be created in Redis. Without this, the tests expect by.version keys to exist but they were never created because artifact_version was None in the data export logic. Fixes test assertions that check for: - ci.benchmarks.redis/.../by.version/{version}/benchmark_end/.../memory_maxmemory Tests fixed: - test_self_contained_coordinator_dockerhub_preload - test_self_contained_coordinator_dockerhub - test_self_contained_coordinator_dockerhub_iothreads - test_self_contained_coordinator_dockerhub_valkey - test_self_contained_coordinator_duplicated_ts * fix(cli): Extract git_version from Docker image tag - Add regex-based version extraction from image names like 'redis:7.4.0' or 'valkey/valkey:7.2.6-bookworm' - Pass extracted git_version to generate_benchmark_stream_request() - Enables by.version Redis TimeSeries keys to be created for CLI-triggered tests - Fixes test_dockerhub_via_cli and test_dockerhub_via_cli_airgap assertion failures * style: Format cli.py with black Apply black formatting to comply with CI code style checks. * debug: Add logging to diagnose artifact_version issue * debug: Add print statements to track artifact_version value * style: Format runner.py with black * fix: Include running_platform in by.version key paths The export_redis_metrics function creates keys with the format: {prefix}/{test}/{by_variant}/benchmark_end/{running_platform}/{setup}/{metric} But tests were expecting keys without running_platform: {prefix}/{test}/{by_variant}/benchmark_end/{setup}/{metric} This fixes all 6 failing tests by adding running_platform to the expected key format. Also removes debug print statements that helped diagnose the issue. * fix: Add JSON module support and correct test metadata This commit fixes stats validation errors: 1. Add JSON module commands to commands.json: - JSON.GET: Return value at path in JSON serialized form - JSON.SET: Sets or updates JSON value at a path 2. Add 'json' group to groups.json with JSON commands 3. Fix test-suite metadata issues: - memtier_benchmark-playbook-rate-limiting-lua-100k-sessions.yml: Remove 'bitmap' from tested-groups (only scripting is tested via EVAL) - memtier_benchmark-playbook-realtime-analytics-membership.yml: Add 'sunion' to tested-commands (SUNION command was used but not listed) - memtier_benchmark-playbook-realtime-analytics-membership-pipeline-10.yml: Add 'sunion' to tested-commands These changes resolve all stats validation errors in CI. * fix missing quote from test case memtier_benchmark-1Mkeys-string-set-with-ex-100B-pipeline-10.yml
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Otherwise what may happen is that a test fails (after a timeout) but some data has been inserted into redis. The following test will find the database to contain unexpected data and fail.