Skip to content

Conversation

@qryxip
Copy link
Member

@qryxip qryxip commented Oct 10, 2025

内容

#1153 で用意したベンチのうちopen_and_close_vvmについて、これだけCodSpeed用の計測では凄まじく(具体的には±25%くらい)ブレが生じていた。そのためCodSpeedでの継続的な計測は不可能と判断し、デフォルトでignoreされるようにしてCodSpeedの対象から外す。

関連 Issue

その他

[追記] このPRをマージし次第、open_and_close_vvm"archive"する。

@codspeed-hq
Copy link

codspeed-hq bot commented Oct 10, 2025

CodSpeed Performance Report

Merging #1169 will degrade performances by 11.05%

Comparing qryxip:pr/test-codspeed-fixup-pr1153-ignore-open-and-close-vvm (204b08c) with main (abef143)1

⚠️ Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

Summary

❌ 2 regressions
✅ 12 untouched
⏩ 10 skipped2

⚠️ Please fix the performance issues or acknowledge them on CodSpeed.

Benchmarks breakdown

Benchmark BASE HEAD Change
unload_and_load_vvm 1.7 s 1.9 s -10.71%
unload_and_load_vvm 1.9 s 2.1 s -11.05%

Footnotes

  1. No successful run was found on main (148dc75) during the generation of this report, so abef143 was used instead as the comparison base. There might be some changes unrelated to this pull request in this report.

  2. 10 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports.

@qryxip
Copy link
Member Author

qryxip commented Oct 10, 2025

うおー君もか、unload_and_load_vvm…。

image

@Hiroshiba Hiroshiba requested a review from Copilot October 11, 2025 02:23
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR addresses instability issues with the open_and_close_vvm benchmark test that was causing significant variance (±25%) in CodSpeed measurements. The change makes this specific benchmark ignored by default in CodSpeed continuous measurements while preserving the ability to run it manually when needed.

  • Adds configuration option to control inclusion of the open_and_close_vvm benchmark
  • Sets the benchmark to be ignored by default to exclude it from CodSpeed measurements
  • Updates documentation to reflect the new configuration option

Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.

Copy link
Member

@Hiroshiba Hiroshiba left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!!

Copy link
Member

@sevenc-nanashi sevenc-nanashi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

遅れました、まぁ問題なさそう。

マージはお任せします。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants