Skip to content

compact: implement native histogram downsampling #8110

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 3 commits into from
Mar 3, 2025
Merged

compact: implement native histogram downsampling #8110

merged 3 commits into from
Mar 3, 2025

Conversation

GiedriusS
Copy link
Member

@GiedriusS GiedriusS commented Feb 17, 2025 •
edited
Loading

Update #6350:

  • Add some tests from me
  • Update to the newest API changes
  • Refactored downsampling loop to be clearer like I've commented
  • Updated Sebastian's tests to have proper "sum"
  • Unify "sum" in the histogram aggrchunks to be like in the float aggrchunks
  • Added a blurb to compactor's documentation
  • Added sample e2e test

Closes #5907.
Closes #6350.

@GiedriusS GiedriusS force-pushed the implement_nativehistogram_downsampling branch 2 times, most recently from c39432a to 73ed78c Compare February 17, 2025 16:03
@GiedriusS
downsample: port other PR
e4e7fa2 
Signed-off-by: Giedrius Statkevičius <[email protected]>
@GiedriusS GiedriusS force-pushed the implement_nativehistogram_downsampling branch from 73ed78c to e4e7fa2 Compare February 17, 2025 16:11
@GiedriusS GiedriusS marked this pull request as ready for review February 18, 2025 11:48
@GiedriusS GiedriusS mentioned this pull request Feb 19, 2025
Merged
yeya24
yeya24 previously approved these changes Feb 21, 2025
View reviewed changes
Copy link
Contributor

@yeya24 yeya24 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. This is epic. Code LGTM. I have 2 questions.

  1. Since we don't aggregate on min and max, does it work with the existing iterator we use in Querier?
  2. Do we need to update ApplyCounterResetIterator to support native histograms?

pkg/compact/downsample/downsample.go Outdated
for _, c := range chks {
if prevEnc != c.Chunk.Encoding() {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If previous encoding is float histogram and current encoding is normal histogram, can we downsample those chunks together since the input is same, float histogram?

@GiedriusS
Copy link
Member Author

GiedriusS commented Mar 3, 2025 •
edited
Loading

(1) min/max are undefined on native histograms so they don't return anything either way: https://github.com/thanos-io/promql-engine/blob/accbf0891a16796f1c1f1ec759d7c60191a4569b/ringbuffer/functions.go#L608-L616.

(2) my understanding is that counter reset hints are not used at the moment because it's unclear how to merge them: prometheus/prometheus#15346. Once this issue is solved, we can try to adjust downsampling.

Will update the PR according to your comments. Aaron tried this branch and sent me a block on
Slack, which supposedly triggers some bugs. I cannot reproduce the errors with that block, waiting for further instructions from Aaron.

@vCra
Copy link
Contributor

vCra commented Mar 3, 2025

I cannot reproduce the errors with that block, waiting for further instructions from Aaron.

This was my mistake - I only updated compactor but not query etc.
Downsampling seemed to work but querying downsampled blocks didn't (I got errors such as Error executing query: sum and count timestamps not aligned or no results).

Actually updating query to use the same image version works well.
Screenshot 2025-03-03 at 10 51 08
Screenshot 2025-03-03 at 10 50 54

@GiedriusS
compact: fix after review
24c3575 
Signed-off-by: Giedrius Statkevičius <[email protected]>
yeya24
yeya24 approved these changes Mar 3, 2025
View reviewed changes
Copy link
Contributor

@yeya24 yeya24 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks LGTM! I think this is good to go

@GiedriusS
Copy link
Member Author

Let's try. I will fix any issues.

@GiedriusS GiedriusS merged commit 03c96d0 into main Mar 3, 2025
22 checks passed
@ahurtaud ahurtaud mentioned this pull request Jun 23, 2025
Closed
openshift-merge-bot bot pushed a commit to stolostron/thanos that referenced this pull request Aug 20, 2025
@acm-observability-bot @didukh86
@MichaHoffmann @dependabot @milinddethe15 @GiedriusS @philipgough @fpetkovski @bluesky6529 @saswatamcode @jnyi @SungJin1212 @yeya24 @rvichery @pedro-stanaka @abelsimonn @coleenquadros @Juneezee @harry671003 @cswpy @rofc @alanprot @dsabsay @nicolastakashi @matej-g @celian-garcia @psliwka @Saumya40-codes @martafolf @miinsun @lachruzam @SuperQ @naohiroo @vCra @yuchen-db @adrmaas @dongjiang1989 @machine424 @dronenb @siavashs @anna-tran @Naman-B-Parlecha @danielblando @gabyfulchic @GregSharpe1 @hczhu @hczhu-db @verejoel @github-actions
[ACM Obs bot] Bump stolostron/thanos to v0.39.1 (#253)
3242420 
* Query-frontend: Fix connection to Redis with TLS.
Issue: https://github.com/thanos-io/thanos/issues/7672

Signed-off-by: didukh86 <[email protected]>
Signed-off-by: didukh86 <[email protected]>
Signed-off-by: didukh86 <[email protected]>
Signed-off-by: Taras Didukh <[email protected]>

* Add record to the  changelog

Signed-off-by: Taras Didukh <[email protected]>

* query: queryable is not respecting limits (#7679)

Signed-off-by: Michael Hoffmann <[email protected]>

* build(deps): bump go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp (#7666)

Bumps [go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp](https://github.com/open-telemetry/opentelemetry-go) from 1.27.0 to 1.29.0.
- [Release notes](https://github.com/open-telemetry/opentelemetry-go/releases)
- [Changelog](https://github.com/open-telemetry/opentelemetry-go/blob/main/CHANGELOG.md)
- [Commits](https://github.com/open-telemetry/opentelemetry-go/compare/v1.27.0...v1.29.0)

---
updated-dependencies:
- dependency-name: go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Signed-off-by: Taras Didukh <[email protected]>

* website: max-height for version-picker dropdown (#7642)

Signed-off-by: milinddethe15 <[email protected]>
Signed-off-by: Taras Didukh <[email protected]>

* build(deps): bump github/codeql-action from 3.26.2 to 3.26.5 (#7667)

Bumps [github/codeql-action](https://github.com/github/codeql-action) from 3.26.2 to 3.26.5.
- [Release notes](https://github.com/github/codeql-action/releases)
- [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md)
- [Commits](https://github.com/github/codeql-action/compare/429e1977040da7a23b6822b13c129cd1ba93dbb2...2c779ab0d087cd7fe7b826087247c2c81f27bfa6)

---
updated-dependencies:
- dependency-name: github/codeql-action
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Signed-off-by: Taras Didukh <[email protected]>

* Bump golang.org/x/time from 0.5.0 to 0.6.0 (#7601)

Bumps [golang.org/x/time](https://github.com/golang/time) from 0.5.0 to 0.6.0.
- [Commits](https://github.com/golang/time/compare/v0.5.0...v0.6.0)

---
updated-dependencies:
- dependency-name: golang.org/x/time
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Signed-off-by: Taras Didukh <[email protected]>

* Remove empty lines

Signed-off-by: Taras Didukh <[email protected]>

* receive: port expanded postings cache from Cortex (#7914)

Port expanded postings cache from Cortex. Huge kudos to @alanprot for
the implementation. I added a TODO item to convert our whole internal
caching infra to be promise based.

Signed-off-by: Giedrius Statkevičius <[email protected]>

* receive/expandedpostingscache: fix race (#7937)

Porting https://github.com/cortexproject/cortex/pull/6369 to our code
base. Add test that fails without the fix.

Signed-off-by: Giedrius Statkevičius <[email protected]>

* receive: Allow specifying a custom gRPC service config via flag (#7907)

Signed-off-by: Philip Gough <[email protected]>

* remove redundant redis config (#7942)

Signed-off-by: Helen Tseng <[email protected]>

* Capnp: Use segment from existing message (#7945)

* Capnp: Use segment from existing message

Signed-off-by: Filip Petkovski <[email protected]>

* Downgrade capnproto

Signed-off-by: Filip Petkovski <[email protected]>

---------

Signed-off-by: Filip Petkovski <[email protected]>

* Fix ver

Signed-off-by: Saswata Mukherjee <[email protected]>

* [Receive] Fix race condition when adding multiple new tenants at once (#7941)

* [Receive] fix race condition

Signed-off-by: Yi Jin <[email protected]>

* add a change log

Signed-off-by: Yi Jin <[email protected]>

* memorize tsdb local clients without race condition

Signed-off-by: Yi Jin <[email protected]>

* fix data race in testing with some concurrent safe helper functions

Signed-off-by: Yi Jin <[email protected]>

* address comments

Signed-off-by: Yi Jin <[email protected]>

---------

Signed-off-by: Yi Jin <[email protected]>

* Update promql-engine for subquery fix (#7953)

Signed-off-by: Saswata Mukherjee <[email protected]>

* Sidecar: Ensure limit param is positive for compatibility with older Prometheus (#7954)

Signed-off-by: Saswata Mukherjee <[email protected]>

* Fix potential deadlock in hedging request (#7962)

Signed-off-by: kade.lee <[email protected]>

* Mark posting group lazy if it has a lot of keys (#7961)

* mark posting group lazy if it has a lot of add keys

Signed-off-by: Ben Ye <[email protected]>

* update docs

Signed-off-by: Ben Ye <[email protected]>

* rename labels

Signed-off-by: Ben Ye <[email protected]>

* changelog

Signed-off-by: Ben Ye <[email protected]>

* change to use max key series ratio

Signed-off-by: Ben Ye <[email protected]>

* update docs

Signed-off-by: Ben Ye <[email protected]>

* mention metrics

Signed-off-by: Ben Ye <[email protected]>

* update docs

Signed-off-by: Ben Ye <[email protected]>

---------

Signed-off-by: Ben Ye <[email protected]>

* sidecar: fix limit mintime (#7970)

Signed-off-by: Michael Hoffmann <[email protected]>

* store gateway: fix merge fetched postings with lazy postings (#7979)

Signed-off-by: Ben Ye <[email protected]>

* api: bump promql engine and fix fallout (#8000)

* bump to new promql-engine version and fix fallout
* new promql-engine makes it possible to provide more options at runtime

Signed-off-by: Michael Hoffmann <[email protected]>
Co-authored-by: Michael Hoffmann <[email protected]>

* receive: fix maxBufferedResponses channel size to avoid deadlock (#7978)

* Fix maxBufferedResponses channel size to avoid deadlock

Fixes #7977

Signed-off-by: Remi Vichery <[email protected]>

* Add changelog entry

Signed-off-by: Remi Vichery <[email protected]>

* adjust line numbers in docs/components/receive.md to match updated code

Signed-off-by: Remi Vichery <[email protected]>

---------

Signed-off-by: Remi Vichery <[email protected]>

* .github: run e2e tests on newer ubuntu

I cannot reproduce chromedp panics locally so trying to see if a newer
Ubuntu version would help.

Signed-off-by: Giedrius Statkevičius <[email protected]>

* Forcing headless and making sure we can run it before creating the context for testing

Signed-off-by: Pedro Tanaka <[email protected]>

* query: distributed engine - allow querying overlapping intervals (commit signed) (#8003)

* chore: add possibility to run individual e2e tests

Signed-off-by: Abel Simon <[email protected]>

* chore: add metric pointer for too old sample logs

Signed-off-by: Abel Simon <[email protected]>

* feat: add flag for distributed queries with overlapping intervals

Signed-off-by: Abel Simon <[email protected]>

* chore: add failing overlapping interval test

Signed-off-by: Abel Simon <[email protected]>

* chore: fix base branch diff

Signed-off-by: Abel Simon <[email protected]>

---------

Signed-off-by: Abel Simon <[email protected]>

* Add labels to rules UI (#8009)

Signed-off-by: Coleen Iona Quadros <[email protected]>

* build(deps): bump golang.org/x/crypto from 0.28.0 to 0.31.0

Bumps [golang.org/x/crypto](https://github.com/golang/crypto) from 0.28.0 to 0.31.0.
- [Commits](https://github.com/golang/crypto/compare/v0.28.0...v0.31.0)

---
updated-dependencies:
- dependency-name: golang.org/x/crypto
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <[email protected]>

* *: replace `golang.org/x/exp` with standard library

These experimental packages are now available in the Go standard
library. Since we upgraded our minimum Go version to 1.23 in PR
https://github.com/thanos-io/thanos/pull/7796, we can replace them with
the standard library:

	1. golang.org/x/exp/slices -> slices [1]
	2. golang.org/x/exp/maps -> maps [2]
	3. golang.org/x/exp/rand -> math/rand/v2 [3]

[1]: https://go.dev/doc/go1.21#slices
[2]: https://go.dev/doc/go1.21#maps
[3]: https://go.dev/doc/go1.22#math_rand_v2

Signed-off-by: Eng Zer Jun <[email protected]>

* Tidy `go.mod` properly

Two sections in total: one for direct dependencies, and one for indirect
dependencies.

Reference: https://github.com/golang/go/issues/56471
Signed-off-by: Eng Zer Jun <[email protected]>

* *: replace `cespare/xxhash` with `cespare/xxhash/v2`

`github.com/cespare/xxhash/v2` is the latest version with bug fixes and
improvements.

Signed-off-by: Eng Zer Jun <[email protected]>

* QFE: Fix @ modifier not being applied correctly on subqueries (#8016)

Signed-off-by: 🌲 Harry 🌊 John 🏔 <[email protected]>

* Bump devcontainer Dockerfile base image from go1.22 to go1.23 (#8031)

Signed-off-by: Pengyu Wang <[email protected]>

* Receiver: cache matchers for series calls (#7353)

* Receiver: cache matchers for series calls

We have tried caching matchers before with a time-based expiration cache, this time we are trying with LRU cache.

We saw some of our receivers busy with compiling regexes and with high CPU usage, similar to the profile of the benchmark I added here:

* Adding matcher cache for method `MatchersToPromMatchers` and a new version which uses the cache.
* The main change is in `matchesExternalLabels` function which now receives a cache instance.

adding matcher cache and refactor matchers

Co-authored-by: Andre Branchizio <[email protected]>

Signed-off-by: Pedro Tanaka <[email protected]>

Using the cache in proxy and tsdb stores (only receiver)

Signed-off-by: Pedro Tanaka <[email protected]>

fixing problem with deep equality

Signed-off-by: Pedro Tanaka <[email protected]>

adding some docs

Signed-off-by: Pedro Tanaka <[email protected]>

Adding benchmark

Signed-off-by: Pedro Tanaka <[email protected]>

undo unecessary changes

Signed-off-by: Pedro Tanaka <[email protected]>

Adjusting metric names

Signed-off-by: Pedro Tanaka <[email protected]>

adding changelog

Signed-off-by: Pedro Tanaka <[email protected]>

wiring changes to the receiver

Signed-off-by: Pedro Tanaka <[email protected]>

Fixing linting

Signed-off-by: Pedro Tanaka <[email protected]>

docs

Signed-off-by: Pedro Tanaka <[email protected]>

* using singleflight to get or set items

Signed-off-by: Pedro Tanaka <[email protected]>

* improve metrics

Signed-off-by: Pedro Tanaka <[email protected]>

* Introduce interface for matchers cache

Signed-off-by: Pedro Tanaka <[email protected]>

* fixing unit test

Signed-off-by: Pedro Tanaka <[email protected]>

* adding changelog

Signed-off-by: Pedro Tanaka <[email protected]>

* fixing benchmark

Signed-off-by: Pedro Tanaka <[email protected]>

* moving matcher cache to storecache package

Signed-off-by: Pedro Tanaka <[email protected]>

* Trying to make the cache more reusable introducing interface

Signed-off-by: Pedro Tanaka <[email protected]>

Fixing problem with wrong initialization

Signed-off-by: Pedro Tanaka <[email protected]>

Moving interface to storecache package

Signed-off-by: Pedro Tanaka <[email protected]>

remove empty file and fix calls to constructor passing nil;

Signed-off-by: Pedro Tanaka <[email protected]>

* Fix false entry on change log

Signed-off-by: Pedro Tanaka <[email protected]>

* Removing default value for registry and rename test file

Signed-off-by: Pedro Tanaka <[email protected]>

* Using fmt.Errf()

Signed-off-by: Pedro Tanaka <[email protected]>

* Remove method that is not on interface anymore

Signed-off-by: Pedro Tanaka <[email protected]>

* Remove duplicate get call

Signed-off-by: Pedro Tanaka <[email protected]>

---------

Signed-off-by: Pedro Tanaka <[email protected]>
Signed-off-by: Pedro Tanaka <[email protected]>

* Updating x/net package (#8034)

Signed-off-by: Roberto O. Fernández Crisial <[email protected]>
Signed-off-by: Roberto O. Fernández Crisial <[email protected]>

* add pool for expanded posting slice (#8035)

* add pool for expanded posting slice

Signed-off-by: Ben Ye <[email protected]>

* check nil postings

Signed-off-by: Ben Ye <[email protected]>

---------

Signed-off-by: Ben Ye <[email protected]>

* receive: unhide tsdb.out-of-order.time-window and tsdb.out-of-order.cap-max (#8032)

* Unhide tsdb.out-of-order.time-window and tsdb.out-of-order.cap-max

Signed-off-by: milinddethe15 <[email protected]>

* make docs

Signed-off-by: milinddethe15 <[email protected]>

* make docs

Signed-off-by: milinddethe15 <[email protected]>

---------

Signed-off-by: milinddethe15 <[email protected]>

* build(deps): bump peter-evans/create-pull-request from 6.1.0 to 7.0.6 (#8028)

Bumps [peter-evans/create-pull-request](https://github.com/peter-evans/create-pull-request) from 6.1.0 to 7.0.6.
- [Release notes](https://github.com/peter-evans/create-pull-request/releases)
- [Commits](https://github.com/peter-evans/create-pull-request/compare/c5a7806660adbe173f04e3e038b0ccdcd758773c...67ccf781d68cd99b580ae25a5c18a1cc84ffff1f)

---
updated-dependencies:
- dependency-name: peter-evans/create-pull-request
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* optimize store gateway bytes limiter reserve with type request (#8025)

Signed-off-by: Ben Ye <[email protected]>

* Fix matcher cache (#8039)

* Fix matcher cache

Signed-off-by: alanprot <[email protected]>

* Simplifying cache interface

Signed-off-by: alanprot <[email protected]>

---------

Signed-off-by: alanprot <[email protected]>

* Optimize sort keys by server in memcache client (#8026)

* optimize sort keys by server in memcache client

Signed-off-by: Ben Ye <[email protected]>

* address comments

Signed-off-by: Ben Ye <[email protected]>

* remove unused mockAddr

Signed-off-by: Ben Ye <[email protected]>

---------

Signed-off-by: Ben Ye <[email protected]>

* add block lifecycle callback (#8036)

Signed-off-by: Ben Ye <[email protected]>

* Fix binary reader download duration histogram (#8017)

* fix binary reader download duration histogram

Signed-off-by: Ben Ye <[email protected]>

* enable native histograms

Signed-off-by: Ben Ye <[email protected]>

* changelog

Signed-off-by: Ben Ye <[email protected]>

---------

Signed-off-by: Ben Ye <[email protected]>

* Matcher cache/series (#8045)

* Add option to cache matcher on the get series call

Signed-off-by: alanprot <[email protected]>

* Adding matcher cacheoption for the store gateway

Signed-off-by: alanprot <[email protected]>

* lint/docs

Signed-off-by: alanprot <[email protected]>

* change desc

Signed-off-by: alanprot <[email protected]>

* change desc

Signed-off-by: alanprot <[email protected]>

* change desc

Signed-off-by: alanprot <[email protected]>

* fix test

Signed-off-by: alanprot <[email protected]>

* Caching only regex matchers

Signed-off-by: alanprot <[email protected]>

* changelog

Signed-off-by: alanprot <[email protected]>

---------

Signed-off-by: alanprot <[email protected]>

* query: fix panic when selecting non-default engine (#8050)

Fix duplicate metrics registration when selecting non-default engine

Signed-off-by: Michael Hoffmann <[email protected]>
Co-authored-by: Michael Hoffmann <[email protected]>

* pkg/cacheutil: Async op fix (#8044)

* Add test for AsyncOperationProcessor stop() behavior

The existing implementation sometimes drops existing operations that are
still on the queue when .stop() is called.

If multiple communications in a select statement can proceed, one is
chosen pseudo-randomly: https://go.dev/ref/spec#Select_statements

This means that sometimes a processor worker will process a remaining
operation, and sometimes it won't.

Signed-off-by: Daniel Sabsay <[email protected]>

* Fix async_op test regarding stop() behavior

Signed-off-by: Daniel Sabsay <[email protected]>

* add header to test file

Signed-off-by: Daniel Sabsay <[email protected]>

---------

Signed-off-by: Daniel Sabsay <[email protected]>
Co-authored-by: Daniel Sabsay <[email protected]>

* make lazy posting series match ratio configurable (#8049)

Signed-off-by: Ben Ye <[email protected]>

* QFE: fixing stats middleware when cache is enabled (#8046)

* QFE: fixing stats middleware when cache is enabled

Signed-off-by: Pedro Tanaka <[email protected]>

* Clarify strange config parameter

Signed-off-by: Pedro Tanaka <[email protected]>

---------

Signed-off-by: Pedro Tanaka <[email protected]>

* [FEATURE] adding otlp endpoint (#7996)

* [FEATURE] adding otlp endpoint

Signed-off-by: Nicolas Takashi <[email protected]>

* [FEATURE] adding otlp endpoint

Signed-off-by: Nicolas Takashi <[email protected]>

* [FEATURE] adding otlp endpoint

Signed-off-by: Nicolas Takashi <[email protected]>

* [FIX] e2e tests for otlp receiver

Signed-off-by: Nicolas Takashi <[email protected]>

* [CHORE] adding otlp flags

Signed-off-by: Nicolas Takashi <[email protected]>

* [DOC] updating docs

Signed-off-by: Nicolas Takashi <[email protected]>

* [CHORE] copying otlptranslator

Signed-off-by: Nicolas Takashi <[email protected]>

* [CHORE] copying otlptranslator tests

Signed-off-by: Nicolas Takashi <[email protected]>

* [CHORE] copying otlptranslator tests

Signed-off-by: Nicolas Takashi <[email protected]>

* [CHORE] copying otlptranslator tests

Signed-off-by: Nicolas Takashi <[email protected]>

* [FIX] lint issues

Signed-off-by: Nicolas Takashi <[email protected]>

* [FIX] lint issues

Signed-off-by: Nicolas Takashi <[email protected]>

* [FIX] lint issues

Signed-off-by: Nicolas Takashi <[email protected]>

* [CHORE] using multi errors

Signed-off-by: Nicolas Takashi <[email protected]>

* [FIX] span naming convention

Signed-off-by: Nicolas Takashi <[email protected]>

* [TEST] adding handler otlp unit test

Signed-off-by: Nicolas Takashi <[email protected]>

* [TEST] upgrade collector version

Signed-off-by: Nicolas Takashi <[email protected]>

* [FIX] golang lint

Signed-off-by: Nicolas Takashi <[email protected]>

* [CHORE] adding allow size bytes limit gate

Signed-off-by: Nicolas Takashi <[email protected]>

* [FIX] unit test otlp endpoint

Signed-off-by: Nicolas Takashi <[email protected]>

* Apply suggestions from code review

Co-authored-by: Saswata Mukherjee <[email protected]>
Signed-off-by: Nicolas Takashi <[email protected]>

* [FIX] unit test otlp endpoint

Signed-off-by: Nicolas Takashi <[email protected]>

* [DOC] updating docs

Signed-off-by: Nicolas Takashi <[email protected]>

* [CHORE] applying pr comments

Signed-off-by: Nicolas Takashi <[email protected]>

* Update pkg/receive/handler_otlp.go

Co-authored-by: Matej Gera <[email protected]>
Signed-off-by: Nicolas Takashi <[email protected]>

* Update cmd/thanos/receive.go

Co-authored-by: Matej Gera <[email protected]>
Signed-off-by: Nicolas Takashi <[email protected]>

* [DOCS] updating

Signed-off-by: Nicolas Takashi <[email protected]>

* [FIX] go mod lint issues

Signed-off-by: Nicolas Takashi <[email protected]>

* Fix TestFromMetrics error comparison

Signed-off-by: Saswata Mukherjee <[email protected]>

---------

Signed-off-by: Nicolas Takashi <[email protected]>
Signed-off-by: Saswata Mukherjee <[email protected]>
Co-authored-by: Saswata Mukherjee <[email protected]>
Co-authored-by: Matej Gera <[email protected]>

* query, rule: make endpoint discovery dynamically reloadable (#7890)

* Removed previously deprecated and hidden flags to configure endpoints ( --rule, --target, ...)
* Added new flags --endpoint.sd-config, --endpoint-sd-config-reload-interval to configure a dynamic SD file
* Moved endpoint set construction into cmd/thanos/endpointset.go for a little cleanup
* Renamed "thanos_(querier/ruler)_duplicated_store_addresses_total" to
  "thanos_(querier/ruler)_duplicated_endpoint_addresses_total"

The new config makes it possible to also set "strict" and "group" flags on the endpoint instead
of only their addresses, making it possible to have file based service discovery for endpoint groups too.

Signed-off-by: Michael Hoffmann <[email protected]>
Signed-off-by: Michael Hoffmann <[email protected]>

* skip match label values for certain matchers (#8084)

Signed-off-by: yeya24 <[email protected]>

* Extend posting-group-max-key-series-ratio for add all posting group (#8083)

Signed-off-by: yeya24 <[email protected]>

* Add a func to convert go-kit log to slog (#7969)

Signed-off-by: SungJin1212 <[email protected]>

* fix: add POST into allowed CORS methods header (#8091)

Signed-off-by: Célian Garcia <[email protected]>

* Fix deadlock in metadata fetcher (#8092)

This is a fix for a bug we encountered in our production deployment at
$WORK, where we experience Thanos Store to spontaneously stop refreshing
its metadata cache every time our (Ceph-based) object storage starts
rate-limiting Thanos's requests too much. This causes the Store to
permanently stop discovering new blocks in the bucket, and keep trying
to access old, long-gone blocks (which have already been compacted and
removed), breaking subsequent queries to the Store and mandating its
manual restart. More details about the bug and the fix below.

Initially, the `GetActiveAndPartialBlockIDs` method spawns fixed number
of 64 goroutines to issue concurrent requests to a remote storage. Each
time the storage causes the `f.bkt.Exists` call to return an error, one
of goroutines exits returning said error, effectively reducing
concurrency of processing remaining block IDs in `metaChan`. While it's
not a big problem in case of one or two errors, it is entirely possible
that, in case of prolonged storage problems, all 64 goroutines quit,
resulting in `metaChan` filling up and blocking the `f.bkt.Iter`
iterator below. This causes the whole method to be stuck indefinitely,
even if the storage becomes fully operational again.

This commit fixes the issue by allowing the iterator to return as soon
as a single processing goroutine errors out, so that the method can
reliably finish, returning the error as intended. Additionally, the
processing goroutines are adjusted as well, to make them all quit early
without consuming remaining items in `metaChan`. While the latter is not
strictly necessary to fix this bug, it doesn't make sense to let any
remaining goroutines keep issuing requests to the storage if the method
is already bound to return nil result along with the first encountered
error.

Signed-off-by: Piotr Śliwka <[email protected]>

* store: lock around iterating over s.blocks (#8088)

Hold a lock around s.blocks when iterating over it. I have experienced a
case where a block had been added to a blockSet twice somehow and it
being somehow removed from s.blocks is the only way it could happen.
This is the only "bad" thing I've been able to spot.

Signed-off-by: Giedrius Statkevičius <[email protected]>

* Update prometheus verison to v3.1.0 (#8090)

Signed-off-by: SungJin1212 <[email protected]>

* build(deps): bump github.com/tjhop/slog-gokit from 0.1.2 to 0.1.3 (#8109)

Bumps [github.com/tjhop/slog-gokit](https://github.com/tjhop/slog-gokit) from 0.1.2 to 0.1.3.
- [Release notes](https://github.com/tjhop/slog-gokit/releases)
- [Changelog](https://github.com/tjhop/slog-gokit/blob/main/.goreleaser.yaml)
- [Commits](https://github.com/tjhop/slog-gokit/compare/v0.1.2...v0.1.3)

---
updated-dependencies:
- dependency-name: github.com/tjhop/slog-gokit
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* *:  bump Go to 1.24 (#8105)

* bump go version to newer 1.24

Signed-off-by: Saumyacodes-40 <[email protected]>

* update .go-version

Signed-off-by: Saumyacodes-40 <[email protected]>

* fix test failing, formatted string func now requires constant string literal

Signed-off-by: Saumyacodes-40 <[email protected]>

* test: point faillint to thanos-community/faillint

Signed-off-by: Saumyacodes-40 <[email protected]>

* fix commit hash

Signed-off-by: Saumyacodes-40 <[email protected]>

* use branch instead of hash

Signed-off-by: Saumyacodes-40 <[email protected]>

* test: update faillint.mod/sum

Signed-off-by: Saumyacodes-40 <[email protected]>

* bump golangci-lint

Signed-off-by: Saumyacodes-40 <[email protected]>

* update .bingo/faillint.mod based on new deps upgrade

Signed-off-by: Saumyacodes-40 <[email protected]>

* address required changes

Signed-off-by: Saumyacodes-40 <[email protected]>

---------

Signed-off-by: Saumyacodes-40 <[email protected]>

* Remove quote from replica label. (#8075)

Signed-off-by: Marta <[email protected]>

* Query: dynamic endpointgroups are allowed (#8113)

This PR fixes a bug where dynamic endpoint groups are silently ignored.

Signed-off-by: Michael Hoffmann <[email protected]>
Co-authored-by: Michael Hoffmann <[email protected]>

* [issue-8106] fix tenant hashring glob with multiple match patterns (#8107)

Signed-off-by: Yi Jin <[email protected]>

* build(deps): bump go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp (#8067)

Bumps [go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp](https://github.com/open-telemetry/opentelemetry-go) from 1.29.0 to 1.34.0.
- [Release notes](https://github.com/open-telemetry/opentelemetry-go/releases)
- [Changelog](https://github.com/open-telemetry/opentelemetry-go/blob/main/CHANGELOG.md)
- [Commits](https://github.com/open-telemetry/opentelemetry-go/compare/v1.29.0...v1.34.0)

---
updated-dependencies:
- dependency-name: go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* query : add missing xincrease/xrate aggregation (#8120)

* Infer max query downsample resolution from promql query (#7012)

* Adjust max_source_resolution automatically based promql queries

Signed-off-by: Ben Ye <[email protected]>

* fix data race

Signed-off-by: yeya24 <[email protected]>

---------

Signed-off-by: Ben Ye <[email protected]>
Signed-off-by: yeya24 <[email protected]>

* Optimize wildcard matchers for .* and .+ (#8131)

* optimize wildcard matchers for .* and .+

Signed-off-by: yeya24 <[email protected]>

* add changelog

Signed-off-by: yeya24 <[email protected]>

---------

Signed-off-by: yeya24 <[email protected]>

* Query: bump promql-engine (#8118)

Bumping PromQL engine, fixing fallback fallout.

Signed-off-by: Michael Hoffmann <[email protected]>
Co-authored-by: Michael Hoffmann <[email protected]>

* build(deps): bump actions/cache from 4.0.2 to 4.2.1 (#8122)

Bumps [actions/cache](https://github.com/actions/cache) from 4.0.2 to 4.2.1.
- [Release notes](https://github.com/actions/cache/releases)
- [Changelog](https://github.com/actions/cache/blob/main/RELEASES.md)
- [Commits](https://github.com/actions/cache/compare/0c45773b623bea8c8e75f6c82b208c3cf94ea4f9...0c907a75c2c80ebcb7f088228285e798b750cf8f)

---
updated-dependencies:
- dependency-name: actions/cache
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* build(deps): bump peter-evans/create-pull-request from 7.0.6 to 7.0.7 (#8123)

Bumps [peter-evans/create-pull-request](https://github.com/peter-evans/create-pull-request) from 7.0.6 to 7.0.7.
- [Release notes](https://github.com/peter-evans/create-pull-request/releases)
- [Commits](https://github.com/peter-evans/create-pull-request/compare/67ccf781d68cd99b580ae25a5c18a1cc84ffff1f...dd2324fc52d5d43c699a5636bcf19fceaa70c284)

---
updated-dependencies:
- dependency-name: peter-evans/create-pull-request
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* compact: implement native histogram downsampling (#8110)

* test/e2e: add native histogram downsampling test case

Signed-off-by: Giedrius Statkevičius <[email protected]>

* downsample: port other PR

Signed-off-by: Giedrius Statkevičius <[email protected]>

* compact: fix after review

Signed-off-by: Giedrius Statkevičius <[email protected]>

---------

Signed-off-by: Giedrius Statkevičius <[email protected]>

* Bump common+client_golang to deal with utf-8 (#8134)

* Bump common+client_golang to deal with utf-8

Signed-off-by: Saswata Mukherjee <[email protected]>

* Fix+Add tests

Signed-off-by: Saswata Mukherjee <[email protected]>

* Bump to 1.21

Signed-off-by: Saswata Mukherjee <[email protected]>

---------

Signed-off-by: Saswata Mukherjee <[email protected]>

* Query: bump promql-engine, fix fallout for distributed mode (#8135)

This PR bumps the thanos promql-engine repository, fixes the fallout and
makes distributed mode respect the user requested partial response
setting.

Signed-off-by: Michael Hoffmann <[email protected]>

* build(deps): bump go.opentelemetry.io/otel/sdk from 1.34.0 to 1.35.0 (#8148)

Bumps [go.opentelemetry.io/otel/sdk](https://github.com/open-telemetry/opentelemetry-go) from 1.34.0 to 1.35.0.
- [Release notes](https://github.com/open-telemetry/opentelemetry-go/releases)
- [Changelog](https://github.com/open-telemetry/opentelemetry-go/blob/main/CHANGELOG.md)
- [Commits](https://github.com/open-telemetry/opentelemetry-go/compare/v1.34.0...v1.35.0)

---
updated-dependencies:
- dependency-name: go.opentelemetry.io/otel/sdk
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* query: Support chain deduplication algorithm (#7808)

Signed-off-by: Michał Mazur <[email protected]>

* rule: Add support for query offset (#8158)

Support Prometheus rule manager upstream "query offset" feature.
* Add support for a default rule query offset via command flag.
* Add per rule group query_offset support.

Fixes: https://github.com/thanos-io/thanos/issues/7596

Signed-off-by: SuperQ <[email protected]>

* update: interactive tests to update non-supported store flags to endpoint in querier (#8157)

Signed-off-by: Saumya Shah <[email protected]>

* e2e: use prom 3 (#8165)

Signed-off-by: Michael Hoffmann <[email protected]>

* Changelog: mark 0.38 as in-progress (#8173)

Signed-off-by: Michael Hoffmann <[email protected]>

* Query: fix endpointset setup (#8175)

This commit fixes an issue where we add non-strict, non-group endpoints
to the endpointset twice, once with resolved addresses from the dns
provider and once with its dns prefix.

Signed-off-by: Michael Hoffmann <[email protected]>

* deps: bump promql-engine (#8181)

Signed-off-by: Michael Hoffmann <[email protected]>

* merge release 0.38.0 to main (#8186)

* Changelog: cut release 0.38-rc.0 (#8174)

Signed-off-by: Michael Hoffmann <[email protected]>

* Changelog: cut release 0.38.0-rc.1 (#8180)

* Query: fix endpointset setup

This commit fixes an issue where we add non-strict, non-group endpoints
to the endpointset twice, once with resolved addresses from the dns
provider and once with its dns prefix.

Signed-off-by: Michael Hoffmann <[email protected]>

* deps: bump promql-engine (#8181)

Signed-off-by: Michael Hoffmann <[email protected]>

* Changelog: cut release 0.38-rc.1

Signed-off-by: Michael Hoffmann <[email protected]>

---------

Signed-off-by: Michael Hoffmann <[email protected]>

* Changelog: cut release 0.38 (#8185)

Signed-off-by: Michael Hoffmann <[email protected]>

---------

Signed-off-by: Michael Hoffmann <[email protected]>

* fix changelog markdown. (#8190)

Signed-off-by: naohiroo <[email protected]>

* Sidecar: increase default prometheus timeout (#8192)

Adjust the default get-config timeout to match the default get-config
interval.

Signed-off-by: Michael Hoffmann <[email protected]>

* Unhide tsdb.enable-native-histograms flag in receive (#8202)

Signed-off-by: Aaron Walker <[email protected]>

* Query: fix partial response for distributed instant query (#8211)

This commit fixes a typo in partial response handling for distributed
instant queries.

Signed-off-by: Michael Hoffmann <[email protected]>

* build(deps): bump github.com/golang-jwt/jwt/v5 from 5.2.1 to 5.2.2 (#8164)

Bumps [github.com/golang-jwt/jwt/v5](https://github.com/golang-jwt/jwt) from 5.2.1 to 5.2.2.
- [Release notes](https://github.com/golang-jwt/jwt/releases)
- [Changelog](https://github.com/golang-jwt/jwt/blob/main/VERSION_HISTORY.md)
- [Commits](https://github.com/golang-jwt/jwt/compare/v5.2.1...v5.2.2)

---
updated-dependencies:
- dependency-name: github.com/golang-jwt/jwt/v5
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* fix Pull iter race between next() and stop()

Signed-off-by: Yuchen Wang <[email protected]>

* add changelog

Signed-off-by: Yuchen Wang <[email protected]>

* add unit test

Signed-off-by: Yuchen Wang <[email protected]>

* fix unit test

Signed-off-by: Yuchen Wang <[email protected]>

* update changelog

Signed-off-by: Yuchen Wang <[email protected]>

* add comments

Signed-off-by: Yuchen Wang <[email protected]>

* simplify unit test

Signed-off-by: Yuchen Wang <[email protected]>

* Update query.md documentation where example uses --query.tenant-default-id flag instead of --query.default-tenant-id (#8210)

Signed-off-by: Andrew Reilly <[email protected]>

* Receive: Ensure forward/replication metrics are incremented in err cases (#8212)

* Ensure forward/replication metrics are incremented in err cases

This commit ensures forward and replication metrics are incremented with
err labels.

This seemed to be missing, came across this whilst working on a
dashboard.

Signed-off-by: Saswata Mukherjee <[email protected]>

* Add changelog

Signed-off-by: Saswata Mukherjee <[email protected]>

---------

Signed-off-by: Saswata Mukherjee <[email protected]>

* receive: remove Get() method from hashring (#8226)

Get() is equivalent to GetN(1) so remove it. It's not used.

Signed-off-by: Giedrius Statkevičius <[email protected]>

* tools: Extend bucket ls options (#8225)

* tools: Extend bucket ls command with min and max time, selector config and timeout options

Signed-off-by: Philip Gough <[email protected]>

* make: docs

Signed-off-by: Philip Gough <[email protected]>

Update cmd/thanos/tools_bucket.go

Co-authored-by: Saswata Mukherjee <[email protected]>
Signed-off-by: Philip Gough <[email protected]>

Update cmd/thanos/tools_bucket.go

Co-authored-by: Saswata Mukherjee <[email protected]>
Signed-off-by: Philip Gough <[email protected]>

---------

Signed-off-by: Philip Gough <[email protected]>
Signed-off-by: Philip Gough <[email protected]>
Co-authored-by: Saswata Mukherjee <[email protected]>

* UI: bump codemirror-promql dependency to latest version (#8230)

* bump codemirror-promql react dep to latest version

Signed-off-by: Saumya Shah <[email protected]>

* fix lint errors, build react-app

Signed-off-by: Saumya Shah <[email protected]>

* sync ui change of input expression

Signed-off-by: Saumya Shah <[email protected]>

* revert build files

Signed-off-by: Saumya Shah <[email protected]>

* build and update few warnings

Signed-off-by: Saumya Shah <[email protected]>

---------

Signed-off-by: Saumya Shah <[email protected]>

* reloader: disable some flaky tests (#8240)

Disabling some flaky tests.

Signed-off-by: Giedrius Statkevičius <[email protected]>

* e2e: disable some more flaky tests (#8241)

These are flaky hence disable them.

Signed-off-by: Giedrius Statkevičius <[email protected]>

* add otlp clientType (#8243)

Signed-off-by: dongjiang <[email protected]>

* test(tools_test.go/Test_CheckRules_Glob): take into consideration RO current dirs while (#8014)

changing files permissions.

The process may not have the needed permissions on the file (not the owner, not root or doesn't have the CAP_FOWNER capability)
to chmod it.
i

Signed-off-by: machine424 <[email protected]>

* docs: volunteer as shepherd (#8249)

* docs: volunteer as shepherd

Release the next version in a few weeks.

Signed-off-by: Giedrius Statkevičius <[email protected]>

* Fix formatting

Signed-off-by: Matej Gera <[email protected]>

---------

Signed-off-by: Giedrius Statkevičius <[email protected]>
Signed-off-by: Matej Gera <[email protected]>
Co-authored-by: Matej Gera <[email protected]>

* shipper: delete unused functions (#8260)

Signed-off-by: Michael Hoffmann <[email protected]>

* feat(promu): add darwin/arm64 (#8263)

* feat(promu): add darwin/arm64

Signed-off-by: Ben Dronen <[email protected]>

* fix(promu): just use darwin

Signed-off-by: Ben Dronen <[email protected]>

---------

Signed-off-by: Ben Dronen <[email protected]>

* receive: implement shuffle sharding (#8238)

See the documentation for details.

Closes #3821.

Signed-off-by: Giedrius Statkevičius <[email protected]>

* docs: fix receive page (#8267)

Fix the docs after the most recent merge.

Signed-off-by: Giedrius Statkevičius <[email protected]>

* Shipper: fix missing meta file errors (#8268)

- fix meta file read error check
- use proper logs for missing meta file vs. other read errors

Signed-off-by: Siavash Safi <[email protected]>

* query: fix query analyze to return appropriate results (#8262)

* call query analysis once querys being exec

Signed-off-by: Saumya Shah <[email protected]>

* refract the analyze logic

Signed-off-by: Saumya Shah <[email protected]>

* send not analyzable warnings instead of returning err

Signed-off-by: Saumya Shah <[email protected]>

* add seperate warnings in query non analyzable state based on engine

Signed-off-by: Saumya Shah <[email protected]>

---------

Signed-off-by: Saumya Shah <[email protected]>

* go.mod: mass update modules (#8277)

Maintenance task: let's update all modules.

Signed-off-by: Giedrius Statkevičius <[email protected]>

* Force sync writes to meta.json in case of host crash (#8282)

* Force sync writes to meta.json in case of host crash

Signed-off-by: Anna Tran <[email protected]>

* Update CHANGELOG for fsync meta.json

Signed-off-by: Anna Tran <[email protected]>

---------

Signed-off-by: Anna Tran <[email protected]>

* query, query-frontend, ruler: Add support for flags to use promQL experimental functions & bump promql-engine (#8245)

* feat: add support for experimental functions, if enabled

Signed-off-by: Saumya Shah <[email protected]>

* fix tests

Signed-off-by: Saumya Shah <[email protected]>

* allow setting enable-feature flag in ruler

Signed-off-by: Saumya Shah <[email protected]>

* add flag info in docs

Signed-off-by: Saumya Shah <[email protected]>

* add CHANGELOG

Signed-off-by: Saumya Shah <[email protected]>

* add hidden flag to throw err on query fallback, red in tests ^_^

Signed-off-by: Saumya Shah <[email protected]>

* bump promql-engine to latest version/commit

Signed-off-by: Saumya Shah <[email protected]>

* format docs

Signed-off-by: Saumya Shah <[email protected]>

---------

Signed-off-by: Saumya Shah <[email protected]>

* fix: maintain expanded state based on current page (#8266)

* fix(sidebar): maintain expanded state based on current page

Signed-off-by: Naman-B-Parlecha <[email protected]>

* fixing changelog

Signed-off-by: Naman-B-Parlecha <[email protected]>

---------

Signed-off-by: Naman-B-Parlecha <[email protected]>
Signed-off-by: Naman-Parlecha <[email protected]>

* query: handle query.Analyze returning nil gracefully (#8199)

* fix: handle analyze returning nil gracefully

Signed-off-by: Saumya Shah <[email protected]>

* update CHANGELOG.md

Signed-off-by: Saumya Shah <[email protected]>

* fix format

Signed-off-by: Saumya Shah <[email protected]>

---------

Signed-off-by: Saumya Shah <[email protected]>

* store: Add --disable-admin-operations Flag to Store Gateway (#8284)

* fix(sidebar): maintain expanded state based on current page

Signed-off-by: Naman-B-Parlecha <[email protected]>

* fixing changelog

Signed-off-by: Naman-B-Parlecha <[email protected]>

* store: --disable-admin-operation flag

Signed-off-by: Naman-B-Parlecha <[email protected]>

docs: Adding Flag details

Signed-off-by: Naman-B-Parlecha <[email protected]>

updated changelog

refactor: changelog

Signed-off-by: Naman-Parlecha <[email protected]>

---------

Signed-off-by: Naman-B-Parlecha <[email protected]>
Signed-off-by: Naman-Parlecha <[email protected]>

* shipper: allow shipper sync to skip corrupted blocks (#8259)

* Allow shipper sync to skip corrupted blocks

Signed-off-by: Daniel Deluiggi <[email protected]>

* Move check to blockMetasFromOldest

Signed-off-by: Daniel Deluiggi <[email protected]>

* Split metrics. Return error

Signed-off-by: Daniel Deluiggi <[email protected]>

* fix test

Signed-off-by: Daniel Deluiggi <[email protected]>

* Reorder shipper contructor variables

Signed-off-by: Daniel Deluiggi <[email protected]>

* Use opts in shipper constructor

Signed-off-by: Daniel Deluiggi <[email protected]>

* Fix typo

Signed-off-by: Daniel Deluiggi <[email protected]>

---------

Signed-off-by: Daniel Deluiggi <[email protected]>

* Receive: Remove migration of legacy storage to multi-tsdb (#8289)

This has been in since 0.13 (~5 years ago). This fixes issues caused when the default-tenant does not have any data and gets churned, resulting in the migration assuming that per-tenant directories are actually blocks, resulting in blocks not being queryable.

Signed-off-by: Aaron Walker <[email protected]>

* Resolve GitHub Actions Failure (#8299)

* update: changing to new prometheus page

Signed-off-by: Naman-B-Parlecha <[email protected]>

* fix: disable-admin-op flag

Signed-off-by: Naman-B-Parlecha <[email protected]>

---------

Signed-off-by: Naman-B-Parlecha <[email protected]>

* Fix: DataRace in TestEndpointSetUpdate_StrictEndpointMetadata test (#8288)

* fix: Fixing Unit Test TestEndpointSetUpdate_StrictEndpointMetadata

Signed-off-by: Naman-B-Parlecha <[email protected]>

* revert: CHANGELOG.md

Signed-off-by: Naman-Parlecha <[email protected]>

---------

Signed-off-by: Naman-B-Parlecha <[email protected]>
Signed-off-by: Naman-Parlecha <[email protected]>

* build(deps): bump github/codeql-action from 3.26.13 to 3.28.19 (#8304)

Bumps [github/codeql-action](https://github.com/github/codeql-action) from 3.26.13 to 3.28.19.
- [Release notes](https://github.com/github/codeql-action/releases)
- [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md)
- [Commits](https://github.com/github/codeql-action/compare/f779452ac5af1c261dce0346a8f964149f49322b...fca7ace96b7d713c7035871441bd52efbe39e27e)

---
updated-dependencies:
- dependency-name: github/codeql-action
  dependency-version: 3.28.19
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* build(deps): bump github.com/alicebob/miniredis/v2 from 2.22.0 to 2.35.0 (#8303)

Bumps [github.com/alicebob/miniredis/v2](https://github.com/alicebob/miniredis) from 2.22.0 to 2.35.0.
- [Release notes](https://github.com/alicebob/miniredis/releases)
- [Changelog](https://github.com/alicebob/miniredis/blob/master/CHANGELOG.md)
- [Commits](https://github.com/alicebob/miniredis/compare/v2.22.0...v2.35.0)

---
updated-dependencies:
- dependency-name: github.com/alicebob/miniredis/v2
  dependency-version: 2.35.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* build(deps): bump golang.org/x/time from 0.11.0 to 0.12.0 (#8302)

Bumps [golang.org/x/time](https://github.com/golang/time) from 0.11.0 to 0.12.0.
- [Commits](https://github.com/golang/time/compare/v0.11.0...v0.12.0)

---
updated-dependencies:
- dependency-name: golang.org/x/time
  dependency-version: 0.12.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Revert capnp to v3.0.0-alpha (#8300)

cef0b02 caused a regression of !7944. This reverts the version upgrade to the previously working version

Signed-off-by: Aaron Walker <[email protected]>

* go.mod: update promql engine + Prom dep (#8305)

Update dependencies. Almost everything works except for
https://github.com/prometheus/prometheus/pull/16252.

Signed-off-by: Giedrius Statkevičius <[email protected]>

* tools: fix tool bucket inspect output arg description (#8252)

* docs: fix tool bucket output arg description

Signed-off-by: gabyf <[email protected]>

* fix(tools_bucket): output description from cvs to csv

Signed-off-by: gabyf <[email protected]>

---------

Signed-off-by: gabyf <[email protected]>

* Updating the documention to reflect the correct flags when using --compact.enable-vertical-compaction.

Signed-off-by: Greg Sharpe <[email protected]>

* Query: limit LazyRetrieval memory buffer size (#8296)

* Limit lazyRespSet memory buffer size using a ring buffer

Signed-off-by: HC Zhu <[email protected]>

* store: make heap a bit more consistent

Add len comparison to make it more consistent.

Signed-off-by: Giedrius Statkevičius <[email protected]>

* Fix linter complains

Signed-off-by: HC Zhu <[email protected]>

---------

Signed-off-by: HC Zhu <[email protected]>
Signed-off-by: Giedrius Statkevičius <[email protected]>
Co-authored-by: HC Zhu (DB) <[email protected]>
Co-authored-by: Giedrius Statkevičius <[email protected]>
Co-authored-by: HC Zhu <[email protected]>

* Add Prometheus counters for pending write requests and series requests in Receive (#8308)

Signed-off-by: HC Zhu <[email protected]>
Co-authored-by: HC Zhu (DB) <[email protected]>

* build(deps): bump github.com/KimMachineGun/automemlimit (#8312)

Bumps [github.com/KimMachineGun/automemlimit](https://github.com/KimMachineGun/automemlimit) from 0.7.2 to 0.7.3.
- [Release notes](https://github.com/KimMachineGun/automemlimit/releases)
- [Commits](https://github.com/KimMachineGun/automemlimit/compare/v0.7.2...v0.7.3)

---
updated-dependencies:
- dependency-name: github.com/KimMachineGun/automemlimit
  dependency-version: 0.7.3
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* query_frontend: only ready if downstream is ready (#8315)

We had an incident in prod where QFE was reporting that it is ready even
though the downstream didn't work due to a misconfigured load-balancer.
In this PR I am proposing sending periodic requests to downstream
to check whether it is working.

TestQueryFrontendTenantForward never worked so I deleted it.

Signed-off-by: Giedrius Statkevičius <[email protected]>

* *: add CHANGELOG, update VERSION (#8320)

Prepare for 0.39.0-rc.0.

Signed-off-by: Giedrius Statkevičius <[email protected]>

* *: release 0.39.0 (#8330)

Signed-off-by: Giedrius Statkevičius <[email protected]>

* cmd/query_frontend: use original roundtripper + close immediately

Let's avoid using all the Cortex roundtripper machinery by using the
downstream roundtripper directly and then close the body immediately as
to not allocate any memory for the body of the response.

Signed-off-by: Giedrius Statkevičius <[email protected]>

* qfe: defer properly

Refactor this check into a separate function so that defer would run at
the end of it and clean up resources properly.

Signed-off-by: Giedrius Statkevičius <[email protected]>

* CHANGELOG: release 0.39.1

Signed-off-by: Giedrius Statkevičius <[email protected]>

* query: add custom values to prompb methods (#8375)

Signed-off-by: Michael Hoffmann <[email protected]>

* fix: querier panic (#8374)

Thanos Query crashes with "concurrent map iteration and map write" panic
in distributed mode when multiple goroutines access the same `annotations.Annotations`
map concurrently.

```
panic: concurrent map iteration and map write
github.com/prometheus/prometheus/util/annotations.(*Annotations).Merge(...)
github.com/thanos-io/promql-engine/engine.(*compatibilityQuery).Exec(...)
```

Here I replaced direct access to `res.Warnings.AsErrors()` with a thread-safe copy:
```go
// Before (unsafe)
warnings = append(warnings, res.Warnings.AsErrors()...)

// After (thread-safe)
safeWarnings := annotations.New().Merge(res.Warnings)
warnings = append(warnings, safeWarnings.AsErrors()...)
```

Signed-off-by: Joel Verezhak <[email protected]>
Co-authored-by: Joel Verezhak <[email protected]>
Signed-off-by: Michael Hoffmann <[email protected]>

* Release 0.39.2

Signed-off-by: Michael Hoffmann <[email protected]>

* Update Containerfile.operator

* update deps

Signed-off-by: Coleen Iona Quadros <[email protected]>

---------

Signed-off-by: didukh86 <[email protected]>
Signed-off-by: didukh86 <[email protected]>
Signed-off-by: Taras Didukh <[email protected]>
Signed-off-by: Michael Hoffmann <[email protected]>
Signed-off-by: dependabot[bot] <[email protected]>
Signed-off-by: milinddethe15 <[email protected]>
Signed-off-by: Taras Didukh <[email protected]>
Signed-off-by: Giedrius Statkevičius <[email protected]>
Signed-off-by: Philip Gough <[email protected]>
Signed-off-by: Helen Tseng <[email protected]>
Signed-off-by: Filip Petkovski <[email protected]>
Signed-off-by: Saswata Mukherjee <[email protected]>
Signed-off-by: Yi Jin <[email protected]>
Signed-off-by: kade.lee <[email protected]>
Signed-off-by: Ben Ye <[email protected]>
Signed-off-by: Michael Hoffmann <[email protected]>
Signed-off-by: Remi Vichery <[email protected]>
Signed-off-by: Pedro Tanaka <[email protected]>
Signed-off-by: Abel Simon <[email protected]>
Signed-off-by: Coleen Iona Quadros <[email protected]>
Signed-off-by: Eng Zer Jun <[email protected]>
Signed-off-by: 🌲 Harry 🌊 John 🏔 <[email protected]>
Signed-off-by: Pengyu Wang <[email protected]>
Signed-off-by: Pedro Tanaka <[email protected]>
Signed-off-by: Roberto O. Fernández Crisial <[email protected]>
Signed-off-by: Roberto O. Fernández Crisial <[email protected]>
Signed-off-by: alanprot <[email protected]>
Signed-off-by: Daniel Sabsay <[email protected]>
Signed-off-by: Nicolas Takashi <[email protected]>
Signed-off-by: yeya24 <[email protected]>
Signed-off-by: SungJin1212 <[email protected]>
Signed-off-by: Célian Garcia <[email protected]>
Signed-off-by: Piotr Śliwka <[email protected]>
Signed-off-by: Saumyacodes-40 <[email protected]>
Signed-off-by: Marta <[email protected]>
Signed-off-by: Michał Mazur <[email protected]>
Signed-off-by: SuperQ <[email protected]>
Signed-off-by: Saumya Shah <[email protected]>
Signed-off-by: naohiroo <[email protected]>
Signed-off-by: Aaron Walker <[email protected]>
Signed-off-by: Yuchen Wang <[email protected]>
Signed-off-by: Andrew Reilly <[email protected]>
Signed-off-by: Philip Gough <[email protected]>
Signed-off-by: dongjiang <[email protected]>
Signed-off-by: machine424 <[email protected]>
Signed-off-by: Matej Gera <[email protected]>
Signed-off-by: Ben Dronen <[email protected]>
Signed-off-by: Siavash Safi <[email protected]>
Signed-off-by: Anna Tran <[email protected]>
Signed-off-by: Naman-B-Parlecha <[email protected]>
Signed-off-by: Naman-Parlecha <[email protected]>
Signed-off-by: Daniel Deluiggi <[email protected]>
Signed-off-by: gabyf <[email protected]>
Signed-off-by: Greg Sharpe <[email protected]>
Signed-off-by: HC Zhu <[email protected]>
Signed-off-by: Joel Verezhak <[email protected]>
Co-authored-by: didukh86 <[email protected]>
Co-authored-by: Taras Didukh <[email protected]>
Co-authored-by: Michael Hoffmann <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Milind Dethe <[email protected]>
Co-authored-by: Giedrius Statkevičius <[email protected]>
Co-authored-by: Philip Gough <[email protected]>
Co-authored-by: Filip Petkovski <[email protected]>
Co-authored-by: bluesky6529 <[email protected]>
Co-authored-by: Saswata Mukherjee <[email protected]>
Co-authored-by: Yi Jin <[email protected]>
Co-authored-by: SungJin1212 <[email protected]>
Co-authored-by: Ben Ye <[email protected]>
Co-authored-by: Michael Hoffmann <[email protected]>
Co-authored-by: Rémi Vichery <[email protected]>
Co-authored-by: Pedro Tanaka <[email protected]>
Co-authored-by: Abel Simon <[email protected]>
Co-authored-by: Coleen Iona Quadros <[email protected]>
Co-authored-by: Eng Zer Jun <[email protected]>
Co-authored-by: Harry John <[email protected]>
Co-authored-by: Pengyu Wang <[email protected]>
Co-authored-by: Roberto O. Fernández Crisial <[email protected]>
Co-authored-by: Alan Protasio <[email protected]>
Co-authored-by: Daniel Sabsay <[email protected]>
Co-authored-by: Daniel Sabsay <[email protected]>
Co-authored-by: Nicolas Takashi <[email protected]>
Co-authored-by: Matej Gera <[email protected]>
Co-authored-by: Célian GARCIA <[email protected]>
Co-authored-by: Piotr Śliwka <[email protected]>
Co-authored-by: Saumya Shah <[email protected]>
Co-authored-by: Marta <[email protected]>
Co-authored-by: 민선 (minnie) <[email protected]>
Co-authored-by: Michał Mazur <[email protected]>
Co-authored-by: Ben Kochie <[email protected]>
Co-authored-by: Naohiro Okada <[email protected]>
Co-authored-by: Aaron Walker <[email protected]>
Co-authored-by: Yuchen Wang <[email protected]>
Co-authored-by: Andrew Reilly <[email protected]>
Co-authored-by: dongjiang <[email protected]>
Co-authored-by: Ayoub Mrini <[email protected]>
Co-authored-by: dronenb <[email protected]>
Co-authored-by: Siavash Safi <[email protected]>
Co-authored-by: Anna Tran <[email protected]>
Co-authored-by: Naman-Parlecha <[email protected]>
Co-authored-by: Daniel Blando <[email protected]>
Co-authored-by: Aaron Walker <[email protected]>
Co-authored-by: gabyf <[email protected]>
Co-authored-by: Greg Sharpe <[email protected]>
Co-authored-by: Hongcheng Zhu <[email protected]>
Co-authored-by: HC Zhu (DB) <[email protected]>
Co-authored-by: HC Zhu <[email protected]>
Co-authored-by: Joel Verezhak <[email protected]>
Co-authored-by: Joel Verezhak <[email protected]>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@yeya24 yeya24 yeya24 approved these changes

Assignees
No one assigned
Labels
size/XXL
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Add Native Histogram Support
3 participants
@GiedriusS @vCra @yeya24