Add Query Resource Based Eviction by eeldaly · Pull Request #7488 · cortexproject/cortex

eeldaly · 2026-05-07T22:07:59Z

What this PR does:
This PR builds on the current resource based throttling infrastructure (#6674) to allow for evicting currently running queries. This is currently only implemented on querier pods but can be extended to other pods similar to resourced based throttling.

Flags
All flags are prefixed with -querier.query-protection.eviction.

threshold.cpu-utilization: Max CPU utilization (0–1) before evicting the heaviest query (default 0)
threshold.heap-utilization: Max heap utilization (0–1) before evicting the heaviest query (default 0)
check-interval: How frequently the evictor checks resource utilization (default 1s)
cooldown-period: Number of check intervals to wait after an eviction before evicting again (default 3)
eviction-metric: Metric used to determine the heaviest query (fetched_samples, fetched_series, fetched_chunks, fetched_chunk_bytes) (default fetched_samples)
min-query-age: Minimum time a query must be running before it becomes eligible for eviction (default 10s)

The evictor will be disabled and will not check every check-interval if both cpu and heap utilization are disabled (set to 0).

How it works
This feature is completely disabled and the registry will not be created if cpu-utilization and heap-utilization are set to 0. If either of them is larger:

Picked up queries will be registered to a registry to track all current queries in a querier
The evictor will check for utilization every check-interval
Once a threshold is breached, all currently running queries who have been evaluted for longer than min-query-age will be evaluated from heaviest based on eviction-metric. The heaviest query will be evicted
We will wait check-interval before checking again if threshold is breached.

Why the current metrics?
The current metrics are not the best to detect the root cause for high heap, however, they are readily available and can be used as a proxy until work in Prometheus/Thanos is done to allow for better metrics. peak_samples is currently only available in query_stats after the query is completed and we have no way of tracking current heap usage by a query. Any new metrics can easily be added to this structure later.

Metrics
cortex_query_evictions_total{resource="cpu|heap", component="querier"}: Counter increments by one for every eviction that occurs. A single query may lead to multiple increments if it retries and ends up evicted again.

note: make doc added the config to store-gateway.md file even though this is not currently implemented on there as it is built on top of resource based throttling.

Checklist

Tests updated
Documentation added
CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]
docs/configuration/v1-guarantees.md updated if this PR introduces experimental flags

Signed-off-by: Essam Eldaly <eeldaly@amazon.com>

friedrichg

Thanks for doing this!

friedrichg · 2026-05-08T17:35:44Z

 	} else {
 		// TODO: Consider wrapping logger to differentiate from querier module logger
-		queryable, _, queryEngine = querier.New(t.Cfg.Querier, t.OverridesConfig, t.Distributor, t.StoreQueryables, rulerRegisterer, util_log.Logger, t.OverridesConfig.RulesPartialData, nil)
+		queryable, _, queryEngine, _ = querier.New(t.Cfg.Querier, t.OverridesConfig, t.Distributor, t.StoreQueryables, rulerRegisterer, util_log.Logger, t.OverridesConfig.RulesPartialData, nil)


No support for rulers. I see. it can be added in a follow up PR.

Signed-off-by: Essam Eldaly <eeldaly@amazon.com>

Signed-off-by: Essam Eldaly <60357054+eeldaly@users.noreply.github.com>

Signed-off-by: Essam Eldaly <eeldaly@amazon.com>

SungJin1212

Thanks for the PR! I left a few comments.

SungJin1212 · 2026-05-18T12:10:37Z

+func NewQueryEvictor(
+	monitor resource.IMonitor,
+	registry *QueryRegistry,
+	cfg configs.EvictionConfig,
+	logger log.Logger,
+	reg prometheus.Registerer,
+	component string,
+) (*QueryEvictor, error) {
+	if !cfg.Enabled() {
+		return nil, nil
+	}
+
+	e := &QueryEvictor{
+		monitor:  monitor,
+		registry: registry,
+		cfg:      cfg,
+		logger:   logger,
+		evictionsTotal: promauto.With(reg).NewCounterVec(prometheus.CounterOpts{
+			Name:        "cortex_query_evictions_total",
+			Help:        "Total number of queries evicted due to resource pressure.",
+			ConstLabels: map[string]string{"component": component},
+		}, []string{"resource"}),
+	}
+
+	e.Service = services.NewBasicService(nil, e.running, nil)
+	return e, nil
+}


nit: NewQueryEvictor() currently appears to return nil error in all code paths.

SungJin1212 · 2026-05-18T12:16:49Z

+			if len(victims) == 0 {
+				continue // no running queries to evict
+			}
+


Would it help to emit a warning or debug log for threshold breached but no evictable query found?
That could make tuning min_query_age much easier in production.

Signed-off-by: Essam Eldaly <eeldaly@amazon.com>

Signed-off-by: Essam Eldaly <60357054+eeldaly@users.noreply.github.com>

justinjung04

Thanks! Few small comments

justinjung04 · 2026-05-21T05:08:15Z

+      # eviction. Supported values: fetched_samples, fetched_series,
+      # fetched_chunks, fetched_chunk_bytes.
+      # CLI flag: -querier.query-protection.eviction.eviction-metric
+      [eviction_metric: <string> | default = "fetched_samples"]


any particular reason you chose fetched_samples as default? is there some data you can share about each metric's correlation to the query heaviness?

Most of the queries I looked at that were causing heavy heap usage on queriers were low scrape interval with high time range. I believe samples is the best metric we currently have to detect both of those dimensions together.

In those queriers that are reaching high heap usage, we see that usually the querier pod is dominated by shards of one heavy query with all 3 of those metrics higher than other queries and any of them would correctly correlate.

Ideally in the future we have an upstream pr and have access to a metric that is more accurate and directly correlates with heap usage.

justinjung04 · 2026-05-21T05:14:31Z

+type ErrQueryEvicted struct{}
+
+func (e *ErrQueryEvicted) Error() string {
+	return status.Error(codes.ResourceExhausted, "resource limit reached").Error()


Could you verify if this triggers retry in query frontend? I belive we do not want evicted queries to be retried?

It does trigger retry. I believe its good to have something that is built for protecting queriers allow for retries as its purpose isnt to judge if a query is good or bad but to protect the querier pod. If an evicted query is never going to succeed, we should be changing our limits or coming up with new ones to cancel it early instead

Add Query Resource Based Eviction

37bb3b8

Signed-off-by: Essam Eldaly <eeldaly@amazon.com>

pull-request-size Bot added the size/XXL label May 7, 2026

dosubot Bot added component/querier type/feature labels May 7, 2026

eeldaly added 5 commits May 7, 2026 15:09

update changelog

f64ba8d

Signed-off-by: Essam Eldaly <eeldaly@amazon.com>

update guarantees

4938f3f

Signed-off-by: Essam Eldaly <eeldaly@amazon.com>

lint

cb9de7d

Signed-off-by: Essam Eldaly <eeldaly@amazon.com>

lint atomic

0a01fbc

Signed-off-by: Essam Eldaly <eeldaly@amazon.com>

lint modernize

1de552c

Signed-off-by: Essam Eldaly <eeldaly@amazon.com>

friedrichg reviewed May 8, 2026

View reviewed changes

eeldaly and others added 6 commits May 11, 2026 08:42

use configs.evictionConfig instead of copy

9bb0269

Signed-off-by: Essam Eldaly <eeldaly@amazon.com>

Panic on eivctor creation failures

5ddc12b

Signed-off-by: Essam Eldaly <eeldaly@amazon.com>

Add support for evicting multiple queries per cycle

63708b6

Signed-off-by: Essam Eldaly <eeldaly@amazon.com>

Doc gen

b4b217b

Signed-off-by: Essam Eldaly <eeldaly@amazon.com>

Merge branch 'master' into query-eviction

225bf62

Signed-off-by: Essam Eldaly <60357054+eeldaly@users.noreply.github.com>

lint

5a3e742

Signed-off-by: Essam Eldaly <eeldaly@amazon.com>

eeldaly requested a review from friedrichg May 11, 2026 18:54

friedrichg approved these changes May 14, 2026

View reviewed changes

dosubot Bot added the lgtm This PR has been approved by a maintainer label May 14, 2026

SungJin1212 reviewed May 18, 2026

View reviewed changes

eeldaly and others added 2 commits May 19, 2026 14:37

Add log. Remove unneeded err

9ea39e9

Signed-off-by: Essam Eldaly <eeldaly@amazon.com>

Merge branch 'master' into query-eviction

25a8665

Signed-off-by: Essam Eldaly <60357054+eeldaly@users.noreply.github.com>

eeldaly requested a review from SungJin1212 May 19, 2026 21:38

SungJin1212 approved these changes May 20, 2026

View reviewed changes

justinjung04 approved these changes May 21, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Query Resource Based Eviction#7488

Add Query Resource Based Eviction#7488
eeldaly wants to merge 14 commits into
cortexproject:masterfrom
eeldaly:query-eviction

eeldaly commented May 7, 2026 •

edited

Loading

Uh oh!

friedrichg left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

friedrichg May 8, 2026

Uh oh!

SungJin1212 left a comment

Uh oh!

SungJin1212 May 18, 2026

Uh oh!

SungJin1212 May 18, 2026

Uh oh!

Uh oh!

justinjung04 left a comment

Uh oh!

justinjung04 May 21, 2026

Uh oh!

eeldaly May 21, 2026

Uh oh!

justinjung04 May 21, 2026

Uh oh!

eeldaly May 21, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

eeldaly commented May 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

friedrichg left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

friedrichg May 8, 2026

Choose a reason for hiding this comment

Uh oh!

SungJin1212 left a comment

Choose a reason for hiding this comment

Uh oh!

SungJin1212 May 18, 2026

Choose a reason for hiding this comment

Uh oh!

SungJin1212 May 18, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

justinjung04 left a comment

Choose a reason for hiding this comment

Uh oh!

justinjung04 May 21, 2026

Choose a reason for hiding this comment

Uh oh!

eeldaly May 21, 2026

Choose a reason for hiding this comment

Uh oh!

justinjung04 May 21, 2026

Choose a reason for hiding this comment

Uh oh!

eeldaly May 21, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

eeldaly commented May 7, 2026 •

edited

Loading