Add fallback-mode option for subset#1308
Open
takanabe wants to merge 1 commit into
Open
Conversation
When the subset API errors or returns isBrainless=true, users can now opt into 'stop' (exit non-zero to halt CI) or 'random-sample' (split locally using --target) instead of the default run-all behavior. The option is hidden and off by default so existing pipelines are unaffected.
Konboi
reviewed
May 29, 2026
| elif self.fallback_mode == FallbackMode.RANDOM_SAMPLE: | ||
| target_fraction = float(self.target) if self.target is not None else 1.0 | ||
| click.echo( | ||
| "Warning: Smart Tests could not retrieve a subset. Falling back to local random sample.", |
Contributor
There was a problem hiding this comment.
Since users may use options other than --target e.g.) --confidence, it might be helpful to include the target percentage in the warning message as well.
Or, if --fallback_mode is specified and the user is using an option other than --target, we could return an error in the validation arera
| sampled = random.sample(test_paths, min(count, len(test_paths))) | ||
| sampled_set = {id(t): t for t in sampled} | ||
| rest = [t for t in test_paths if id(t) not in sampled_set] | ||
| return cls(subset=sampled, rest=rest, subset_id='', summary={}, is_brainless=False, is_observation=False) |
Contributor
There was a problem hiding this comment.
Is is_brainless=False intentionally?
| result = self.cli(*self._subset_args(rest_file.name, ("--fallback-mode", "random-sample")), mix_stderr=False) | ||
| self.assert_success(result) | ||
| all_tests = result.stdout.strip().split("\n") + Path(rest_file.name).read_text().strip().split("\n") | ||
| all_tests = [t for t in all_tests if t] |
| # TODO(Konboi): split subset isn't provided for smart-tests initial release | ||
| # if split: | ||
| # click.echo("subset/{}".format(subset_result.subset_id)) | ||
| if subset_result.is_brainless: |
Contributor
There was a problem hiding this comment.
In Brainless mode, we're already splitting requests between subset and rest on the server side.
I can understand the stop behavior, but for sampling, do you mean re-sampling the tests on the CLI side?
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Background
When the subset API errors or returns isBrainless=true, users can now opt into 'stop' (exit non-zero to halt CI) or 'random-sample' (split locally using --target) instead of the default run-all behavior. The option is hidden and off by default so existing pipelines are unaffected.
Changes
--fallback-modeoption for subset command