Support for running multiple YAML configuration files concurrently via multithreading

[20millot]

After reviewing a few related issues ( #608, #11 ), I noticed this question has come up before without a direct answer regarding configuration-level concurrency.
While a PR like #654 introduced multithreading, that implementation only parallelizes source and target queries within a single YAML file. The overall execution flow remains strictly sequential, running through multiple YAML configuration files one after another unless deployed via Kubernetes.
Given that running distinct YAML configurations appears well-suited for parallel execution, is there an architectural reason or underlying constraint preventing native multithreading across multiple YAML files?
This would be helpful to know before potentially attempting an implementation. Any context or advice on this matter would be greatly appreciated!

[hippi345]

The parallelism is process-level. With --kube-completions it reads JOB_COMPLETION_INDEX (or CLOUD_RUN_TASK_INDEX), sorts the yaml files and runs just the one at that index; without it, it loops. Keeping each validation in its own process keeps the DB connections and memory separate, which I'd guess is why there's no threading.

Those are just env vars though, so you can fan out locally:

DIR=/path/to/yamls/
N=$(ls "$DIR"/*.yaml | wc -l)
seq 0 $((N-1)) | xargs -P 4 -I{} env JOB_COMPLETION_INDEX={} JOB_COMPLETION_COUNT=$N \
  data-validation configs run -kc --config-dir "$DIR"

-P caps concurrency. The index has to stay below the file count or it raises.