Add num_threads config to cap CPU inference threads and disable pinning - #43
Open
anaghdin wants to merge 2 commits into
Open
Add num_threads config to cap CPU inference threads and disable pinning#43anaghdin wants to merge 2 commits into
anaghdin wants to merge 2 commits into
Conversation
Introduce a NUM_THREADS SDK config key that lets callers cap the OpenVINO CPU inference thread pool per instance (ov::inference_num_threads). When num_threads > 0, also disable OpenVINO CPU pinning (ov::hint::enable_cpu_pinning(false)). Rationale: with inference_num_threads set, the CPU plugin enables thread pinning by default and deterministically binds every process to the same core block. Independent concurrent encodes then all collide on those few cores while the rest of the machine sits idle. Disabling pinning lets the OS scheduler spread jobs across the machine. In a 40-job sweep this raised throughput from ~258 to ~782 FPS. A value of 0 (default) preserves the prior uncapped OpenVINO behavior. Changes: - ivsr_sdk: add NUM_THREADS key, parse it in ivsr_init, and thread it through parse_engine_config to emit inference_num_threads + enable_cpu_pinning(false). - ivsr_ffmpeg_plugin: add ffmpeg patch 0004 exposing a num_threads option in the ivsr DNN backend, plumbed to the SDK as NUM_THREADS.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Introduce a NUM_THREADS SDK config key that lets callers cap the OpenVINO CPU inference thread pool per instance (ov::inference_num_threads). When num_threads > 0, also disable OpenVINO CPU pinning (ov::hint::enable_cpu_pinning(false)).
Rationale: with inference_num_threads set, the CPU plugin enables thread pinning by default and deterministically binds every process to the same core block. Independent concurrent encodes then all collide on those few cores while the rest of the machine sits idle. Disabling pinning lets the OS scheduler spread jobs across the machine. In a 40-job sweep this raised throughput from ~258 to ~782 FPS. A value of 0 (default) preserves the prior uncapped OpenVINO behavior.
Changes: