Repository navigation
Conversation
bb5f578 to
8d1b7ef
Compare
|
Added @boblodgett as a reviewer since I'm curious whether the |
…hreads, profiling recipe - Geometric mean computed in log space. The direct product of the 71 per-case results overflows double once the geomean exceeds about 21,900 ops/CPU-sec, which turned the overall and per-protocol summaries into infinity. - --case-filter REGEX runs only the matching cases. Skipped cases return null from MeasureAsync; CpuTimeBenchmarks.cs and generate_serde_benchmarks.py carry the matching nullable result type. - --breakdown links a probe handler above the pipeline and below every handler through RuntimePipelineCustomizerRegistry, and reports per-stage ns/op and B/op in both directions, with the stub transport timed separately (MockHttpHandler stamps its own entry and exit). --breakdown-csv appends the rows to a CSV file. It requires --threads 1 and --transport stub: the probes record into static slots, and only the in-process stub stamps the transport stage. - --threads N runs N concurrent workers per case on the shared client and request, as a regression check for SDK state shared between calls. It is not the cross-SDK metric. Warmup is split across the workers, and the run stops at --min-iterations x N total ops or --min-cpu-seconds x N process CPU. The 12 cases whose shared request holds a MemoryStream are skipped when N > 1. --threads 1 (the default) runs the existing loop unchanged. - profiling/profile.sh and perf_report.py: a steady-state perf sampling profile of one case on Linux, keeping only the samples taken after tiered compilation finishes, with per-stage and per-thread summaries and collapsed stacks. Diagnostic only. - README documents the new options next to the --transport options from #4552.
8d1b7ef to
33780d0
Compare
@dscpinheiro other SDKs don't need to implement it. It's a .NET-only regression check for state shared between concurrent calls on one client (locks, caches, pools), and it's never part of the cross-SDK score; without it the runner behaves exactly as before. It's how we checked that the perf PRs don't add contention: at 16 threads, |
|
@boblodgett scored-path cost of the
−0.6%, inside the base runs' 1.7% spread, so no measurable cost (the change is two static-bool checks per call). Geomean of the 71 cases, stub, net10.0, 100k warmup, 3 interleaved rounds per side, dev host (not metal). |
Description
Test-only changes to the serde ops/CPU-sec runner. The SDK isn't touched.
doubleonce the geomean passes about 21,900 ops/CPU-sec and prints infinity. Both the overall and the per-protocol geomeans are now computed in log space, as ocs-sop.md (Geometric Mean Calculation) specifies. Below the overflow the value is unchanged.--case-filter REGEX: runs only the matching cases.--breakdown/--breakdown-csv: per-stage ns/op and B/op for each pipeline handler in both directions, with the stub transport timed separately. Requires--threads 1and--transport stub.--threads N: N concurrent workers per case on a shared client, as a regression check for state shared between calls. Not the cross-SDK metric; the default (1) runs the existing loop unchanged.profiling/profile.sh+perf_report.py: a steady-stateperfsampling profile of one case on Linux, keeping only samples after tiered compilation finishes. Diagnostic only.--transportoptions from Add a selectable HTTPS loopback transport to the ops/CPU-sec mode #4552. The profiling build recipe is tested on awsJson cases.With no new options, the measured loop is unchanged; the only addition on the default path is a disabled-probe check (one static bool) at entry and exit of
MockHttpHandler.SendAsync.Motivation and Context
The overflow made summaries unusable as results improved. The breakdown and profile show where per-call CPU goes, which is how the other PRs in this series were found and checked.
Testing
Built the runner; ran stub mode,
--breakdown(+CSV),--threads 2, and checked that each invalid combination exits 2 (--breakdown --transport https,--breakdown --threads 2,--threadswithout--cpu-time,--case-filter/--breakdown/--breakdown-csvwithout--cpu-time,--breakdown-csvwithout--breakdown, missing or invalid values for--case-filter/--threads/--breakdown-csv, and a--case-filterthat matches no case). The warmup default (100,000) is unchanged.Breaking Changes Assessment
None. Test code only.
Dry-runs
Test-only change under
sdk/test/Performance.e2c4cb0e-523d-48d5-88d5-183ac33bd05ffa1208b4-23b1-44de-b776-9014c72515e8Checklist
License