@dkorchevgithub adding lines suggested by @EtoDemerzel0427 to fix "g… - #13

Open

dkorchevgithub wants to merge 796 commits into

dkorchevgithub:masterfrom

mlcommons:master

This pull request is big! We're only showing the most recent 250 commits

Commits on Jun 24, 2025

[DLRMv2] Updating DLRMv2 dataset size (#2222)
keithachorn-intel
authored
064db01
Remove incorrect MLC Commands (#2218)
anandhu-eng
authored
e983036

Commits on Jun 25, 2025

Fix Typo in Interactive Latencies (#2147)

authored
0a3570e

Commits on Jul 2, 2025

Host speech2text artifacts in MLC storage bucket (#2223)
pgmpablo157321
authored
fdac4dc

Commits on Jul 7, 2025

Add missing interactive configurations (#2224)

pgmpablo157321
and
mrmhodak
authored
62a0dfd
Increment version to 5.0.23
mrmhodak
committed
20628f0

Commits on Jul 8, 2025

Update compliance table (#2243)
pgmpablo157321
authored
6a5fcc5
Add deepseek dataset sources (#2242)
pgmpablo157321
authored
7ef8b14
Adding mlperf.conf setting for Whisper (#2238)

keithachorn-intel
and
arjunsuresh
authored
b25eac3
Increment version to 5.0.24
mrmhodak
committed
745b9fe
Fix Docs (#2229)

authored
7482011
Incorrect Regex for RougeLSum (#2230)
hvagadia
authored
ae1320c
update eval_accuracy.py and deepseek thresholds (#2233)
viraatc
authored
24767db
Add llama3.1-8b-edge as a separated benchmark (#2231)
pgmpablo157321
authored
e05fda5
Increment version to 5.0.25
arjunsuresh
committed
1bc3e99

Commits on Jul 15, 2025

accuracy (#2259)
wu6u3tw
authored
bb0e01a
fix regex (#2260)
wu6u3tw
authored
bcb600e

Commits on Jul 16, 2025

Update download path for llama3.1_8b dataset (#2261)
pgmpablo157321
authored
7d86e6b
Update README.md (#2255)

keithachorn-intel
and
hanyunfan
authored
5232291

Commits on Jul 18, 2025

Update version generate_final_report.py (#2269)
pgmpablo157321
authored
1446b35

Commits on Jul 21, 2025

Add interactive scenario in the TEST06, bump loadgen version to 5.1 (#2272)
nvzhihanj
authored
c57507b

Commits on Jul 22, 2025

Commits on Jul 23, 2025

Update documentation (#2279)
anandhu-eng
authored
35d9836
Address issue that logger.info not captured by stdout; remove redundant logging (#2278)

nvzhihanj
and
arjunsuresh
authored
50de991

Commits on Jul 25, 2025

fix: update ds-r1 truncate max-output-len to 20k (was 32k) (#2290)
viraatc
authored
b9ed3c7

Commits on Jul 29, 2025

Fix llama3.1-8b metric and dataset (#2300)
pgmpablo157321
authored
c54683d
Add interactive scenario to llama3.1 models (#2299)
pgmpablo157321
authored
c308e42
Allow more flexible datatypes in measurements file (#2298)

pgmpablo157321
and
hanyunfan
authored
4a01733
Update evaluation.py (#2303)

taran2210
and
arjunsuresh
authored
33a0c34
Only require Server or Interactive for closed (#2304)

pgmpablo157321
and
arjunsuresh
authored
01bc11a

Commits on Jul 30, 2025

Commits on Jul 31, 2025

Only check for token latency requirements for server scenario (#2313)
pgmpablo157321
authored
47b91b9

Commits on Aug 1, 2025

Use server SUT for SingleStream (#2314)
pgmpablo157321
authored
6ad8e26

Commits on Aug 4, 2025

Update the default value for repository arg (#2317)
anandhu-eng
authored
1b64700

Commits on Aug 6, 2025

Commits on Aug 7, 2025

Allow lowercase 'interactive' as scenario name (#2315)
psyhtest
authored
70d58ad

Commits on Aug 20, 2025

Commits on Sep 1, 2025

hide long time untested implementations from docs (#2328)
anandhu-eng
authored
65e3228

Commits on Sep 10, 2025

Initial draft for SCC 25 documentation (#2331)
anandhu-eng
authored
46beea8
fix for fstring (#2332)
anandhu-eng
authored
75a2012

Commits on Sep 11, 2025

Updation of automation run commands - v5.1_dev (#2333)
anandhu-eng
authored
938be6d

Commits on Sep 14, 2025

Fixes for docs (#2334)
anandhu-eng
authored
2035f3f

Commits on Sep 16, 2025

Update submission_checker.py | Fixes 2325 (#2326)

arjunsuresh
and
hanyunfan
authored
5fbf01a

Commits on Sep 18, 2025

Fixes for scc doc (#2339)
anandhu-eng
authored
a6bd784
Update link to AMD readme for SCC (#2340)
anandhu-eng
authored
1d6b4c3

Commits on Sep 24, 2025

Add additional information for dataset and model downloads (#2343)
anandhu-eng
authored
8c07d7c

Commits on Sep 29, 2025

Provide correct syntax of how batch size is provided for nvidia implementation (#2335)
anandhu-eng
authored
7c8c84d

Commits on Sep 30, 2025

Fix lookup of required accuracy delta. (#2337)
psyhtest
authored
b15073f

Commits on Oct 6, 2025

Commits on Oct 7, 2025

Automation Docs: Add information about launching docker in privileged… (#2352)

anandhu-eng
and
arjunsuresh
authored
dc8ca5b
Remove Nvidia folder from compliance tree (#2354)
pgmpablo157321
authored
d9efabf
Provide help - insufficient max locked memory error for Nvidia runs (#2355)

anandhu-eng
and
mrmhodak
authored
f4d495b
Update SUT_VLLM.py (#2349)

viraatc
and
mrmhodak
authored
fbed09d
Fix minor typos in reference_mlperf_perf.sh and reference_mlperf_accuracy.sh (#2327)

authored
eee6352

Commits on Oct 13, 2025

[LoadGen] Time to Output Token -> Time per Output Token (#2360)
wangshangsam
authored
11e469d
Increment version to 5.1.1
hanyunfan
committed
0e556f1
[Whisper] Add labels' in the whisper output (#2252)

wu6u3tw
and
hanyunfan
authored
0b8ca03

Commits on Oct 20, 2025

Commits on Oct 21, 2025

Replaced shell commands with Python for Windows compliance script compatibility (#2344)

authored
6776245

Commits on Nov 11, 2025

[VLM] Offline scenario, performance-only mode of the reference implementation (#2381)

wangshangsam
and
github-actions[bot]
authored
808e2d7

Commits on Nov 15, 2025

update docs with submission instruction + hyperlinks for heading (#2390)
anandhu-eng
authored
ceafe86

Commits on Nov 18, 2025

[VLM] Server Scenario, performance and accuracy (#2388)
johncalesp
authored
32b6d66

Commits on Nov 25, 2025

Submission dir update (#2366)

pgmpablo157321
and
anandhu-eng
authored
ee809ab

Commits on Nov 26, 2025

Add note on LoadGen PyPI auto-build workflow (#2398)
anandhu-eng
authored
59c59dd
Increment version to 5.1.2
arjunsuresh
committed
06c742a
Revert changes used for testing (#2401)
anandhu-eng
authored
353bfb8

Commits on Dec 2, 2025

[VLM] Accuracy Evaluation (#2393)

authored
fff953c

Commits on Dec 9, 2025

Commits on Dec 18, 2025

Update loadgen to 6.0; Bulk update 6.0 checker bits (#2415)
nvzhihanj
authored
003f4fb

Commits on Dec 19, 2025

feat: add MTP to ds-r1 ref. impl (#2403)
viraatc
authored
c098f80
Increment version to 6.0.1
arjunsuresh
committed
abed98c

Commits on Dec 22, 2025

Commits on Dec 23, 2025

Commits on Jan 6, 2026

[VLM] Add a CLI plugin system for `mlperf-inf-mm-q3vl benchmark` (#2420)

wangshangsam
and
github-actions[bot]
authored
f690570
[DLRMv3] Reference Implementation (#2410)
LinjianMa
authored
7a9fc82
Increment version to 6.0.4
arjunsuresh
committed
9374959
[GPT-OSS-120B] Reference implementation (#2395)

authored
5523589

Commits on Jan 7, 2026

Commits on Jan 9, 2026

Remove Rclone download instructions for datasets supported by R2 Downloader (#2358)

authored
dd0b647

Commits on Jan 12, 2026

YOLO dev branch (#2382)

authored
4d88832
Add SingleStream text2video loadgen integration (#2427)

authored
ef20641
add compliance check 07 (#2425)

viraatc
and
hanyunfan
authored
3cbcc3e
Add support for endpoints logs (#2431)

authored
26864ba
Folder structure fixes to checker and samples (#2433)

pgmpablo157321
and
hanyunfan
authored
9b77f35

Commits on Jan 13, 2026

Fix submission_checker: accept --skip-dataset-size-check (#2438)
tanvi-mlcommons
authored
6f86c96
Update seeds for inference v6.0 (#2437)
pgmpablo157321
authored
f131a0d
Increment version to 6.0.5
arjunsuresh
committed
9142ef0

Commits on Jan 14, 2026

Commits on Jan 15, 2026

Moving sys import (#2447)
keithachorn-intel
authored
fa33fa8

Commits on Jan 16, 2026

Add YOLO accuracy targets (#2451)
anandhu-eng
authored
7527760

Commits on Jan 20, 2026

Add accuracy_sample_count (#2414)

authored
a8d4d78
Increment version to 6.0.6
pgmpablo157321
committed
65ef999

Commits on Jan 21, 2026

Change dataset access link to new URL (#2458)
v-shobhit
authored
1ee18b2

Commits on Jan 22, 2026

MLCFlow automation: Add model and dataset download commands (#2455)

anandhu-eng
and
arjunsuresh
authored
948e534
Update v6.0 benchmark table (#2460)

pgmpablo157321
and
hanyunfan
authored
fb1d0c0

Commits on Jan 27, 2026

Temporary turn off retinanet GitHub action (#2467)
anandhu-eng
authored
3586bd1
[DLRMv3] Add compliance test (#2453)

LinjianMa
and
hanyunfan
authored
37c4ac0
Fix PyTorch version issue for Llama2-70b and Mixtral-8x7b (#2466)

ssam18
and
arjunsuresh
authored
02a2dbb
Add yolo run commands to inference docs (#2461)
anandhu-eng
authored
9abe5c0
Quick fix: Add user conf to yolo reference implementation (#2446)

authored
b233567
Increment version to 6.0.7
arjunsuresh
committed
0c1353f

Commits on Jan 28, 2026

Add token latency tracking for gpt-oss-120b-interactive (#2470)
v-shobhit
authored
b3002b8
Increment version to 6.0.8
mrmhodak
committed
236aa64
Update submission checker (#2469)

pgmpablo157321
and
hanyunfan
authored
cff5f1b
Fixed fps in Wan2.2-T2V-A14B inference script (#2441)

authored
bb03736
Add compliance tests for Text-to-Video (#2456)

authored
1c84e1b
Proposal T2V: Save videos as videos instead of numpy arrays (#2459)

authored
7d040db
Increment version to 6.0.9
mrmhodak
committed
6520e39
Added Wan2.2-T2V to submission checker (#2457)

authored
7373829
Update README.md (#2473)
TheKanter
authored
a4e012a

Commits on Jan 29, 2026

submission checker updates for gpt-0ss (#2454)
viraatc
authored
053dba7
GPT-OSS-120B accuracy checker fixes (#2476)

authored
b84c0d0
Fix import path for TEST-01 (#2478)

anandhu-eng
and
pgmpablo157321
authored
5133d3c
Add dlrm-v3 support to submission checker (#2477)

zihaok
and
pgmpablo157321
authored
79b4d77
Fixing submission checker and compliance for yolo (#2474)

authored
4757cd1
[PATCH][Submission][VLM] Fix submission for VLMs (#2475)

johncalesp
and
github-actions[bot]
authored
1d1c6ea

Commits on Feb 2, 2026

fix: gpt-oss TEST09 sample-count (#2479)
viraatc
authored
3f25191
Fix typo in text-to-video model name (#2471)

authored
6bb0d6c

Commits on Feb 3, 2026

fix test09 (#2481)
viraatc
authored
986d8b6
Change text to video folder name (#2485)
pgmpablo157321
authored
6c6bbed
[Wan22] update complience test04 num to reduce the runtime (#2480)

wu6u3tw
and
hanyunfan
authored
a671384

Commits on Feb 5, 2026

Update README.md (#2491)
viraatc
authored
f6b0fbe
Push submission checker fix for GPT-OSS-120b (#2492)

authored
52235a4

Commits on Feb 6, 2026

update submodule path (#2496)
wu6u3tw
authored
4b4a3ff
Update README.md (#2500)
TheKanter
authored
3cf21b2
Add missing fields for gpt-oss-120b-interactive (#2499)

v-shobhit
and
hanyunfan
authored
80aaa82
Increment version to 6.0.10
hanyunfan
committed
0b4aa6f
Minor T2V bugfixes (#2495)

Akshat-Tripathi
and
hanyunfan
authored
1e04ea4

Commits on Feb 10, 2026

Update columns of final report (#2526)
pgmpablo157321
authored
d810a68
Quick fix: Capitalize scenarios in final report (#2528)
pgmpablo157321
authored
e4c303e
Revert TEST01 for Text-to-video benchmark (#2529)

Akshat-Tripathi
and
hanyunfan
authored
d4513ee

Commits on Feb 11, 2026

Urgent: Fix for compliance tests(old submission checker) (#2532)
anandhu-eng
authored
73d7e4b
update for submission checker (#2531)
wu6u3tw
authored
9e3d105
Quick fix: map scenario correctly (#2537)
pgmpablo157321
authored
2f836ea

Commits on Feb 12, 2026

Add scenario check to submission checker (#2538)
pgmpablo157321
authored
104d485
Quick fix for report script (#2539)
pgmpablo157321
authored
d33bb4d

Commits on Feb 13, 2026

Add YOLO models to final report generation (#2540)
arjunsuresh
authored
d757424

Commits on Feb 17, 2026

Activate compliance check + fix incorrect errors (#2542)
pgmpablo157321
authored
b795264

Commits on Feb 19, 2026

Compliance check fix (#2544)

pgmpablo157321
and
github-actions[bot]
authored
5a03e7a

Commits on Feb 20, 2026

Quick fix for summary table (#2545)
pgmpablo157321
authored
9d8d499

Commits on Mar 1, 2026

Fix hyperlink generation in final report(code -> src) (#2557)
anandhu-eng
authored
d7cde22

Commits on Mar 4, 2026

Fix report metrics (#2551)

authored
394195a

Commits on Mar 6, 2026

Update README.md for YOLOv11l (#2552)
manpreetssokhi
authored
a0c968e

Commits on Mar 10, 2026

Enable Bert GH action (#2550)
anandhu-eng
authored
cde9790

Commits on Mar 17, 2026

Add total accelerators to summary (#2561)
pgmpablo157321
authored
8f77c6e
Update Llama3.1b interactive unit (#2564)

tanvi-mlcommons
and
pgmpablo157321
authored
6d0cf9d

Commits on Mar 27, 2026

Commits on Mar 30, 2026

Update compliance/TEST04/verify_performance.py to use context-manager (#2510)

authored
fba0a0f
Update language/gpt-j/utils.py to use context manager (#2511)

authored
f8cc184
Remove Python2 Future Annotations (#2512)

authored
c70a7ba
Increment version to 6.0.13
pgmpablo157321
committed
bdc87e2
Remove use of eval in vision/classification_and_detection/tools/calibrate_torchvision_model.py (#2514)

authored
6292e4c
Remove use of eval in language/mixtral-8x7b/evaluate_mbxp.py (#2515)

authored
c94154a
Update text_to_image/tools/coco_generate_calibration.py to use pathlib and shutil (#2517)

authored
20d318e
Remove malformed string literal in repository check (#2519)

authored
3a3111a
Specify wildcard imports in medical imaging task (#2521)

arav-agarwal2
and
pgmpablo157321
authored
9d135bc
Update wildcard import for classification task code. (#2523)

arav-agarwal2
and
pgmpablo157321
authored
f065cb6
Update retired benchmarks wildcard imports (#2524)

authored
144c395
Update .gitignore (#2488)

authored
36c8d3a
Fix DLRMv3 fbgemm_gpu loading with hardcoded Bazel paths (#2439)

authored
2ee7190

Commits on Mar 31, 2026

[Wan22] update video idx (#2530)

authored
175ea7b

Commits on Apr 10, 2026

Initial shellcheck pass (#2568)
arav-agarwal2
authored
824914b
Increment version to 6.0.14
pgmpablo157321
committed
534cbfd

Commits on Apr 22, 2026

Initial gitleaks implementation (#2580)

arav-agarwal2
and
github-actions[bot]
authored
3ecc2e0

Commits on May 4, 2026

Create auto-close-old-issues-prs.yml (#2571)

hanyunfan
and
anandhu-eng
authored
7b11eeb

Commits on May 13, 2026

Commits on May 20, 2026

Initial pre-commit for formatting (#2589)
arav-agarwal2
authored
7305637
Use GitLeaks CLI for scanning (#2590)

arav-agarwal2
and
github-actions[bot]
authored
e59ce58

Commits on Jun 2, 2026

Update VLM Reamde for v6.1 Specifications (#2591)

Victor49152
and
wangshangsam
authored
bdc4116

Commits on Jun 16, 2026

[VLM]Update Calibration Dataset in V6.1 (#2600)

Victor49152
and
wangshangsam
authored
7a7db59

Commits on Jun 23, 2026

Add private ids to checker and final report (#2592)

pgmpablo157321
and
hanyunfan
authored
c0a402f

Commits on Jun 24, 2026

Add nameplate power YAML check for v6.1+ power submissions (#2606)

authored
f96a9af

Commits on Jun 25, 2026

E2E RAG reference implementation (#2602)

authored
564cec7
Increment version to 6.0.15
pgmpablo157321
committed
3546c66

Commits on Jul 1, 2026

Loadgen++/endpoints integration with submission checker (#2609)

authored
d1515ab
Add constants for v6.1 (#2610)

anandhu-eng
and
pgmpablo157321
authored
4b790a6
feat(edge-agentic): add BFCL v4 edge-agentic reference implementation… (#2598)

Palanivelg
and
pgmpablo157321
authored
553b252

Commits on Jul 2, 2026

Commits on Jul 7, 2026

Commits on Jul 8, 2026

[e2e-rag] Fix error when running retrieval with reranker (#2621)
fadara01
authored
393d8ef
Set power YAML template as optional (#2615)

authored
37b3811

Commits on Jul 10, 2026

Commits on Jul 17, 2026

[Wan-2.2 T2V] Update sample IDs to use random_ids.txt (#2632)
wu6u3tw
authored
da738a5

Commits on Jul 20, 2026

docs(edge-agentic): fix endpoints example link (10->11) and add calibration dataset reference (#2635)
Palanivelg
authored
3695ef0

Commits on Jul 21, 2026

Update endpoints structure (#2630)

authored
2af078b
Fix DeepSeek-R1 R2 downloader link to point at correct checkpoint (#2638)

authored
c998c0e

Commits on Jul 23, 2026

E2e rag robustness fixes (#2625)

authored
2fe63e0
Update submission_structure.md (#2639)
hanyunfan
authored
63220f3
Submission checker: support e2e-rag-qna and e2e-rag-db (#2640)

authored
d8260b2

Commits on Jul 24, 2026

Add pre-commit setup documentation (#2597)

authored
da26cb1
Increment version to 6.0.17
arav-agarwal2
committed
d6147c7

Commits on Jul 27, 2026

Update the default resolved value of sampling parameter (#2644)
Victor49152
authored
81ebebe
Initial ScanOSS Licensing Checker (#2583)

authored
279ea35
Initial PR with codecov report logging. (#2581)

authored
49469eb

Commits on Jul 28, 2026

Move autopep8 formatting from happening on the PR to being checked on the repo. (#2643)

arav-agarwal2
and
hanyunfan
authored
cd92cfe

Commits on Jul 29, 2026

Commits on Jul 30, 2026

Add MLPerf Inference v6.1 benchmarks table (#2648)
pgmpablo157321
authored
a468775

Commits on Aug 12, 2026

Harden COCO dataset bootstrap: HTTPS + SHA-256 verification (#2572)

kiwaku
and
hanyunfan
authored
af8f224

Commits on Aug 13, 2026

Remove llama2-70b-99 (#2651)
pgmpablo157321
authored
44cc3a3
Fix Qwen3-VL latency constrain (#2650)

pgmpablo157321
and
hanyunfan
authored
985f2d1
Fix metric overwrite for endpoints submissions (#2649)

pgmpablo157321
and
hanyunfan
authored
1b9bc00

Commits on Aug 18, 2026

Update display metric for E2E-QNA (#2653)
pgmpablo157321
authored
d2e600b

Commits on Aug 19, 2026

Update submission checker to handle interactive constraints of q3vl from endpoints (#2652)

Victor49152
and
pgmpablo157321
authored
46e6a0b

Commits on Aug 20, 2026

Require only one of server or interactive (#2654)
pgmpablo157321
authored
b66003e

Commits on Aug 24, 2026

Quick fix for scenario mapping (#2655)
pgmpablo157321
authored
ced6a9f

Commits on Aug 26, 2026

Quick fix: Add interactive to metric overwrite (#2656)
pgmpablo157321
authored
cfb0df1

Commits on Sep 2, 2026

Quick fix: add edge qwen 27b in final report (#2659)
pgmpablo157321
authored
a16585a
Fix (#2660)

authored
ecef039
Sync Docs branch with master (#2657)

authored
f98c6a7
Update model names in report (#2661)
pgmpablo157321
authored
3fbc329