Issues

• SGLang provider cannot load OLMo 3 checkpoints saved by transformers 5: Olmo3Config has no max_position_embeddings

  allenai/olmo-eval#518 · Filter by author armancohan opened on Oct 28, 2026

• Don't score harness failures in SWE-bench Verified

  allenai/olmo-eval#487 · Filter by author gregorybchris opened on Oct 16, 2026

• Remove blocking HTTP execution from remaining sandbox command paths

  allenai/olmo-eval#483 · Filter by author abhishekraok opened on Oct 15, 2026

• Make the sandbox restart budget configurable or reset it after successful work

  allenai/olmo-eval#482 · Filter by author abhishekraok opened on Oct 15, 2026

• Align top-level DISPLAY_ONLY summaries with uploaded suite scores

  allenai/olmo-eval#481 · Filter by author abhishekraok opened on Oct 15, 2026

• Reconcile per-instance truncation and per-output cap-hit diagnostics

  allenai/olmo-eval#479 · Filter by author abhishekraok opened on Oct 15, 2026

• Count empty scoring text consistently across reasoning tasks

  allenai/olmo-eval#478 · Filter by author abhishekraok opened on Oct 15, 2026

• Truncated reasoning traces are scored: unclosed <think> output reaches the scorer

  allenai/olmo-eval#472 · Filter by author abhishekraok opened on Oct 13, 2026
  1

• ifeval_mt: letter_frequency items without let_relation are scored against a random relation

  allenai/olmo-eval#468 · Filter by author abhishekraok opened on Oct 13, 2026

• openai_agents scaffold: Qwen3.5 thinking mode ends agent runs after one tool call

  allenai/olmo-eval#461 · Filter by author gregorybchris opened on Oct 12, 2026

• LiveCodeBench: a stalled execution quarantines every podman sandbox and later samples are silently scored 0

  allenai/olmo-eval#457 · Filter by author abhishekraok opened on Oct 8, 2026
  1

• IFEval per-instance strict metrics are stored as loose in predictions

  allenai/olmo-eval#456 · Filter by author abhishekraok opened on Oct 8, 2026
  1

• Share the long-context task base, class factory and JSONL/hub helpers between RULER and HELMET

  allenai/olmo-eval#454 · Filter by author jopetty opened on Oct 6, 2026

• Nested suites are flattened to a plain task mean inside average-of-averages suites

  allenai/olmo-eval#448 · Filter by author pdasigi opened on Oct 4, 2026
  1

• CTC suite: measure parity against the reference grid

  allenai/olmo-eval#443 · Filter by author PrasannS opened on Oct 2, 2026

• olmo_core_vlm: refuse checkpoints with RoPE scaling (Molmo2-O-7B's YaRN gives a CUDA assert)

  allenai/olmo-eval#442 · Filter by author jason718 opened on Oct 2, 2026

• Qwen3-VL degenerates into repetition loops with transformers 5.17.0 (clean on 5.7.0)

  allenai/olmo-eval#441 · Filter by author jason718 opened on Oct 2, 2026
  1

• TaskConfig.get_data_source(split=None) should keep a DataSource's own split

  allenai/olmo-eval#425 · Filter by author PrasannS opened on Sep 28, 2026

• Synchronous scoring path aborts a whole task on one scorer exception

  allenai/olmo-eval#424 · Filter by author PrasannS opened on Sep 28, 2026

• LCB silent zero without sandbox: fail the task when its grader has no sandbox

  allenai/olmo-eval#423 · Filter by author abhishekraok opened on Sep 26, 2026
  1

• HF Molmo2 path fails on transformers 5.12: create_causal_mask() rejects input_embeds

  allenai/olmo-eval#416 · Filter by author jason718 opened on Sep 26, 2026
  1

• Pin Hub dataset revisions for the remaining vision tasks

  allenai/olmo-eval#415 · Filter by author jason718 opened on Sep 25, 2026

• Port mm_olmo's GroundingEvaluatorV5 and the balanced SA-Co v5 benchmark

  allenai/olmo-eval#402 · Filter by author jason718 opened on Sep 23, 2026
  1

• Reasoning-model responses truncated: 19-37% cap-hit on Qwen3.5, and ifeval_ood's 2048 budget never completes

  allenai/olmo-eval#399 · Filter by author abhishekraok opened on Sep 21, 2026
  1

• gpt-oss harmony format is scored verbatim: IFEval off by 45pp, IFBench by 21pp

  allenai/olmo-eval#398 · Filter by author abhishekraok opened on Sep 21, 2026