Skip to content

ORACLE-RESULTS.md

Latest commit

 

History

History
141 lines (104 loc) · 9.63 KB

ORACLE-RESULTS.md

File metadata and controls

141 lines (104 loc) · 9.63 KB
Raw

Oracle results: Unsga3 vs pymoo UNSGA3

Recorded 2026-08-10 (post DTLZ2 gap fix) · pymoo 0.6.2 · Unsga3 0.1.2.

Protocol

Knob Value
Algorithm U-NSGA-III / pymoo UNSGA3
SBX η=30, p_c=1, p_var=0.5
PM η=20, p_m=1/n
Refs Das–Dennis
IGD mean nearest Euclidean distance (pymoo IGD)
Duplicates eliminated (pymoo default)
Seeds 1 (table); DTLZ2 multi-seed below

Reproduce:

# Python
pip install pymoo
python tools/oracle/run_pymoo_oracle.py --problem zdt1 --partitions 12 --pop 52 --gens 100 --seed 1
python tools/oracle/run_pymoo_oracle.py --problem zdt2 --partitions 12 --pop 52 --seed 1
python tools/oracle/run_pymoo_oracle.py --problem dtlz2 --partitions 12 --pop 92 --gens 150 --seed 1
# DTLZ2 default is n_var=12 (k=10), matching Dtlz2Problem. pymoo's own default is n_var=10 (k=8).
# Pass --n-var 10 only to reproduce the historical mismatched column.

# C#
dotnet run --project tools/OracleCompare -c Release -- --problem zdt1 --partitions 12 --pop 52 --gens 100 --seed 1
dotnet run --project tools/OracleCompare -c Release -- --problem zdt2 --partitions 12 --pop 52 --seed 1 --pymoo-mode
dotnet run --project tools/OracleCompare -c Release -- --problem dtlz2 --partitions 12 --pop 92 --gens 150 --seed 1 --pymoo-mode

Omitted --gens on --problem zdt2 is 250 (quality protocol). --gens 100 is an early-stress snapshot. ZDT1 stays 100; DTLZ2 stays 150.

ZDT2 protocol honesty (matches unsga3-bend)

C# never published a hard ZDT2 oracle / Wilcoxon table. The unpublished Wilcoxon harness used gens=100 and RankNicheDistance. That budget collapses on Bend, C#, and pymoo (axis pile near f1≈0). gens=250 + PymooCompatible is the quality A/B bar (15-seed: 0/15 collapse on both stacks; see unsga3-bend PR #17 / docs/ZDT2_COLLAPSE.md). RankNicheDistance remains an optional Wilcoxon mating mode — do not silently switch all ZDT defaults to it. This repo does not invent a ZDT2 IGD table here.

Results (seed=1)

Problem Settings pymoo IGD C# default IGD C# PymooCompatible IGD Verdict
ZDT1 p=12, pop=52, 100 gen 0.0629 (res.F, n=13) 0.0514 (full ND front, n=52) — Different sets. Not an algorithm ranking.
DTLZ2 p=12, pop=92, 150 gen, mismatched k 0.00350 (n=91, pymoo n_var=10, k=8) 0.0070 (n=92, n_var=12) 0.00403 (n=92, n_var=12) Historical pair only. Not a same-problem ratio.

ZDT fronts (same run, different sets)

C# OracleCompare scores the full feasible non-dominated front (here n=52) against ParetoFronts.Zdt1(500). pymoo's oracle scores res.F, the survival niche set (here n=13, one per Das–Dennis direction), against pymoo's 100-point pareto_front(). The published 0.0514 vs 0.0629 pair is those two reporters. It is not evidence that the algorithm is better by ~0.011 IGD.

Seed 1 remeasured 2026-09-22, pymoo 0.6.2. The C# console reprinted the published scalar.

Set Reference front n IGD
C# non-dominated front library 500-point ZDT1 52 0.051430749249856716 (console 0.0514307)
C# non-dominated front pymoo 100-point PF 52 0.05119280568479224
pymoo final population, non-dominated pymoo 100-point PF 52 0.05378307132263516
pymoo res.F pymoo 100-point PF 13 0.0628633417931784
pymoo res.F library 500-point ZDT1 13 0.06276449352608372
pymoo population ND library 500-point ZDT1 52 0.05381261662420749

On the shared 100-point PF, the full-front pair is C# 0.05119280568479224 and pymoo 0.05378307132263516. One seed cannot carry a ranking. Switching the C# front from the 500-point sampler to that 100-point PF changes its IGD by 0.051430749249856716 − 0.05119280568479224 = 2.37943565064476×10⁻⁴, which is much smaller than the 13-versus-52 gap on pymoo's own PF (0.0628633417931784 − 0.05378307132263516 = 0.00908027047054324).

ReferenceDirectionThinning.OnePerDirection keeps the raw objective vector closest (perpendicular distance) to each Das–Dennis direction. On this C# front that helper kept 13 points and scored 0.06357076535717451 against the 100-point PF. That set is not res.F. The 15-seed pymoo column is still res.F, so its median ratio inherits the same asymmetry. Do not rewrite WILCOXON-RESULTS.md until those seeds are re-run on a shared front definition.

DTLZ2 multi-seed (C# PymooCompatible, same protocol)

Seed IGD
1 0.00403
2 0.00567
3 0.00513
4 0.00478
5 0.00466
mean ~0.00485

Those five C# seeds are Dtlz2Problem(k: 10) (n_var=12). The 0.0035 figure they were compared with is pymoo at n_var=10 (k=8). That is not a same-problem band. The current 15-seed file is the matched n_var=12 re-run (see below).

DTLZ2 n_var (known mismatch, seed 1 remeasured)

Dtlz2Problem(nObjectives: 3, k: 10) builds n = 12. Deb et al. suggest k = 10. pymoo 0.6.2 get_problem("dtlz2", n_obj=3) defaults to n_var=10 (k = 8). The harness used to omit n_var, so the historical seed-1 pair below compares those two dimensions. tools/oracle/run_pymoo_oracle.py and tools/oracle/run_multiseed_wilcoxon.py pass n_var=12. WILCOXON-RESULTS.md is that matched 15-seed re-run.

Published mismatched seed 1 (already in the Wilcoxon table; not re-interpreted as parity):

Solver n_var k IGD
C# PymooCompatible 12 10 0.00403168
pymoo default 10 8 0.00349879

Seed 1 remeasured 2026-09-22 with pymoo 0.6.2 after the oracle passes n_var=12. Console figures are the G6 print; the second number is the meta-file value. Front sizes are what each reporter wrote (res.F vs full non-dominated front).

Solver n_var k Console IGD Meta IGD Front
C# PymooCompatible 12 10 0.00403168 0.004031675764658275 92
pymoo n_var=12 12 10 0.00308392 0.003083921253245871 91

Ratio of the two meta IGDs: 0.004031675764658275 / 0.003083921253245871 = 1.30732. That is one seed, and the fronts still differ by one point (92 vs 91). Seed 1 of the matched 15-seed table is this console pair. The 15-seed median ratio is 1.16183 (WILCOXON-RESULTS.md).

Root cause of the old ~5× DTLZ2 gap (fixed)

Deep-dive vs pymoo HyperplaneNormalization / ReferenceDirectionSurvival (pymoo 0.6.2):

Bug Effect Fix
ASF weights inverted Extreme points landed on mid-edges (0,√½,√½) instead of axes (1,0,0) → wrong hyperplane intercepts → distorted niche association Preferred axis weight = 1, others = 1e-6 (divide form ≡ pymoo multiply form)
Ideal recomputed only on current pop Lost historical ideal Persistent ideal / worst across generations
Extremes from whole pop Contaminated by dominated points Extremes from ND front only + persist prior extremes
No duplicate elimination Extra clones, uneven niches eliminateDuplicates=true (default), decision-vector key

How we diagnosed it

  1. Front quality: C# was already on the unit sphere (|r−1|≈0.004) — convergence was fine.
  2. Diversity: pymoo had 0 empty niches / max 1 per niche; C# had empty niches + max_d ≈ 0.13 on PF.
  3. Synthetic ASF unit test: inverted weights selected mid-edge points; corrected weights match pymoo axis extremes.

Shipping bars (CI)

Test Bar
ZDT1 seed=1, 100 gen, default tournament IGD ≤ 1.5 × 0.0629
ZDT2 seed=2, 250 gen, default RankNicheDistance IGD < 0.75 (loose CI smoke, not oracle parity)
DTLZ2 seed=1, 150 gen, pymoo-mode IGD ≤ 2 × 0.00350. The 0.00350 scalar is the mismatched n_var=10 run. 3× still passed a regression to about 2.9×. This bar does not claim same-problem equivalence.
DTLZ2 short smoke (80 gen) IGD < 0.15

ZDT2 quality A/B is gens=250 + PymooCompatible (not the loose smoke bar). ZDT1 / DTLZ2 shipping bars are unchanged.

Known remaining deltas (intentional / minor)

Item Status
IGD mean-distance aligned
ASF / axis intercepts aligned
Collapsed nadir (span ≤ 1e-6) delta: nadir = ideal + 1 after the worst-of-pop fallback. pymoo 0.6.2 stops at worst-of-pop. Locked by Collapsed_span_sets_nadir_to_ideal_plus_one ({2, 2+1e-8} → nadir 3).
Infeasible points feasible hyperplane. Ideal, worst, and ASF use feasible objectives when any feasible member exists. An all-infeasible generation does not move the hyperplane. Survival niches the feasible set and fills a shortfall by ascending CV. The old whole-pool fixture (feasible (1, 1) vs infeasible (0, 0) → ideal (0, 0)) is replaced. OSY, TNK, and C1-DTLZ1 have self-tests. 15-seed IGD is in NEW-SURFACES-RESULTS.md.
Mating re-association delta: after survival, PrepareForSelection normalizes the survivors again and overwrites niche ids. pymoo keeps the survival ids. Fixture: PrepareForSelection_overwrites_survival_niche_ids.
Persistent ideal + ND extremes aligned
TournamentMode.PymooCompatible implemented
Duplicate elimination implemented (default on)
RNG path / batch niche pick order residual ~10–50% IGD noise (expected)
Multi-seed Mann–Whitney / Wilcoxon (n=15) done — WILCOXON-RESULTS.md
Exact bit-identical fronts not a goal (different RNG streams)