EvalEngine as an EvalPort-compatible grader

[adhabnr-ux]

Hi — I maintain EvalPort, an open JSON Schema spec for portable LLM evaluation data — test suites, test cases, and result sets — with Python/TS SDKs that validate against the public schemas, meant to be a shared format across eval tools rather than tied to any one of them.

EvalEngine.evaluate(input, actual_output, **kwargs) -> (score, reason) (griptape/engines/eval/eval_engine.py) is about as close a match to an EvalPort grader as I've seen — an EvalPort grader result (schema: https://github.com/adhabnr-ux/evalport/blob/main/spec/schemas/grader.json) resolves to exactly {score, reason, passed} per test case, and your criteria/evaluation_steps LLM-as-judge approach is a legitimate implementation of an EvalPort grader, not just a lookalike.

The idea: a thin adapter that runs an EvalPort suite's test cases through EvalEngine, one call per test case, and writes the (score, reason) pairs straight into an EvalPort ResultSet (schema: https://github.com/adhabnr-ux/evalport/blob/main/spec/schemas/resultset.json):

score, reason = eval_engine.evaluate(input=tc["input"], actual_output=actual_output)
grader_result = {"grader_id": eval_engine.name, "type": "llm_judge", "score": score, "passed": score >= 0.7, "reason": reason}

That would let anyone with an EvalPort suite use Griptape's EvalEngine as a grader without writing framework-specific glue, and Griptape users could export their eval results in a format other EvalPort-speaking tools understand.

Per your CONTRIBUTING guide this is squarely a "have an idea" discussion rather than an issue — following your "Griptape Extensions" pattern, I'd expect this to start life as a standalone extension (griptape-evalport or similar) rather than upstream code, so mostly wanted to float the idea and see if it's something the extension pattern is meant for, or if there's already prior art here I'm missing. No pressure — happy to just leave this as a note. Thanks for EvalEngine, it's a clean, minimal design.

Full spec, for reference: https://github.com/adhabnr-ux/evalport/blob/main/SPEC.md