Add low-precision floating point data types explainer

[mklimenko-nv]

This PR is a first step to add low-precision floating point data types to the WebNN specification. It's a follow-up to #930 with a conservative addition of bfloat16 and float8 types only, with a potential follow-up for microscaled data types.

[fdwr]

Tis a nice document, thanks for preparing, and I support adding low precision data types (I may just have "are we sure xyz" questions 🙂).

[fdwr]

How certain are we that this specific float8 is the float8 bit allocation that we want to codify into the API without any clarifying suffix? Will we need another float8 in the future, one that would need a distinguishing suffix (making it odd to have float8 and float8suffixed)? After all, we already determined that IEEE float16 (float16m10e5s1) was insufficient for all models, warranting float16m7e8s1.

Just to double check, the flavor of float8 proposed is this one ⭐, being IEEE-compliant, with full infinity, NaN, and distinct positive/negative zero?

float8m3e4s1_t ⭐     { uint8_t mantissa: 3; uint8_t exponent: 4; uint8_t sign: 1;} // infinities, NaNs
float8m2e5s1_t        { uint8_t mantissa: 2; uint8_t exponent: 5; uint8_t sign: 1;} // infinities, NaNs
float8m3e4s1fnuz_t    { uint8_t mantissa: 3; uint8_t exponent: 4; uint8_t sign: 1;} // no infinities, NaN as -0
float8m2e5s1fnuz_t    { uint8_t mantissa: 2; uint8_t exponent: 5; uint8_t sign: 1;} // no infinities, NaN as -0
float8m4e3s1_t        { uint8_t mantissa: 4; uint8_t exponent: 3; uint8_t sign: 1;} // infinities, NaN's
float8m0e8s0fn_t      { uint8_t mantissa: 8; uint8_t exponent: 0; uint8_t sign: 0;} // no infinities, NaN via all ones (purely range scaling)
float8m3e4b11s1fnuz_t { uint8_t mantissa: 3; uint8_t exponent: 4; uint8_t sign: 1;} // exp bias 11, no infinities, NaN as -0
float8m3e4s1fn_t      { uint8_t mantissa: 3; uint8_t exponent: 4; uint8_t sign: 1;} // infinities, NaNs

[mklimenko-nv]

I agree with the float16 vs bfloat16 use cases, but with float8 I find it important to maintain a balance between being explicit and user-friendly. We see that the e4m3 variant is used primarily in inference with e5m2 being more popular in training. For the sake of completeness, it is possible to specify the suffix upfront, I'll update the explainer.

On the _fnuz subvariants and other flavors my personal stance is that they're vendor-specific and would overcomplicate the specification.

[fdwr]

with float8 I find it important to maintain a balance between being explicit and user-friendly

Yeah, I'm okay with no suffix (I don't want web devs to have to type out cryptic suffixes either :b), but I'm just double checking the confidence percentage that this float8 is the float8, and if the answer is "yes, because this is what most current/upcoming hardware prefers, and this is what the models need for inference", then so be it. All the ML libraries I surveyed showed support for that type (along with others).

On the _fnuz subvariants and other flavors my personal stance...

I too would prefer the more IEEE-like float8, consistent with WebNN's existing float16 and float32 (which have infinity, NaN, and dual-sign zero), and though there's no official IEEE float8, m3e4s1 is one of the first examples shown here.

[fdwr]

dequantizeLinear already supports blockwise scales? See emulated decomposition:

function dequantizeLinear(builder, input, scale, zeroPoint, options)
{
    // output = (input - zeroPoint) * scale
    const floatInput = builder.cast(input, scale.dataType);
    const floatZeroPoint = builder.cast(zeroPoint, scale.dataType);
    const upsampledScale = blockwiseExpand(builder, scale, input.shape);
    const upsampledZeroPoint = blockwiseExpand(builder, floatZeroPoint, input.shape);
    return builder.mul(builder.sub(floatInput, upsampledZeroPoint), upsampledScale);
}

I think we had a Phi-3 WebNN demo running some time ago that used block scales (though, it's now Phi 4, and I'm not sure if it's been kept up to date with the API) .

If this was unclear from the documentation, we should clarify it.

[fdwr]

Aha, supporting links:

• Support block-wise quantization #779
• https://chromium-review.googlesource.com/c/chromium/src/+/5964816

[mklimenko-nv]

I missed it, thanks for pointing out

[fdwr]

Might some wording like this help?

https://github.com/webmachinelearning/webnn/compare/main...fdwr:webnn:user/dwayner/dequantizeLinearClarifyBlockwiseNature?expand=1

[anssiko]

I observe general support and no concerns, see also meeting minutes.

@mklimenko-nv, any pending changes you'd still like to integrate into this v1 PR? Otherwise, editors are free to merge this by our next meeting.

[mklimenko-nv]

@anssiko, I've made some minor modifications to the explainer based on the discussion above (thanks @fdwr!).

[anssiko]

@mklimenko-nv thank you, it looks like all feedback has been addressed. The editor are free to merge this explainer PR.

@mklimenko-nv are you interested in taking the first stab at the spec PR for this?

[mklimenko-nv]

@anssiko, yeah, sounds good to me, let me prepare a spec PR for this. Thanks!