Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion Directory.Build.props
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@
<Copyright>Copyright © Bruno Capuano $([System.DateTime]::Now.Year)</Copyright>
<PackageLicenseExpression>MIT</PackageLicenseExpression>
<PackageIcon>nuget_logo.png</PackageIcon>
<PublishedSiblingPackageVersion>0.22.0</PublishedSiblingPackageVersion>
<PublishedSiblingPackageVersion>0.23.0</PublishedSiblingPackageVersion>
<ManagePublishedSiblingPackageVersion
Condition="'$(MSBuildProjectName)' == 'ElBruno.LocalLLMs'
Or '$(MSBuildProjectName)' == 'ElBruno.LocalLLMs.Rag'
Expand Down
1 change: 1 addition & 0 deletions ElBruno.LocalLLMs.slnx
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,7 @@
<Project Path="src/samples/RagChatbot/RagChatbot.csproj" />
<Project Path="src/samples/LocalDecisions/LocalDecisions.csproj" />
<Project Path="src/samples/DecisionCalibration/DecisionCalibration.csproj" />
<Project Path="src/samples/SharedDecisions/SharedDecisions.csproj" />
<Project Path="src/samples/ZeroCloudRag/ZeroCloudRag.csproj" />
<Project Path="src/samples/OpenAiServer/OpenAiServer.csproj" />
<Project Path="src/samples/MagenticBrainAgent/MagenticBrainAgent.csproj" />
Expand Down
4 changes: 2 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,11 +17,11 @@ Run local LLMs in .NET through `IChatClient` — the same interface you'd use fo

> The last 5 notable additions to the library. Updated with each NuGet release.

- 🎯 **`ElBruno.LocalLLMs.Decisions`** — new package for local **System One** decision models. Typed `choice`, `score` and yes/no answers with full probability distributions in a single forward pass (~600 ms on CPU), running [Laya](https://github.com/NandhaKishorM/laya) **fully in-process on ONNX Runtime** — no server, no Python. Built for routing, triage, moderation and intent detection, where a chat model is slow and overkill. Call `services.AddLocalDecisions()` and inject `IDecisionClient`. See the [Local Decisions Guide](docs/decisions-guide.md) and the [LocalDecisions sample](src/samples/LocalDecisions/).
- 🔌 **`ElBruno.AI.Decisions` provider** — `ElBruno.LocalLLMs.Decisions` now implements the shared [`ElBruno.AI.Decisions`](https://www.nuget.org/packages/ElBruno.AI.Decisions) `IDecisionClient`, so the in-process ONNX Laya model drops in next to the Foundry, Ollama and Jev providers and into `DecisionRoutingChatClient`. The rich API moved to `ILayaDecisionClient`. **Breaking** for code that injected the v0.22.0 `IDecisionClient`. See the [SharedDecisions sample](src/samples/SharedDecisions/).
- 🎯 **`ElBruno.LocalLLMs.Decisions`** — new package for local **System One** decision models. Typed `choice`, `score` and yes/no answers with full probability distributions in a single forward pass (~600 ms on CPU), running [Laya](https://github.com/NandhaKishorM/laya) **fully in-process on ONNX Runtime** — no server, no Python. Built for routing, triage, moderation and intent detection, where a chat model is slow and overkill. Call `services.AddLocalDecisions()` and inject `ILayaDecisionClient`. See the [Local Decisions Guide](docs/decisions-guide.md) and the [LocalDecisions sample](src/samples/LocalDecisions/).
- ⬆️ **.NET 10 only** — every project now single-targets `net10.0`. .NET 8 reaches end of support in November 2026, so multi-targeting was dropped ahead of it. **Breaking:** consumers still on .NET 8 must stay on `v0.21.0` or upgrade to .NET 10.
- 🧩 **`ElBruno.LocalLLMs.BlazorComponents`** — new Razor Class Library with 7 ready-to-use Blazor components: `ModelStatusCard` (download progress bar + actions), `ModelGallery` (filterable grid), `ModelSelector` (two-way-bindable dropdown), `ChatBox` (streaming token display), `EnvironmentDashboard` (CPU/CUDA/DirectML badges), `LocalLLMHealthBadge` (nav-bar status dot), and `RagPlayground`. Call `services.AddLocalLLMsBlazorComponents()` to register. See the [Blazor Components Guide](docs/blazor-components.md) and the [BlazorDemo sample](src/samples/BlazorDemo/).
- 🧠 **GPT-OSS 20B support** — OpenAI's open-weight MoE model (Apache-2.0) now runs locally via the official `onnxruntime/gpt-oss-20b-onnx` artifacts. Adds the **Harmony** prompt format, channel-aware output filtering (chain-of-thought is stripped, never shown to users), Harmony tool calling, and a `ReasoningEffort` option. Two model IDs: `gpt-oss-20b` (CPU INT4) and `gpt-oss-20b-cuda`. See the [GptOssChat sample](src/samples/GptOssChat/). Also fixes a token-duplication bug that repeated the final token of every generation.
- 🚀 **`v0.20.12`** — Corrects sibling-package assembly versions, hardens vision token probing against model context limits, and verifies Fara smart image resizing for screenshot workflows.

## Features

Expand Down
13 changes: 13 additions & 0 deletions docs/CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,19 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

---

## [0.23.0] - 2026-10-09

### Changed
- **BREAKING: `ElBruno.LocalLLMs.Decisions` is now a provider for the shared [`ElBruno.AI.Decisions`](https://www.nuget.org/packages/ElBruno.AI.Decisions) abstractions.** `LayaOnnxDecisionClient` implements `ElBruno.AI.Decisions.IDecisionClient` (`ChooseAsync`, `ScoreAsync`, `AssessAsync`), so it can replace or sit next to the Foundry, Ollama and Jev providers and plug into `DecisionRoutingChatClient`. The v0.22.0 `ElBruno.LocalLLMs.Decisions.IDecisionClient` is renamed `ILayaDecisionClient` and now extends the shared interface; it keeps `EvaluateAsync` batching and the `CalibrationClamped` results. `DecisionException` is now the shared type from `ElBruno.AI.Decisions`. Migration: inject `ILayaDecisionClient` instead of the old `IDecisionClient`, or inject `ElBruno.AI.Decisions.IDecisionClient` to stay provider-neutral.
- `AddLocalDecisions()` registers `ILayaDecisionClient` and the concrete client, and registers the shared `IDecisionClient` with `TryAdd` so an existing provider is not displaced.
- Neutral results clamp probabilities into `[0, 1]`, since the shared result types reject values slightly outside that range.

### Added
- **`SharedDecisions` sample** — provider-neutral decisions plus `DecisionRoutingChatClient` routing.
- Dependency on `ElBruno.AI.Decisions` 0.6.2 (early access).

---

## [0.22.0] - 2026-09-30

### Changed
Expand Down
37 changes: 33 additions & 4 deletions docs/decisions-guide.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ which is exactly where a chat model is the wrong tool.

## Decision model vs. chat model

| | Chat model (`IChatClient`) | Decision model (`IDecisionClient`) |
| | Chat model (`IChatClient`) | Decision model (`ILayaDecisionClient`) |
|---|---|---|
| Output | free text, token by token | one typed answer + probabilities |
| Passes | autoregressive, one token at a time | a single forward pass |
Expand Down Expand Up @@ -225,6 +225,35 @@ It is far too small to flip a routing decision, but large enough to matter if yo
Batching is still the right default: four questions in one pass cost about what one costs. Just
treat the probabilities as approximate, which the calibration caveats above already require.

## Using with ElBruno.AI.Decisions

`LayaOnnxDecisionClient` is a provider for the shared [`ElBruno.AI.Decisions`](https://www.nuget.org/packages/ElBruno.AI.Decisions) abstractions, alongside its Foundry, Ollama and Jev providers. It is the in-process ONNX one: no server and no network call at inference time.

```csharp
using ElBruno.AI.Decisions;

builder.Services.AddLocalDecisions();

// Provider-neutral: swap the registration to change providers, nothing else changes.
IDecisionClient decisions = provider.GetRequiredService<IDecisionClient>();

ChoiceDecision team = await decisions.ChooseAsync(ticket, "Which team should handle this?", teams);
AssessmentDecision angry = await decisions.AssessAsync(ticket, "The customer is frustrated.");
```

The same client also plugs into `DecisionRoutingChatClient`, which picks the `IChatClient` that answers each message.

Two interfaces are registered, both resolving to the same singleton:

| Inject | When |
|---|---|
| `ElBruno.AI.Decisions.IDecisionClient` | You want to stay provider-neutral. Registered with `TryAdd`, so the first provider registered keeps this slot. |
| `ILayaDecisionClient` | You want Laya extras: `EvaluateAsync` batching into one forward pass, `CalibrationClamped`, `ProbabilityResult.IsTrue`. It extends `IDecisionClient`. |

The neutral results clamp every probability into `[0, 1]`, because the shared types reject values a few ulps outside that range. `CalibrationClamped` is only available on the rich results.

`DecisionException` now comes from `ElBruno.AI.Decisions`. The shared package is early access (0.6.x), so expect its surface to move.

## Dependency injection

```csharp
Expand All @@ -235,7 +264,7 @@ builder.Services.AddLocalDecisions(options =>
});
```

Then inject `IDecisionClient`. The client is thread-safe and registered as a singleton, because
Then inject `ILayaDecisionClient` (or the provider-neutral `IDecisionClient`). The client is thread-safe and registered as a singleton, because
loading the model is expensive and the loaded session is safe to share. Registration does not
touch the network — the model is loaded lazily on the first call.

Expand All @@ -251,11 +280,11 @@ touch the network — the model is loaded lazily on the first call.

## Testing

`IDecisionClient` is a plain interface, so unit tests can implement it directly with whatever
`IDecisionClient` and `ILayaDecisionClient` are plain interfaces, so unit tests can implement them directly with whatever
distributions the test needs — no model download and no ONNX session:

```csharp
internal sealed class StubDecisionClient : IDecisionClient { /* ... */ }
internal sealed class StubDecisionClient : ILayaDecisionClient { /* ... */ }
```

## Error handling
Expand Down
23 changes: 23 additions & 0 deletions docs/samples.md
Original file line number Diff line number Diff line change
Expand Up @@ -566,6 +566,29 @@ for what to do about both.

---

## SharedDecisions: Provider-Neutral Decisions and Routing

Uses the local Laya model through the shared `ElBruno.AI.Decisions` `IDecisionClient`, then hands the same client to `DecisionRoutingChatClient` so the decision model picks which `IChatClient` answers a message.

```bash
dotnet run --project src/samples/SharedDecisions

# or against a model directory you already have
dotnet run --project src/samples/SharedDecisions -- D:\models\laya
```

Expected output (probabilities vary slightly by CPU):

```text
Team: billing (confidence 98.3%)
Frustrated: 85.3%
[billing agent] I'll look into the duplicate charge.
```

See the [Local Decisions Guide](decisions-guide.md#using-with-elbrunoaidecisions).

---

## Next Steps

- 📖 [Getting Started](getting-started.md) — full setup guide with GPU configuration
Expand Down
8 changes: 4 additions & 4 deletions src/ElBruno.LocalLLMs.Decisions/DecisionClientExtensions.cs
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
namespace ElBruno.LocalLLMs.Decisions;

/// <summary>
/// Convenience overloads for <see cref="IDecisionClient"/>.
/// Convenience overloads for <see cref="ILayaDecisionClient"/>.
/// </summary>
public static class DecisionClientExtensions
{
Expand All @@ -20,7 +20,7 @@ public static class DecisionClientExtensions
/// <c>"payment, invoice and refund problems"</c>. Prefer the dictionary overload when accuracy matters.
/// </remarks>
public static Task<ChoiceResult> ChooseAsync(
this IDecisionClient client,
this ILayaDecisionClient client,
string state,
IEnumerable<string> options,
string? instructions = null,
Expand All @@ -43,10 +43,10 @@ public static Task<ChoiceResult> ChooseAsync(
/// <returns><see langword="true"/> when the probability meets the configured threshold.</returns>
/// <remarks>
/// Discarding the probability discards the information you need to tune the threshold later.
/// Prefer <see cref="IDecisionClient.AskAsync"/> unless you are certain a boolean is enough.
/// Prefer <see cref="ILayaDecisionClient.AskAsync"/> unless you are certain a boolean is enough.
/// </remarks>
public static async Task<bool> IsTrueAsync(
this IDecisionClient client,
this ILayaDecisionClient client,
string state,
string proposition,
CancellationToken cancellationToken = default)
Expand Down
16 changes: 0 additions & 16 deletions src/ElBruno.LocalLLMs.Decisions/DecisionException.cs

This file was deleted.

11 changes: 8 additions & 3 deletions src/ElBruno.LocalLLMs.Decisions/DecisionServiceExtensions.cs
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.DependencyInjection.Extensions;

namespace ElBruno.LocalLLMs.Decisions;

Expand All @@ -8,8 +9,8 @@ namespace ElBruno.LocalLLMs.Decisions;
public static class DecisionServiceExtensions
{
/// <summary>
/// Registers an <see cref="IDecisionClient"/> that runs a Laya decision model in-process
/// through ONNX Runtime.
/// Registers a Laya decision model that runs in-process through ONNX Runtime, as the shared
/// <see cref="IDecisionClient"/> provider and as <see cref="ILayaDecisionClient"/>.
/// </summary>
/// <param name="services">The service collection.</param>
/// <param name="configureOptions">Optional action to configure the model source and threshold.</param>
Expand All @@ -18,6 +19,8 @@ public static class DecisionServiceExtensions
/// The client is registered as a singleton because loading the model is expensive and the
/// loaded session is safe to share. The model is downloaded lazily on first use, so
/// registration does not block startup.
/// The shared <see cref="IDecisionClient"/> uses <c>TryAdd</c>, like the other ElBruno.AI.Decisions providers,
/// so the first provider registered keeps that slot; <see cref="ILayaDecisionClient"/> is always available.
/// </remarks>
/// <example>
/// <code>
Expand All @@ -39,7 +42,9 @@ public static IServiceCollection AddLocalDecisions(
options.Validate();

services.AddSingleton(options);
services.AddSingleton<IDecisionClient>(_ => new LayaOnnxDecisionClient(options));
services.AddSingleton(_ => new LayaOnnxDecisionClient(options));
services.AddSingleton<ILayaDecisionClient>(provider => provider.GetRequiredService<LayaOnnxDecisionClient>());
services.TryAddSingleton<IDecisionClient>(provider => provider.GetRequiredService<LayaOnnxDecisionClient>());

return services;
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,8 @@
</PropertyGroup>

<ItemGroup>
<Using Include="ElBruno.AI.Decisions" />
<PackageReference Include="ElBruno.AI.Decisions" Version="0.6.2" />
<PackageReference Include="ElBruno.HuggingFace.Downloader" Version="1.4.4" />
<PackageReference Include="Microsoft.Extensions.DependencyInjection.Abstractions" Version="10.0.12" />
<PackageReference Include="Microsoft.ML.OnnxRuntime" Version="1.28.0" />
Expand Down
Original file line number Diff line number Diff line change
@@ -1,7 +1,8 @@
namespace ElBruno.LocalLLMs.Decisions;

/// <summary>
/// Makes typed decisions about text using a local System One model.
/// The in-process Laya decision client: the provider-neutral <see cref="IDecisionClient"/> from
/// <c>ElBruno.AI.Decisions</c> plus the extras that only a local model can offer.
/// </summary>
/// <remarks>
/// <para>
Expand All @@ -13,8 +14,12 @@ namespace ElBruno.LocalLLMs.Decisions;
/// Use it for routing, triage, moderation gates, intent detection and similar classification work,
/// and hand the result to a generative model when you actually need prose.
/// </para>
/// <para>
/// Depend on <see cref="IDecisionClient"/> when you want to swap providers (Foundry, Ollama, Jev).
/// Depend on this interface when you want request batching and <c>CalibrationClamped</c>.
/// </para>
/// </remarks>
public interface IDecisionClient
public interface ILayaDecisionClient : IDecisionClient
{
/// <summary>
/// Selects one of <paramref name="options"/> for the supplied text.
Expand Down
39 changes: 37 additions & 2 deletions src/ElBruno.LocalLLMs.Decisions/LayaOnnxDecisionClient.cs
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
namespace ElBruno.LocalLLMs.Decisions;

/// <summary>
/// An <see cref="IDecisionClient"/> that runs a Laya decision model in-process through ONNX Runtime.
/// An <see cref="IDecisionClient"/> provider that runs a Laya decision model in-process through ONNX Runtime.
/// </summary>
/// <remarks>
/// <para>
Expand All @@ -16,7 +16,7 @@ namespace ElBruno.LocalLLMs.Decisions;
/// A single instance is safe to share and expensive to create, so register it as a singleton.
/// </para>
/// </remarks>
public sealed class LayaOnnxDecisionClient : IDecisionClient, IDisposable
public sealed class LayaOnnxDecisionClient : ILayaDecisionClient, IDisposable
{
private readonly DecisionOptions _options;
private readonly SemaphoreSlim _initializationLock = new(1, 1);
Expand Down Expand Up @@ -75,6 +75,41 @@ public async Task<ProbabilityResult> AskAsync(
return result.Probability("ask");
}

/// <inheritdoc />
async Task<ChoiceDecision> IDecisionClient.ChooseAsync(
string input,
string instructions,
IReadOnlyDictionary<string, string?> options,
CancellationToken cancellationToken)
{
DecisionRequestValidation.Choice(input, instructions, options);
var result = await ChooseAsync(input, options, instructions, cancellationToken).ConfigureAwait(false);
return NeutralDecisionMapper.ToDecision(result);
}

/// <inheritdoc />
async Task<ScoreDecision> IDecisionClient.ScoreAsync(
string input,
string instructions,
IReadOnlyList<string> rubric,
CancellationToken cancellationToken)
{
DecisionRequestValidation.Score(input, instructions, rubric);
var result = await ScoreAsync(input, rubric, instructions, cancellationToken).ConfigureAwait(false);
return NeutralDecisionMapper.ToDecision(result);
}

/// <inheritdoc />
async Task<AssessmentDecision> IDecisionClient.AssessAsync(
string input,
string proposition,
CancellationToken cancellationToken)
{
DecisionRequestValidation.Assess(input, proposition);
var result = await AskAsync(input, proposition, cancellationToken).ConfigureAwait(false);
return NeutralDecisionMapper.ToDecision(result);
}

/// <inheritdoc />
public async Task<DecisionResult> EvaluateAsync(
DecisionRequest request,
Expand Down
22 changes: 22 additions & 0 deletions src/ElBruno.LocalLLMs.Decisions/NeutralDecisionMapper.cs
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
namespace ElBruno.LocalLLMs.Decisions;

/// <summary>
/// Converts Laya's rich results into the provider-neutral <c>ElBruno.AI.Decisions</c> types.
/// </summary>
internal static class NeutralDecisionMapper
{
internal static ChoiceDecision ToDecision(ChoiceResult result) =>
new(result.Choice, ClampDistribution(result.Probabilities), Clamp(result.Confidence));

internal static ScoreDecision ToDecision(ScoreResult result) =>
new(result.Probabilities.OrderBy(pair => pair.Key).Select(pair => Clamp(pair.Value)).ToArray());

internal static AssessmentDecision ToDecision(ProbabilityResult result) =>
new(Clamp(result.Probability));

// Softmax output can land a few ulps outside [0,1]; the shared types reject that outright.
private static double Clamp(double value) => Math.Clamp(value, 0.0, 1.0);

private static Dictionary<string, double> ClampDistribution(IReadOnlyDictionary<string, double> probabilities) =>
probabilities.ToDictionary(pair => pair.Key, pair => Clamp(pair.Value), StringComparer.Ordinal);
}
Loading
Loading