fix(oracle): emit tool_calls in streaming responses

[fede-kamel]

Summary

Fixes the OCI streaming chunk transform so that tool_calls are actually emitted to clients. Without this fix, agents using streamed tool calling against OCI models receive finish_reason: "tool_calls" with no tool data — see #1538.

What changed

src/providers/oracle/chatComplete.ts (+ supporting types in types/ChatDetails.ts):

• Streaming chunk transform — split the OCI message chunk that bundles toolCalls together with finishReason into:
  1. a tool_calls delta chunk
  2. the finish-reason chunk
  3. a data: [DONE] marker
• OracleStreamState — tracks stable tool-call indices across streamed chunks (via a seen-id set) so the OpenAI-format tool_calls[].index stays consistent.
• Request transform — propagate toolCalls (assistant) and toolCallId (tool) onto the OCI request envelope so tool-call round-trips are correctly wired in.
• Non-streaming response transform — extract toolCalls into OpenAI-shape tool_calls, with an id fallback for providers (e.g. Gemini) that omit it.

Tests

src/providers/oracle/chatComplete.test.ts — focused on the new behaviour:

• request shaping (toolCalls on assistant, toolCallId on tool, omitted otherwise)
• non-stream extraction (incl. synthesised id when provider omits one)
• streaming chunk shape (tool_calls delta → finish chunk → DONE)
• index stability across multiple parallel tool calls
• [DONE] and ping passthroughs

Scope note (re: prior review)

This PR has been trimmed per @narengogi's review on the previous version. The earlier branch bundled embeddings, multi-vendor handler dispatch, GPT-5 maxCompletionTokens routing, multimodal content types, and unrelated changes to open-ai-base/index.ts, utils/env.ts, requestBody.ts, and jest.config.js. All of that is out — this PR is now scoped solely to the streaming tool-call fix referenced in #1538 plus the minimum supporting types and tests.

The split-off items now live in their own focused PRs:

• Cohere embeddings → feat(oracle): add Cohere embeddings support #1634 (approved)
• GPT-5 maxCompletionTokens routing → fix(oracle): route maxTokens to maxCompletionTokens for GPT-5 family #1635
• Multi-vendor handler dispatch → will follow in a separate PR once the current queue clears

The older kitchen-sink branch (#1540) has been closed in favour of these focused PRs.

Verification

• npm run build ✓
• npm run format:check ✓
• Unit tests pass for the new test file (chatComplete.test.ts)
• Verified against live OCI (inference.generativeai.us-chicago-1.oci.oraclecloud.com/20231130/actions/chat) for streaming tool calls on Llama and Grok families.

Closes #1538

[fede-kamel]

Note on src/providers/open-ai-base/index.ts change

While working on the Oracle provider, I ran tsc --noEmit to verify type safety and discovered a pre-existing TypeScript error in open-ai-base/index.ts:

src/providers/open-ai-base/index.ts(72,23): error TS2339: Property 'default' does not exist on type 'ParameterConfig | ParameterConfig[]'.

This error existed on main before my changes. The code works at runtime (JavaScript ignores types), but TypeScript couldn't properly narrow the union type when accessing via index.

Original code:

if (Object.hasOwn(baseParams, key) && !Array.isArray(baseParams[key])) {
  baseParams[key].default = defaultValues?.[key];  // TS error
}

Fixed code:

const param = baseParams[key];
if (param && !Array.isArray(param)) {
  (param as { default?: any }).default = defaultValues?.[key];  // ✅
}

I included this fix since I was already touching the codebase and it improves type safety. If you prefer, I can revert this change and keep it Oracle-only.

[fede-kamel]

Issue Discovery: OCI Streaming Tool Calls

While testing portkey-strands (Strands Agents SDK integration with Portkey), I discovered a critical issue with OCI GenAI streaming responses for tool/function calling.

The Problem

When using OCI GenAI models through the current production Portkey gateway, streaming responses do not properly return tool call data. The model indicates it wants to call a tool (finish_reason: "tool_calls"), but the actual tool call information (function name, arguments) is missing from the streamed chunks.

Test Results (before this PR):

Provider	Model	Basic Chat	Tool Use
OpenAI	gpt-4o	✅ Pass	✅ Pass
Anthropic	claude-sonnet-4-6	✅ Pass	✅ Pass
OCI	meta.llama-3.3-70b	✅ Pass	✅ Pass
OCI	xai.grok-4-1-fast-reasoning	✅ Pass	✅ Passl
OCI	cohere.command-a	✅ Pass

Actual Error

# Model returns end_turn with empty content instead of tool_use
AgentResult(
    stop_reason='end_turn',  # Should be 'tool_use'
    message={'role': 'assistant', 'content': []},  # Tool call data missing
    ...
)

The streaming response returns stopReason: 'end_turn' instead of 'tool_use', and the contentBlockStart events for tool calls are never emitted.

Why This PR Fixes It

This PR includes proper streaming state management for tool calls:

1. Tool call accumulation - Properly buffers tool call chunks across SSE events
2. Index tracking - Maintains tool call indices for parallel tool calls
3. Finish reason mapping - Correctly maps OCI's finish reasons to OpenAI-compatible format
4. Content block emission - Emits proper contentBlockStart/contentBlockDelta/contentBlockStop events for tools

Verification

After applying this PR's changes, all OCI models pass tool calling tests:

• Single tool calls
• Multiple sequential tools
• Parallel tool calls (3+ simultaneous)
• Streaming tool calls

This enables full agentic workflows with OCI GenAI models through Portkey.

[fede-kamel]

cc @VisargD - This PR addresses a critical issue affecting agentic workflows with OCI GenAI models. The streaming tool call bug prevents Strands Agents (and likely other agent frameworks) from working properly with OCI models through Portkey.

Would appreciate a review when you have time!

[fede-kamel]

Tracking issue created: #1538

This documents the critical streaming tool call bug and explains why keeping the Oracle GenAI integration up to date is important for enterprise customers.

[fede-kamel]

Rebased check done: branch is already up to date with main (no conflicts).

Validation run:

• Unit: tests/unit/src/providers/oracle/chatComplete.test.ts, tests/unit/src/providers/oracle/modelConfig.test.ts (PASS)
• Integration (OCI profile LUIGI_FRA_API): tests/integration/src/providers/oracle/run-tests.sh ... all (PASS; chat+embed suites)

Re-pinging for review.

[fede-kamel]

Friendly follow-up on this one when you have a moment: @VisargD @narengogi\n\nWould appreciate a review. Thank you!

[fede-kamel]

@narengogi @VisargD

Maintainer note: keeping OCI support current is strategically important here, not just additive. OCI is increasingly exposing new model families and provider-specific behaviors behind a single enterprise surface, and that surface changes fast: model availability, request/response shapes, auth expectations, token parameters, streaming/tool-call behavior, and embedding support all evolve over time.

If the OCI provider lags, Portkey users in Oracle environments end up with capability gaps or subtle incompatibilities even when the same models are available elsewhere. Keeping OCI up to date means Portkey stays credible as a gateway layer for Oracle-hosted GPT/Gemini/Llama/Grok/Cohere workloads, reduces integration drift for enterprise customers, and avoids pushing teams into provider-specific forks or direct OCI workarounds.

This PR moves OCI much closer to parity with the broader provider matrix, which is exactly the kind of maintenance surface that compounds in value if it is kept current.

[fede-kamel]

Hey @VisargD @narengogi — friendly ping on this one. It's been open for about 3 weeks now.

Beyond the technical additions (full OCI GenAI support, streaming tool call fix, 61 unit tests), there's a broader Portkey + Oracle partnership story here: Portkey becomes the gateway layer for enterprise teams running GenAI on OCI, and Oracle customers get unified API access to all their OCI-hosted models (Llama, Gemini, GPT, Grok, Cohere) through a gateway they may already be using for OpenAI/Anthropic.

That's a compelling value prop for both sides — Portkey expands its enterprise reach, Oracle gets first-class representation in the provider matrix.

Happy to jump on a call if it's easier to walk through the implementation. Would really appreciate a review when you get a chance!

[fede-kamel]

Friendly ping — this PR has been open for ~57 days without a review. Happy to address feedback or rebase further if needed. Let me know if there's anything I can clarify. Thanks!

[narengogi]

@fede-kamel I'm reviewing this now

[narengogi]

please revert these changes if they are not relevant to your PR
If baseParams[key] is an empty string "" or the number 0, the this change will skip it, whereas the first version would have processed it.

[narengogi]

too many cahnges, please trim to only necessary changes and raise a separate PR if cleaner

adding new providers, extending existing ones is rather simple as you can see here:
https://github.com/Portkey-AI/gateway/pull/1510/changes

[narengogi]

what are these changes for?

can you please limit your changes to adding a new provider, these changes touch a lot of irrelevant code blocks, this will increase the testing scope and affects core logic in the gateway

[fede-kamel]

Hi @narengogi — thanks for the review, this is really helpful.

Inline comments — agreed, will revert:

• src/providers/open-ai-base/index.ts: you're right, the falsy-skip behavior change isn't related to this PR. Reverting.
• src/utils/env.ts: agreed — those changes aren't strictly needed for adding the provider. Reverting.

On "too many changes" — quick clarification before I rework:

I want to make sure I split this in the way that's most useful for you. The current PR bundles a few things that I think should each stand on their own — could you confirm the plan below works, or steer me differently?

1. This PR (fix(oracle): emit tool_calls in streaming responses #1537), trimmed: bare-minimum Oracle provider — chatComplete only, no handlers/* dispatch, no embed, no modelConfig. Match the latitude PR shape: provider-dir adds + ~6 lines of registration (globals.ts, providers/index.ts, providers/types.ts). Tests trimmed to the essentials.
2. Separate PR: OCI streaming tool-call fix (this is a behavior fix, not a provider add — happy to extract it).
3. Separate PR: Oracle embeddings.
4. Separate PR (only if useful): the multi-vendor handler dispatch (handlers/cohere.ts, handlers/meta.ts, etc.) — needed because OCI exposes Cohere, Meta, OpenAI, Gemini, Grok models behind one endpoint with different request/response shapes. If you'd rather I keep it inline in chatComplete.ts à la single-vendor providers, let me know.
5. feat(oracle): Add rerank and guardrails support #1540 (rerank/guardrails) rebases on top of the trimmed fix(oracle): emit tool_calls in streaming responses #1537.

Two specific questions:

• Is the multi-vendor handler dispatch acceptable in some form, or do you want it inline?
• Anything in particular about test scope you'd like trimmed (the chatComplete suite is the heaviest at ~1k lines)?

Will hold off on the rework until you confirm direction. Thanks!

[fede-kamel]

@narengogi — split per your review.

This PR is now scoped to only the OCI streaming tool-call fix (the bug from #1538). I dropped:

• embeddings (embed.ts, embed handler wiring)
• multi-vendor handler dispatch (handlers/*, modelConfig.ts)
• GPT-5 maxCompletionTokens routing
• multimodal content types (document/video/file)
• unrelated changes to src/providers/open-ai-base/index.ts, src/utils/env.ts, src/types/requestBody.ts, jest.config.js

What's left (force-pushed): 3 files, +387/−20, all under src/providers/oracle/:

• chatComplete.ts — fix the streaming chunk transform + propagate tool_calls/tool_call_id through request and non-stream response
• types/ChatDetails.ts — add toolCalls / toolCallId fields on Message
• chatComplete.test.ts — focused unit tests (request shaping, non-stream extraction, streaming chunk shape, parallel index stability, DONE/ping passthroughs)

The rest will land as separate PRs:

1. Oracle Cohere embeddings
2. Oracle multi-vendor handler dispatch (Cohere/Meta/OpenAI/Gemini/Grok request shapes behind one OCI endpoint)
3. Oracle GPT-5 maxCompletionTokens (small, inline)

I'll open them after this one merges (or in parallel if you prefer — let me know). #1540 (rerank/guardrails) will rebase on top of whichever lands first.

Re your inline comments on the previous version: yes, both src/providers/open-ai-base/index.ts and src/utils/env.ts are reverted here — they're not in this PR's diff anymore.

Ready for review.

[fede-kamel]

Split is complete. Sibling PRs opened:

• feat(oracle): add Cohere embeddings support #1634 — feat(oracle): add Cohere embeddings support (+266 / −0)
• fix(oracle): route maxTokens to maxCompletionTokens for GPT-5 family #1635 — fix(oracle): route maxTokens to maxCompletionTokens for GPT-5 family (+63 / −1)

This PR (#1537) remains scoped to the streaming tool-call fix only (+387 / −20).

The originally-bundled multi-vendor handler dispatch (handlers/*, modelConfig.ts) was dropped — those files weren't actually wired into chatComplete.ts on the prior branch, so they were dead code rather than a real feature split. If/when a real handler-dispatch mechanism is needed, it can land as its own focused PR.

Sibling PRs are independent and can land in any order.

[fede-kamel]

End-to-end test against live OCI surfaced two real bugs in this PR — both fixed in b83acee0:

Bug 1 — [DONE] got mangled. I had the finish-chunk transform return string[] to split the tool_calls delta and the finish chunk into separate SSE events. But src/handlers/streamHandler.ts::readStream (the path used by SSE-style providers like OCI) doesn't handle string[] returns — it just yields whatever the transform returned, and an array gets coerced to a comma-joined string. Result: the final tail looked like …,data: [DONE] instead of properly separated SSE events.

Fixed by concatenating the (optional) tool_calls delta + finish chunk + data: [DONE]\n\n into a single string with proper \n\n boundaries.

Bug 2 — index-thrash on continuation chunks. OCI streams tool calls progressively: the first chunk for a tool carries id + name, then subsequent chunks carry argument fragments with no id. My seenToolCallIds.has(tc.id) check was hitting has(undefined) on every continuation, which is always false, so it incremented the tool index on every fragment. A single tool call's argument fragments ended up split across indices [0, 1, 1, 1] instead of all staying at 0.

Fixed by anchoring the index on id when it's present and inheriting the current index when it's not. Also dropped id / name from the emitted delta when OCI doesn't send them in that chunk, to match OpenAI's wire shape.

E2E verification (live OCI, us-chicago-1, meta.llama-3.3-70b-instruct, streaming + tool calling):

signal	before fix	after fix
tool_calls deltas streamed	✅ 4	✅ 4
finish_reason: tool_calls chunk	✅	✅
data: [DONE] terminator	❌ malformed (,data: [DONE])	✅
tool_call indices across fragments	❌ [0, 1, 1, 1]	✅ [0, 0, 0, 0]

Test added: keeps continuation chunks attached to the same tool index when id is omitted.

The non-streaming path and request shaping were unaffected and continue to pass.

[fede-kamel]

@roh26it @narengogi @VisargD — flagging this for partner-level attention.

Status across the OCI split:

PR	Scope	Rebased on main	Mergeable	E2E verified
#1537	Streaming tool-call fix (the bug from #1538)	✅ 0 behind	✅	✅ live OCI us-chicago-1, meta.llama-3.3-70b-instruct, streaming + tools
#1634	Cohere embeddings	✅ 0 behind	✅	✅ cohere.embed-multilingual-v3.0, batch input, dim 1024
#1635	GPT-5 maxCompletionTokens routing	✅ 0 behind	✅	✅ openai.gpt-5, HTTP 200 (was 400 before fix)

All three are split per @narengogi's review on the original bundled PR, each diff is small and focused (#1537: +387/−20, #1634: +266/−0, #1635: +63/−1), all green on type-check / formatting / build, all changes scoped under src/providers/oracle/ plus a one-line embed route addition for #1634.

The partner ask, plainly:

The current published OCI provider in Portkey doesn't represent the quality bar of the OCI Generative AI product — streamed tool-calling is broken (finish_reason: tool_calls with empty content, breaks every agent framework), GPT-5 returns 400, Cohere embeddings aren't exposed. Oracle customers running agentic workloads on OCI through Portkey hit these failures today and have been routing around the gateway as a result.

I'm coming at this as the Oracle-side engineer working on these integrations, and the goal here is straightforward: bring the OCI surface in Portkey up to the same parity bar the rest of the provider matrix gets. The PRs are deliberately small and focused so they're cheap to review — none of them touch shared gateway code, all the changes are inside the Oracle provider directory.

Would really appreciate a review pass from the team. Happy to jump on a call to walk through the OCI streaming wire format and the GPT-5 / GPT-OSS family quirks if it's useful — there are a few more (verbosity, reasoning_effort, structured outputs) that would naturally follow these as a continued partnership.

Thanks!

[fede-kamel]

@roh26it @narengogi @VisargD — gentle follow-up.

This work has been in flight since late February. The original bundled PR sat for ~57 days before the first review, and after I split it per @narengogi's feedback into the three small focused PRs (#1537, #1634, #1635), we're back in a holding pattern.

I'm not flagging this to be impatient — I'm flagging it because the cost of the delay is now landing on Oracle customers using Portkey for OCI Generative AI workloads. Streaming tool-calls don't work, GPT-5 returns 400, and embeddings aren't exposed. Every week these stay unmerged is another week those customers either work around the gateway or pick something else for OCI traffic, which doesn't serve either side of this.

The PRs are deliberately easy to review:

• fix(oracle): emit tool_calls in streaming responses #1537 — +387 / −20, single bug fix, scoped entirely to src/providers/oracle/
• feat(oracle): add Cohere embeddings support #1634 — +266 / −0, additive embeddings only
• fix(oracle): route maxTokens to maxCompletionTokens for GPT-5 family #1635 — +63 / −1, two-file param-routing fix

All three are rebased on main, mergeable, type-clean, formatted, and e2e-verified against live OCI.

Would the team be able to commit to a review timeline this week? Even a "we'll get to this on X" would help me plan the follow-up work (verbosity / reasoning_effort / structured outputs) without piling more PRs into the same backlog.

Happy to make any change you want to land these faster — including jumping on a 15-minute call to walk through the implementation if that's the lower-friction path. Really would like to get this over the line.

Thanks!

[fede-kamel]

Full e2e verification across all three PRs — 23/25 PASS against live OCI (us-chicago-1).

Combined the three branches into a single test runner, built the gateway, and exercised every model family OCI exposes plus every code path the PRs touch:

Group A — non-streaming chat (6 cases)

Model	Result
meta.llama-3.3-70b-instruct	✅ 2 tokens, 'ok'
google.gemini-2.5-flash	✅ 1 token, 'ok'
xai.grok-3-mini	✅ 1 token, 'ok'
openai.gpt-4o	✅ 2 tokens, 'Ok.'
openai.gpt-5	✅ 138 tokens, 'ok'
cohere.command-a-03-2025	❌ 400 "Chat request type does not match serving model" — known issue, see note below

Group B — streaming text (5 cases)

Model	Result
meta.llama-3.3-70b-instruct	✅ 16 events, finish + DONE
google.gemini-2.5-flash	✅ 3 events, finish + DONE
xai.grok-3-mini	✅ 386 events, finish + DONE
openai.gpt-4o	✅ 15 events, finish + DONE
cohere.command-a-03-2025	❌ same Cohere-shape issue

Group C — non-stream tool calls (4 cases)

Model	tool_calls	finish_reason
meta.llama-3.3-70b-instruct	✅ get_weather({"city": "Tokyo"})	tool_calls
google.gemini-2.5-flash	✅ get_weather({"city":"Tokyo"})	stop
xai.grok-3-mini	✅ get_weather({"city":"Tokyo"})	tool_calls
openai.gpt-4o	✅ get_weather({"city":"Tokyo"})	tool_calls

Group D — streaming tool calls (this PR's core fix)

Model	tool_deltas	finish	[DONE]	indices
meta.llama-3.3-70b-instruct	4	tool_calls	✅	[0] ← stable across continuations
google.gemini-2.5-flash	1	stop	✅	[0]
xai.grok-3-mini	1	tool_calls	✅	[0]
openai.gpt-4o	6	tool_calls	✅	[0]

Group E — embeddings (#1634)

Case	Result
Single string input	✅ 1 embedding, dim 1024
Batch of 3 strings	✅ 3 embeddings, indices [0, 1, 2], dim 1024
input_type=search_document	✅ accepted, returned valid embedding

Group F — GPT-5 maxCompletionTokens routing (#1635)

Case	Result
gpt-5 + max_tokens: 200 (gateway must rewrite)	✅ 200 OK, content returned
gpt-5 + max_completion_tokens: 200 (passthrough)	✅ 200 OK
meta.llama-* + max_tokens: 50 (negative regression — must NOT rewrite)	✅ Llama still uses maxTokens, no spurious rewriting

---

About the Cohere chat failures

cohere.command-a-03-2025 chat hits 400 "Chat request type does not match serving model" because OCI's Cohere chat endpoint requires apiFormat: COHERE with a different request shape (message + chatHistory instead of OpenAI's messages array). The existing OracleChatDetailsConfig on main always sends apiFormat: GENERIC. This pre-dates these PRs — it's the same on main today.

This is not in scope for any of the three PRs. The bundled multi-vendor handler dispatch in the original #1537 would have addressed it, but per the trim-down request that's out, and Cohere chat parity should follow as its own focused PR once these three land. Cohere embeddings still work fine (Group E above) — it's only the chat endpoint shape that needs the separate fix.

---

Bottom line

23/25 across the live surface, with the only failures being a pre-existing Cohere chat-shape limitation that none of these three PRs change. All three PRs are independently mergeable and don't depend on each other.

[fede-kamel]

Thanks for the approval @narengogi! What's the next step from here to land this in a release? Happy to add a changelog entry, rebase, or anything else the merge process needs — just let me know what's missing.

Also worth flagging: #1635 (GPT-5 maxCompletionTokens routing) touches the same chatComplete.ts, so once this lands I'll rebase that one on top. Happy to coordinate merge order if useful.