Skip to content

schema: classify transcription models + add gpt-transcribe + audio media types #1121

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
paultancre-bt wants to merge 2 commits into
base: main
Choose a base branch
from
Open

schema: classify transcription models + add gpt-transcribe + audio media types #1121

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
d7fe65d
schema: classify transcription models + add gpt-transcribe
paultancre-bt Aug 17, 2026
7fedd19
schema: accept audio media types for transcription models
paultancre-bt Aug 17, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 15 additions & 6 deletions packages/proxy/schema/media-types.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
import type { ModelFormat, ModelName } from "./models";
import { getAvailableModels, type ModelFormat } from "./models";

const IMAGE_MEDIA_TYPES = [
"image/jpeg",
Expand Down Expand Up @@ -39,6 +39,10 @@ const AUDIO_MEDIA_TYPES = [
"audio/mpeg",
"audio/mp4",
"audio/webm",
"audio/flac",
"audio/ogg",
"audio/m4a",
"audio/x-m4a",
] as const;

const VIDEO_MEDIA_TYPES = [
Expand Down Expand Up @@ -132,22 +136,27 @@ export const ModelFormatMediaTypes: {
/**
* Overrides for specific models to support additional media types.
*/
export const ModelMediaTypeOverrides: {
[model in ModelName]?: MediaTypeSupport;
} = {
export const ModelMediaTypeOverrides: Partial<
Record<string, MediaTypeSupport>
> = {
// will be useful for gpt-audio
};

export function getSupportedMediaTypes(
format: ModelFormat,
model?: ModelName,
model?: string,
): Set<string> {
const baseSupport = { ...ModelFormatMediaTypes[format] };

if (model && ModelMediaTypeOverrides[model]) {
Object.assign(baseSupport, ModelMediaTypeOverrides[model]);
}

// Transcription models take an audio file regardless of provider format.
if (model && getAvailableModels()[model]?.transcription) {
Object.assign(baseSupport, toMediaTypeSupport(AUDIO_MEDIA_TYPES));
Comment on lines +155 to +157

@chatgpt-codex-connector chatgpt-codex-connector Bot Aug 18, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Move transcription upload shaping to the gateway

This model-aware media filtering determines which files the transcription request path accepts, so it is request/upload shaping in the deprecated proxy. The root AGENTS.md assigns that behavior to the parent gateway/ implementation and permits only the catalog JSON exception here; keeping active behavior in this package risks the gateway and deprecated proxy diverging. Move this logic to the paired gateway change and retain only the catalog update here.

AGENTS.md reference: AGENTS.md:L6-L15

Useful? React with 👍 / 👎.

Comment on lines +155 to +157

@chatgpt-codex-connector chatgpt-codex-connector Bot Aug 18, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Restrict transcription models to transcription inputs

When a flagged OpenAI transcription model is selected, baseSupport already contains PDFs and images, and this assignment merely adds audio types. Consequently, calls such as getSupportedMediaTypes("openai", "whisper-1") advertise application/pdf and image/jpeg, allowing the upload validator to accept files that the transcription endpoint will reject. In the active gateway implementation, short-circuit to the transcription-specific media set instead of unioning it with the format defaults.

AGENTS.md reference: AGENTS.md:L6-L12

Useful? React with 👍 / 👎.

Comment on lines +155 to +157

@chatgpt-codex-connector chatgpt-codex-connector Bot Aug 18, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Accept standard MIME types for supported containers

When a browser reports a supported .mp4, .webm, or .mpeg upload using its standard video/mp4, video/webm, or video/mpeg MIME type, this branch merges only AUDIO_MEDIA_TYPES, so the transcription helper rejects the file despite supporting those container formats. The aliases already exist in VIDEO_MEDIA_TYPES; include the applicable container MIME aliases in the transcription-specific set in the active gateway implementation.

AGENTS.md reference: AGENTS.md:L6-L12

Useful? React with 👍 / 👎.

}

return new Set(
Object.entries(baseSupport)
.filter(([_, supported]) => supported)
Expand All @@ -158,7 +167,7 @@ export function getSupportedMediaTypes(
export function isMediaTypeSupported(
mediaType: string,
format: ModelFormat,
model?: ModelName,
model?: string,
): boolean {
return getSupportedMediaTypes(format, model).has(mediaType);
}
17 changes: 17 additions & 0 deletions packages/proxy/schema/model_list.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -940,9 +940,19 @@
"azure"
]
},
"gpt-transcribe": {
"format": "openai",
"flavor": "chat",
"transcription": true,
"displayName": "GPT Transcribe",
"available_providers": [
"openai"
]
},
"gpt-4o-mini-transcribe": {
"format": "openai",
"flavor": "chat",
"transcription": true,
"input_cost_per_mil_tokens": 1.25,
"output_cost_per_mil_tokens": 5,
"displayName": "GPT-4o mini Transcribe",
Expand Down Expand Up @@ -983,6 +993,7 @@
"gpt-4o-transcribe": {
"format": "openai",
"flavor": "chat",
"transcription": true,
"input_cost_per_mil_tokens": 2.5,
"output_cost_per_mil_tokens": 10,
"max_input_tokens": 16000,
Expand All @@ -995,6 +1006,7 @@
"gpt-4o-transcribe-diarize": {
"format": "openai",
"flavor": "chat",
"transcription": true,
"input_cost_per_mil_tokens": 2.5,
"output_cost_per_mil_tokens": 10,
"max_input_tokens": 16000,
Expand All @@ -1007,6 +1019,7 @@
"gpt-4o-mini-transcribe-2025-12-15": {
"format": "openai",
"flavor": "chat",
"transcription": true,
"input_cost_per_mil_tokens": 1.25,
"output_cost_per_mil_tokens": 5,
"displayName": "GPT-4o mini Transcribe (2025-12-15)",
Expand Down Expand Up @@ -1048,6 +1061,7 @@
"gpt-4o-mini-transcribe-2025-03-20": {
"format": "openai",
"flavor": "chat",
"transcription": true,
"input_cost_per_mil_tokens": 1.25,
"output_cost_per_mil_tokens": 5,
"displayName": "GPT-4o mini Transcribe (2025-03-20)",
Expand Down Expand Up @@ -2401,6 +2415,7 @@
"whisper-1": {
"format": "openai",
"flavor": "chat",
"transcription": true,
"available_providers": [
"openai",
"azure"
Expand Down Expand Up @@ -5806,6 +5821,7 @@
"accounts/fireworks/models/whisper-v3": {
"format": "openai",
"flavor": "chat",
"transcription": true,
"max_input_tokens": 4096,
"max_output_tokens": 4096,
"available_providers": [
Expand All @@ -5815,6 +5831,7 @@
"accounts/fireworks/models/whisper-v3-turbo": {
"format": "openai",
"flavor": "chat",
"transcription": true,
"max_input_tokens": 4096,
"max_output_tokens": 4096,
"available_providers": [
Expand Down
6 changes: 6 additions & 0 deletions packages/proxy/schema/models.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,12 @@ export type ModelEndpointType = (typeof ModelEndpointType)[number];
export const ModelSchema = z.object({
format: z.enum(ModelFormats),
flavor: z.enum(ModelFlavors),
transcription: z
.boolean()
.nullish()
.describe(
"The model is a speech-to-text model served on /audio/transcriptions, not chat/completions.",
),
multimodal: z.boolean().nullish(),
input_cost_per_token: z.number().nullish(),
output_cost_per_token: z.number().nullish(),
Expand Down
Loading