OS
Linux
GPU Library
CUDA 12.x
Python version
3.12
Describe the bug
I use the openai client and use the response_format parameter to pass the json_schema information, hoping that the tabbyAPI will respond strictly according to the json_schema I provide, but the output content is not exactly divisored by json_schema, but some errors are randomly generated.Make this feature unavailable.``
Reproduction steps
tabbyAPI :latest; exl2:0.2.8;ubuntu 22.04.
code ：
response_format={ "type": "json_schema", "json_schema": {"name": "data", "schema": json_schema}, } client = openai.OpenAI(base_url='http://host:5000/v1',api_key='sk-000') response = client.chat.completions.create( model="calme-3.2-instruct-78b-exl2", messages=[ {"role": "system", "content": system_prompt}, {"role": "user", "content": user_prompt} ], response_format=response_format ) result = response.choices[0].message.content # remove ```json and ``` result = result.replace('```json', '').replace('```', '')
prompt:
output format
`
{
"identified_needs": [
{
"type": "",
"description": "",
"importance_score": 8,
"importance_reasoning": "",
"source": ""
},
// .....
],
"analysis_summary": ""
}`
json_schema:
{ "type": "object", "properties": { "identified_needs": { "type": "array", "items": { "type": "object", "properties": { "type": { "type": "string", "enum": ["功能性需求", "非功能性需求"] }, "description": { "type": "string", "description": "需求描述" }, "importance_score": { "type": "integer", "description": "重要性评分", "minimum": 1, "maximum": 10 }, "importance_reasoning": { "type": "string", "description": "重要性理由" }, "source": { "type": "string", "description": "需求来源" } }, "required": [ "type", "description", "importance_score", "importance_reasoning", "source" ] } }, "analysis_summary": { "type": "string", "description": "分析总结" } }, "required": [ "identified_needs", "analysis_summary" ] }
However, in the response json, keys that do not match json_schema appear randomly. For example:
{ "identified_needs": [ { "type": "非功能性需求", "description": "产品防水清洗改进，主要针对功能类型产品的清洁问题", "importance_score": 9, "importance_reasoning": "防水和清洁是影响产品使用和维护的关键因素，直接关系到用户体验和产品寿命。", "source": "3. 功能类型的产品清洗问题目前是无法解决的痛点；" }, { "type": "功能性需求", "description": "吮吸和震动功能类型的感应类型产品创新", "importence_score": 8, "importance_reasoning": "用户反馈对既有功能的需要改进和对新型感应功能的兴趣，显示出对功能性差异化的关注。", "source": "3. 吮吸和震动类型的产品，希望供应商考虑往感应类型的方向进行产品开发，做出更多功能差异化，这一点也是普遍对功能感兴趣的用户的需求；" }, { "type": "非功能性需求", "description": "小型手办娃娃产品开发", "importance_score": 7, "importance_reasoning": "用户对于小型手办娃娃的兴趣显现了一种新的非功能性需求，可能与消费者对于便携性和角色关联性的需求相关。", "source": "6. 用户对小型手办娃娃的产品比较感兴趣；" } ], "analysis_summary": "" }
As in the example response above, there are random importance_score and importence_score in the response, but my json_schema declares that the need for importance_score is required.This inconspicuous mistake caused the business to fail.
Expected behavior
It is expected to perform decoding constraints as efficiently and accurately as mlc-ai/xgrammar.And the response content may not contain ```json``
Logs
No response
Additional context
No response
Acknowledgements
OS
Linux
GPU Library
CUDA 12.x
Python version
3.12
Describe the bug
I use the openai client and use the response_format parameter to pass the json_schema information, hoping that the tabbyAPI will respond strictly according to the json_schema I provide, but the output content is not exactly divisored by json_schema, but some errors are randomly generated.Make this feature unavailable.``
Reproduction steps
tabbyAPI :latest; exl2:0.2.8;ubuntu 22.04.
code ：
response_format={ "type": "json_schema", "json_schema": {"name": "data", "schema": json_schema}, } client = openai.OpenAI(base_url='http://host:5000/v1',api_key='sk-000') response = client.chat.completions.create( model="calme-3.2-instruct-78b-exl2", messages=[ {"role": "system", "content": system_prompt}, {"role": "user", "content": user_prompt} ], response_format=response_format ) result = response.choices[0].message.content # remove ```json and ``` result = result.replace('```json', '').replace('```', '')prompt:
output format
`
{ "identified_needs": [ { "type": "", "description": "", "importance_score": 8, "importance_reasoning": "", "source": "" }, // ..... ], "analysis_summary": "" }`
json_schema:
{ "type": "object", "properties": { "identified_needs": { "type": "array", "items": { "type": "object", "properties": { "type": { "type": "string", "enum": ["功能性需求", "非功能性需求"] }, "description": { "type": "string", "description": "需求描述" }, "importance_score": { "type": "integer", "description": "重要性评分", "minimum": 1, "maximum": 10 }, "importance_reasoning": { "type": "string", "description": "重要性理由" }, "source": { "type": "string", "description": "需求来源" } }, "required": [ "type", "description", "importance_score", "importance_reasoning", "source" ] } }, "analysis_summary": { "type": "string", "description": "分析总结" } }, "required": [ "identified_needs", "analysis_summary" ] }However, in the response json, keys that do not match json_schema appear randomly. For example:
{ "identified_needs": [ { "type": "非功能性需求", "description": "产品防水清洗改进，主要针对功能类型产品的清洁问题", "importance_score": 9, "importance_reasoning": "防水和清洁是影响产品使用和维护的关键因素，直接关系到用户体验和产品寿命。", "source": "3. 功能类型的产品清洗问题目前是无法解决的痛点；" }, { "type": "功能性需求", "description": "吮吸和震动功能类型的感应类型产品创新", "importence_score": 8, "importance_reasoning": "用户反馈对既有功能的需要改进和对新型感应功能的兴趣，显示出对功能性差异化的关注。", "source": "3. 吮吸和震动类型的产品，希望供应商考虑往感应类型的方向进行产品开发，做出更多功能差异化，这一点也是普遍对功能感兴趣的用户的需求；" }, { "type": "非功能性需求", "description": "小型手办娃娃产品开发", "importance_score": 7, "importance_reasoning": "用户对于小型手办娃娃的兴趣显现了一种新的非功能性需求，可能与消费者对于便携性和角色关联性的需求相关。", "source": "6. 用户对小型手办娃娃的产品比较感兴趣；" } ], "analysis_summary": "" }As in the example response above, there are random importance_score and importence_score in the response, but my json_schema declares that the need for importance_score is required.This inconspicuous mistake caused the business to fail.
Expected behavior
It is expected to perform decoding constraints as efficiently and accurately as mlc-ai/xgrammar.And the response content may not contain ```json``
Logs
No response
Additional context
No response
Acknowledgements