Skip to content

test(agents): contract tests for _parse_native_tool_call shapes#6035

Open
alvinttang wants to merge 1 commit into
crewAIInc:mainfrom
alvinttang:test/native-tool-call-contract
Open

test(agents): contract tests for _parse_native_tool_call shapes#6035
alvinttang wants to merge 1 commit into
crewAIInc:mainfrom
alvinttang:test/native-tool-call-contract

Conversation

@alvinttang
Copy link
Copy Markdown

@alvinttang alvinttang commented Jun 4, 2026

Refs #4972.

@andrewclayman flagged on the issue that this kind of provider-specific tool-call bug really wants a small contract test per shape. The original fix landed in 25fcf39, but there's no unit-level coverage that would have caught the truthy-default regression at parse time. This PR adds that net.

What this locks in

Every native tool-call shape _parse_native_tool_call handles, asserted on the parsed tuple before pydantic validation or tool dispatch runs:

  • OpenAI-style object (function.arguments JSON string)
  • Gemini-style function_call object (Struct mapping coerced to dict)
  • Anthropic-style object (name + input on the call object itself)
  • Dict variants: OpenAI nested function, Bedrock toolUseId+input, mixed payloads with empty function + populated input
  • Negatives: empty input dict stays empty, fully missing args returns empty dict, unknown shape returns None

RED (simulated regression)

Reintroduced the original truthy-default bug locally — func_info.get("arguments", "{}") instead of func_info.get("arguments") — and reran the suite:

FAILED test_dict_bedrock_converse_format            - AssertionError: '{}' != {'search_query': 'test'}
FAILED test_dict_bedrock_with_empty_input_dict_stays_empty
FAILED test_dict_missing_input_and_function_returns_empty_args
FAILED test_dict_variants_preserve_args[tool_call1-expected_args1]
FAILED test_dict_variants_preserve_args[tool_call2-expected_args2]
5 failed, 8 passed in 1.13s

The five failures point straight at the dict-branch fall-through.

GREEN (current main)

13 passed in 1.04s

Pure test-only PR. No production code changed.

Summary by CodeRabbit

  • Tests
    • Added comprehensive test coverage for native tool call parsing across multiple LLM provider formats (OpenAI, Gemini, Anthropic, Bedrock), including edge cases, missing data scenarios, and regression coverage to ensure robust tool call handling.

Locks in the wire-format contract for every native tool-call shape
CrewAgentExecutor parses so provider-specific argument bugs (crewAIInc#4972 —
Bedrock Converse 'input' silently dropped) can't quietly regress.

Coverage:
- OpenAI-style object (function.arguments JSON string)
- Gemini-style function_call object (Struct mapping coerced to dict)
- Anthropic-style object (name + input on the call object itself)
- Dict variants: OpenAI nested function, Bedrock toolUseId+input,
  mixed payloads with empty function + populated input
- Negative cases: empty input dict stays empty, fully missing args
  returns empty dict, unknown shape returns None

Each assertion checks the parsed (call_id, func_name, func_args) tuple
before pydantic validation or tool dispatch runs, so a future
truthy-default regression (the original crewAIInc#4972 cause) fails fast at
parse time with a clear assertion message.

Refs crewAIInc#4972
@coderabbitai
Copy link
Copy Markdown

coderabbitai Bot commented Jun 4, 2026

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 22141e9f-3a37-4fac-91a8-b0cbab65004d

📥 Commits

Reviewing files that changed from the base of the PR and between 051fa0c and f6eb6ff.

📒 Files selected for processing (1)
  • lib/crewai/tests/agents/test_parse_native_tool_call.py

📝 Walkthrough

Walkthrough

This PR adds a comprehensive contract test module for CrewAgentExecutor._parse_native_tool_call, verifying correct parsing of provider-specific tool call formats (OpenAI, Gemini, Anthropic, Bedrock) into a standardized (call_id, func_name, func_args) tuple, with edge case coverage for missing ids, null arguments, and malformed inputs.

Changes

Native Tool Call Parsing Contract Tests

Layer / File(s) Summary
Module setup and helper
lib/crewai/tests/agents/test_parse_native_tool_call.py
Module documentation describing contract expectations, required imports (pytest, SimpleNamespace, mocks, CrewAgentExecutor), and _parse() helper for invoking the method under test.
Object-shaped provider format tests
lib/crewai/tests/agents/test_parse_native_tool_call.py
Tests for OpenAI-style objects (including id synthesis when missing), Gemini-style function_call objects (with args=None{} coercion), and Anthropic-style objects using name/input extraction with idcall_id mapping.
Dict-shaped format and edge case tests
lib/crewai/tests/agents/test_parse_native_tool_call.py
Tests for dict-based OpenAI format, Bedrock Converse dict regression (argument extraction from input when function-side is empty), missing argument field handling, unrecognized shape rejection, and parametrized validation of dict variants across mixed representations.

Estimated Code Review Effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

Poem

🐰 A test suite hops in, checking every wire,
OpenAI, Gemini, Anthropic fire!
Bedrock formats blend, ids bloom and grow,
Contract verified—parsers now know! 🌟

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 75.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title directly and clearly describes the main change: adding contract tests for the _parse_native_tool_call method's shape handling.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant