Skip to content

feat: add Caesar as a web search provider - #284

Open
TF-Caesar wants to merge 1 commit into
virattt:mainfrom
TF-Caesar:add-caesar-search-provider
Open

TF-Caesar wants to merge 1 commit into
virattt:mainfrom
TF-Caesar:add-caesar-search-provider

Conversation

@TF-Caesar

Copy link
Copy Markdown

Adds Caesar as a fifth web_search backend alongside Exa, Perplexity, Tavily, and LangSearch.

What makes it worth a slot: Caesar returns the passages it selected for your query inline with each search result, so for most questions the agent does not need a second round trip to read the page. The other four providers return a title, a URL, and a snippet; getting the text that actually answers the question is a follow-up fetch.

What that looks like

Caesar's snippet field is the page's meta description. Its passages field is query-conditioned. Same document, two different queries, zero overlap in the passages returned:

query: "work stealing balancing load across worker threads"
  -> "That notified processor will steal half the tasks in the batch, and in turn notify another..."

query: "atomic reference counting overhead in the old scheduler"
  -> "There are many outstanding references to the task structure: the scheduler and each waker..."

Both from tokio.rs/blog/2019-10-scheduler. So the provider surfaces passages and falls back to snippet when a result has none. That is the same preference LangSearch's provider already makes when it takes summary over its own shorter snippet, so I followed that rather than inventing a new field.

Concretely, on why did tokio avoid the atomic increment in wake_by_ref across 5 results:

text handed to the agent top result answers the question
meta description 1,148 chars no
query-selected passages 4,690 chars yes

Each result also carries a publication date, surfaced as published when the document has one. For a research agent that reasons over filings and news, knowing a source is from 2019 rather than last week seemed worth keeping.

Tradeoff, stated plainly

Passages make each response roughly 4x larger than a snippet-only mapping, so a web_search call costs more context tokens. I think that is the right trade for this agent, because the alternative is spending a fetch plus the whole page. If you would rather keep responses small, the mapper is one function (contentFor) and can prefer snippet instead.

Scope

Additive. The only existing files touched are the ones any new provider touches: the SearchProviderId union and SEARCH_PROVIDERS in src/utils/env.ts, registration in src/tools/registry.ts (guarded by process.env.CAESAR_API_KEY, exactly like the other four), the /search menu entry, the barrel export, and env.example. No existing provider is modified and no shared type is widened.

CAESAR_API_KEY is required and sent as Authorization: Bearer <key>. Missing key throws [Caesar API] CAESAR_API_KEY is not set, mirroring search_langsearch.

Tested

$ bun test
 81 pass  0 fail

$ tsc --noEmit
(clean)

src/tools/search/caesar.test.ts covers the Bearer header, the missing-key throw, passage preference, the snippet fallback when a result has no passages, empty-passage handling, alternate url/snippet field names, and a 401.

Beyond the mocked units I ran a real query through the built tool:

$ bun -e 'import { caesarSearch } from "./src/tools/search/caesar.ts";
  const r = JSON.parse(await caesarSearch.invoke({ query: "why did tokio avoid the atomic increment in wake_by_ref" }));
  console.log(r.data.results[0]);'

title    : Making the Tokio scheduler 10x faster | Tokio
published : 2019-10-13T00:00:00Z
chars    : 1208
snippet  : "It was observed that when Waker::wake is called, often times the original waker
            reference is no longer needed. This allows for reusing the atomic reference count
            when pushing the task into the run queue..."

That is the answer to the question, returned by the search call itself.


Disclosure: I work on GTM for Caesar. I wrote this to match how langsearch.ts is built rather than as a vendor drop-in, and I am happy to adjust the mapping, the priority placement, or the published field to fit your conventions.

Adds Caesar (https://docs.trycaesar.com) alongside Exa, Perplexity, Tavily,
and LangSearch. Mirrors the LangSearch provider: raw fetch against a base
URL, the shared web_search tool and WEB_SEARCH_DESCRIPTION, formatToolResult
mapping, and a required CAESAR_API_KEY sent as a Bearer token.

Caesar returns query-selected passages inline with each search result, so
the mapper surfaces those rather than the page's meta description, the same
way the LangSearch provider prefers `summary` over its short `snippet`. It
also carries a publication date, which is surfaced as `published` when the
document has one.

- src/tools/search/caesar.ts: provider (passage-preferring, tolerant mapping)
- src/tools/search/caesar.test.ts: bun tests (Bearer, missing key, passages,
  fallback, 401)
- src/tools/search/index.ts, src/tools/registry.ts: export + register
- src/utils/env.ts: SearchProviderId + SEARCH_PROVIDERS entry
- src/components/select-list.ts: /search provider menu entry
- env.example: document CAESAR_API_KEY
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant