Add Content-Signal and markdown content negotiation - #201
Draft
injoon5 wants to merge 1 commit into
Draft
Conversation
Declare ai-train/search/ai-input preferences in robots.txt, and serve text/markdown (with x-markdown-tokens) when agents send Accept: text/markdown. Prerendered routes are rewritten to /api/markdown on Vercel so static HTML is not served in that case. Co-authored-by: Injoon Oh <injoon5@icloud.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Deploying web with
|
| Latest commit: |
e033ef4
|
| Status: | 🚫 Build failed. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Declares AI content-use preferences and serves markdown to agents that ask for it.
Content Signals
static/robots.txtnow includes:Search indexing stays on. Model training is opted out.
ai-input=yesmatches the new markdown-for-agents path — agents may ground on the page.The same value is sent as a
Content-Signalheader on markdown responses.Markdown negotiation
Requests with
Accept: text/markdown(preferred overtext/html) get a markdown document instead of HTML:Content-Type: text/markdown; charset=utf-8Vary: Acceptx-markdown-tokens(4 chars/token estimate)?lang=en/ cookie for English)/now, and/healthare generated from metadata, not scraped HTMLSSR routes are handled in
hooks.server.js. Prerendered pages never hit that hook on Vercel, sovercel.jsonrewritesAccept: text/markdownto/api/markdown?path=….Tests
Unit coverage for Accept parsing, robots.txt lockstep with the shared constant, listing/article helpers, and the
/api/markdownhandler.