feat: optimize markdown urls for agents #4057
Draft
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
|
Deployment failed with the following error: Learn More: https://vercel.link/3Fpeeb1 |
|
Deployment failed with the following error: |
|
Hi @ruf-io check this out! |
vburlak
approved these changes
Oct 23, 2025
ruf-io
reviewed
Oct 24, 2025
| ]; | ||
|
|
||
| const hasAIAgentUserAgent = aiAgentPatterns.some((pattern) => | ||
| userAgent.toLowerCase().includes(pattern) |
There was a problem hiding this comment.
@saimonkat do the models really put their name in the user agent? I did not see that in testing
There was a problem hiding this comment.
@ruf-io the User-Agent detection is actually a fallback mechanism. The primary detection is the Accept header. User-Agent patterns serve as a fallback for specific agents (for example, Claude Code actually uses Claude-Test-Request), but the primary indicator is the Accept header.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR optimizes markdown URLs for AI agents by:
Technical Implementation
src/middleware.js- Intercepts AI agent requests and redirects to GitHub raw URLssrc/utils/ai-agent-detection.js- Detects AI agents and maps URLs to markdown file pathssrc/constants/content.js- Centralized configuration for content sectionsSupported Sections
/docs/*→content/docs/*.md/postgresql/*→content/postgresql/*.md/guides/*→content/guides/*.md/branching/*→content/branching/*.md/programs/*→content/pages/programs/*.md/use-cases/*→content/pages/use-cases/*.mdExcluded from Optimization
Tests
Tests
Full testing report by Claude Code
Successfully implemented direct markdown serving for AI agents, achieving 99% bandwidth reduction and 99% token savings with 100% content accuracy.
Test Results
1. AI Agent Detection ✅
User-Agent patterns tested (all working):
Accept header detection:
Accept: text/html→ HTMLAccept: application/json, text/plain, application/xml→ MarkdownAccept: */*→ HTML (default)Route coverage verified:
/docs/introduction→ serves markdown directly/guides/node→ serves markdown directly/postgresql/postgresql-extensions→ serves markdown directly/docs/changelog→ serves HTML (excluded, as expected)/guides→ serves HTML (excluded, as expected)/branching→ serves HTML (excluded, as expected)2. File Size Comparison
Result: Markdown is 82x smaller than HTML
3. Token Usage & Cost Savings
Projected savings at scale:
4. Content Accuracy ✅
Markdown version contains:
Result: 100% content match between HTML and Markdown versions
Summary
Preview