feat: add llms files #4416
feat: add llms files #4416
Conversation
dishafaujdar
requested review from
Mayaleeeee,
akshatnema,
anshgoyalevil,
asyncapi-bot-eve,
derberg and
sambhavgupta0705
as code owners
✅ Deploy Preview for asyncapi-website ready!Built without sensitive environment variables
To edit notification comments on pull requests, go to your Netlify project configuration. |
WalkthroughAdds two TypeScript scripts that generate llms.txt and llms-full.txt, integrates their invocation into the build pipeline, adds corresponding public text outputs, and includes unit tests for both scripts. Changes
Sequence Diagram(s)sequenceDiagram
autonumber
participant Index as scripts/index.ts
participant LTxt as buildLlmsTxt()
participant LFull as buildLlmsFull()
participant FS as File System
participant Config as ../config/posts.json
Index->>LTxt: invoke buildLlmsTxt()
LTxt->>FS: write ./public/llms.txt
FS-->>LTxt: write complete
LTxt-->>Index: resolve
Index->>LFull: invoke buildLlmsFull()
LFull->>Config: dynamic import posts.json
Config-->>LFull: posts data
LFull->>FS: write ./public/llms-full.txt
FS-->>LFull: write complete
LFull-->>Index: resolve
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~25 minutes Suggested reviewers
Poem
Pre-merge checks and finishing touches✅ Passed checks (3 passed)
✨ Finishing touches
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 5
🧹 Nitpick comments (2)
components/Calendar.tsx (1)
54-54: Trim trailing whitespace.Minor tidy-up.
- </div> + </div>scripts/index.ts (1)
37-38: Consider isolating LLMS generation failures from the main build.Optional guard so docs build isn’t blocked by non-critical LLMS outputs.
- await buildLlmsTxt(); - await buildLlmsFull(); + try { + await buildLlmsTxt(); + await buildLlmsFull(); + } catch (err) { + console.error('LLMS build failed:', err); + }
📜 Review details
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
⛔ Files ignored due to path filters (1)
package-lock.jsonis excluded by!**/package-lock.json
📒 Files selected for processing (4)
components/Calendar.tsx(2 hunks)scripts/build-llms-full.ts(1 hunks)scripts/build-llms.ts(1 hunks)scripts/index.ts(2 hunks)
🧰 Additional context used
🧠 Learnings (2)
📚 Learning: 2024-11-01T09:35:23.912Z
Learnt from: akshatnema
PR: asyncapi/website#3101
File: scripts/build-rss.js:4-6
Timestamp: 2024-11-01T09:35:23.912Z
Learning: Converting `getAllPosts()` to asynchronous in `scripts/build-rss.js` causes the existing build system to fail.
Applied to files:
scripts/build-llms-full.tsscripts/index.ts
📚 Learning: 2024-11-10T18:16:22.273Z
Learnt from: vishvamsinh28
PR: asyncapi/website#3284
File: scripts/build-post-list.js:49-53
Timestamp: 2024-11-10T18:16:22.273Z
Learning: In `scripts/build-post-list.js`, the `walkDirectories` function depends on sequential execution because it mutates `resultObj`. Parallelizing the loop with `Promise.all` causes tests to fail.
Applied to files:
scripts/index.ts
🧬 Code graph analysis (2)
scripts/build-llms-full.ts (1)
types/scripts/build-posts-list.ts (1)
Result(37-42)
scripts/index.ts (2)
scripts/build-llms.ts (1)
buildLlmsTxt(15-56)scripts/build-llms-full.ts (1)
buildLlmsFull(28-144)
🪛 GitHub Actions: PR testing - if Node project
scripts/build-llms-full.ts
[error] 10-10: ModuleNotFoundError: Cannot find module '../config/posts.json' from 'scripts/build-llms-full.ts' during 'npm test --if-present'.
🪛 Biome (2.1.2)
components/Calendar.tsx
[error] 35-35: Expected an expression but instead found '<<'.
Expected an expression here.
(parse)
[error] 35-35: Expected an expression but instead found '<<'.
Expected an expression here.
(parse)
[error] 36-36: expected , but instead found className
Remove className
(parse)
[error] 36-36: expected , but instead found =
Remove =
(parse)
[error] 36-36: expected , but instead found twMerge
Remove twMerge
(parse)
[error] 36-36: expected , but instead found (
Remove (
(parse)
[error] 36-36: expected ) but instead found ,
Remove ,
(parse)
[error] 36-36: expected , but instead found )
Remove )
(parse)
[error] 36-37: Expected a JSX attribute but instead found '======='.
Expected a JSX attribute here.
(parse)
[error] 38-38: expected > but instead found <
Remove <
(parse)
[error] 43-44: Unexpected token. Did you mean {'>'} or >?
(parse)
[error] 44-44: Unexpected token. Did you mean {'>'} or >?
(parse)
[error] 44-44: Unexpected token. Did you mean {'>'} or >?
(parse)
[error] 44-44: Unexpected token. Did you mean {'>'} or >?
(parse)
[error] 44-44: Unexpected token. Did you mean {'>'} or >?
(parse)
[error] 44-44: Unexpected token. Did you mean {'>'} or >?
(parse)
[error] 44-44: Unexpected token. Did you mean {'>'} or >?
(parse)
🪛 GitHub Actions: Run Cypress E2E Tests
components/Calendar.tsx
[error] 35-74: Merge conflict marker encountered in Calendar.tsx. Remove conflict markers (<<<<<<< HEAD, =======, >>>>>>> upstream/master) and resolve the merge conflict.
⏰ Context from checks skipped due to timeout of 180000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (1)
- GitHub Check: Lighthouse CI
🔇 Additional comments (1)
scripts/build-llms.ts (1)
32-32: No change required — default branch ismaster.
Default branch is "master" and the spec file is at spec/asyncapi.md, so the existing blob/master/spec/asyncapi.md link is correct.
components/Calendar.tsx
Outdated
| <<<<<<< HEAD | ||
| <div className={twMerge('overflow-hidden rounded-md border border-gray-200 bg-white p-4 h-full', className)}> | ||
| ======= | ||
| <div | ||
| className={twMerge( | ||
| 'overflow-hidden rounded-md border border-gray-200 bg-white p-4 h-full flex flex-col gap-2', | ||
| className | ||
| )} | ||
| > | ||
| >>>>>>> upstream/master | ||
| <Heading level={HeadingLevel.h2} typeStyle={HeadingTypeStyle.mdSemibold}> |
There was a problem hiding this comment.
💡 Verification agent
🧩 Analysis chain
Resolve merge conflict markers; keep the flex wrapper.
Conflict markers break the build and E2E. Prefer the upstream/master wrapper to preserve layout.
Apply this diff:
-<<<<<<< HEAD
- <div className={twMerge('overflow-hidden rounded-md border border-gray-200 bg-white p-4 h-full', className)}>
-=======
<div
className={twMerge(
'overflow-hidden rounded-md border border-gray-200 bg-white p-4 h-full flex flex-col gap-2',
className
)}
>
->>>>>>> upstream/masterRun to ensure no other conflict markers remain:
🏁 Script executed:
#!/bin/bash
rg -n '^(<<<<<<<|=======|>>>>>>>)' -SLength of output: 803
Resolve merge conflict markers; keep upstream/master flex wrapper
- components/Calendar.tsx — remove the merge markers at lines 35–44 and keep the upstream/master div (the twMerge variant with 'flex flex-col gap-2') so layout is preserved.
- rg also matched lines starting with '=======' in: markdown/blog/asyncapi-and-apicurio-for-asynchronous-apis.md (lines 30,156,186,211) and LICENSE-docs (lines 3,55,376) — inspect and remove any true conflict separators.
- After fixes, run: rg -n '^(<<<<<<<|=======|>>>>>>>)' -S to confirm no markers remain.
🧰 Tools
🪛 Biome (2.1.2)
[error] 35-35: Expected an expression but instead found '<<'.
Expected an expression here.
(parse)
[error] 35-35: Expected an expression but instead found '<<'.
Expected an expression here.
(parse)
[error] 36-36: expected , but instead found className
Remove className
(parse)
[error] 36-36: expected , but instead found =
Remove =
(parse)
[error] 36-36: expected , but instead found twMerge
Remove twMerge
(parse)
[error] 36-36: expected , but instead found (
Remove (
(parse)
[error] 36-36: expected ) but instead found ,
Remove ,
(parse)
[error] 36-36: expected , but instead found )
Remove )
(parse)
[error] 36-37: Expected a JSX attribute but instead found '======='.
Expected a JSX attribute here.
(parse)
[error] 38-38: expected > but instead found <
Remove <
(parse)
[error] 43-44: Unexpected token. Did you mean {'>'} or >?
(parse)
[error] 44-44: Unexpected token. Did you mean {'>'} or >?
(parse)
[error] 44-44: Unexpected token. Did you mean {'>'} or >?
(parse)
[error] 44-44: Unexpected token. Did you mean {'>'} or >?
(parse)
[error] 44-44: Unexpected token. Did you mean {'>'} or >?
(parse)
[error] 44-44: Unexpected token. Did you mean {'>'} or >?
(parse)
[error] 44-44: Unexpected token. Did you mean {'>'} or >?
(parse)
🪛 GitHub Actions: Run Cypress E2E Tests
[error] 35-74: Merge conflict marker encountered in Calendar.tsx. Remove conflict markers (<<<<<<< HEAD, =======, >>>>>>> upstream/master) and resolve the merge conflict.
🤖 Prompt for AI Agents
In components/Calendar.tsx around lines 35–45 there are unresolved Git conflict
markers; remove the conflict markers and keep the upstream/master div variant
that includes twMerge('overflow-hidden rounded-md border border-gray-200
bg-white p-4 h-full flex flex-col gap-2', className) so the flex layout is
preserved, deleting the HEAD version and all <<<<<<<, =======, >>>>>>> lines;
also inspect the other files flagged by rg
(markdown/blog/asyncapi-and-apicurio-for-asynchronous-apis.md at lines
~30,156,186,211 and LICENSE-docs at lines ~3,55,376) and remove any true
conflict separators there as well; after edits run rg -n
'^(<<<<<<<|=======|>>>>>>>)' -S to confirm no conflict markers remain.
There was a problem hiding this comment.
Actionable comments posted: 0
♻️ Duplicate comments (3)
scripts/build-llms-full.ts (3)
9-12: Fix JSON import: replace dynamic JSON import with fs+path (avoids ESM JSON import and bundler pitfalls).Dynamic
import('../config/posts.json')commonly fails in CI/tests (needsassert { type: 'json' }, bundler hooks, or experimental flags). Read via FS with an absolute path.Apply:
-import fs from 'fs'; +import { promises as fs } from 'node:fs'; +import { dirname, resolve } from 'node:path'; +import { fileURLToPath } from 'node:url'; @@ -async function getAllPosts(): Promise<Result> { - const posts = (await import('../config/posts.json')).default as Result; - return posts; -} +async function getAllPosts(): Promise<Result> { + const __filename = fileURLToPath(import.meta.url); + const __dirname = dirname(__filename); + const jsonPath = resolve(__dirname, '../config/posts.json'); + const raw = await fs.readFile(jsonPath, 'utf8'); + return JSON.parse(raw) as Result; +}Also applies to: 1-2
135-144: Fix “Total URLs” undercount; write asynchronously and ensure ./public exists; make output path CWD‑safe.Current count is computed before the final block and sync write may race/build stall; also missing mkdir. Compute after assembling, append,
mkdir -p, and write via async API using a path relative to this file.-# END OF RESOURCE LIST -# Total URLs: ${content.split('\n').filter(line => line.includes('http')).length} -`; - - // Write the file - fs.writeFileSync('./public/llms-full.txt', content, 'utf8'); - - const urlCount = content.split('\n').filter(line => line.includes('http')).length; - console.log(`✅ llms-full.txt generated successfully with ${urlCount} URLs at ./public/llms-full.txt`); +# END OF RESOURCE LIST +`; + + // Compute final URL count and append + const urlCount = content + .split('\n') + .filter((line) => /^https?:\/\//.test(line.trim())) + .length; + content += `# Total URLs: ${urlCount}\n`; + + // Ensure output dir and write (path resolved relative to this file) + const __filename = fileURLToPath(import.meta.url); + const __dirname = dirname(__filename); + const outDir = resolve(__dirname, '../public'); + const outFile = resolve(outDir, 'llms-full.txt'); + await fs.mkdir(outDir, { recursive: true }); + await fs.writeFile(outFile, content, 'utf8'); + console.log(`✅ llms-full.txt generated successfully with ${urlCount} URLs at ${outFile}`);Also applies to: 60-60
91-103: Guard non-array post types and skip docsTree.Prevents accidental inclusion of nav trees/objects.
- Object.keys(posts).forEach(postType => { - if (!['docs', 'blog', 'about'].includes(postType)) { - const postArray = (posts as any)[postType]; - if (postArray && postArray.length > 0) { + Object.keys(posts).forEach((postType) => { + if (!['docs', 'blog', 'about', 'docsTree'].includes(postType)) { + const postArray = (posts as any)[postType]; + if (Array.isArray(postArray) && postArray.length > 0) { const sectionTitle = postType.charAt(0).toUpperCase() + postType.slice(1); content += `\n## ${sectionTitle}\n`; postArray.forEach((item: any) => { content += `${base}${item.slug} - ${item.title}\n`; }); } } });
🧹 Nitpick comments (4)
scripts/build-llms-full.ts (4)
72-81: Harden blog sorting for missing/invalid dates.
new Date(x).getTime()can beNaN, making comparator unstable.- posts.blog - .sort((a: any, b: any) => new Date(b.date).getTime() - new Date(a.date).getTime()) + posts.blog + .sort((a: any, b: any) => { + const tb = Date.parse(b?.date ?? ''); + const ta = Date.parse(a?.date ?? ''); + return (isNaN(tb) ? 0 : tb) - (isNaN(ta) ? 0 : ta); // desc + }) .forEach((post: any) => { - const date = post.date ? ` (${post.date.split('T')[0]})` : ''; + const date = + post?.date && !isNaN(Date.parse(post.date)) + ? ` (${new Date(post.date).toISOString().split('T')[0]})` + : '' content += `${base}${post.slug} - ${post.title}${date}\n`; });
31-31: Make base URL configurable (works in previews and local runs).- const base = 'https://www.asyncapi.com'; + const base = (process.env.SITE_URL ?? 'https://www.asyncapi.com').replace(/\/$/, '');
127-129: Correct v2 vs v3 note for accuracy.v2 already has
channels; the change is how operations are modeled.-# v3.0 uses 'channels' and 'operations' objects instead of v2's inline channel definitions +# In v2, operations live inline on each channel item via `publish`/`subscribe`. +# In v3.0, operations are first-class under a top-level `operations` object and channels reference them.
65-69: Optional: removeanyby using the Details type for stronger checks.- .forEach((doc: any) => { + .forEach((doc: { slug: string; title: string }) => { content += `${base}${doc.slug} - ${doc.title}\n`; }); @@ - .forEach((post: any) => { + .forEach((post: { slug: string; title: string; date?: string }) => { const date = post.date ? ` (${post.date.split('T')[0]})` : ''; content += `${base}${post.slug} - ${post.title}${date}\n`; }); @@ - posts.about.forEach((page: any) => { + posts.about.forEach((page: { slug: string; title: string }) => { content += `${base}${page.slug} - ${page.title}\n`; }); @@ - postArray.forEach((item: any) => { + postArray.forEach((item: { slug: string; title: string }) => { content += `${base}${item.slug} - ${item.title}\n`; });Also applies to: 75-80, 86-88, 98-100
📜 Review details
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
⛔ Files ignored due to path filters (1)
package-lock.jsonis excluded by!**/package-lock.json
📒 Files selected for processing (1)
scripts/build-llms-full.ts(1 hunks)
🧰 Additional context used
🧠 Learnings (4)
📚 Learning: 2024-11-01T09:35:23.912Z
Learnt from: akshatnema
PR: asyncapi/website#3101
File: scripts/build-rss.js:4-6
Timestamp: 2024-11-01T09:35:23.912Z
Learning: Converting `getAllPosts()` to asynchronous in `scripts/build-rss.js` causes the existing build system to fail.
Applied to files:
scripts/build-llms-full.ts
📚 Learning: 2025-01-18T08:44:43.614Z
Learnt from: akshatnema
PR: asyncapi/website#3423
File: tests/index.test.js:2-7
Timestamp: 2025-01-18T08:44:43.614Z
Learning: In the AsyncAPI website project, JavaScript test files must include the .ts extension when importing TypeScript files (e.g., `require('../scripts/build-rss.ts')`). This is enforced by the project's configuration which uses `moduleResolution: "bundler"` in tsconfig.json and TypeScript-aware ESLint plugins. The .ts extensions are required even though the files are imported using CommonJS require statements.
Applied to files:
scripts/build-llms-full.ts
📚 Learning: 2024-11-10T18:16:22.273Z
Learnt from: vishvamsinh28
PR: asyncapi/website#3284
File: scripts/build-post-list.js:49-53
Timestamp: 2024-11-10T18:16:22.273Z
Learning: In `scripts/build-post-list.js`, the `walkDirectories` function depends on sequential execution because it mutates `resultObj`. Parallelizing the loop with `Promise.all` causes tests to fail.
Applied to files:
scripts/build-llms-full.ts
📚 Learning: 2025-01-18T08:44:43.614Z
Learnt from: akshatnema
PR: asyncapi/website#3423
File: tests/index.test.js:2-7
Timestamp: 2025-01-18T08:44:43.614Z
Learning: In the AsyncAPI website project, JavaScript test files must include the .ts extension when importing TypeScript files (e.g., `require('../scripts/build-rss.ts')`). This is a project-specific requirement enforced by the linting rules and build setup, even though it differs from typical Node.js behavior.
Applied to files:
scripts/build-llms-full.ts
🧬 Code graph analysis (1)
scripts/build-llms-full.ts (1)
types/scripts/build-posts-list.ts (1)
Result(37-42)
⏰ Context from checks skipped due to timeout of 180000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (1)
- GitHub Check: Lighthouse CI
🔇 Additional comments (2)
scripts/build-llms-full.ts (2)
28-33: No action required — buildLlmsFull is awaited in scripts/index.ts.
Foundawait buildLlmsFull()at scripts/index.ts:38; no changes needed.
41-46: Verify external URLs/branches and confirm definitions endpointsFile: scripts/build-llms-full.ts (lines 41–46)
- GitHub spec link uses blob/master — the file is reachable at that path; confirm the repo default branch (master vs main) and update the link if the default is now main. (github.com)
- Definitions JSON endpoints under https://www.asyncapi.com/definitions/... (3.0.0, 2.6.0, 2.5.0, 2.4.0) could not be fetched by the automated check — run curl/HEAD from CI or locally to confirm they exist or redirect. ()
- Case studies path is /casestudies (not /case‑studies). (asyncapi.com)
- Slack entry asyncapi.slack.com is reachable. (asyncapi.slack.com)
- v3 and v2 spec reference pages exist at /docs/reference/specification/v3.0.0 and /docs/reference/specification/v2.6.0. (asyncapi.com)
Also applies to lines: 47–55, 56–59, 116–121, 123–124, 130–134.
|
⚡️ Lighthouse report for the changes in this PR:
Lighthouse ran on https://deploy-preview-4416--asyncapi-website.netlify.app/ |
There was a problem hiding this comment.
Actionable comments posted: 0
🧹 Nitpick comments (3)
components/Calendar.tsx (3)
50-50: Strip trailing whitespace to satisfy lintersExtraneous spaces after the closing div can trip formatters.
- </div> + </div>
32-33: Potential runtime crash when eventsData is undefined
eventsData?.filter(...).lengthcan throw because.lengthis accessed onundefinedwheneventsDatais absent. Use.some()with nullish coalescing.- const eventsExist = eventsData?.filter((event: IEvent) => moment(event.date).isAfter(currentDate)).length > 0; + const eventsExist = eventsData?.some((event: IEvent) => moment(event.date).isAfter(currentDate)) ?? false;
46-46: Avoid array index as React keyUse a stable, unique key to prevent reconciliation bugs.
- <li key={index} data-testid='Calendar-list-item'> + <li key={event.url || `${event.date}-${index}`} data-testid='Calendar-list-item'>
📜 Review details
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (1)
components/Calendar.tsx(1 hunks)
⏰ Context from checks skipped due to timeout of 180000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (1)
- GitHub Check: Lighthouse CI
🔇 Additional comments (1)
components/Calendar.tsx (1)
35-40: Merge markers resolved and flex wrapper preserved — LGTM; confirm no real conflict markers remain
- rg matched long '=' lines at: markdown/blog/asyncapi-and-apicurio-for-asynchronous-apis.md:30,156,186,211 and LICENSE-docs:3,55,376 — these are repeated '=' lines (matched by the previous regex), not explicit '<<<<<<<'/'>>>>>>>' markers.
- Run this targeted check to be sure: rg -n '^(<<<<<<<|>>>>>>>)' -S — if it returns nothing, mark resolved.
There was a problem hiding this comment.
Actionable comments posted: 0
♻️ Duplicate comments (2)
scripts/build-llms.ts (1)
53-56: Async FS + mkdir — resolved.You switched to async fs and ensure ./public exists. This addresses the earlier ENOENT/blocking I/O concern.
scripts/build-llms-full.ts (1)
11-17: Good: robust FS + absolute path for posts.json.Switching from dynamic import to fs+path fixes ModuleNotFoundError in tests/build.
🧹 Nitpick comments (7)
scripts/build-llms.ts (3)
18-18: Allow base URL override (keep canonical as default).Make the base host configurable for previews/self-hosts while defaulting to asyncapi.com.
Apply:
- const base = 'https://www.asyncapi.com'; + const base = (process.env.SITE_URL || 'https://www.asyncapi.com').replace(/\/$/, '');
29-33: Prefer docs site for spec reference (GitHub branch can drift).Add the v3 docs reference (keep GitHub as secondary if you want).
## Specifications & Schemas ${base}/definitions/3.0.0/asyncapi.json - AsyncAPI 3.0.0 JSON Schema ${base}/definitions/2.6.0/asyncapi.json - AsyncAPI 2.6.0 JSON Schema -https://github.com/asyncapi/spec/blob/master/spec/asyncapi.md - Latest specification +${base}/docs/reference/specification/v3.0.0 - Latest specification ( docs ) +https://github.com/asyncapi/spec - Specification source ( GitHub )
1-1: Minor: use node:fs/promises and absolute output path (CWD‑safe).Avoid CWD assumptions and use the promises module directly.
-import fs from 'fs'; +import { mkdir, writeFile } from 'node:fs/promises'; +import { resolve, dirname } from 'node:path'; +import { fileURLToPath } from 'node:url'; @@ - await fs.promises.mkdir('./public', { recursive: true }); - await fs.promises.writeFile('./public/llms.txt', llmsContent, 'utf8'); + const __filename = fileURLToPath(import.meta.url); + const __dirname = dirname(__filename); + const outDir = resolve(__dirname, '../public'); + const outFile = resolve(outDir, 'llms.txt'); + await mkdir(outDir, { recursive: true }); + await writeFile(outFile, llmsContent, 'utf8');Also applies to: 54-55
scripts/build-llms-full.ts (4)
79-85: Guard against invalid/missing blog dates in sort.If a post lacks a valid date, the comparator can return NaN leading to unstable order.
- .sort((a: any, b: any) => new Date(b.date).getTime() - new Date(a.date).getTime()) + .sort((a: any, b: any) => { + const ta = Number.isFinite(new Date(a?.date ?? '').getTime()) ? new Date(a.date).getTime() : 0; + const tb = Number.isFinite(new Date(b?.date ?? '').getTime()) ? new Date(b.date).getTime() : 0; + return tb - ta; + })
110-120: Use stable join link for Slack.Workspaces URLs can require auth; prefer the public join URL on the site.
-https://asyncapi.slack.com - Official Slack community +${base}/slack - Official Slack community
143-146: Count URLs precisely (not just lines containing “http”).This avoids under/over‑count and is resilient if multiple URLs appear on a line.
- const urlCount = content.split('\n').filter((line) => line.includes('http')).length; + const urlCount = (content.match(/https?:\/\/\S+/g) || []).length;
147-150: CWD‑independent output path.Resolve ./public relative to this script to avoid surprises in different runners.
- await fs.promises.mkdir('./public', { recursive: true }); - await fs.promises.writeFile('./public/llms-full.txt', content, 'utf8'); - console.log(`✅ llms-full.txt generated successfully with ${urlCount} URLs at ./public/llms-full.txt`); + const outDir = resolve(dirname(fileURLToPath(import.meta.url)), '../public'); + const outFile = resolve(outDir, 'llms-full.txt'); + await fs.promises.mkdir(outDir, { recursive: true }); + await fs.promises.writeFile(outFile, content, 'utf8'); + console.log(`✅ llms-full.txt generated successfully with ${urlCount} URLs at ${outFile}`);
📜 Review details
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (2)
scripts/build-llms-full.ts(1 hunks)scripts/build-llms.ts(1 hunks)
🧰 Additional context used
🧠 Learnings (4)
📚 Learning: 2024-11-01T09:35:23.912Z
Learnt from: akshatnema
PR: asyncapi/website#3101
File: scripts/build-rss.js:4-6
Timestamp: 2024-11-01T09:35:23.912Z
Learning: Converting `getAllPosts()` to asynchronous in `scripts/build-rss.js` causes the existing build system to fail.
Applied to files:
scripts/build-llms-full.ts
📚 Learning: 2025-01-18T08:44:43.614Z
Learnt from: akshatnema
PR: asyncapi/website#3423
File: tests/index.test.js:2-7
Timestamp: 2025-01-18T08:44:43.614Z
Learning: In the AsyncAPI website project, JavaScript test files must include the .ts extension when importing TypeScript files (e.g., `require('../scripts/build-rss.ts')`). This is enforced by the project's configuration which uses `moduleResolution: "bundler"` in tsconfig.json and TypeScript-aware ESLint plugins. The .ts extensions are required even though the files are imported using CommonJS require statements.
Applied to files:
scripts/build-llms-full.ts
📚 Learning: 2024-11-10T18:16:22.273Z
Learnt from: vishvamsinh28
PR: asyncapi/website#3284
File: scripts/build-post-list.js:49-53
Timestamp: 2024-11-10T18:16:22.273Z
Learning: In `scripts/build-post-list.js`, the `walkDirectories` function depends on sequential execution because it mutates `resultObj`. Parallelizing the loop with `Promise.all` causes tests to fail.
Applied to files:
scripts/build-llms-full.ts
📚 Learning: 2025-01-18T08:44:43.614Z
Learnt from: akshatnema
PR: asyncapi/website#3423
File: tests/index.test.js:2-7
Timestamp: 2025-01-18T08:44:43.614Z
Learning: In the AsyncAPI website project, JavaScript test files must include the .ts extension when importing TypeScript files (e.g., `require('../scripts/build-rss.ts')`). This is a project-specific requirement enforced by the linting rules and build setup, even though it differs from typical Node.js behavior.
Applied to files:
scripts/build-llms-full.ts
🧬 Code graph analysis (1)
scripts/build-llms-full.ts (1)
types/scripts/build-posts-list.ts (1)
Result(37-42)
🔇 Additional comments (4)
scripts/build-llms-full.ts (3)
96-108: Nice: skip non‑array post types (docsTree).Prevents accidental inclusion of nav trees/objects. LGTM.
33-37: Confirmed — both buildLlmsTxt() and buildLlmsFull() are awaited in the build orchestrator.
scripts/index.ts calls await buildLlmsTxt(); and await buildLlmsFull(); (lines 37–38).
127-130: Replace invalid repo link; spec uses “master”File: scripts/build-llms-full.ts lines 127-130 — the URL github.com/asyncapi/asyncapi/tree/master/examples is incorrect (no repo named "asyncapi" in the org). (github.com)
Examples are in asyncapi/spec/tree/master/examples and the spec repo’s default branch is master. Update the first URL to asyncapi/spec/tree/master/examples or confirm the intended repo. (github.com)
Likely an incorrect or invalid review comment.
scripts/build-llms.ts (1)
41-47: Verified: 'casestudies' and 'tools' routes exist.
Found: pages/casestudies/index.tsx, pages/casestudies/[id].tsx, pages/tools/index.tsx (also pages/[lang]/tools). No renaming required.
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (11)
scripts/build-llms-full.ts (11)
1-4: Prefer node: specifiers and fs.promises for clarity and ESM correctness.Use built‑in module specifiers and import promises API directly; simplifies calls and avoids CJS interop edge cases.
-import fs from 'fs'; -import type { Result } from '@/types/scripts/build-posts-list'; -import { dirname, resolve } from 'path'; +import { promises as fs } from 'node:fs'; +import type { Result } from '@/types/scripts/build-posts-list'; +import { dirname, resolve } from 'node:path'; import { fileURLToPath } from 'url';And update below:
- const raw = await fs.promises.readFile(jsonPath, 'utf8'); + const raw = await fs.readFile(jsonPath, 'utf8');- await fs.promises.mkdir('./public', { recursive: true }); - await fs.promises.writeFile('./public/llms-full.txt', content, 'utf8'); + await fs.mkdir(outDir, { recursive: true }); + await fs.writeFile(outPath, content, 'utf8');
11-17: Harden read/parse with actionable error messages.Wrap IO/JSON parse to aid CI debugging.
async function getAllPosts(): Promise<Result> { const __filename = fileURLToPath(import.meta.url); const __dirname = dirname(__filename); const jsonPath = resolve(__dirname, '../config/posts.json'); - const raw = await fs.promises.readFile(jsonPath, 'utf8'); - return JSON.parse(raw) as Result; + try { + const raw = await fs.readFile(jsonPath, 'utf8'); + return JSON.parse(raw) as Result; + } catch (err) { + const msg = err instanceof Error ? err.message : String(err); + throw new Error(`Failed to read/parse posts.json at ${jsonPath}: ${msg}`); + } }Does this script also run from your compiled output (if you compile scripts), or only via tsx/ts-node? If compiled, confirm that ../config/posts.json is available relative to the emitted scripts.
33-38: Make base URL configurable to support previews/staging.Allow override via env while defaulting to production.
-export async function buildLlmsFull(): Promise<void> { +export async function buildLlmsFull(): Promise<void> { console.log('Building llms-full.txt...'); - const base = 'https://www.asyncapi.com'; + const base = (process.env.SITE_URL || 'https://www.asyncapi.com').replace(/\/$/, ''); const posts = await getAllPosts();Set SITE_URL in Netlify/Vercel preview so links match the deploy-preview domain if desired.
45-51: Avoid hard‑coding “master” branch in GitHub links.Use HEAD to be branch‑agnostic.
-https://github.com/asyncapi/spec/blob/master/spec/asyncapi.md - Latest specification source +https://github.com/asyncapi/spec/blob/HEAD/spec/asyncapi.md - Latest specification source
67-75: Normalize URLs when concatenating base + slug.Guard against missing/duplicate slashes by using URL().
- posts.docs + posts.docs .sort((a, b) => (a.slug ?? '').localeCompare(b.slug ?? '')) .forEach((doc: any) => { - content += `${base}${doc.slug} - ${doc.title}\n`; + const href = new URL(doc.slug ?? '/', base).toString(); + content += `${href} - ${doc.title}\n`; });
77-86: Handle missing/invalid dates in blog sort to avoid NaN comparator.Current comparator can return NaN leading to unstable ordering.
- posts.blog - .sort((a: any, b: any) => new Date(b.date).getTime() - new Date(a.date).getTime()) + const toEpoch = (d?: string) => (d ? Date.parse(d) || 0 : 0); + posts.blog + .sort((a: any, b: any) => toEpoch(b.date) - toEpoch(a.date)) .forEach((post: any) => { - const date = post.date ? ` (${post.date.split('T')[0]})` : ''; - content += `${base}${post.slug} - ${post.title}${date}\n`; + const href = new URL(post.slug ?? '/', base).toString(); + const date = post.date ? ` (${(post.date as string).split('T')[0]})` : ''; + content += `${href} - ${post.title}${date}\n`; });
88-94: Stabilize About page order.Sort for deterministic output.
- posts.about.forEach((page: any) => { - content += `${base}${page.slug} - ${page.title}\n`; - }); + posts.about + .sort((a: any, b: any) => (a.slug ?? '').localeCompare(b.slug ?? '')) + .forEach((page: any) => { + const href = new URL(page.slug ?? '/', base).toString(); + content += `${href} - ${page.title}\n`; + });
96-108: Dynamic sections: type‑safety and deterministic order.Avoid any and keep sorting consistent.
- Object.keys(posts).forEach((postType) => { + (Object.keys(posts) as string[]).forEach((postType) => { if (!['docs', 'blog', 'about', 'docsTree'].includes(postType)) { - const postArray = (posts as any)[postType]; - if (Array.isArray(postArray) && postArray.length > 0) { + const postArray = (posts as any)[postType]; + if (Array.isArray(postArray) && postArray.length > 0) { const sectionTitle = postType.charAt(0).toUpperCase() + postType.slice(1); content += `\n## ${sectionTitle}\n`; - postArray.forEach((item: any) => { - content += `${base}${item.slug} - ${item.title}\n`; - }); + postArray + .sort((a: any, b: any) => (a.slug ?? '').localeCompare(b.slug ?? '')) + .forEach((item: any) => { + const href = new URL(item.slug ?? '/', base).toString(); + content += `${href} - ${item.title}\n`; + }); } } });
127-130: Avoid hard‑coded “master” in example links.Prefer HEAD to survive default‑branch changes.
-https://github.com/asyncapi/asyncapi/tree/master/examples - Official AsyncAPI document examples -https://github.com/asyncapi/spec/tree/master/examples - Specification examples +https://github.com/asyncapi/asyncapi/tree/HEAD/examples - Official AsyncAPI document examples +https://github.com/asyncapi/spec/tree/HEAD/examples - Specification examplesConfirm the first link actually has an examples folder; if not, drop it and keep only the spec repo.
135-139: Use a public Slack join URL instead of workspace URL.Direct workspace URL often requires being logged in.
-https://asyncapi.slack.com - Official Slack community +https://www.asyncapi.com/slack - Official Slack community
143-151: Robust URL counting and path resolution for output.Count actual URLs (regex) and write using an absolute path relative to this script to avoid CWD issues.
- // Calculate URL count after all content is assembled - const urlCount = content.split('\n').filter((line) => line.includes('http')).length; + // Calculate URL count after all content is assembled + const urlCount = (content.match(/\bhttps?:\/\/\S+/g) ?? []).length; content += `# Total URLs: ${urlCount}\n`; - // Ensure output dir exists and write the file - await fs.promises.mkdir('./public', { recursive: true }); - await fs.promises.writeFile('./public/llms-full.txt', content, 'utf8'); - console.log(`✅ llms-full.txt generated successfully with ${urlCount} URLs at ./public/llms-full.txt`); + // Ensure output dir exists and write the file (relative to repo, not CWD) + const outDir = resolve(dirname(fileURLToPath(import.meta.url)), '../public'); + const outPath = resolve(outDir, 'llms-full.txt'); + await fs.mkdir(outDir, { recursive: true }); + await fs.writeFile(outPath, content, 'utf8'); + console.log(`✅ llms-full.txt generated successfully with ${urlCount} URLs at ${outPath}`);
📜 Review details
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (1)
scripts/build-llms-full.ts(1 hunks)
🧰 Additional context used
🧠 Learnings (3)
📚 Learning: 2024-11-01T09:35:23.912Z
Learnt from: akshatnema
PR: asyncapi/website#3101
File: scripts/build-rss.js:4-6
Timestamp: 2024-11-01T09:35:23.912Z
Learning: Converting `getAllPosts()` to asynchronous in `scripts/build-rss.js` causes the existing build system to fail.
Applied to files:
scripts/build-llms-full.ts
📚 Learning: 2025-01-18T08:44:43.614Z
Learnt from: akshatnema
PR: asyncapi/website#3423
File: tests/index.test.js:2-7
Timestamp: 2025-01-18T08:44:43.614Z
Learning: In the AsyncAPI website project, JavaScript test files must include the .ts extension when importing TypeScript files (e.g., `require('../scripts/build-rss.ts')`). This is enforced by the project's configuration which uses `moduleResolution: "bundler"` in tsconfig.json and TypeScript-aware ESLint plugins. The .ts extensions are required even though the files are imported using CommonJS require statements.
Applied to files:
scripts/build-llms-full.ts
📚 Learning: 2024-11-10T18:16:22.273Z
Learnt from: vishvamsinh28
PR: asyncapi/website#3284
File: scripts/build-post-list.js:49-53
Timestamp: 2024-11-10T18:16:22.273Z
Learning: In `scripts/build-post-list.js`, the `walkDirectories` function depends on sequential execution because it mutates `resultObj`. Parallelizing the loop with `Promise.all` causes tests to fail.
Applied to files:
scripts/build-llms-full.ts
🧬 Code graph analysis (1)
scripts/build-llms-full.ts (1)
types/scripts/build-posts-list.ts (1)
Result(37-42)
scripts/build-llms-full.ts
Outdated
| // Add GitHub repositories and tools | ||
| content += `\n## GitHub Repositories & Tools | ||
| https://github.com/asyncapi/asyncapi - Main AsyncAPI specification repository | ||
| https://github.com/asyncapi/generator - AsyncAPI code/docs generator | ||
| https://github.com/asyncapi/parser - AsyncAPI document parser | ||
| https://github.com/asyncapi/modelina - AsyncAPI model/type generator | ||
| https://github.com/asyncapi/cli - AsyncAPI CLI tool | ||
| https://github.com/asyncapi/studio - AsyncAPI visual editor | ||
| https://github.com/asyncapi/website - This website's source code | ||
| https://github.com/asyncapi/community - Community resources and governance | ||
| ## Migration Guides & Best Practices |
There was a problem hiding this comment.
💡 Verification agent
🧩 Analysis chain
Verify all GitHub repo links (some repos moved/renamed/archived).
Run a quick status check to catch 404s and branch issues.
🏁 Script executed:
#!/bin/bash
set -euo pipefail
urls=(
"https://github.com/asyncapi/asyncapi"
"https://github.com/asyncapi/generator"
"https://github.com/asyncapi/parser"
"https://github.com/asyncapi/modelina"
"https://github.com/asyncapi/cli"
"https://github.com/asyncapi/studio"
"https://github.com/asyncapi/website"
"https://github.com/asyncapi/community"
)
for u in "${urls[@]}"; do
code=$(curl -s -o /dev/null -w "%{http_code}" "$u")
echo "$code $u"
done | awk '$1!="200"{print "⚠️", $0} $1=="200"{print "✅", $0}'Length of output: 128
🏁 Script executed:
#!/bin/bash
set -euo pipefail
urls=(
"https://github.com/asyncapi/asyncapi"
"https://github.com/asyncapi/generator"
"https://github.com/asyncapi/parser"
"https://github.com/asyncapi/modelina"
"https://github.com/asyncapi/cli"
"https://github.com/asyncapi/studio"
"https://github.com/asyncapi/website"
"https://github.com/asyncapi/community"
)
for u in "${urls[@]}"; do
out="$(curl -sSL -o /dev/null -w '%{http_code} %{url_effective}' "$u")" || out="000 $u"
code="${out%% *}"
final="${out#* }"
if [ "$code" = "200" ]; then
printf "✅ %s %s\n" "$code" "$final"
else
printf "⚠️ %s %s\n" "$code" "$final"
fi
doneLength of output: 349
Update GitHub links — some redirect/are archived
scripts/build-llms-full.ts (lines 110–121): replace the two redirected/archived links:
- https://github.com/asyncapi/asyncapi → https://github.com/asyncapi/spec
- https://github.com/asyncapi/parser → https://github.com/asyncapi-archived-repos/parser-go (archived)
Other listed links (generator, modelina, cli, studio, website, community) returned 200 and are fine.
🤖 Prompt for AI Agents
In scripts/build-llms-full.ts around lines 110 to 121, two GitHub URLs are
outdated/redirected; replace https://github.com/asyncapi/asyncapi with
https://github.com/asyncapi/spec and replace https://github.com/asyncapi/parser
with https://github.com/asyncapi-archived-repos/parser-go so the document points
to the correct spec repo and the archived parser repository.
There was a problem hiding this comment.
Actionable comments posted: 1
📜 Review details
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
⛔ Files ignored due to path filters (1)
package-lock.jsonis excluded by!**/package-lock.json
📒 Files selected for processing (1)
package.json(1 hunks)
package.json
Outdated
| "gray-matter": "^4.0.3", | ||
| "he": "^1.2.0", | ||
| "i18next": "^23.12.2", | ||
| "i18next": "^25.5.2", |
There was a problem hiding this comment.
🧩 Analysis chain
Major bump to i18next v25 — verify next-i18next compatibility or hold at v24
- next-i18next 15.x explicitly added support for i18next v24; there’s no documented guarantee for v25 yet. Expect possible peer/version mismatches or subtle runtime breakages (e.g., language change semantics) unless verified. (classic.yarnpkg.com)
- i18next v25 includes potentially breaking changes; validate your app flows and any custom i18n utils. (i18next.com)
Recommended paths (pick one):
- Safe path: keep i18next on latest v24.x and (optionally) bump next-i18next to ≥15.4.x.
- Adopt v25: ensure next-i18next explicitly supports v25 or prove compatibility via tests.
Proposed diff (safe path):
- "i18next": "^25.5.2",
+ "i18next": "^24.2.2",Follow‑ups:
- If proceeding with v25, update next-i18next to the latest 15.4.x and run e2e smoke on locale switching, SSR/SSG pages, and dynamic namespace loads.
Run locally to surface peer/compat issues:
References:
- next-i18next changelog notes “support i18next v24”. (classic.yarnpkg.com)
- i18next v25 migration guide outlines potential breaking changes. (i18next.com)
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Show declared versions
jq -r '.dependencies | {i18next, "next-i18next", "react-i18next"}' package.json
# Install and print resolved versions
npm ci
npm ls i18next next-i18next react-i18next
# Inspect next-i18next peer deps (if any published)
npm view next-i18next@latest peerDependencies
# Quick grep for i18next changeLanguage usages (sensitive to v25 semantics)
rg -nP --type=ts --type=tsx '\bchangeLanguage\s*\(' -C2
# Optional: run a locale switch SSR smoke (project-specific)
npm run test:e2e || trueLength of output: 2167
Block merge — confirm compatibility before upgrading i18next to v25
- package.json currently declares i18next "^25.5.2" while next-i18next is "^15.3.0" (jq output from verification). next-i18next releases explicitly document support up to i18next v24 (15.4.0 changelog); no documented v25 support found. (github.com)
- i18next v25 contains potentially breaking changes (notably changeLanguage semantics) that can affect language switching, SSR/SSG and dynamic namespace loads — validate before upgrading. (i18next.com)
- npm ci failed in the sandbox due to lockfile out-of-sync (missing/invalid entries), so I could not fully resolve installed versions — run npm install locally and re-run dependency verification (npm ls) and e2e smoke tests.
Action (pick one):
- Safe path: pin i18next to latest v24. Proposed diff (File: package.json, line ~81):
- "i18next": "^25.5.2",
+ "i18next": "^24.2.2",- If you want to adopt v25: confirm an upstream next-i18next release that explicitly supports i18next v25, upgrade next-i18next to that release, update the lockfile (npm install), add react-i18next if required, then run smoke/e2e tests for language switching, SSR/SSG, and dynamic namespace loads.
📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| "i18next": "^25.5.2", | |
| "i18next": "^24.2.2", |
🤖 Prompt for AI Agents
In package.json around line 81 the i18next dependency is set to "^25.5.2" which
is potentially incompatible with next-i18next "^15.3.0" (which documents support
only up to v24); replace that entry by pinning i18next to the latest v24
release, run npm install to update package-lock.json, run npm ls to verify
dependency tree, and then execute full smoke/e2e tests (language switching,
SSR/SSG, dynamic namespace loading) to confirm no regressions; if you instead
want v25, first upgrade next-i18next to a release that explicitly supports
i18next v25, add any required peers (e.g., react-i18next), then run npm install,
npm ls and the same tests before merging.
|
@derberg on it. |
- Add build-llms.ts for generating llms.txt index file - Add build-llms-full.ts for comprehensive resource list - Integrate LLM file generation into build process - Help LLMs discover accurate AsyncAPI v3.0 documentation
dishafaujdar
force-pushed
the
feature/add-llms-files
branch
from
845704e to
79ab3aa
Compare
There was a problem hiding this comment.
- package-lock.json changes are still in PR, please remove
- please do not use fs sync functions, make sure to use async
- generated txt files do not seem to be complient with how standards is described and how people do it. Did you read https://llmstxt.org/ and looked at https://llmstxt.site/ to see some examples?
|
@dishafaujdar the pipeline is failing |
|
@sambhavgupta0705 can idea how I can fix it? |
|
@dishafaujdar You would have to fix the tests for checks to pass. You can run |
|
@anshgoyalevil Hey, can you please review it once? I've added the tests for both the files and run them, there's no error. |
There was a problem hiding this comment.
Actionable comments posted: 0
🧹 Nitpick comments (4)
tests/scripts/build-llms-full.test.ts (2)
7-7: Fix indentation for consistency.The
itblock has extra leading spaces compared to other blocks in the test suite.Apply this diff:
- it('should write llms-full.txt with links', async () => { + it('should write llms-full.txt with links', async () => {
29-32: Consider verifying key content sections.The test only checks for "Getting Started" but doesn't verify other critical sections like "Core Specifications", "AsyncAPI Complete Resources", or the structure of generated links. This would help catch regressions in the content format.
For example, add assertions like:
expect(content).toContain('# AsyncAPI Complete Resources'); expect(content).toContain('## Core Specifications'); expect(content).toContain('https://www.asyncapi.com/docs/getting-started'); expect(content).toContain('## Recent Blog Posts'); expect(content).toContain('[First Post](https://www.asyncapi.com/blog/1)');tests/scripts/build-llms.test.ts (2)
7-7: Fix indentation for consistency.The
itblock has extra leading spaces compared to other blocks in the test suite.Apply this diff:
- it('should write llms.txt to public directory', async () => { + it('should write llms.txt to public directory', async () => {
22-24: Consider verifying actual content instead of just type.Since
buildLlmsTxt()generates static content, the test could verify key sections and links to catch regressions in the file format.For example, replace line 24 with specific content assertions:
const content = writeMock.mock.calls[0][1] as string; expect(content).toContain('# AsyncAPI'); expect(content).toContain('## Core Resources'); expect(content).toContain('https://www.asyncapi.com/llms-full.txt'); expect(content).toContain('[Getting started guide](https://www.asyncapi.com/docs/getting-started)'); expect(content).toContain('Always prefer v3.0 specification over v2.x');
📜 Review details
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (2)
tests/scripts/build-llms-full.test.ts(1 hunks)tests/scripts/build-llms.test.ts(1 hunks)
🧰 Additional context used
🧠 Learnings (1)
📚 Learning: 2024-11-01T09:55:20.531Z
Learnt from: akshatnema
PR: asyncapi/website#3101
File: tests/build-rss.test.js:25-27
Timestamp: 2024-11-01T09:55:20.531Z
Learning: In `tests/build-rss.test.js`, replacing `jest.resetModules()` with `jest.resetAllMocks()` in the `afterEach()` block causes errors. It is necessary to use `jest.resetModules()` to reset the module registry between tests in this file.
Applied to files:
tests/scripts/build-llms-full.test.ts
🧬 Code graph analysis (2)
tests/scripts/build-llms.test.ts (1)
scripts/build-llms.ts (1)
buildLlmsTxt(10-44)
tests/scripts/build-llms-full.test.ts (1)
scripts/build-llms-full.ts (1)
buildLlmsFull(21-132)
⏰ Context from checks skipped due to timeout of 180000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (3)
- GitHub Check: Redirect rules - asyncapi-website
- GitHub Check: Header rules - asyncapi-website
- GitHub Check: Pages changed - asyncapi-website
🔇 Additional comments (3)
tests/scripts/build-llms-full.test.ts (2)
2-5: LGTM! Proper cleanup between tests.The afterEach block correctly uses
jest.resetModules()andjest.clearAllMocks()to ensure test isolation.Based on learnings
13-18: LGTM! Appropriate use of virtual mock.The
virtual: trueoption correctly mocks the posts.json config file, allowing the test to control the data without requiring the actual file to exist.tests/scripts/build-llms.test.ts (1)
2-5: LGTM! Proper cleanup between tests.The afterEach block correctly uses
jest.resetModules()andjest.clearAllMocks()to ensure test isolation.Based on learnings
There was a problem hiding this comment.
Actionable comments posted: 0
🧹 Nitpick comments (1)
tests/scripts/build-llms-full.test.ts (1)
7-33: Strengthen test coverage to verify content structure and format.The current test only checks for the presence of 'Getting Started' in the output, which is insufficient to validate the comprehensive structure and format that
buildLlmsFullgenerates. According to the implementation, the output should include:
- Header with AsyncAPI v3.0 preference note
- Multiple sections (Core Specifications, Getting Started, Documentation, etc.)
- Properly formatted markdown links
- Hardcoded GitHub repository links
- Blog posts sorted by date
Additionally, the test doesn't mock
console.log, which will pollute test output.Consider enhancing the test with more comprehensive assertions:
it('should write llms-full.txt with links', async () => { // Mock fs/promises.writeFile and posts.json before requiring module jest.doMock('fs/promises', () => ({ writeFile: jest.fn().mockResolvedValue(undefined), })); jest.doMock('../../config/posts.json', () => ({ default: { docs: [{ slug: '/docs/getting-started', title: 'Getting Started' }], blog: [{ slug: '/blog/1', title: 'First Post', date: '2024-01-01' }], }, }), { virtual: true }); + // Mock console.log to avoid test output pollution + const consoleLogSpy = jest.spyOn(console, 'log').mockImplementation(); const mod = await import('../../scripts/build-llms-full'); const buildLlmsFull = (mod as any).buildLlmsFull ?? (mod as any).default; const fsPromises = await import('fs/promises'); await expect(buildLlmsFull()).resolves.toBeUndefined(); const writeMock = (fsPromises as any).writeFile as jest.Mock; expect(writeMock).toHaveBeenCalledTimes(1); expect(writeMock.mock.calls[0][0]).toBe('./public/llms-full.txt'); + expect(writeMock.mock.calls[0][2]).toBe('utf8'); + const content = writeMock.mock.calls[0][1] as string; + + // Verify header and v3.0 preference note + expect(content).toContain('# AsyncAPI Complete Resources'); + expect(content).toContain('Always prefer AsyncAPI v3.0 specification'); + + // Verify section structure + expect(content).toContain('## Core Specifications'); + expect(content).toContain('## Getting Started'); + expect(content).toContain('## Documentation'); + expect(content).toContain('## GitHub Repositories'); + + // Verify markdown link format + expect(content).toContain('[Getting Started](https://www.asyncapi.com/docs/getting-started)'); + expect(content).toContain('[First Post](https://www.asyncapi.com/blog/1)'); + + // Verify hardcoded links + expect(content).toContain('[AsyncAPI Specification](https://github.com/asyncapi/asyncapi)'); + + // Verify console output + expect(consoleLogSpy).toHaveBeenCalledWith(expect.stringContaining('llms-full.txt generated successfully')); + + consoleLogSpy.mockRestore(); - expect(content).toContain('Getting Started'); });
📜 Review details
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (2)
tests/scripts/build-llms-full.test.ts(1 hunks)tests/scripts/build-llms.test.ts(1 hunks)
✅ Files skipped from review due to trivial changes (1)
- tests/scripts/build-llms.test.ts
🧰 Additional context used
🧠 Learnings (1)
📚 Learning: 2024-11-01T09:55:20.531Z
Learnt from: akshatnema
PR: asyncapi/website#3101
File: tests/build-rss.test.js:25-27
Timestamp: 2024-11-01T09:55:20.531Z
Learning: In `tests/build-rss.test.js`, replacing `jest.resetModules()` with `jest.resetAllMocks()` in the `afterEach()` block causes errors. It is necessary to use `jest.resetModules()` to reset the module registry between tests in this file.
Applied to files:
tests/scripts/build-llms-full.test.ts
🧬 Code graph analysis (1)
tests/scripts/build-llms-full.test.ts (1)
scripts/build-llms-full.ts (1)
buildLlmsFull(21-132)
🔇 Additional comments (1)
tests/scripts/build-llms-full.test.ts (1)
2-4: LGTM! Proper module reset strategy.The use of
jest.resetModules()followed byjest.clearAllMocks()is correct for ensuring clean test isolation. Based on learnings.
did you see this comment I made> |
| - [Case studies](https://www.asyncapi.com/casestudies): Real-world usage examples | ||
|
|
||
| ## Optional | ||
| - [GitHub specification](https://github.com/asyncapi/spec/blob/master/spec/asyncapi.md): Source specification on GitHub |
There was a problem hiding this comment.
not needed imho
more important to mention https://github.com/asyncapi/bindings in core resources
There was a problem hiding this comment.
Actionable comments posted: 1
♻️ Duplicate comments (3)
scripts/build-llms-full.ts (3)
8-12: Critical: Dynamic import of posts.json breaks tests.The pipeline is failing with
Cannot find module '../config/posts.json'during test execution. Dynamic JSON imports are unreliable in test environments.Despite a past review suggesting this fix (marked "Addressed in commit 4ca90e4"), the code still uses dynamic import and tests are still failing.
Apply this diff to use filesystem reads with proper path resolution:
-import { writeFile } from 'fs/promises'; +import { writeFile, readFile } from 'fs/promises'; +import { dirname, resolve } from 'path'; +import { fileURLToPath } from 'url'; import type { Result } from '@/types/scripts/build-posts-list'; /** * Asynchronously retrieves all posts from the posts configuration file. */ async function getAllPosts(): Promise<Result> { - const posts = (await import('../config/posts.json')).default as Result; - - return posts; + const __filename = fileURLToPath(import.meta.url); + const __dirname = dirname(__filename); + const jsonPath = resolve(__dirname, '../config/posts.json'); + const raw = await readFile(jsonPath, 'utf8'); + return JSON.parse(raw) as Result; }
94-104: Update outdated GitHub repository URLs.Multiple repository URLs need correction:
- Line 94:
asyncapi/asyncapishould beasyncapi/spec(main specification repository)- Line 96:
asyncapi/parseris archived; link either to the archived repo explicitly or to current parser implementations- Line 103: References
asyncapi/asyncapi/tree/master/examplesshould useasyncapi/specThese issues were identified in a past review but remain unaddressed.
Apply this diff to fix the repository links:
content += '\n## GitHub Repositories\n'; - content += `- [AsyncAPI Specification](https://github.com/asyncapi/asyncapi): Main specification repository + content += `- [AsyncAPI Specification](https://github.com/asyncapi/spec): Main specification repository - [AsyncAPI Generator](https://github.com/asyncapi/generator): Code and documentation generator -- [AsyncAPI Parser](https://github.com/asyncapi/parser): AsyncAPI document parser library +- [AsyncAPI Parser JS](https://github.com/asyncapi/parser-js): AsyncAPI document parser library (JavaScript) - [AsyncAPI CLI](https://github.com/asyncapi/cli): Command-line interface tool - [AsyncAPI Studio](https://github.com/asyncapi/studio): Visual AsyncAPI editor - [Modelina](https://github.com/asyncapi/modelina): Model/type generation library - [AsyncAPI Bindings](https://github.com/asyncapi/bindings): Protocol bindings specifications ## Examples and Templates -- [Specification Examples](https://github.com/asyncapi/asyncapi/tree/master/examples): Official AsyncAPI document examples +- [Specification Examples](https://github.com/asyncapi/spec/tree/master/examples): Official AsyncAPI document examples - [Template Examples](https://github.com/asyncapi/spec/tree/master/examples): Additional specification examples
113-125: Add type safety check to prevent runtime errors with non-array post types.The loop assumes all post types are arrays, but
ResultincludesdocsTreewhich is aNavTreeobject. While the current code happens to fail safely (.lengthon object returnsundefined), it should explicitly guard against non-array types.Past review suggested this fix but it was not applied.
Apply this diff to add proper type guards:
// Add other important resources from posts Object.keys(posts).forEach((postType) => { - if (!['docs', 'blog'].includes(postType)) { + if (!['docs', 'blog', 'docsTree'].includes(postType)) { const postArray = (posts as any)[postType]; - if (postArray && postArray.length > 0) { + if (Array.isArray(postArray) && postArray.length > 0) { postArray.forEach((item: any) => { if (item.slug && item.title) { content += `- [${item.title}](${base}${item.slug})\n`;
🧹 Nitpick comments (1)
scripts/build-llms-full.ts (1)
127-133: Consider ensuring output directory exists before writing.The script writes to
./public/llms-full.txtwithout first ensuring the directory exists. While this likely works in practice (thepublic/directory is created by other build steps), it's a best practice to ensure the directory exists for robustness.Add directory creation before the write:
+ import { writeFile, mkdir } from 'fs/promises'; + // Write the file + await mkdir('./public', { recursive: true }); await writeFile('./public/llms-full.txt', content, 'utf8');
📜 Review details
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (6)
public/llms-full.txt(1 hunks)public/llms.txt(1 hunks)scripts/build-llms-full.ts(1 hunks)scripts/build-llms.ts(1 hunks)tests/scripts/build-llms-full.test.ts(1 hunks)tests/scripts/build-llms.test.ts(1 hunks)
🚧 Files skipped from review as they are similar to previous changes (4)
- tests/scripts/build-llms-full.test.ts
- scripts/build-llms.ts
- public/llms.txt
- tests/scripts/build-llms.test.ts
🧰 Additional context used
🧬 Code graph analysis (1)
scripts/build-llms-full.ts (1)
types/scripts/build-posts-list.ts (1)
Result(37-42)
🪛 GitHub Actions: PR testing - if Node project
scripts/build-llms-full.ts
[error] 9-9: Cannot find module '../config/posts.json' from 'scripts/build-llms-full.ts'. During step 'npm test --if-present' the tests failed due to missing posts.json dependency.
🪛 LanguageTool
public/llms-full.txt
[uncategorized] ~98-~98: If this is a compound adjective that modifies the following noun, use a hyphen.
Context: ...roglia_project_interview) - [Inside the Open Source Climb: Aayush’s Path to AsyncAPI Leader...
(EN_COMPOUND_ADJECTIVE_INTERNAL)
| - [Introduction](https://www.asyncapi.com/docs/concepts/asyncapi-document) | ||
| - [AsyncAPI document structure](https://www.asyncapi.com/docs/concepts/asyncapi-document/structure) |
There was a problem hiding this comment.
Duplicate entries detected in generated output.
The generated file contains duplicate "Introduction" entries:
- Lines 26-27: Both point to
/docs/concepts/asyncapi-document - Lines 43-44: Both point to
/docs/tutorials/getting-started
These duplicates originate from the generator logic in scripts/build-llms-full.ts. The script likely includes the same documentation pages multiple times when filtering and assembling content sections.
Also applies to: 43-44
🤖 Prompt for AI Agents
In public/llms-full.txt around lines 26-27 and 43-44 the generated output
contains duplicate links because scripts/build-llms-full.ts is assembling the
same documentation pages multiple times; update scripts/build-llms-full.ts to
deduplicate entries when collecting/concatenating pages (e.g., track seen
URLs/slugs in a Set and skip repeats while preserving original order), ensure
any filtering/flattening steps collapse duplicates before writing the file, and
add a unit or build-time assertion that the final list contains only unique
links so future runs won't reintroduce duplicates.
6 participants
feat: #4408
Summary
This PR adds automated generation of LLM discovery files to help AI systems find accurate AsyncAPI documentation and prefer v3.0 over v2.x specifications.
Changes
build-llms.tsfor generatingllms.txtindex filebuild-llms-full.tsfor a comprehensive resource listindex.ts)Files Generated
public/llms.txt- Small index pointing to key resourcespublic/llms-full.txt- Complete list of all AsyncAPI resourcesSummary by CodeRabbit
New Features
Documentation
Tests
Chores