Does Stagehand integrate with LlamaIndex?

[rishwanth-aiseberg]

I have structured extraction workflow.. given an url.... but the present url scrapers (Jina, FireCrawl etc...) are not robust to real world scenarios.
Can i integrate Stagehand with [LlamaIndex Workflows](https://docs.llamaindex.ai/en/stable/module_guides/workflow/)

[EbrahimSolomon]

Yes. You can integrate Stagehand into LlamaIndex Workflows by wrapping Stagehand calls in a Workflow node/tool. The pattern is:

1. Use Stagehand (running on Browserbase) to robustly render + extract from a target URL (handle JS, lazy content, anti-bot, etc.).
2. Return structured JSON that matches a Pydantic schema you define.
3. Wire it into a LlamaIndex Workflow so downstream nodes (cleaning, dedupe, DB upsert, etc.) work on the validated object.

Call Stagehand via a tiny HTTP microservice:
Keep Stagehand in Node/TypeScript (where it shines with Playwright + Browserbase), expose a single HTTP endpoint, and call it from your Python Workflow.

1. Stagehand microservice (Node/TS, sketch):
   // server.ts (Express/Fastify) — minimal sketch
   import express from "express";
   import { stagehandExtract } from "./stagehandExtract"; // your Stagehand logic

const app = express();
app.use(express.json());

app.post("/extract", async (req, res) => {
const { url, schema } = req.body; // schema: JSONSchema/Pydantic-like
try {
const data = await stagehandExtract({ url, schema });
res.json({ ok: true, data });
} catch (e:any) {
res.status(500).json({ ok: false, error: e?.message ?? "extract failed" });
}
});

app.listen(process.env.PORT ?? 3000, () => {
console.log("stagehand service up");
});
Your stagehandExtract uses Stagehand + Browserbase project/key to open a browser, navigate, wait for network idle / selectors, then returns JSON that matches schema.

2. LlamaIndex Workflow node (Python):
   from llama_index.core.workflow import Workflow, StartEvent, step
   from pydantic import BaseModel
   import requests, tenacity

1) Define the structured output you expect

class Product(BaseModel):
title: str
price: float
currency: str | None = None
url: str

2) A helper that calls your Stagehand service with retries

@tenacity.retry(wait=tenacity.wait_exponential(multiplier=1, min=1, max=20),
stop=tenacity.stop_after_attempt(3),
reraise=True)
def run_stagehand(url: str) -> dict:
# NOTE: replace with your service URL
r = requests.post(
"http://localhost:3000/extract",
json={
"url": url,
"schema": Product.model_json_schema() # pass JSON schema down
},
timeout=60
)
r.raise_for_status()
payload = r.json()
if not payload.get("ok"):
raise RuntimeError(payload.get("error","stagehand returned error"))
return payload["data"]

3) Build a minimal Workflow

class CrawlEvent(StartEvent):
urls: list[str]

class CrawlWorkflow(Workflow):
@step()
def fetch_and_validate(self, ev: CrawlEvent):
results: list[Product] = []
for u in ev.urls:
raw = run_stagehand(u)
# Pydantic validation — guarantees structure downstream
results.append(Product(**raw))
return {"items": [p.model_dump() for p in results]}

wf = CrawlWorkflow()
out = wf.run(CrawlEvent(urls=[
"https://example.com/product/123",
"https://example.com/product/456",
]))
print(out)

This gives you:

Browser-grade robustness (Stagehand/Playwright)

Typed outputs (Pydantic)

Composable orchestration (LlamaIndex Workflows)

Notes:
Schema-first: always pass a schema (Pydantic JSON Schema) to Stagehand so extraction is constrained and less brittle.

Retries & timeouts: wrap with tenacity (as shown).

Anti-bot: run via Browserbase with proper headers, viewport, slowMo if needed; wait for content (e.g., page.wait_for_selector(...)).

Determinism: add deterministic “locators” (CSS/xpath) + small post-processing to stabilize output.

Parallelism: fan-out the URLs with a map step or asyncio in the node; cap concurrency to what your Browserbase project allows.