CrawlWise: GEO Agent System

CrawlWise is an end-to-end Generative Engine Optimization (GEO) agent system that audits and improves webpages for SEO and AI-readiness. It combines a FastAPI backend, a modern React/Tailwind frontend, the PocketFlow agent framework, and the AlchemystAI LLM Proxy for robust, production-grade content analysis and generation.

---

🚀 What is CrawlWise?

• Input: A user submits a public URL (plus optional keywords and tone).
• Process:
  1. The backend crawls the page (using Crawl4AI), extracts content and metadata.
  2. PocketFlow orchestrates a multi-step agent pipeline:
     • Audit Node: Analyzes structure, SEO/GEO issues, and provides recommendations (via AlchemystAI LLM).
     • Enhancement Node: Generates improved intro, meta tags, and FAQ (via AlchemystAI LLM).
  3. The frontend displays results in a clean, animated UI.
• Output: A structured JSON with audit and improvement suggestions.

---

🛠️ Tech Stack

📽️ Demo

demo.mp4

Click here to watch the demo video if the playback does't work

---

🛠️ Tech Stack

• Backend: FastAPI, PocketFlow, Crawl4AI, LangChain, AlchemystAI LLM Proxy
• Frontend: React, TailwindCSS
• Agent Orchestration: PocketFlow
• LLM Provider: AlchemystAI

---

🏗️ Architecture

flowchart LR
    A["User (Frontend)"] --> B["FastAPI Backend"]
    B --> C["Crawl4AI Crawler"]
    C --> D["Extracted Content & Metadata"]
    D --> E["PocketFlow Agent Pipeline"]
    subgraph E ["PocketFlow Agent Pipeline"]
        F["Audit Node (LLM via AlchemystAI)"] --> H["Audit JSON"]
        G["Enhancement Node (LLM via AlchemystAI)"] --> I["Improvements JSON"]
    end
    D --> F
    H --> G
    H --> J["API Response"]
    I --> J
    J --> A
    F -.-> K["AlchemystAI LLM Proxy"]
    G -.-> K

Loading

User submits a URL via the frontend. The backend crawls the page, runs the PocketFlow agent pipeline (audit and enhancement nodes, both using AlchemystAI LLM), and returns results to the frontend for display.

---

🌐 Key Links

• Framework: https://github.com/The-Pocket/PocketFlow
• AlchemystAI LLM Proxy: https://getalchemystai.com

---

⚡ Quickstart

1. Clone the repo

   git clone <your-fork-or-this-repo>
   cd CrawlWise

2. Set up environment
   • Create a .env file in src/ with your AlchemystAI API key:

     ALCHEMYST_API_KEY=sk-...  # Get yours at https://getalchemystai.com

3. Install backend dependencies

   cd src
   pip install -r requirements.txt
   # or
   pip install .

4. Run the backend

   uvicorn main:app --reload

5. Run the frontend

   cd frontend
   npm install
   npm run dev

---

🐳 Docker Deployment

CrawlWise supports Docker for easy deployment and production use.

1. Build the Docker image:

   docker build -t crawlwise .

2. Run the container:

   docker run -p 8000:8000 --env-file src/.env crawlwise

   • The backend will be available at http://localhost:8000.
   • Make sure your .env file with ALCHEMYST_API_KEY is present and passed in.

You can also deploy the frontend separately (e.g., on Netlify or Vercel) and point it to the backend API.

---

🔑 Get an AlchemystAI API Key

• Sign up at https://getalchemystai.com
• Copy your API key and add it to your .env as ALCHEMYST_API_KEY

---

📡 Example API Usage

Request:

POST /api/v1/audit
{
  "agent": "seo",
  "url": "https://example.com/your-blog-post",
  "keywords": ["LLM SEO", "AI search"],
  "tone": "professional"
}

Response:

{
  "audit": {
    "structure": ["Missing H1", "Too many H3s"],
    "issues": ["No meta title", "No structured FAQ section"],
    "recommendations": ["Add a meta title using keywords", "Use structured data"]
  },
  "improvements": {
    "intro": "In this article, we explore how to use large language models for modern SEO...",
    "meta": {
      "title": "Mastering SEO for LLMs in 2025",
      "description": "Learn how to optimize your content for AI-driven search engines using LLM strategies."
    },
    "faqs": [
      {
        "question": "What is Generative Engine Optimization (GEO)?",
        "answer": "GEO is the practice of optimizing your content for inclusion in AI-generated answers."
      }
    ]
  }
}

---

🤖 How It Works (Agent Pipeline)

• PocketFlow orchestrates the agent pipeline:
  1. CrawlExtractNode: Fetches and parses the webpage.
  2. AuditContentNode: Audits content/metadata for SEO/GEO (via LLM).
  3. GenerateEnhancementsNode: Generates improved intro, meta, FAQ (via LLM).
• All LLM calls are routed through the AlchemystAI Proxy for reliability and cost efficiency.

---

👥 Contributing

• PRs and issues welcome!
• See PocketFlow for agent framework docs and patterns.

---

License

MIT