RAG All-in-one

Hello there! 👋

I'm Long Le, a Machine Learning Engineer passionate about building AI systems. This repository is my collection of RAG (Retrieval-Augmented Generation) resources to help you build powerful AI applications.

Feel free to connect with me on social media to discuss AI, machine learning, or this project:

|

Introduction

RAG All-in-one is a guide to building Retrieval-Augmented Generation (RAG) applications. It offers a collection of tools, libraries, and frameworks for RAG systems, with explanations of key components and recommendations for effective implementation.

RAG Architecture Diagram

RAG Components ✅

Component	Description
📄 Document Ingestor	Tools for ingesting and processing raw documents. Document loaders, parsers, and preprocessing tools
🤖 Agent Framework	End-to-end frameworks for building RAG applications. Unified solutions for RAG implementation
📀 Database	Databases optimized for storing and searching vector embeddings. Vector storage, similarity search, and indexing
💻 LLM	Large Language Models for generating responses. LLM providers and frameworks
📝 Embedding	Models and services for creating text embeddings. Embedding models and APIs
🖥️ LLM Observability	Tools for monitoring and analyzing LLM performance. Logging, tracing, and analytics
📕 Prompt Techniques	Methods for effective prompt engineering. Prompt templates and frameworks
🤔 Evaluation	Tools for assessing RAG system performance. Metrics and evaluation frameworks

Document Ingestor

Tools and libraries for ingesting various document formats, extracting text, and preparing data for further processing.

Library	Description	Link
LangChain Document Loaders	Comprehensive set of document loaders for various file types	GitHub
LlamaIndex Parser	Flexible document parsing and chunking capabilities for various file formats	GitHub
Docling	Document processing tool that parses diverse formats with advanced PDF understanding and AI integrations	GitHub
Unstructured	Library for pre-processing and extracting content from raw documents	Github
PyPDF	Library for reading and manipulating PDF files	GitHub
PyMuPDF	A Python binding for MuPDF, offering fast PDF processing capabilities	GitHub
MegaParse	Versatile parser for text, PDFs, PowerPoint, and Word documents with lossless information extraction	GitHub
Adobe PDF Extract	A service provided by Adobe for extracting content from PDF documents	Link
Azure AI Document Intelligence	A service provided by Azure for extracting content including text, tables, images from PDF documents	Link

Agent Framework

End-to-end frameworks that provide integrated solutions for building RAG applications.

Library	Description	Link
LangChain	Framework for building applications with LLMs and integrating with various data sources	GitHub
LlamaIndex	Data framework for building RAG systems with structured data	GitHub
Haystack	End-to-end framework for building NLP pipelines	GitHub
SmolAgents	A barebones library for agents	GitHub
txtai	Open-source embeddings database for semantic search and LLM workflows	GitHub
Pydantic AI	Agent Framework / shim to use Pydantic with LLMs	GitHub
OpenAI Agent	A lightweight, powerful framework for multi-agent workflows	GitHub

Vector Database

Databases optimized for storing and efficiently searching vector embeddings/text documents.

Database	Description	Link
FAISS	Efficient similarity search library from Facebook AI Research	GitHub
Milvus	Open-source vector database	GitHub
Qdrant	Vector similarity search engine	GitHub
Chroma	Open-source embedding database designed for RAG applications	GitHub
Weaviate	Open-source vector search engine	GitHub
LanceDB	Developer-friendly, embedded retrieval engine for multimodal AI	GitHub
Pinecone	Managed vector database for semantic search	Link
MongoDB	General-purpose document database	Link
Elasticsearch	Search and analytics engine that can store documents	Link

LLM

Large Language Models and platforms for generating responses based on retrieved context.

LLM	Description	Link
OpenAI API	Access to GPT models through API	Link
Claude	Anthropic's Claude series of LLMs	Link
Hugging Face LLM Models	Platform for open-source NLP models	Link
LLaMA	Meta's open-source large language model	Link
Mistral	Open-source and commercial models	Link
Cohere	API access to generative and embedding models	Link
DeepSeek	Advanced large language models for various applications	Link
Qwen	Alibaba Cloud's large language model accessible via API	Link
Ollama	Run open-source LLMs locally	Link

Embedding

Models and services for creating vector representations of text.

Embedding Solution	Description	Link
OpenAI Embeddings	API for text-embedding-ada-002 and newer models	Link
Sentence Transformers	Python framework for state-of-the-art sentence embeddings	Link
Cohere Embed	Specialized embedding models API	Link
Hugging Face Embeddings	Various embedding models	Link
E5 Embeddings	Microsoft's text embeddings	Link
BGE Embeddings	BAAI general embeddings	Link

LLM Observability

Tools for monitoring, analyzing, and improving LLM applications.

Library	Description	Link
Langfuse	Open source LLM engineering platform	GitHub
Opik/Comet	Debug, evaluate, and monitor LLM applications with tracing, evaluations, and dashboards	GitHub
Phoenix/Arize	Open-source observability for LLM applications	GitHub
Helicone	Open source LLM observability platform. One line of code to monitor, evaluate, and experiment	GitHub
Openlit	Open source platform for AI Engineering: OpenTelemetry-native LLM Observability, GPU Monitoring, Guardrails, Evaluations, Prompt Management, Vault, Playground	GitHub
Lunary	The production toolkit for LLMs. Observability, prompt management and evaluations.	GitHub
Langtrace	OpenTelemetry-based observability tool for LLM applications with real-time tracing and metrics	GitHub

Prompt Techniques

Methods and frameworks for effective prompt engineering in RAG systems.

Open Source Prompt Engineering Tools

Library	Description	Link
Prompt Engineering Guide	Comprehensive guide to prompt engineering	GitHub
DSPy	Framework for programming language models instead of prompting	GitHub
Guidance	Language for controlling LLMs	GitHub
LLMLingua	Prompt compression library for faster LLM inference	GitHub
Promptify	NLP task prompt generator for GPT, PaLM and other models	GitHub
PromptSource	Toolkit for creating and sharing natural language prompts	GitHub
Promptimizer	Library for optimizing prompts	GitHub
Selective Context	Context compression tool for doubling LLM content processing	GitHub
betterprompt	Testing suite for LLM prompts before production	GitHub

Documentation & Services

Resource	Description	Link
OpenAI Prompt Engineering	Official guide to prompt engineering from OpenAI	Link
LangChain Prompts	Templates and composition tools for prompts	Link
PromptPerfect	Tool for optimizing prompts	Link

Evaluation

Tools and frameworks for assessing and improving RAG system performance.

Library	Description	Link
FastChat	Open platform for training, serving, and evaluating LLM-based chatbots	Github
OpenAI Evals	Framework for evaluating LLMs and LLM systems	GitHub
RAGAS	Ultimate toolkit for evaluating and optimizing RAG systems	GitHub
Promptfoo	Open-source tool for testing and evaluating prompts	GitHub
DeepEval	Comprehensive evaluation library for LLM applications	GitHub
Giskard	Open-source evaluation and testing for ML & LLM systems	Github
PromptBench	Unified evaluation framework for large language models	Github
TruLens	Evaluation and tracking for LLM experiments with RAG-specific metrics	GitHub
EvalPlus	Rigorous evaluation framework for LLM4Code	Github
LightEval	All-in-one toolkit for evaluating LLMs	Github
LangTest	Test suite for comparing LLM models on accuracy, bias, fairness and robustness	Github
AgentEvals	Evaluators and utilities for measuring agent performance	Github