Merge pull request #20 from mongodb-developer/add_voyageai

Voyage AI doc update

Author: ajosh0504

docs/10-key-concepts/3-components-of-rag.mdx

@@ -6,12 +6,47 @@ RAG systems have two main components: **Retrieval** and **Generation**.
 
 Retrieval mainly involves processing your data and constructing a knowledge base in a way that you are able to efficiently retrieve relevant information from it. It typically involves three main steps:
 
-* **Chunking**: Break down large pieces of information into smaller segments or chunks.
+### Chunking
 
-* **Embedding**: Convert a piece of information such as text, images, audio, video, etc. into an array of numbers a.k.a. vectors.
+Chunking is the process of breaking down large pieces of mainly text into smaller segments or chunks. At retrieval time, only the chunks that are most relevant to the user queries are retrieved. This helps reduce generation costs since fewer tokens are used. It also helps reduce hallucinations by focusing the attention of LLMs on the most relevant information, rather than having them sift through a large body of text to identify the relevant parts.
 
-* **Vector Search**: Retrieve the most relevant documents from the knowledge base based on embedding similarity with the query vector.
+There are several common chunking methodologies for RAG:
+* **Fixed token with overlap**: Splits text into chunks consisting of a fixed number of tokens, with some overlap between chunks to avoid context loss at chunk boundaries.
+
+![](/img/screenshots/10-key-concepts/fixed-token.png)
+
+* **Recursive with overlap**: First splits text on a set of characters and recursively merges them into tokens until the desired chunk size is reached. This has the effect of keeping related text in the same chunk to the extent possible. 
+
+![](/img/screenshots/10-key-concepts/recursive.png)
+
+* **Semantic**: Creates splits at semantic boundaries, typically identified using an LLM.
+
+
+![](/img/screenshots/10-key-concepts/semantic.png)
+
+
+### Embedding
+
+Convert a piece of information such as text, images, audio, video, etc. into an array of numbers a.k.a. vectors. Read more about embeddings [here](https://mongodb-developer.github.io/vector-search-lab/docs/key-concepts/embeddings).
+
+### Vector Search:
+
+Retrieve the most relevant documents from a knowledge base based on their similarity to the embedding of the query vector. Read more about vector search and how it works in MongoDB [here](https://mongodb-developer.github.io/vector-search-lab/docs/key-concepts/vector-search).
 
 ## Generation
 
-Generation involves crafting a prompt that contains all the instructions and information required by the LLM to generate accurate answers to user queries.
+Generation involves passing the information retrieved using vector search, system prompts, the user query and information from past interactions (memory) to the LLM for it to generate context-aware responses to user questions.
+
+Here's an example of what a RAG prompt might look like:
+
+```
+# System prompt
+Answer the question based on the context below. If no context is provided, respond with I DON’T KNOW.
+# Retrieved context
+Context: <CONTEXT> 
+# Memory
+User: Give me a summary of Q4 earnings for ABMD.
+Assistant: <ANSWER>
+# User question
+Question: What were some comments made by the CEO?
+```

docs/20-dev-env/1-dev-env-setup.mdx

@@ -14,7 +14,7 @@ import Screenshot from "@site/src/components/Screenshot";
 
     <Screenshot url="https://play.instruqt.com" src="img/screenshots/20-dev-env/1-dev-env-setup/instruqt/1-resume-sandbox.png" alt="Resume sandbox" />
 
-    In the Explorer menu, navigate to `genai-devday-notebooks` > `notebooks` > `ai-rag-lab.ipynb` This is the Jupyter Notebook you will be using throughout this lab.
+    In the Explorer menu, navigate to `genai-devday-notebooks` > `labs` > `ai-rag-lab.ipynb` This is the Jupyter Notebook you will be using throughout this lab.
 
     <Screenshot url="https://play.instruqt.com" src="img/screenshots/20-dev-env/1-dev-env-setup/instruqt/2-nav-notebook.png" alt="Navigate to the notebook" />
 
@@ -29,7 +29,7 @@ import Screenshot from "@site/src/components/Screenshot";
 
     <Screenshot url="https://github.com/codespaces" src="img/screenshots/20-dev-env/1-dev-env-setup/codespaces/1-resume-codespace.png" alt="Resume codespace" />
 
-    Give the codespace a few seconds to restart. When files appear in the Explorer tab, click on the file named `ai-rag-lab.ipynb` under `notebooks`. This is the Jupyter Notebook you will be using throughout this lab.
+    Give the codespace a few seconds to restart. When files appear in the Explorer tab, click on the file named `ai-rag-lab.ipynb` under `labs`. This is the Jupyter Notebook you will be using throughout this lab.
 
     <Screenshot url="https://github.com/codespaces" src="img/screenshots/20-dev-env/1-dev-env-setup/codespaces/2-nav-notebook.png" alt="Navigate to the notebook" />
   </TabItem>
@@ -89,7 +89,7 @@ You will also see the default databases in the cluster appear under **Connection
 
 You will be filling code in a Jupyter Notebook during this lab, so let's get set up with that next!
 
-Within the sandbox, click on the files icon in the left navigation bar of the IDE. In the Explorer menu, navigate to `genai-devday-notebooks` > `notebooks` > `ai-rag-lab.ipynb` to open the Jupyter Notebook for this lab.
+Within the sandbox, click on the files icon in the left navigation bar of the IDE. In the Explorer menu, navigate to `genai-devday-notebooks` > `labs` > `ai-rag-lab.ipynb` to open the Jupyter Notebook for this lab.
 
 <Screenshot url="https://play.instruqt.com" src="img/screenshots/20-dev-env/1-dev-env-setup/instruqt/2-nav-notebook.png" alt="Navigate to the notebook" />
 
@@ -143,7 +143,7 @@ You will also see the default databases in the cluster appear under **Connection
 
 You will be filling code in a Jupyter Notebook during this lab, so let's get set up with that next!
 
-Within the codespace, click on the files icon in the left navigation bar of the IDE. In the Explorer menu, under `notebooks`, click on the file named `ai-rag-lab.ipynb` to open the Jupyter Notebook for this lab.
+Within the codespace, click on the files icon in the left navigation bar of the IDE. In the Explorer menu, under `labs`, click on the file named `ai-rag-lab.ipynb` to open the Jupyter Notebook for this lab.
 
 <Screenshot url="https://github.com/codespaces" src="img/screenshots/20-dev-env/1-dev-env-setup/codespaces/2-nav-notebook.png" alt="Navigate to the notebook" />
 
@@ -161,10 +161,10 @@ To run the lab locally, follow the steps below:
 git clone https://github.com/mongodb-developer/genai-devday-notebooks.git
 ```
 
-* `cd` into the `notebooks` directory of the cloned repository:
+* `cd` into the `labs` directory of the cloned repository:
 
 ```
-cd genai-devday-notebooks/notebooks
+cd genai-devday-notebooks/labs
 ```
 
 * Create and activate a Python virtual environment:

docs/20-dev-env/2-setup-pre-reqs.mdx

@@ -1,26 +1,40 @@
 # 👐 Setup prerequisites
 
-Run the cells under **Step 1: Setup prerequisites** section in the notebook.
+Set the passkey provided by your workshop instructor, and run the cells under the **Step 1: Setup prerequisites** section in the notebook.
 
-:::info
+### Expired passkey OR don't have a passkey
 
-Additional steps **if you are running the lab locally**:
+Passkeys are provided to you at MongoDB Developer Days to easily get API keys for LLM and embedding APIs that are used in the workshop. These passkeys are valid for 3 days after the workshop.
 
-* Spin up a MongoDB Atlas cluster and obtain its connection string:
+Once the passkey expires, or if you weren't at a MongoDB Developer Day recently, you will need to obtain the following API keys for the workshop:
+
+**VoyageAI**
+
+* Follow the steps here to [obtain a Voyage AI API key](https://docs.voyageai.com/docs/api-key-and-installation#authentication-with-api-keys).
+* Set the `VOYAGE_API_KEY` environment variable in the notebook as follows:
+
+```python
+os.environ["VOYAGE_API_KEY"] = "your-voyageai-api-key"
+```
+
+### If you are running the lab locally
+
+If you aren't using Instruqt or GitHub Codespaces to run the lab and instead running it locally, you will need to do the following additional steps:
+
+* Spin up a free MongoDB Atlas cluster and obtain its connection string:
 
     * Register for a [free MongoDB Atlas account](https://www.mongodb.com/cloud/atlas/register) if you don't already have one
     * [Create a new database cluster](https://www.mongodb.com/docs/guides/atlas/cluster)
     * [Obtain the connection string](https://www.mongodb.com/docs/guides/atlas/connection-string) for your database cluster
 
-* Set the `MONGODB_URI` variable to the connection string for your cluster as follows:
+* Set the `MONGODB_URI` variable in the notebook as follows:
 
 ```python
-MONGODB_URI = "<your_connection_string>"
+MONGODB_URI = "your_connection_string"
 ```
 
-* Manually set the value of the `SERVERLESS_URL` variable as follows:
+* Manually set the `PROXY_ENDPOINT` variable in the notebook as follows:
 
 ```python
-SERVERLESS_URL = "https://vtqjvgchmwcjwsrela2oyhlegu0hwqnw.lambda-url.us-west-2.on.aws/"
-```
-:::
+PROXY_ENDPOINT = "https://vtqjvgchmwcjwsrela2oyhlegu0hwqnw.lambda-url.us-west-2.on.aws/"
+```

docs/30-prepare-the-data/2-chunk-data.mdx

@@ -0,0 +1,75 @@
+# 👐 Chunk and embed the data
+
+Since we are working with large documents, we first need to break them up into smaller chunks. Then, to make each chunk searchable using vector search, we need to add embeddings to them.
+
+In this workshop, we will use _voyage-context-3_ from Voyage AI to produce contextualized embeddings for the chunks.
+
+Fill in any `<CODE_BLOCK_N>` placeholders and run the cells under the **Step 3: Chunk and embed the data** section in the notebook to chunk and embed the articles we loaded.
+
+The answers for code blocks in this section are as follows:
+
+**CODE_BLOCK_1**
+
+<details>
+<summary>Answer</summary>
+<div>
+```python
+text_splitter.split_text(text)
+```
+</div>
+</details>
+
+**CODE_BLOCK_2**
+
+<details>
+<summary>Answer</summary>
+<div>
+```python
+vo.contextualized_embed(inputs=[content], model="voyage-context-3", input_type=input_type)
+```
+</div>
+</details>
+
+**CODE_BLOCK_3**
+
+<details>
+<summary>Answer</summary>
+<div>
+```python
+get_chunks(doc, "body")
+```
+</div>
+</details>
+
+**CODE_BLOCK_4**
+
+<details>
+<summary>Answer</summary>
+<div>
+```python
+get_embeddings(chunks, "document")
+```
+</div>
+</details>
+
+**CODE_BLOCK_5**
+
+<details>
+<summary>Answer</summary>
+<div>
+```python
+chunk_doc["body"] = chunk
+```
+</div>
+</details>
+
+**CODE_BLOCK_6**
+
+<details>
+<summary>Answer</summary>
+<div>
+```python
+chunk_doc["embedding"] = embedding
+```
+</div>
+</details>

docs/30-prepare-the-data/2-chunk-embed-data.mdx

@@ -8,7 +8,7 @@ Fill in any `<CODE_BLOCK_N>` placeholders and run the cells under the **Step 5:
 
 The answers for code blocks in this section are as follows:
 
-**CODE_BLOCK_5**
+**CODE_BLOCK_7**
 
 <details>
 <summary>Answer</summary>

docs/30-prepare-the-data/3-embed-data.mdx

@@ -2,19 +2,4 @@
 
 To retrieve documents from MongoDB using vector search, you must configure a vector search index on the collection into which you ingested your data. In this lab, you will programmatically create vector search indexes using MongoDB's Python driver.
 
-Fill in any `<CODE_BLOCK_N>` placeholders and run the cells under the **Step 6: Create a vector search index** section in the notebook to create a vector search index.
-
-The answers for code blocks in this section are as follows:
-
-**CODE_BLOCK_6**
-
-<details>
-<summary>Answer</summary>
-<div>
-
-```python
-create_index(collection, ATLAS_VECTOR_SEARCH_INDEX_NAME, model)
-```
-
-</div>
-</details>
+Run the cells under the **Step 5: Create a vector search index** section in the notebook to create a vector search index.

docs/30-prepare-the-data/4-ingest-data.mdx renamed to docs/30-prepare-the-data/3-ingest-data.mdx

@@ -2,22 +2,22 @@
 
 Now let's run some vector search queries against the data present in MongoDB. 
 
-Fill in any `<CODE_BLOCK_N>` placeholders and run the cells under the **Step 7: Perform vector search on your data** section in the notebook to run vector search queries against your data.
+Fill in any `<CODE_BLOCK_N>` placeholders and run the cells under the **Step 6: Perform vector search on your data** section in the notebook to run vector search queries against your data.
 
 The answers for code blocks in this section are as follows:
 
-**CODE_BLOCK_7**
+**CODE_BLOCK_8**
 
 <details>
 <summary>Answer</summary>
 <div>
 ```python
-get_embedding(user_query)
+get_embeddings([user_query], "query")
 ```
 </div>
 </details>
 
-**CODE_BLOCK_8**
+**CODE_BLOCK_9**
 
 <details>
 <summary>Answer</summary>
@@ -37,6 +37,9 @@ get_embedding(user_query)
         "$project": {
             "_id": 0,
             "body": 1,
+            "metadata.productName": 1, 
+            "metadata.contentType": 1,
+            "updated": 1,
             "score": {"$meta": "vectorSearchScore"}
         }
     }
@@ -45,7 +48,7 @@ get_embedding(user_query)
 </div>
 </details>
 
-**CODE_BLOCK_9**
+**CODE_BLOCK_10**
 
 <details>
 <summary>Answer</summary>