update README and create-search-app.sh

Author: sewenew

Chinese.md

@@ -211,7 +211,7 @@ Module 'LLM' loaded from /path/to/libredis-llm.so
 
 #### Search Application
 
-创建一个search（查询）应用，让它基于vector store中保存的数据来回答问题。
+创建一个search（查询）应用，让它基于vector store中保存的私有数据来回答问题。
 
 - 调用LLM.CREATE-LLM创建一个LLM模型
 - 调用LLM.CREATE-VECTOR-STORE创建一个带LLM支持的vector store
@@ -232,6 +232,8 @@ Module 'LLM' loaded from /path/to/libredis-llm.so
 "The author of redis-llm is sewenew."
 ```
 
+[这里](https://github.com/sewenew/redis-llm/tree/main/examples/search-application)包含了基于Redis命令文档和redis-plus-plus文档构建私有数据检索的例子。
+
 #### Chat Application
 
 创建一个chat（聊天）应用，该应用能帮助LLM记住很长的聊天历史。

README.md

@@ -15,7 +15,7 @@
 - [Terminology](#terminology)
     - [LLM](#llm)
     - [Prompt](#prompt)
-    - [Vector Store](#vector-store-2)
+    - [Vector Store](#vector-store-1)
     - [Application](#application)
 - [Commands](#commands)
     - [LLM.CREATE-LLM](#llmcreate-llm)
@@ -211,7 +211,7 @@ Create a *hello world* application with LLM and prompt template (You must call L
 
 #### Search Application
 
-Create a search application which can answer questions based on data stored in the vector store.
+Create a search application which can answer questions based on your private data stored in the vector store.
 
 - Call LLM.CREATE-LLM to create an LLM model.
 - Call LLM.CREATE-VECTOR-STORE to create a vector store with LLM support.
@@ -232,6 +232,8 @@ Create a search application which can answer questions based on data stored in t
 "The author of redis-llm is sewenew."
 ```
 
+Check [this](https://github.com/sewenew/redis-llm/tree/main/examples/search-application) for examples on building a private data searcher on Redis commands and redis-plus-plus.
+
 #### Chat Application
 
 Create a chat application which help LLM remember a long conversation history.

examples/search-application/README.md

@@ -0,0 +1,77 @@
+# create search application
+
+You can use redis-llm to create a search application which can answer questions based on your private data stored in the vector store. Normally, you need to run the following steps:
+
+- Call LLM.CREATE-LLM to create an LLM model.
+- Call LLM.CREATE-VECTOR-STORE to create a vector store with LLM support.
+- Call LLM.ADD to add your private data to the store.
+- Call LLM.CREATE-SEARCH to create a search application with LLM and vector store.
+- Call LLM.RUN to ask questions on your private data.
+
+I splitted [Redis' commands documentation](https://github.com/redis/redis-doc/tree/master/commands), and [redis-plus-plus (C++ Redis client)'s README.md](https://github.com/sewenew/redis-plus-plus/blob/master/README.md) into segmentations. For each segmentation, I saved it into a file, and called OpenAI embedding API (model: text-embedding-ada-002) to generate their embeddings. These data and embeddings are located at *examples/search-application/embeddings.zip*. You can unzip it to add into vector store.
+
+```
+unzip embeddings.zip
+```
+
+```
+ls embeddings
+data				redis-cmds-embedding		redis-plus-plus-doc-embeddings
+```
+
+You can find all segmentation files under *data* directory. *redis-cmds-embedding* and *redis-plus-plus-doc-embeddings* are embeddings files. Each embedding file contains pre-generated embeddings. Each line of the file contains data file name and data embedding, separated with "\t".
+
+You can use *examples/search-application/create-search-app.sh* script to run create LLM model and vector store, add pre-genereated embeddings to the store, and create a search application. The following is the usage:
+
+```Shell
+create-search-app.sh embedding-file data-path store-key llm-key openai-api-key search-key
+```
+
+- *embedding-file*: Path to embedding file. e.g. examples/search-application/embeddings/redis-plus-plus-doc-embeddings
+- *data-path*: Path to data directory, e.g. examples/search-application/embeddings/data.
+- *store-key*: Redis key for the created vector store.
+- *llm-key*: Redis key for the created LLM model.
+- *openai-api-key*: Your OpenAI API key.
+- *search-key*: Redis key for the created search application.
+
+Ensure you've already installed `redis-cli`, and launched Redis server on localhost with redis-llm module loaded:
+
+```
+redis-cli module list
+1) 1) "name"
+   2) "LLM"
+   3) "ver"
+   4) (integer) 1
+```
+
+## Create a search application to ask question on Redis commands' documentation
+
+Run the following command to load Redis documenation and embeddings:
+
+```
+cd /path/to/redis-llm/examples/search-application
+
+sh create-search-app.sh embeddings/redis-cmds-embedding embeddings/data redis-cmd-store openai-model $YOUR-OPENAI-API-KEY redis-cmd-searcher
+```
+
+Ask questions on how to use Redis commands:
+
+```
+redis-cli llm.run redis-cmd-searcher 'How to run set command?'
+```
+
+## Create a search application to ask question on redis-plus-plus' documentation
+
+Run the following command to load redis-plus-plus' documenation and embeddings:
+
+```
+cd /path/to/redis-llm/examples/search-application
+
+sh create-search-app.sh embeddings/redis-plus-plus-doc-embeddings embeddings/data redis-plus-plus-doc-store openai-model $YOUR-OPENAI-API-KEY redis-plus-plus-searcher
+```
+
+Ask questions on redis-plus-plus:
+
+```
+redis-cli llm.run redis-plus-plus-searcher 'Who is the author of redis-plus-plus?'
+```

docker/create-search-app.sh renamed to examples/search-application/create-search-app.sh

@@ -20,19 +20,19 @@ redis_cli=redis-cli
 
 if [ $# != 6 ]
 then
-    echo "Usage: create-search-app.sh embedding-file data-path store-key openai-key openai-api-key search-key"
+    echo "Usage: create-search-app.sh embedding-file data-path store-key llm-key openai-api-key search-key"
     exit -1
 fi
 
 embedding_file=$1
 data_path=$2
 store_key=$3
-openai_key=$4
+llm_key=$4
 openai_api_key=$5
 search_key=$6
 
-$redis_cli llm.create-llm "$openai_key" --params "{\"api_key\":\"$openai_api_key\"}"
-$redis_cli llm.create-vector-store "$store_key" --llm "$openai_key"
+$redis_cli llm.create-llm "$llm_key" --params "{\"api_key\":\"$openai_api_key\"}"
+$redis_cli llm.create-vector-store "$store_key" --llm "$llm_key"
 
 while read line; do
     key=$(echo "$line" | cut -f1)
@@ -41,4 +41,4 @@ while read line; do
     cat $path | $redis_cli -x llm.add "$store_key" --embedding "$embedding"
 done <"$embedding_file"
 
-$redis_cli llm.create-search "$search_key" --llm "$openai_key" --vector-store "$store_key"
+$redis_cli llm.create-search "$search_key" --llm "$llm_key" --vector-store "$store_key"