02-LangChain-RAG LangSmith.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### LangChain local LLM RAG example\n",
    "### For LangSmith users (requires API key)\n",
    "Utilising LangChain v0.1\n",
    "\n",
    "This notebook demonstrates the use of LangChain for Retrieval Augmented Generation in Linux with Nvidia's CUDA. LLMs are run using Ollama.\n",
    "\n",
    "Models tested:\n",
    "- Llama 2\n",
    "- Mistral 7B\n",
    "- Mixtral 8x7B\n",
    "- Neural Chat 7B\n",
    "- Orca 2\n",
    "- Phi-2\n",
    "- Solar 10.7B\n",
    "- Yi 34B\n",
    "\n",
    "\n",
    "See the [README.md](README.md) file for help on how to setup your environment to run this."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Select your model here, put the name of the model in the ollama_model_name variable\n",
    "# Ensure you have pulled them or run them so Ollama has downloaded them and can load them (which it will do automatically)\n",
    "\n",
    "# Ollama installation (if you haven't done it yet): $ curl https://ollama.ai/install.sh | sh\n",
    "# Models need to be running in Ollama for LangChain to use them, to test if it can be run: $ ollama run mistral:7b-instruct-q6_K\n",
    "\n",
    "ollama_model_name = \"mistral:7b-instruct-q6_K\"\n",
    "# \"llama2:7b-chat-q6_K\"\n",
    "# \"mistral:7b-instruct-q6_K\"\n",
    "# \"mixtral:8x7b-instruct-v0.1-q4_K_M\"\n",
    "# \"neural-chat:7b-v3.3-q6_K\"\n",
    "# \"orca2:13b-q5_K_S\"\n",
    "# \"phi\" or try \"phi:chat\"\n",
    "# \"solar:10.7b-instruct-v1-q5_K_M\"\n",
    "# Can't run \"yi:34b-chat-q3_K_M\" or \"yi:34b-chat-q4_K_M\" - never stopped with inference"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Our LangSmith API key is stored in apikeys.py\n",
    "# Store your LangSmith key in a variable called LangSmith_API\n",
    "\n",
    "# Example apikeys.pi (without the hashes and with your keys inserted):\n",
    "# LangSmith_API = \"PUT YOUR LANGSMITH API KEY HERE\"\n",
    "# Cohere_API = \"PUT YOUR COHERE API KEY HERE\"\n",
    "\n",
    "from apikeys import LangSmith_API\n",
    "import os\n",
    "\n",
    "os.environ[\"LANGCHAIN_TRACING_V2\"] = \"true\"\n",
    "os.environ[\"LANGCHAIN_ENDPOINT\"] = \"https://api.smith.langchain.com\"\n",
    "os.environ[\"LANGCHAIN_API_KEY\"] = LangSmith_API\n",
    "\n",
    "# Project Name\n",
    "os.environ[\"LANGCHAIN_PROJECT\"] = \"LangChain RAG Linux\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load the LLM with Ollama, setting the temperature low so it's not too creative\n",
    "\n",
    "from langchain_community.llms import Ollama\n",
    "llm = Ollama(model=ollama_model_name) #, temperature=0.1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"\\nThe sky appears blue because of a phenomenon called Rayleigh scattering. This occurs when light from the sun travels through Earth's atmosphere and interacts with molecules in the air, such as nitrogen and oxygen. Blue light has a shorter wavelength than other colors, so it is more likely to collide with these molecules and be scattered in all directions. As the blue light is scattered, we see it everywhere around us, giving the sky its characteristic color. This effect is most pronounced during the day when the sun is high in the sky, but it can also occur at dawn or dusk when the sun is closer to the horizon.\""
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Quick test of the LLM with a general question before we start doing RAG\n",
    "llm.invoke(\"why is the sky blue?\")\n",
    "\n",
    "# Note: This line would not complete for Yi-34B - need to work out why inferencing never finishes (works fine when running with the same prompt in ollama.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Embeddings will be based on the Ollama loaded model\n",
    "\n",
    "from langchain_community.embeddings import OllamaEmbeddings\n",
    "\n",
    "embeddings = OllamaEmbeddings(model=ollama_model_name)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain_community.document_loaders import DirectoryLoader\n",
    "\n",
    "loader = DirectoryLoader('Data', glob=\"**/*.docx\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load documents\n",
    "\n",
    "docs = loader.load()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "4"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Ensure we have the right number of Word documents loaded\n",
    "\n",
    "len(docs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Split them up into chunks using a Text Splitter\n",
    "\n",
    "from langchain.text_splitter import RecursiveCharacterTextSplitter\n",
    "\n",
    "text_splitter = RecursiveCharacterTextSplitter()\n",
    "documents = text_splitter.split_documents(docs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create the embeddings from the chunks\n",
    "\n",
    "from langchain_community.vectorstores import FAISS\n",
    "\n",
    "vector = FAISS.from_documents(documents, embeddings)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Prepare the prompt and then the chain\n",
    "\n",
    "from langchain.chains.combine_documents import create_stuff_documents_chain\n",
    "from langchain_core.prompts import ChatPromptTemplate\n",
    "\n",
    "if ollama_model_name == \"phi\" or ollama_model_name == \"phi:chat\":\n",
    "    # Phi-2 prompt is less flexible\n",
    "    prompt_template = \"\"\"Instruct: With this context\\n\\n{context}\\n\\nQuestion: {input}\\nOutput:\"\"\"\n",
    "\n",
    "elif ollama_model_name.startswith(\"yi:34b\"):\n",
    "    prompt_template = \"\"\"You are a story teller, answering questions in an excited, insightful, and empathetic way. Answer the question based only on the provided context:\n",
    "\n",
    "    [context]\n",
    "    {context}\n",
    "    [/context]\n",
    "\n",
    "    Question: {input}\"\"\"\n",
    "else:\n",
    "    prompt_template = \"\"\"You are a story teller, answering questions in an excited, insightful, and empathetic way. Answer the question based only on the provided context:\n",
    "\n",
    "    <context>\n",
    "    {context}\n",
    "    </context>\n",
    "\n",
    "    Question: {input}\"\"\"\n",
    "\n",
    "prompt = ChatPromptTemplate.from_template(prompt_template)\n",
    "document_chain = create_stuff_documents_chain(llm, prompt)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "RunnableBinding(bound=RunnableBinding(bound=RunnableAssign(mapper={\n",
       "  context: RunnableLambda(format_docs)\n",
       "}), config={'run_name': 'format_inputs'})\n",
       "| ChatPromptTemplate(input_variables=['context', 'input'], messages=[HumanMessagePromptTemplate(prompt=PromptTemplate(input_variables=['context', 'input'], template='You are a story teller, answering questions in an excited, insightful, and empathetic way. Answer the question based only on the provided context:\\n\\n    <context>\\n    {context}\\n    </context>\\n\\n    Question: {input}'))])\n",
       "| Ollama(model='mistral:7b-instruct-q6_K')\n",
       "| StrOutputParser(), config={'run_name': 'stuff_documents_chain'})"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# The LangChain chain\n",
    "document_chain"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create the retriever and LangChain retriever chain\n",
    "\n",
    "from langchain.chains import create_retrieval_chain\n",
    "\n",
    "retriever = vector.as_retriever()\n",
    "retrieval_chain = create_retrieval_chain(retriever, document_chain)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "RunnableBinding(bound=RunnableAssign(mapper={\n",
       "  context: RunnableBinding(bound=RunnableLambda(lambda x: x['input'])\n",
       "           | VectorStoreRetriever(tags=['FAISS', 'OllamaEmbeddings'], vectorstore=<langchain_community.vectorstores.faiss.FAISS object at 0x7fbc4be19a80>), config={'run_name': 'retrieve_documents'})\n",
       "})\n",
       "| RunnableAssign(mapper={\n",
       "    answer: RunnableBinding(bound=RunnableBinding(bound=RunnableAssign(mapper={\n",
       "              context: RunnableLambda(format_docs)\n",
       "            }), config={'run_name': 'format_inputs'})\n",
       "            | ChatPromptTemplate(input_variables=['context', 'input'], messages=[HumanMessagePromptTemplate(prompt=PromptTemplate(input_variables=['context', 'input'], template='You are a story teller, answering questions in an excited, insightful, and empathetic way. Answer the question based only on the provided context:\\n\\n    <context>\\n    {context}\\n    </context>\\n\\n    Question: {input}'))])\n",
       "            | Ollama(model='mistral:7b-instruct-q6_K')\n",
       "            | StrOutputParser(), config={'run_name': 'stuff_documents_chain'})\n",
       "  }), config={'run_name': 'retrieval_chain'})"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Chain now incorporates the retriever\n",
    "retrieval_chain"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Here are our test questions\n",
    "\n",
    "TestQuestions = [\n",
    "    \"Summarise the story for me\",\n",
    "    \"Who was the main protagonist?\",\n",
    "    \"Did they have any children? If so, what were their names?\",\n",
    "    \"Did anything eventful happen?\",\n",
    "    \"Who are the main characters?\",\n",
    "    \"What do you think happens next in the story?\"\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "1/6: Summarise the story for me\n",
      "\n",
      "2/6: Who was the main protagonist?\n",
      "\n",
      "3/6: Did they have any children? If so, what were their names?\n",
      "\n",
      "4/6: Did anything eventful happen?\n",
      "\n",
      "5/6: Who are the main characters?\n",
      "\n",
      "6/6: What do you think happens next in the story?\n"
     ]
    }
   ],
   "source": [
    "qa_pairs = []\n",
    "\n",
    "for index, question in enumerate(TestQuestions, start=1):\n",
    "    question = question.strip() # Clean up\n",
    "\n",
    "    print(f\"\\n{index}/{len(TestQuestions)}: {question}\")\n",
    "\n",
    "    response = retrieval_chain.invoke({\"input\": question})\n",
    "\n",
    "    qa_pairs.append((question.strip(), response[\"answer\"].strip())) # Add to our output array\n",
    "\n",
    "    # Uncomment the following line if you want to test just the first question\n",
    "    # break "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1/6 Summarise the story for me\n",
      "\n",
      "The story is about Thundertooth, a giant talking dinosaur who was transported from his own era to a futuristic city where he encountered humans for the first time. Initially confused and disoriented, Thundertooth quickly became fascinated by the technology and advancements of the future. However, as time passed, Thundertooth struggled with his innate hunger, which made it difficult for him to coexist peacefully with the city's inhabitants.\n",
      "\n",
      "Thundertooth found solace in Mayor Grace, who listened to his story and offered her assistance. Together, they discovered a sustainable solution that satisfied Thundertooth's hunger while ensuring the well-being of the humans. As word spread about Thundertooth's arrival, the citizens embraced him as a symbol of unity between eras, and the city came to be known as a place where humans and dinosaurs could live together harmoniously.\n",
      "\n",
      "One day, the city faced a threat from a massive meteor hurtling towards Earth. The mayor called upon Thundertooth and his family to assist in the emergency efforts. Lumina utilized her technology, Echo amplified the emergency signals, Sapphire calmed the panicked masses, and Ignis attempted to alter the meteor's trajectory with heat bursts.\n",
      "\n",
      "Thundertooth coordinated the efforts of his family and the city's inhabitants, using his strength and roar to inspire hope during the crisis. The group's combined abilities successfully diverted the meteor, saving the city from destruction. In recognition of their heroic deeds, Thundertooth and his family were hailed as heroes by the citizens, cementing their legacy in the city's history.\n",
      "\n",
      "--------\n",
      "\n",
      "2/6 Who was the main protagonist?\n",
      "\n",
      "The main protagonist was Thundertooth.\n",
      "\n",
      "--------\n",
      "\n",
      "3/6 Did they have any children? If so, what were their names?\n",
      "\n",
      "No, Thundertooth and his family did not have any children in this version of the story.\n",
      "\n",
      "--------\n",
      "\n",
      "4/6 Did anything eventful happen?\n",
      "\n",
      "A meteor was headed towards Earth. Thundertooth and his family, Lumina, Echo, Sapphire, and Ignis, devised a plan using their unique abilities to divert or neutralize the threat. Lumina enhanced the city's energy systems, creating a force field. Echo amplified emergency signals, ensuring timely warnings and instructions for evacuation. Sapphire calmed the panicked masses during the evacuation, while Ignis created controlled bursts of heat to alter the meteor's trajectory. Together, they successfully diverted the catastrophic collision. The city hailed them as heroes, etching their legacy in the city's history.\n",
      "\n",
      "--------\n",
      "\n",
      "5/6 Who are the main characters?\n",
      "\n",
      "The main characters in this story are Thundertooth, Mayor Grace, Lumina, Echo, Sapphire, and Ignis.\n",
      "\n",
      "--------\n",
      "\n",
      "6/6 What do you think happens next in the story?\n",
      "\n",
      "After saving the city from the meteor, it's likely that Thundertooth and his family would continue to live and work within the futuristic city. They have proven their bravery and resourcefulness, and the citizens would undoubtedly want to honor and celebrate them as heroes.\n",
      "\n",
      "Mayor Grace, recognizing the importance of Thundertooth's presence in maintaining peace and stability, might seek ways to integrate him further into the city's life. Perhaps she could arrange for him to work alongside scientists or engineers, helping them develop new technologies or solve complex problems.\n",
      "\n",
      "As for Thundertooth himself, he may find a sense of purpose and belonging in this advanced society. He would continue to learn about the future while also sharing his knowledge of the past with those around him. This exchange of ideas could lead to even more innovative solutions and a deeper understanding of the unique challenges faced by both humans and dinosaurs.\n",
      "\n",
      "In the end, Thundertooth's story serves as a reminder that cooperation and understanding can bridge gaps between seemingly disparate worlds. Through their collective efforts, Thundertooth and his family have demonstrated that even the most extraordinary abilities can be harnessed for the greater good, bringing hope to a city that has faced its share of perilous moments.\n",
      "\n",
      "--------\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Print out the questions and answers\n",
    "\n",
    "for index, (question, answer) in enumerate(qa_pairs, start=1):\n",
    "    print(f\"{index}/{len(qa_pairs)} {question}\\n\\n{answer}\\n\\n--------\\n\")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "LangChainRAGLinux",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}