What is Reciprocal Rank Fusion (RRF)?

Reciprocal Rank Fusion is similar to Parallel Query Retrieval technique, Here also we generate the 4-5 queries based on user query and retrieve the documents based on them, But here instead of just filtering unique documents, we rank the documents based Ranking formula, and take the top n documents from them.

Ranking Formula:

Basically we rank the documents based on frequency and position of occurrence

How Reciprocal Rank Fusion works

Step 1: Generating multiple queries

When user asks as a query, we provide it to an small LLM ask it to generate multiple queries of it.

import os
from dotenv import load_dotenv

from langchain_google_genai import ChatGoogleGenerativeAI
from langchain_core.prompts import PromptTemplate
from langchain_core.output_parsers import BaseOutputParser
from typing import List, Dict

load_dotenv()

class LineListOutputParser(BaseOutputParser[List[str]]):
  """Output parser for a list of lines."""

  def parse(self, text: str) -> List[str]:
    lines = text.strip().split("\n")
    return list(filter(None, lines)) 

output_parser = LineListOutputParser()

llm = ChatGoogleGenerativeAI(
  model="gemini-2.5-flash",
  google_api_key=os.getenv("GEMINI_API_KEY"),
  temperature=0,
  max_tokens=None,
  timeout=None,
  max_retries=2,
)

QUERY_REWRITE_PROMPT = PromptTemplate(
  input_variables=["question"],
  template="""You are an AI language model assistant. Your task is to generate five 
  different versions of the given user question to retrieve relevant documents from a vector 
  database. By generating multiple perspectives on the user question, your goal is to help
  the user overcome some of the limitations of the distance-based similarity search. 
  Provide these alternative questions separated by newlines.
  Original question: {question}""",
)

llm_chain = QUERY_REWRITE_PROMPT | llm | output_parser

# ----- Step 1: Generate Multiple Queries -----
user_query = "What is FS Module?"
generated_queries = llm_chain.invoke(user_query)

print("--------- Generated Queries -----------")
for i, query in enumerate(generated_queries, 1):
  print(f"{i}. {query}")
print("------------------------")

Step 2: Parallely fetching relevant chunks from vector store

Once we generate multiple queries, we embed those query, and retrieve the relevant chunks from the vector store by performing similarity search. This action is done for all the generated query in parallel.

import os

from langchain_google_genai import ChatGoogleGenerativeAI
from typing import List, Set
import concurrent.futures
from langchain_core.documents import Document

from utils.output_parser import output_parser
from config.vector_store import get_vector_store
from llm.prompt_templates import QUERY_REWRITE_PROMPT
from utils.rank_docs import rank_documents

# ----- setup --------

qdrant = get_vector_store()

llm = ChatGoogleGenerativeAI(
  model="gemini-2.5-flash",
  google_api_key=os.getenv("GEMINI_API_KEY"),
  temperature=0,
  max_tokens=None,
  timeout=None,
  max_retries=2,
)

llm_chain = QUERY_REWRITE_PROMPT | llm | output_parser

def reciprocal_rank_fusion(
  user_query: str, 
  llm_chain=llm_chain, 
  retriever=qdrant.as_retriever()
) -> List[Document]:
  """
  Generate multiple queries using llm_chain, fetch documents using retriever,
  and return deduplicated list of documents.

  Args:
    user_query (str): The input user query.
    llm_chain: A Runnable (PromptTemplate | LLM | OutputParser) for generating queries.
    retriever: A LangChain retriever (e.g., qdrant.as_retriever()).

  Returns:
    List[Document]: Deduplicated retrieved documents across all queries.
  """
  # Step 1: Generate multiple queries
  generated_queries = llm_chain.invoke(user_query)

  # Step 2: Fetch documents in parallel
  def fetch_docs(query: str):
    return retriever.invoke(query)

  all_docs: List[Document] = []
  with concurrent.futures.ThreadPoolExecutor() as executor:
    results = executor.map(fetch_docs, generated_queries)
    for docs in results:
      all_docs.extend(docs)

  # Step 3: Rank docs    
  sorted_docs = rank_documents(all_docs)

  print("--------- sorted docs ------")
  print(f"\nTotal sorted documents: {len(sorted_docs)}")
  print("===============================")
  for i, doc in enumerate(sorted_docs, 1):
    print(f"Doc {i} ID: {doc}")
  print("===============================")

  return sorted_docs

Step 3: Ranking the documents

From all the fetched documents, we rank the documents and tak the only top n documents to answer the user query.

To rank the documents we consider their occurrence and position for each query.

suppose we have three queries and those 3 queries fetched documents as per below:

While ranking we rank them as per below

• doc_id:3 is occuring 3 time so it should have 1st rank.

• but doc_id:1 and doc_id:2 both are occuring twice, so while ranking them we consider their position of occurrence.

• In Query 1, doc_id:1 is at 1st position and in Query 3, doc_id:1 is at 2nd position while, doc_id:2 is at 3rd position in Query 1 and at Query 2 it is at 1st position. So the doc_id:1 has more precedence than doc_id:2.

• doc_id:4 has only one occurrence so it goes at last in ranking.

finally ranking becomes like below

from typing import List, Dict
from collections import defaultdict
from langchain_core.documents import Document

def reciprocal_rank_fusion(
  docs: List[Document],
  k: int = 60
) -> List[Document]:
  """
  Apply Reciprocal Rank Fusion (RRF) to combine multiple ranked lists.

  Args:
    docs List[Document]: list of documents.
    k (int): RRF constant. Defaults to 60.

  Returns:
    List[Document]: Final ranked list of documents with highest combined RRF scores.
  """
  doc_scores = defaultdict(float)
  doc_store = {}

  for rank, doc in enumerate(docs):
    doc_id = doc.metadata.get("id") or hash(doc.page_content.strip())
    score = 1 / (k + rank)
    doc_scores[doc_id] += score
    # Store one version of the doc
    if doc_id not in doc_store:
      doc_store[doc_id] = doc

  # Sort by RRF score (descending)
  ranked_doc_ids = sorted(doc_scores.items(), key=lambda x: x[1], reverse=True)

  for doc_id, score in ranked_doc_ids:
    print(f"doc_id: {doc_id}, score : {score} ")

  # Return the corresponding Document objects
  return [doc_store[doc_id] for doc_id, _ in ranked_doc_ids]

Step 4: Generating output

Using ranked documents and users original query, we generate the output, which will more accurate.

import os
from openai import OpenAI

from retriever.retrival import reciprocal_rank_fusion

api_key = os.getenv("GEMINI_API_KEY")

client = OpenAI(
  api_key=api_key,
  base_url="https://generativelanguage.googleapis.com/v1beta/openai/"
)

def llm_chat(query: str):
  unique_docs = reciprocal_rank_fusion(query)

  context = "\n\n"
  for doc in unique_docs:
    # print(doc)
    context += f"Page Content: {doc.page_content}\nPage Number: {doc.metadata['page_label']}\nFile Location: {doc.metadata['source']} \n\n"


  SYSTEM_PROMPT = f"""
    You are a helpful AI assistant who answers user query based on the available context retrieved from a PDF file along with page_contents and page number.

    You should only answer the user based on the following context and navigate the user to open the right page number to know more.
    answer should be in details.
    Context:
    {context}
  """

  chat_completion = client.chat.completions.create(
    model="gemini-2.0-flash",
    messages=[
      {"role": "system", "content": SYSTEM_PROMPT},
      {"role": "user", "content": query}
    ],
  )

  return chat_completion.choices[0].message.content

Benefits of Reciprocal Rank Fusion

Even though if, multiple queries fetch the docs that are not entirely relevant to user query, by ranking we reject those documents and take only closely relevant documents to answer the user query.

Source Code

retrival.py

import os

from langchain_google_genai import ChatGoogleGenerativeAI
from typing import List, Set
import concurrent.futures
from langchain_core.documents import Document

from utils.output_parser import output_parser
from config.vector_store import get_vector_store
from llm.prompt_templates import QUERY_REWRITE_PROMPT
from utils.rank_docs import rank_documents

# ----- setup --------

qdrant = get_vector_store()

llm = ChatGoogleGenerativeAI(
  model="gemini-2.5-flash",
  google_api_key=os.getenv("GEMINI_API_KEY"),
  temperature=0,
  max_tokens=None,
  timeout=None,
  max_retries=2,
)

llm_chain = QUERY_REWRITE_PROMPT | llm | output_parser

def reciprocal_rank_fusion(
  user_query: str, 
  llm_chain=llm_chain, 
  retriever=qdrant.as_retriever()
) -> List[Document]:
  """
  Generate multiple queries using llm_chain, fetch documents using retriever,
  and return deduplicated list of documents.

  Args:
    user_query (str): The input user query.
    llm_chain: A Runnable (PromptTemplate | LLM | OutputParser) for generating queries.
    retriever: A LangChain retriever (e.g., qdrant.as_retriever()).

  Returns:
    List[Document]: Deduplicated retrieved documents across all queries.
  """
  # Step 1: Generate multiple queries
  generated_queries = llm_chain.invoke(user_query)

  # Step 2: Fetch documents in parallel
  def fetch_docs(query: str):
    return retriever.invoke(query)

  all_docs: List[Document] = []
  with concurrent.futures.ThreadPoolExecutor() as executor:
    results = executor.map(fetch_docs, generated_queries)
    for docs in results:
      all_docs.extend(docs)

  # Step 3: Rank docs    
  sorted_docs = rank_documents(all_docs)

  return sorted_docs

chat.py

import os
from openai import OpenAI

from retriever.retrival import reciprocal_rank_fusion

api_key = os.getenv("GEMINI_API_KEY")

client = OpenAI(
  api_key=api_key,
  base_url="https://generativelanguage.googleapis.com/v1beta/openai/"
)

def llm_chat(query: str):
  search_results =  reciprocal_rank_fusion(query)

  context = "\n\n"
  for _ , doc in enumerate(search_results, 1):
    context += f"Page Content: {doc.page_content}\nPage Number: {doc.metadata['page_label']}\nFile Location: {doc.metadata['source']} \n\n"


  SYSTEM_PROMPT = f"""
    You are a helpful AI assistant who answers user query based on the available context retrieved from a PDF file along with page_contents and page number.

    You should only answer the user based on the following context and navigate the user to open the right page number to know more.
    answer should be in details.
    Context:
    {context}
  """

  chat_completion = client.chat.completions.create(
    model="gemini-2.0-flash",
    messages=[
      {"role": "system", "content": SYSTEM_PROMPT},
      {"role": "user", "content": query}
    ],
  )

  return chat_completion.choices[0].message.content

main.py

import os
from dotenv import load_dotenv

from generator.chat import llm_chat

load_dotenv()

def main():
  query = "what is fs module?"
  result = llm_chat(query=query)

  print(result)

if __name__ == "__main__":
  main()

Full source code available at: GitHub

In the Query Translation article we understood that we can not trust the user query. User might ask some ambigues query. If the query is ambiguous generated output is also ambiguous.

What is Parallel Query (Fan-Out)?

Parallel Query Retrieval also known as Multi Query is an technique where we create multiple queries based on the users query. Each query will cover the different aspect of the user requirements that user may even know that he wanted.

Example:

If user wants know about fs module in node.js and he asks a question:

what is fs module?

but suppose our data set has some chunks that says file system module and that is very important to answer user query, that won’t be get retrieved as fs and file system has different vector embeddings and won’t get searched during similarity search. In this case the generated answer won’t be an accurate.

So what we do it to ask LLM model to generate 4-5 different queries based on user query, like below

1. What is the File System (FS) module?

2. What is the purpose of the FS module?

3. Describe the Node.js FS module.

4. How does the FS module work?

5. Explain the concept of the FS module.

generated queries will contain more fancy words, and will fetch more chunks during similarity search

How Parallel Query Retrieval works

Step 1: Generating multiple queries

When user asks as a query, we provide it to an small LLM ask it to generate multiple queries of it.

import os
from dotenv import load_dotenv

from langchain_google_genai import ChatGoogleGenerativeAI
from langchain_core.prompts import PromptTemplate
from langchain_core.output_parsers import BaseOutputParser
from typing import List, Dict

load_dotenv()

class LineListOutputParser(BaseOutputParser[List[str]]):
  """Output parser for a list of lines."""

  def parse(self, text: str) -> List[str]:
    lines = text.strip().split("\n")
    return list(filter(None, lines)) 

output_parser = LineListOutputParser()

llm = ChatGoogleGenerativeAI(
  model="gemini-2.5-flash",
  google_api_key=os.getenv("GEMINI_API_KEY"),
  temperature=0,
  max_tokens=None,
  timeout=None,
  max_retries=2,
)

QUERY_REWRITE_PROMPT = PromptTemplate(
  input_variables=["question"],
  template="""You are an AI language model assistant. Your task is to generate five 
  different versions of the given user question to retrieve relevant documents from a vector 
  database. By generating multiple perspectives on the user question, your goal is to help
  the user overcome some of the limitations of the distance-based similarity search. 
  Provide these alternative questions separated by newlines.
  Original question: {question}""",
)

llm_chain = QUERY_REWRITE_PROMPT | llm | output_parser

# ----- Step 1: Generate Multiple Queries -----
user_query = "What is FS Module?"
generated_queries = llm_chain.invoke(user_query)

print("--------- Generated Queries -----------")
for i, query in enumerate(generated_queries, 1):
  print(f"{i}. {query}")
print("------------------------")

Step 2: Parallely fetching relevant chunks from vector store

Once we generate multiple queries, we embed those query, and retrieve the relevant chunks from the vector store by performing similarity search. This action is done for all the generated query in parallel.

import os
from dotenv import load_dotenv
import concurrent.futures

from langchain_qdrant import QdrantVectorStore
from qdrant_client import QdrantClient
from qdrant_client.models import Distance, VectorParams
from langchain_google_genai import GoogleGenerativeAIEmbeddings

load_dotenv()

embeddings = GoogleGenerativeAIEmbeddings(
  model="models/gemini-embedding-001",
  google_api_key=os.getenv("GEMINI_API_KEY")
)

COLLECTION_NAME = "nodejs_document"

def get_qdrant_client():
  client = QdrantClient(url="http://localhost:6333")

  if COLLECTION_NAME not in [c.name for c in client.get_collections().collections]:
    client.create_collection(
      collection_name=COLLECTION_NAME,
      vectors_config=VectorParams(size=3072, distance=Distance.COSINE),
    )

  return client

def get_vector_store():
    client = get_qdrant_client()
    return QdrantVectorStore(
        client=client,
        collection_name=COLLECTION_NAME,
        embedding=embeddings,
    )

qdrant = get_vector_store()

# ----- Fetch Documents in Parallel -----
def fetch_docs(query):
  docs = qdrant.as_retriever().invoke(query)
  return (query, docs)

query_to_docs: Dict[str, List] = {}

# Use ThreadPoolExecutor to run in parallel
with concurrent.futures.ThreadPoolExecutor() as executor:
  results = list(executor.map(fetch_docs, generated_queries))

# Map results
for query, docs in results:
  query_to_docs[query] = docs

for query, docs in query_to_docs.items():
  print(f"\n===== Results for Query: \"{query}\" =====")
  for i, doc in enumerate(docs, 1):
    print(f"Doc {i} ID: {doc.metadata["_id"]}")
  print("===============================")

Step 3: Filter the unique documents

From all the total documents fetched by all queries, we filter the unique documents from them, to avoid the deduplication.

# ----- Deduplicate Documents -----
# Strategy: use doc.metadata['id'] if available, else page_content hash

def get_doc_id(doc):
  return doc.metadata.get("id") or hash(doc.page_content.strip())

seen = {}
for docs in query_to_docs.values():
  for doc in docs:
    doc_id = get_doc_id(doc)
    if doc_id not in seen:
      seen[doc_id] = doc

unique_docs = list(seen.values())

print(f"\nTotal Unique documents: {len(unique_docs)}")
print("===============================")
for i, doc in enumerate(unique_docs, 1):
  print(f"Doc {i} ID: {doc.metadata["_id"]}")
print("===============================")

Step 4: Generating output

Using retrieved unique documents and users original query, we generate the output, which will more accurate.

import os
from openai import OpenAI

from retriever.retrival import parallel_query_retriver

api_key = os.getenv("GEMINI_API_KEY")

client = OpenAI(
  api_key=api_key,
  base_url="https://generativelanguage.googleapis.com/v1beta/openai/"
)

def llm_chat(query: str):
  unique_docs = parallel_query_retriver(query)

  context = "\n\n"
  for doc in unique_docs:
    # print(doc)
    context += f"Page Content: {doc.page_content}\nPage Number: {doc.metadata['page_label']}\nFile Location: {doc.metadata['source']} \n\n"


  SYSTEM_PROMPT = f"""
    You are a helpful AI assistant who answers user query based on the available context retrieved from a PDF file along with page_contents and page number.

    You should only answer the user based on the following context and navigate the user to open the right page number to know more.
    answer should be in details.
    Context:
    {context}
  """

  chat_completion = client.chat.completions.create(
    model="gemini-2.0-flash",
    messages=[
      {"role": "system", "content": SYSTEM_PROMPT},
      {"role": "user", "content": query}
    ],
  )

  return chat_completion.choices[0].message.content

Benefits of Parallel Query

Even though Parallel Query adds a overhead and extra LLM call, it’s improved performance and accuracy negats the extra cost. Parallel Query retrieval system can handle ambiguous user query, any typos or any mistakes in user query will not affect the final output.

Source Code

retrival.py

import os
from dotenv import load_dotenv

from langchain_qdrant import QdrantVectorStore
from langchain_google_genai import ChatGoogleGenerativeAI
from langchain_core.prompts import PromptTemplate
from langchain.retrievers.multi_query import MultiQueryRetriever
from langchain_core.output_parsers import BaseOutputParser
from typing import List, Dict, Set
import concurrent.futures
from langchain_core.documents import Document

from utils.output_parser import output_parser
from config.vector_store import get_vector_store
from llm.prompt_templates import QUERY_REWRITE_PROMPT

# ----- setup --------

qdrant = get_vector_store()

llm = ChatGoogleGenerativeAI(
  model="gemini-2.5-flash",
  google_api_key=os.getenv("GEMINI_API_KEY"),
  temperature=0,
  max_tokens=None,
  timeout=None,
  max_retries=2,
)

llm_chain = QUERY_REWRITE_PROMPT | llm | output_parser

def parallel_query_retriver(
  user_query: str, 
  llm_chain=llm_chain, 
  retriever=qdrant.as_retriever()
) -> List[Document]:
  """
  Generate multiple queries using llm_chain, fetch documents using retriever,
  and return deduplicated list of documents.

  Args:
    user_query (str): The input user query.
    llm_chain: A Runnable (PromptTemplate | LLM | OutputParser) for generating queries.
    retriever: A LangChain retriever (e.g., qdrant.as_retriever()).

  Returns:
    List[Document]: Deduplicated retrieved documents across all queries.
  """
  # Step 1: Generate multiple queries
  generated_queries = llm_chain.invoke(user_query)

  # Step 2: Fetch documents in parallel
  def fetch_docs(query: str):
    return retriever.invoke(query)

  all_docs: List[Document] = []
  with concurrent.futures.ThreadPoolExecutor() as executor:
    results = executor.map(fetch_docs, generated_queries)
    for docs in results:
      all_docs.extend(docs)

  # Step 3: Deduplicate docs
  seen: Set[str] = set()
  unique_docs: List[Document] = []

  for doc in all_docs:
    doc_id = doc.metadata.get("id") or hash(doc.page_content.strip())
    if doc_id not in seen:
      seen.add(doc_id)
      unique_docs.append(doc)

  return unique_docs

chat.py

import os
from openai import OpenAI

from retriever.retrival import parallel_query_retriver

api_key = os.getenv("GEMINI_API_KEY")

client = OpenAI(
  api_key=api_key,
  base_url="https://generativelanguage.googleapis.com/v1beta/openai/"
)

def llm_chat(query: str):
  search_results =  parallel_query_retriver(query)

  context = "\n\n"
  for doc in search_results:
    # print(doc)
    context += f"Page Content: {doc.page_content}\nPage Number: {doc.metadata['page_label']}\nFile Location: {doc.metadata['source']} \n\n"


  SYSTEM_PROMPT = f"""
    You are a helpful AI assistant who answers user query based on the available context retrieved from a PDF file along with page_contents and page number.

    You should only answer the user based on the following context and navigate the user to open the right page number to know more.
    answer should be in details.
    Context:
    {context}
  """

  chat_completion = client.chat.completions.create(
    model="gemini-2.0-flash",
    messages=[
      {"role": "system", "content": SYSTEM_PROMPT},
      {"role": "user", "content": query}
    ],
  )

  return chat_completion.choices[0].message.content

main.py

import os
from dotenv import load_dotenv

from generator.chat import llm_chat

load_dotenv()

def main():
  query = "what is fs module?"
  result = llm_chat(query=query)

  print(result)

if __name__ == "__main__":
  main()

Full source code available at: GitHub

https://understanding-gen-ai.hashnode.dev/rag-retrieval-augmented-generation-explained

To solve a business use case a simple RAG won’t do anything, we need something optimized, accurate and reliable RAG system. There are various strategies to optimise the RAG and the most common is Query Translation.

Let’s understand that, user can ask anything to AI, user may not precise about what he wants and may ask irrelevant or half baked query, which might generate an unexpected output, and hence decrease the user experience. In this case user asked an wrong query and will blame the system that it is not generating the correct response. So we can not rely on the user query. Also we can not guide all the users on how they can ask the more accurate query.

Example:

If our system generates correct output for below query

“What is Amir Khan’s character name in 3 idiots?”

but suppose user asks something like

“Who is Rancho?”

Our system might not generate the correct response as it becomes generic question.

So, the solution here is Query Translation. as the name says we translate the users query in a such a way that it is clearly understandable to LLM so that it can generate accurate response.

There are different techniques in Query Translation as per below:

1. Parallel Query Retrieval (Fan-Out)

2. Reciprocal Rank Fusion

3. Query Decomposition (Chain of Thoughts)

4. Step-Back Prompting

5. HyDE (Hypothetical Document Embeddings)

Let’s briefly understand each technique one by one.

Parallel Query Retrieval (Fan-Out)

In this technique we take user query, ask one LLM model to generate 4-5 different queries based on the user query, then we use this 5 queries to retrieve the documents from vector store. As these 5 queries will bring the more context will also retrieve the more chunks from vector store, Then we filter the unique documents from all the retrieved documents. Using this filtered documents and original user query we ask LLM to answer the user query which will generate more accurate response, as it will have more context.

This is also known as multi-query

Read More: Parallel Query Retrieval in RAG: In-Depth Guide with Code Examples

Reciprocal Rank Fusion

This technique is similar to Parallel Query Retrieval. Here also we generate multiple queries based on user query, retrieve documents based on those multiple queries. But here instead of just filtering unique documents we rank them using ranking formula and take the top n documents to feed the LLM to generate the response.

Ranking Formula:

Basically we rank the documents based on frequency and position of occurrence

Read More: Reciprocal Rank Fusion (RRF) in RAG: Complete Guide with Working Code

Query Decomposition (Chain of Thoughts)

In this techniques, we generate the sub-query based on users query, we retrieve the documents and generate it’s answer is isolation. In same we we generate multiple sub-query and it’s answers, and lastly using this Q&A pairs we answer the users original query.

Step Back Prompting

Its similar to basic RAG but, In this technique we ask LLM to generate broader view question of users query. As broader view will fetch the more context about the query.

HyDE (Hypothetical Document Embeddings)

In this technique we ask LLM model to generate the hypothetical document which will cover all the aspects to answer the user query. Then using this generated document we retrieve the documents from our vector store, This will fetch all the relevant documents from the store as it covers all the aspects. Answer generated based on those retrieved documents will be more precise and accurate.

In production application there are many steps involved in RAG system and Query Translation one of them. There is no right way to pick up the best technique, it totally depends on the use case. Sometime you might need to combine multiple techniques together to solve your problem.

Summary

In this article, we delve into advanced strategies for optimizing Retrieval Augmented Generation (RAG) systems, focusing on improving accuracy and reliability. While a simple RAG setup may not suffice for complex business use cases, techniques like Query Translation can enhance performance. Key methods discussed include Parallel Query Retrieval, Reciprocal Rank Fusion, Query Decomposition, Step Back Prompting, and HyDE. These strategies involve transforming user queries to ensure precise document retrieval and response generation. The choice of technique depends on specific applications, and sometimes combining multiple methods is necessary for optimal results.

In this blog we will talk about RAG, how it works, what are different aspects of RAG system, different techniques involved in it, also we will build our own basic pdf chat RAG.

What is RAG?

As you know, When you talk with any AI model, it gives you the the responses based on it’s pretrained data. But what if I want model to answer based on my data which resides in by dataset which can be anything like, documents(pdf, word, excel), database or even website. It is won’t happen directly as model don’t have context of our data, it’s not trained on it.

So, we need to give context to model. But we can’t provide everything to model for every query users asks, it will go outside of its context windows and also increase the cost and irrelevant data to userquery might decrease the accuracy. Here RAG solves our problem.

How RAG Works?

When user asks query, we first search the relevant information from our datasource. Then using this information we ask LLM to generate the response, which will be based on our data. But is not that simple,

The above diagram show the typical RAG system. As you can see in the above diagram you can see that RAG consists of 3 parts:

1. Indexing / Ingestion

2. Retrieval

3. Generation

PS: Real-world RAG system will be more complex, This is just for understanding of RAG.

let’s understand each part one by one.

Indexing:

Indexing also referred as Ingestion, means loading the data from your dataset into vector store. This step in only performed for first time or when the dataset is changed. Generally we will have seperate script or pipeline for this.

In Indexing we first split the large dataset into smaller chunks. Then this chunks are converted into vector embeddings using any embedding model. Then this embeddings are stored into a vector store.

Retrieval

In retrieval part, we retrive the relevant documents from vector store as per the user query. Retrieval itself has two steps in it.

First, Indexing: We Index the user query with a same embedding model that we used in the Indexing part, and generate it’s vector embeddings.

Second, Retrieval: Based on this vector embeddings of user query we perform the similarity search operation in the vector store, and fetch the top k similar documents from it.

Generation

In this part, we take the user query and retrieved relevant data and feed it to the LLM Chat model and generate the response.

This generated response will based on our provided context to LLM.

Basic RAG

Now once you have understood about the RAG, lets build basic pdf chat RAG.

We will take one pdf (nodejs.pdf), index it, store it’s embeddings in vector store (QdrantDB) and we will chat over it. We will be using langchain for this.

For this example, We will using self hosted QdrantDB using docker. Below is docker-compose.yml file for it. You can user by running docker compose up command.

services:
  qdrant:
    image: qdrant/qdrant
    ports:
      - 6333:6333

Step 1: Create qdrant client:

#qdrant.py

from qdrant_client import QdrantClient

qdrant_client = QdrantClient(url="http://localhost:6333")

Step 2: create an embedder

#embedder.py

import os
from dotenv import load_dotenv
from langchain_google_genai import GoogleGenerativeAIEmbeddings

load_dotenv()

embeddings = GoogleGenerativeAIEmbeddings(
  model="models/gemini-embedding-001",
  google_api_key=os.getenv("GEMINI_API_KEY")
)

Step 3: Now, create a ingestion pipeline

#Ingestion.py

from pathlib import Path

from langchain_community.document_loaders import PyPDFLoader
from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain_qdrant import QdrantVectorStore
from qdrant_client.http.models import Distance, VectorParams
from langchain_google_genai._common import GoogleGenerativeAIError

from embedder import embeddings
from qdrant import qdrant_client

print("-------- Starting Ingestion ------------")

pdf_path = Path(__file__).parent / "nodejs.pdf"
print("File Path:", pdf_path)

loader = PyPDFLoader(file_path=pdf_path)
docs = loader.load()
print("Doc length : ", len(docs))

splitter = RecursiveCharacterTextSplitter(
  chunk_size=1000,
  chunk_overlap=200
)

chunks = splitter.split_documents(documents=docs)
print("Chunks length : ", len(chunks))

qdrant_client.create_collection(
  collection_name="nodejs_document",
  vectors_config=VectorParams(size=3072, distance=Distance.COSINE)
)

qdrant = QdrantVectorStore(
  client=qdrant_client,
  collection_name="nodejs_document",
  embedding=embeddings
)

qdrant.add_documents(documents=batch)

print("----------- Ingestion completed ------------")

Step 4: Create a retrieval and generation pipeline

#reterival.py

import os
from dotenv import load_dotenv

from langchain_qdrant import QdrantVectorStore
from langchain_google_genai import ChatGoogleGenerativeAI
from langchain_core.prompts import ChatPromptTemplate

from qdrant import qdrant_client
from embedder import embeddings

load_dotenv()

qdrant = QdrantVectorStore(
  client=qdrant_client,
  collection_name="nodejs_document",
  embedding=embeddings
)

llm = ChatGoogleGenerativeAI(
  model="gemini-2.5-flash",
  google_api_key=os.getenv("GEMINI_API_KEY"),
  temperature=0,
  max_tokens=None,
  timeout=None,
  max_retries=2,
)

prompt = ChatPromptTemplate.from_messages(
  [
    (
      "system",
      """You are an helpfull AI assistant, you answwr user query based on 
         revlevant context data provided to you.
      Relevant Context data :
      {relavent_chunks}""",
    ),
    ("human", "{input}"),
  ]
)

chain = prompt | llm

query = input("> ")
relavent_chunks = qdrant.similarity_search(query=query)

ai_msg = chain.invoke({
  "relavent_chunks": relavent_chunks,
  "input": query
})

print(ai_msg.content)

As this is basic RAG we will be doing retrieval and generation in same pipeline.

relavent_chunks = qdrant.similarity_search(query=query)

The above single line acts as a retriever in this case.

qdrant.similarity_search(query=query)

The above function from langchain will do both the indexing of user query, and fetch the relevant chunks from the vector store.

Summary

In today's Gen AI landscape, mastering Retrieval-Augmented Generation (RAG) is crucial. RAG is a powerful method enabling AI models to provide responses using specific user data rather than just pretrained data. It involves three main components: Indexing/Ingestion, Retrieval, and Generation. This article explains how RAG systems work, outlining the steps to create a basic PDF chat RAG using tools like QdrantDB, Langchain, and Docker. By indexing data into vector stores and retrieving relevant information, we can enhance AI's contextual responses, making RAG an essential skill for developers in AI-driven industries.

AI Agents

AI agents are not a secret agents like James Bond. They are just a piece of code, which will fo some task for us.

All the pre trained LLM models are very good in understating user query, logical reasoning and thinking, but you ask them to perform any task they can not perform it, it’s like brain without hands and legs which can think but can not act. Agentic AI is like giving hands and legs to brain so that it can perform any action or task for us.

In Agentic AI workflow, we build some tools which acts as hands and legs of model and can perform any actions. We gave it’s access to LLM model, LLM model does the NLP (Natural Language Processing) of the user query, and based on the query it uses appropriate tool to perform the action asked by the user.

Building An Weather AI Agent

To build AI Agents we have to build an tool which will perform the action that we need.

Here we need an tool which make an API call to weather API and get a real-time current weather information for us.

def get_weather(city: str):
  print("🔨: get_weather tool called, input : ", city)

  url = f"https://wttr.in/{city}?format=%C+%t"
  response = requests.get(url)

  if response.status_code == 200:
    return response.text

  return "Something went wrong"

Now we have tell model that we have a tool which makes an and gets the real-time weather information. we provide this information in SYSTEM_PROMPT

avaliable_tools = {
  "get_weather": {
    "fn": get_weather,
    "description": "This function takes a city as a input and retuns the weather infomation for it."
  }
}

tool_descriptions = "\n".join(
  f"- {tool_name}: {avaliable_tools[tool_name]['description']}" for tool_name in avaliable_tools
)

system_prompt = f"""
  You are an helpfull weather AI assistant, you help in user to finding the current weather of their location.
  If user asks anything besids weather info, kindly reject the question.

  once user asks his query, you break down the query, analyze it, you plan how to answer it,
  use the approprate tool from the list to get required data, wait for it, and once you get data from tool
  use it to answer user query. include some funy message in output

  Follow the steps in sequence:
    analyze > plan > action > observe > output

  Rules:
    - follow strict JSON output format.
    - perform one step at a time and wait for next step
    - carefully analyze the user query

  Output JSON Format:
    {{
      "step": "string",
      "content": "string",
      "function": "string", # Name of the function in action step
      "input": "string", # Input paramenter for the function
      "result": string" # Output of the function call
    }}

  Avaliable tools:
    {tool_descriptions}

  Example:
    Input: What is weather in Pune?
    Output: {{"step": "analyze", "content": "User is instreated in knowing the weather information of pune"}}
    Output: {{"step": "plan", "content": "In order to get weather information I need to call get_weather tool with pune as parameter"}}
    Output: {{"step": "action", "function": "get_weather", "input": "pune"}}
    Output: {{"step": "observe", "result": "raniy, 20 degree Celcuis"}}
    Output: {{"step": "output", "content": "Curent weather of pune is raniy with 20 degree Celcuis, don't forget to take your umbrella!!"}}
"""

Notice in the above system_prompt we have asked model to follow the strict JSON output and also asked him to follow the certain steps, here in this case analyze > plan > action > observe > output. Also provided model an available tools list and example demonstrating how he should approach for the answer.

When we ask AI a query which is related to weather, it will give us a tool name to call with it’s input parameter during an action step, By checking which step currently we are in, we only have call the function, decided by model. Note: model only provide us the tool name and we only have to invoke is, model can not invoke the tool.

while True:
    response = client.chat.completions.create(
      model="gemini-2.5-flash",
      response_format={"type": "json_object"},
      messages=messages
    )

    parsed_response = json.loads(response.choices[0].message.content)
    messages.append({"role": "assistant", "content": json.dumps(parsed_response)})

    if parsed_response.get("step") == "output":
      print(f"🤖: {parsed_response.get("content")}")
      break

    if parsed_response.get("step") == "action":
      function_name = parsed_response.get("function")
      function_input = parsed_response.get("input")

      if function_name in avaliable_tools:
        result = avaliable_tools[function_name]["fn"](function_input)
        messages.append({"role": "assistant", "content": json.dumps({"step": "observe", "result": result})})
        continue
      else:
        print(f"❌ Invalid function name")

Notice if step is action we are extracting function_name and function_input from the models response and invoking the function from avaliable_tools list with function_input and appending the function’s output in the messages list to give the context to model.

Complete Code:

import os
import json
import requests

from openai import OpenAI
from dotenv import load_dotenv

load_dotenv()

client = OpenAI()

def get_weather(city: str):
  print("🔨: get_weather tool called, input : ", city)

  url = f"https://wttr.in/{city}?format=%C+%t"
  response = requests.get(url)

  if response.status_code == 200:
    return response.text

  return "Something went wrong"

avaliable_tools = {
  "get_weather": {
    "fn": get_weather,
    "description": "This function takes a city as a input and retuns the weather infomation for it."
  }
}

tool_descriptions = "\n".join(
  f"- {tool_name}: {avaliable_tools[tool_name]['description']}" for tool_name in avaliable_tools
)

system_prompt = f"""
  You are an helpfull weather AI assistant, you help in user to finding the current weather of their location.
  If user asks anything besids weather info, kindly reject the question.

  once user asks his query, you break down the query, analyze it, you plan how to answer it,
  use the approprate tool from the list to get required data, wait for it, and once you get data from tool
  use it to answer user query. include some funy message in output

  Rules:
    - follow strict JSON output format.
    - perform one step at a time and wait for next step
    - carefully analyze the user query

  Output JSON Format:
    {{
      "step": "string",
      "content": "string",
      "function": "string", # Name of the function in action step
      "input": "string", # Input paramenter for the function
      "result": string" # Output of the function call
    }}

  Avaliable tools:
    {tool_descriptions}

  Example:
    Input: What is weather in Pune?
    Output: {{"step": "analyze", "content": "User is instreated in knowing the weather information of pune"}}
    Output: {{"step": "plan", "content": "In order to get weather information I need to call get_weather tool with pune as parameter"}}
    Output: {{"step": "action", "function": "get_weather", "input": "pune"}}
    Output: {{"step": "observe", "result": "raniy, 20 degree Celcuis"}}
    Output: {{"step": "output", "content": "Curent weather of pune is raniy with 20 degree Celcuis, don't forget to take your umbrella!!"}}
"""

messages = []
messages.append({"role": "system", "content": system_prompt})

print("Welcome 🙋🏼‍♂️, Know the weather of your location!⛅")
while True:
  query = input("> ")
  messages.append({"role": "user", "content": query})

  if query.lower() in ['exit', 'quit'] :
    print("Good Bye!! 👋🏼")
    break

  while True:
    response = client.chat.completions.create(
      model="gemini-2.5-flash",
      response_format={"type": "json_object"},
      messages=messages
    )

    parsed_response = json.loads(response.choices[0].message.content)
    messages.append({"role": "assistant", "content": json.dumps(parsed_response)})

    if parsed_response.get("step") == "output":
      print(f"🤖: {parsed_response.get("content")}")
      break

    if parsed_response.get("step") == "action":
      function_name = parsed_response.get("function")
      function_input = parsed_response.get("input")

      if function_name in avaliable_tools:
        result = avaliable_tools[function_name]["fn"](function_input)
        messages.append({"role": "assistant", "content": json.dumps({"step": "observe", "result": result})})
        continue
      else:
        print(f"❌ Invalid function name")

    print(f"🧠: {parsed_response.get("content")}")

Above is the complete for a simple real-time weather AI Agent code. You can directly copy and run it. Make sure you provide the OpenAI API Key.

Now, once you understand the Agentic AI concept, we just have to build various tools and give their access to model. Use case this can be anything, we can build function which can read/write in database, a function which can even write a code for us, any function which fulfills your requirement.

You are just one function away to build your own personalized AI Agent.

Summary

The article introduces AI Agents and Agentic AI workflows, explaining their function as code pieces that execute tasks, akin to giving hands and legs to a brain (LLM models). It outlines steps to build an AI agent with a real-time weather example, showcasing how to create a tool for weather data retrieval via API calls. The guide emphasizes a structured workflow: analyze, plan, action, observe, and output. The article also highlights the capability of creating personalized AI agents by developing various functions that can interact with databases, generate code, or meet specific needs, contingent on integrating them with AI models like OpenAI's.

Before learning about Prompting techniques, let’s understand few key concepts.

What is Prompt?

Prompt is a piece of text or input given to AI model, We can say that it’s nothing but initial tokens.

System Prompt

System prompt is set of predefined instructions which sets the initial context. System prompt are guidelines given to the model by its developer or administrator to shape the behaviour of model, how AI responds, its tone.

Prompting Techniques

Zero Shot Prompting

In this prompting technique we directly ask question to LLM, we didn’t provide any background context or give any example to LLM

from openai import OpenAI
from dotenv import load_dotenv

load_dotenv()

client = OpenAI()

responce = client.chat.completions.create(
  model="gpt-4",
  messages=[
   { "role": "user", "content": "What is 2 + 2?"}
  ]
)

print(responce.choices[0].message.content)

Few Shot Prompting

In this prompting technique we sets context to LLM, give few examples for it. Which will help LLM to answer the query efficiently.

from dotenv import load_dotenv
from openai import OpenAI

load_dotenv()

client = OpenAI()

system_prompt = """
You are an AI Assistant who is specialized in maths.
You should not answer any query that is not related to maths.

For a given query help user to solve that along with explanation.

Example:
Input: 2 + 2
Output: 2 + 2 is 4 which is calculated by adding 2 with 2.

Input: 3 * 10
Output: 3 * 10 is 30 which is calculated by multipling 3 by 10. Funfact you can even multiply 10 * 3 which gives same result.

Input: Why is sky blue?
Output: Bruh? You alright? Is it maths query?
"""

result = client.chat.completions.create(
    model="gpt-4",
    messages=[
        { "role": "system", "content": system_prompt },
        { "role": "user", "content": "what is a mobile phone?" }
    ]
)

print(result.choices[0].message.content)

Chain of Thoughts

In this type of technique we encourage model to break down the user query and think over it 3-5 times before answering the question.

import json

from dotenv import load_dotenv
from openai import OpenAI

load_dotenv()

client = OpenAI()

system_prompt = """
You are an AI assistant who is expert in breaking down complex problems and then resolve the user query.

For the given user input, analyse the input and break down the problem step by step.
Atleast think 5-6 steps on how to solve the problem before solving it down.

The steps are you get a user input, you analyse, you think, you again think for several times and then return an output with explanation and then finally you validate the output as well before giving final result.

Follow the steps in sequence that is "analyse", "think", "output", "validate" and finally "result".

Rules:
1. Follow the strict JSON output as per Output schema.
2. Always perform one step at a time and wait for next input
3. Carefully analyse the user query

Output Format:
{{ step: "string", content: "string" }}

Example:
Input: What is 2 + 2.
Output: {{ step: "analyse", content: "Alright! The user is intersted in maths query and he is asking a basic arthermatic operation" }}
Output: {{ step: "think", content: "To perform the addition i must go from left to right and add all the operands" }}
Output: {{ step: "output", content: "4" }}
Output: {{ step: "validate", content: "seems like 4 is correct ans for 2 + 2" }}
Output: {{ step: "result", content: "2 + 2 = 4 and that is calculated by adding all numbers" }}

"""

messages = [
    { "role": "system", "content": system_prompt },
]


query = input("> ")
messages.append({ "role": "user", "content": query })


while True:
    response = client.chat.completions.create(
        model="gpt-4o",
        response_format={"type": "json_object"},
        messages=messages
    )

    parsed_response = json.loads(response.choices[0].message.content)
    messages.append({ "role": "assistant", "content": json.dumps(parsed_response) })

    if parsed_response.get("step") != "output":
        print(f"🧠: {parsed_response.get("content")}")
        continue

    print(f"🤖: {parsed_response.get("content")}")
    break

Persona-based Prompting

In this type of technique, The model is instructed to respond as if it were a particular character or professional. We assign model some personality and expect model to answer the questions in tone of that person.

from dotenv import load_dotenv
from openai import OpenAI

load_dotenv()

client = OpenAI()

system_prompt = """
  Your are Hitesh sir, who is expert in Technologies like MERN Stack, Python, Gen AI
  You lot to talk about new and emerging techology, You talk in Hinglish (Hindi + English)
  like "Hannji kaise ho!", Hitesh uses "Hann ji!!" in most of sentences and love to drink Chai,
  Hitesh Sir have two youtube channels named ChaiAurCode and Hitesh Choudhri.

  Hitesh Sir guides students in their learning journy and loves to solve their doubts. keep answer short and sweet.


  Converation with Hitesh Sir looks like this:
  1.  
    Hitesh: Hnjii, to aap Ai se darney waalo me se hain ya use karney waalo me se hai?
    Student: Hum to AI ka use karne wale hai.
    Hitesh: To kal he humne chai aur code youtube channel pai ek n8 wala course dala hai wo dekha ki nahi?
    Student: Ji sir, dekha na badhiya hai.
    Hitesh: Glad you found this content valuable, more such videos coming!

  2. 
    Student: Without formal education can I get job in coding
    Hitesh: Bilkul lekin dekhate hai, dekho bahot sari company bolti hai ki hume education ki requirement nahi hai
            lekin agr unke pass 5000 application aa gayi to kaise filter karenge wo log, kaise filter kare ki aapko 
            le ya na le, tak lagta hai criteria, ta ki bhid ko chatana hai, agar aapke pass formal education nahi hai
            to aapko un logo se jyada mehant karni padegi, unse jyada aapko apne aap ko proof karna padega jinke pass
            education bhi hai aur, aap jitni mehnat bhi kr rahe hai.

  Some of Hitesh Sir's tweets 
    1. Jb protein ka paani aa hi gya h to, gol gappe bna do na uske. Per puchka 3gm 😂
    2. Ye flight travel se hi dr lagne laga h ab to. Affordable vs Zindgi me to Zindgi hi select krenge na. Kya hi chal h, fielding lagi hui h hr jgh🫣
    3. Ab @coolifyio jaise projects ki hum baat nhi krenge to kon krega Self host coolify on @Hostinger ChaiAurCode YouTube channel pe available h video, enjoy
"""

result = client.chat.completions.create(
    model="gpt-4",
    messages=[
        { "role": "system", "content": system_prompt },
        { "role": "user", "content": "what is a mobile phone?" }
    ]
)

print(result.choices[0].message.content)

Self-Consistency Prompting:

In this type of technique, we ask model to generate multiple outputs and ask him to choose the most relevant answer.

import json

from dotenv import load_dotenv
from openai import OpenAI

load_dotenv()

client = OpenAI()

system_prompt = """
  You are an helpfull AI assistant, you help is solving users query.
  User can ask you about any thing, you have carefully understand the user input and
  generate at least 4 to 5 responces carefully analyze the responce cross check each on them with
  user query and give user the best sutiable and relevant responce.

  The steps are you get an user input, you analyze it, you generate 4-5 responces, you analyze the responces and query, and finally you give the output

  Follow the steps in squence that is "analyze", "think", "reponces", "responce_analysis" and "output"

  Rules:
    1. Follow the strict JSON output as per Output schema.
    2. Always perform one step at a time and wait for next input
    3. Carefully analyse the user query

  Output Format:
    {{ step: "string", content: "string" }}

  Example:
  Input: what is greater 9.8 or 9.11
  Output: {{ step: "analyze", content: "The user is asking about what is greater 9.8 or 9.11?"}}
  Output: {{ step: "think", content: "To answer this question I must think is aspect of fileds, mathematic, writing, finical, time"}}
  Output: {{ step: "responce", content: [
      "If you are taking about mathematical terms 9.8 is greater that 9.11",
      "If you are asking in therms of book chapter 9.11 is greater than 9.8 as chapter 9.11 comes after 9.8",
      "User is asking about what is greater 9.8 or 9.11, 9.8 can be considered as 9.80, and 80 is greater than 11, so 9.8 is greater.",
      "If we look in aspect of time, 9 hrs 8 min is less than 9 hrs 11 min, so in this case 9.11 is greater."
    ]
  }}
  Output: {{ step: "responce_analysis", content: "after thinking in all the aspects like mathematic, writing, and time, 
          also noticed that user hasn't provided the aspecct he is looking, I will assume he maths aspect is most comman in this of questions 
          so the answer will be 9.8 is greater that 9.11"}}
  Output: {{ step: output" content: "9.8 is greater that 9.11"}}

"""

messages = [
    { "role": "system", "content": system_prompt },
]


query = input("> ")
messages.append({ "role": "user", "content": query })


while True:
    response = client.chat.completions.create(
        model="gpt-4o",
        response_format={"type": "json_object"},
        messages=messages
    )

    parsed_response = json.loads(response.choices[0].message.content)
    messages.append({ "role": "assistant", "content": json.dumps(parsed_response) })

    if parsed_response.get("step") != "output":
        print(f"🧠: {parsed_response.get("content")}")
        continue

    print(f"🤖: {parsed_response.get("content")}")
    break

Instruction Prompting

The model is explicitly instructed to follow a particular format or guideline.

Direct Answer Prompting

The model is asked to give a concise and direct response without explanation.

Role-Playing Prompting

The model assumes a specific role and interacts accordingly.

Contextual Prompting

The prompt includes background information to improve response quality.

Multimodal Prompting

The model is given a combination of text, images, or other modalities to generate a response.

Everyone has used Chat GPT, but have you wondered how the GPT answers your query?, how it writes a code that fits to your requirements?, How it generates the content?, Let’s understand this.

Well, the answer of this is in the name of Chat GPT - GPT (Generative Pretrained Transformer), it is a Transformer which is Pertained with high volume of data and it is generative in nature. Let’s understander this transformer in detail.

Chat GPT is interface using which we can chat with GPT model with added functionality.

Transformer

Ref: https://research.google/pubs/attention-is-all-you-need/

The transformer you see in the above image is proposed by Google, It was originally developed for Google translator, This also used in LLM training. We won’t going the depth of transform, but only going to understand how it works

Phase 1 : Input Embedding

Tokenization

When enter a query to GPT, your query is first tokenized based on some predertimed algorithms. Every LLM has its own tokenizer.

Tokenization is nothing but breaking down the text or input data into smaller chunks called tokens. This tokens are often converted into numerical for better understating of LLMs. Tokenization is first step in making data understandable for AI

tiktoken is library used by OpenAI to convert words into tokens,

import tiktoken

encoder = tiktoken.encoding_for_model('gpt-4o')
print("Vocab Size: ", encoder.n_vocab) # (200,019)

text="The cat sat on the mat"

encoded = encoder.encode(text)
print("Encoded Tokens: ", encoded) # [976, 9059, 10139, 402, 290, 2450]

decoded = encoder.decode(encoded)
print("Decoded Text: ", decoded) # "The cat sat on the mat"

Vector Embedding

Once the your input query is tokenized, this data is transformed into multidimensional representation where each dimension represent the similar meaning and close relation among the words. This numerical repreparation of data points representing their meaning and relation is known as Vector Embedding. This enables LLM’s to process data efficiently.

The position of data points in space reflects its meaning and relation with other data points. The words with similar meaning will be placed closer in the space while words with dissimilar meaning will apart from each other.

Example:

King and Queen are similar words and has relation in between them, also the words Man and Woman has same meaning and same relation between them, when we plot this words on 3D space, it will looks something like below.

On the graph there will be same distance between King to Queen and Man to Woman, King to Man and Queen to Woman.

https://projector.tensorflow.org/ You can go to this link to visualize how vector embeddings looks like.

Vector Embeddings helps us find the semantic meaning of the words.

from google import genai
from dotenv import load_dotenv
import os

load_dotenv()

client = genai.Client(api_key=os.getenv("GEMINI_API_KEY"))

text="The cat sat on the mat"

result = client.models.embed_content(
  model="gemini-embedding-001",
  contents=text
)

print("Embeded Text: ", result.embeddings)

#  values=[
#     -0.022850024,
#     0.012087576,
#     -0.0012127935,
#     -0.07871191,
#     0.0022645553,
#     <... 3067 more items ...>,
#   ]

Phase 2 : Positional Encoding

Once the vector embeddings are generated then this embeddings are encoded based on their position. In This process the words in sentences are provided with the information about their sequence in the sentence. As the transformer process the all the input tokens parallely, and don’t capture the sequence of the words. This encoding is added to the word embeddings to allow the model to understand the relationships between words based on their position in the sentence.

Without positional information, the model would treat all words the same regardless of their position, making it difficult to understand context and relationships. Positional encoding involves adding a vector, representing the position of a token in the sequence, to the token's embedding vector.

Example

Consider the sentence “The Cat sat on the mat”, if we don’t consider the positional encoding sentence might become as “The mat sat on the Cat”, which will entirely change the meaning of the sentence.

Phase 3 : Self Attention

Self Attention is the mechanism that allows the transformer to highlight the importance of the different words of input sentence when processing the words. It helps the model to understand the relation between the words by considering the entire context rather than just relaying on static embeddings. Self Attention allows the model to dynamically adjust the representation of each word based on its relationship with all other words in the sequence, leading to a more context-aware understanding of the input.

Example:

Consider we have two sentences -

1. The river bank

2. The ICICI bank

In this above sentences bank is common word, but it has different meaning in both the sentences. In Self Attention it has been made aware that in first sentence it is associated with river and in second sentence it is associated with ICICI, so the word back adjust its embedding to be more precise about which back are we talking about.

It can done in single headed or multi headed format.

Both Encoder and Decoder of LLM works on the nearly same principle.

Extra key concepts

Knowledge cutoff

All the LLM’s are trained with past data and can not answer the query related to real-time data. Every model has a Knowledge cutoff, the data till the data model is trained on.

When you ask the GPT what is current weather of Pune, It won’t be able to answer it, as it does not have current data.

But Chat GPT can answer it, as it uses Agentic AI workflows to call Weather API and gets the real-time weather information. This is one of the added functionality

Temperature

It is a parameter of model that controls the randomness and creativity of the model's output

Softmax

The softmax is function that is used in the final layer of a neural network model for classification of tasks, it converts raw output scores into probabilities. it also known as logits

Vocab Size:

Number of unique token available in the tokenization dictionary