[DO NOT MERGE] Siddhi retriever

comps/retrievers/multimodal_langchain/redis/README.md

@@ -0,0 +1,121 @@
+# Retriever Microservice
+
+This retriever microservice is a highly efficient search service designed for handling and retrieving embedding vectors from multimodal data. It operates by receiving an embedding vector as input and conducting a similarity search against vectors stored in a VectorDB database. Users must specify the VectorDB's URL and the index name, and the service searches within that index to find documents with the highest similarity to the input vector.
+
+The service primarily utilizes similarity measures in vector space to rapidly retrieve contentually similar documents. The vector-based retrieval approach is particularly suited for handling large datasets, offering fast and accurate search results that significantly enhance the efficiency and quality of information retrieval.
+
+Overall, this microservice provides robust backend support for applications requiring efficient similarity searches, playing a vital role in scenarios such as recommendation systems, information retrieval, or any other context where precise measurement of document similarity is crucial.
+
+## 🚀1. Start Microservice with Python (Option 1)
+
+To start the retriever microservice, you must first install the required python packages. 
+
+### 1.1 Install Requirements
+
+```bash
+pip install -r requirements.txt
+```
+### 1.2 Setup VectorDB Service
+
+You need to setup your own VectorDB service (Redis in this example), and ingest your knowledge documents into the vector database.
+
+As for Redis, you could start a docker container using the following commands.
+Remember to ingest data into it manually.
+
+```bash
+docker run -d --name="redis-vector-db" -p 6379:6379 -p 8001:8001 redis/redis-stack:7.2.0-v9
+```
+### 1.3 Ingest images or video
+
+Upload a video or images using the dataprep microservice, instructions can be found  [here](https://github.com/opea-project/GenAIComps/tree/main/comps/dataprep/redis/multimodal_langchain/README.md).
+
+### 1.4 Start Retriever Service
+
+```bash
+python langchain_multimodal/retriever_redis.py
+```
+
+## 🚀2. Start Microservice with Docker (Option 2)
+
+### 2.1 Setup Environment Variables
+
+```bash
+export your_ip=$(hostname -I | awk '{print $1}')
+export REDIS_URL="redis://${your_ip}:6379"
+export INDEX_NAME=${your_index_name}
+```
+
+### 2.2 Build Docker Image
+
+```bash
+cd ../../../../
+docker build -t opea/multimodal-retriever-redis:latest --build-arg https_proxy=$https_proxy --build-arg http_proxy=$http_proxy -f comps/retrievers/multimodal_langchain/redis/docker/Dockerfile .
+```
+
+To start a docker container, you have two options:
+
+- A. Run Docker with CLI
+- B. Run Docker with Docker Compose
+
+You can choose one as needed.
+
+### 2.3 Run Docker with CLI (Option A)
+
+```bash
+docker run -d --name="multimodal-retriever-redis-server" -p 7000:7000 --ipc=host -e http_proxy=$http_proxy -e https_proxy=$https_proxy -e REDIS_URL=$REDIS_URL -e INDEX_NAME=$INDEX_NAME opea/multimodal-retriever-redis:latest
+```
+
+### 2.4 Run Docker with Docker Compose (Option B)
+
+```bash
+cd docker
+docker compose -f docker_compose_retriever.yaml up -d
+```
+
+## 🚀3. Consume Retriever Service
+
+### 3.1 Consume Embedding Service
+
+To consume the Retriever Microservice, you can generate a mock embedding vector of length 512 with Python.
+
+```bash
+your_embedding=$(python -c "import random; embedding = [random.uniform(-1, 1) for _ in range(512)]; print(embedding)")
+curl http://${your_ip}:7000/v1/multimodal_retrieval \
+  -X POST \
+  -d "{\"text\":\"What is the revenue of Nike in 2023?\",\"embedding\":${your_embedding}}" \
+  -H 'Content-Type: application/json'
+```
+
+You can set the parameters for the retriever.
+
+```bash
+your_embedding=$(python -c "import random; embedding = [random.uniform(-1, 1) for _ in range(512)]; print(embedding)")
+curl http://localhost:7000/v1/multimodal_retrieval \
+  -X POST \
+  -d "{\"text\":\"What is the revenue of Nike in 2023?\",\"embedding\":${your_embedding},\"search_type\":\"similarity\", \"k\":4}" \
+  -H 'Content-Type: application/json'
+```
+
+```bash
+your_embedding=$(python -c "import random; embedding = [random.uniform(-1, 1) for _ in range(512)]; print(embedding)")
+curl http://localhost:7000/v1/multimodal_retrieval \
+  -X POST \
+  -d "{\"text\":\"What is the revenue of Nike in 2023?\",\"embedding\":${your_embedding},\"search_type\":\"similarity_distance_threshold\", \"k\":4, \"distance_threshold\":1.0}" \
+  -H 'Content-Type: application/json'
+```
+
+```bash
+your_embedding=$(python -c "import random; embedding = [random.uniform(-1, 1) for _ in range(512)]; print(embedding)")
+curl http://localhost:7000/v1/multimodal_retrieval \
+  -X POST \
+  -d "{\"text\":\"What is the revenue of Nike in 2023?\",\"embedding\":${your_embedding},\"search_type\":\"similarity_score_threshold\", \"k\":4, \"score_threshold\":0.2}" \
+  -H 'Content-Type: application/json'
+```
+
+```bash
+your_embedding=$(python -c "import random; embedding = [random.uniform(-1, 1) for _ in range(512)]; print(embedding)")
+curl http://localhost:7000/v1/multimodal_retrieval \
+  -X POST \
+  -d "{\"text\":\"What is the revenue of Nike in 2023?\",\"embedding\":${your_embedding},\"search_type\":\"mmr\", \"k\":4, \"fetch_k\":20, \"lambda_mult\":0.5}" \
+  -H 'Content-Type: application/json'
+```

comps/retrievers/multimodal_langchain/redis/__init__.py

@@ -0,0 +1,2 @@
+# Copyright (C) 2024 Intel Corporation
+# SPDX-License-Identifier: Apache-2.0

comps/retrievers/multimodal_langchain/redis/docker/Dockerfile

@@ -0,0 +1,29 @@
+# Copyright (C) 2024 Intel Corporation
+# SPDX-License-Identifier: Apache-2.0
+
+FROM langchain/langchain:latest
+
+ARG ARCH="cpu"
+
+RUN apt-get update -y && apt-get install -y --no-install-recommends --fix-missing \
+    libgl1-mesa-glx \
+    libjemalloc-dev \
+    vim
+
+RUN useradd -m -s /bin/bash user && \
+    mkdir -p /home/user && \
+    chown -R user /home/user/
+
+COPY comps /home/user/comps
+
+USER user
+
+RUN pip install --no-cache-dir --upgrade pip && \
+    if [ ${ARCH} = "cpu" ]; then pip install torch torchvision --index-url https://download.pytorch.org/whl/cpu; fi && \
+    pip install --no-cache-dir -r /home/user/comps/retrievers/multimodal_langchain/redis/requirements.txt
+
+ENV PYTHONPATH=$PYTHONPATH:/home/user
+
+WORKDIR /home/user/comps/retrievers/multimodal_langchain/redis
+
+ENTRYPOINT ["python", "retriever_redis.py"]

comps/retrievers/multimodal_langchain/redis/docker/docker_compose_retriever.yaml

@@ -0,0 +1,23 @@
+# Copyright (C) 2024 Intel Corporation
+# SPDX-License-Identifier: Apache-2.0
+
+version: "1.0"
+
+services:
+  retriever:
+    image: opea/multimodal-retriever-redis:latest
+    container_name: multimodal-retriever-redis-server
+    ports:
+      - "7000:7000"
+    ipc: host
+    environment:
+      no_proxy: ${no_proxy}
+      http_proxy: ${http_proxy}
+      https_proxy: ${https_proxy}
+      REDIS_URL: ${REDIS_URL}
+      INDEX_NAME: ${INDEX_NAME}
+    restart: unless-stopped
+
+networks:
+  default:
+    driver: bridge

comps/retrievers/multimodal_langchain/redis/multimodal_config.py

@@ -0,0 +1,80 @@
+# Copyright (C) 2024 Intel Corporation
+# SPDX-License-Identifier: Apache-2.0
+
+import os
+
+current_file_path = os.path.abspath(__file__)
+parent_dir = os.path.dirname(current_file_path)
+
+def get_boolean_env_var(var_name, default_value=False):
+    """Retrieve the boolean value of an environment variable.
+    Args:
+    var_name (str): The name of the environment variable to retrieve.
+    default_value (bool): The default value to return if the variable
+    is not found.
+    Returns:
+    bool: The value of the environment variable, interpreted as a boolean.
+    """
+    true_values = {"true", "1", "t", "y", "yes"}
+    false_values = {"false", "0", "f", "n", "no"}
+
+    # Retrieve the environment variable's value
+    value = os.getenv(var_name, "").lower()
+
+    # Decide the boolean value based on the content of the string
+    if value in true_values:
+        return True
+    elif value in false_values:
+        return False
+    else:
+        return default_value
+
+
+# Check for openai API key
+#if "OPENAI_API_KEY" not in os.environ:
+#    raise Exception("Must provide an OPENAI_API_KEY as an env var.")
+
+
+# Whether or not to enable langchain debugging
+DEBUG = get_boolean_env_var("DEBUG", False)
+# Set DEBUG env var to "true" if you wish to enable LC debugging module
+if DEBUG:
+    import langchain
+
+    langchain.debug = True
+
+
+# Embedding model
+EMBED_MODEL = os.getenv("EMBED_MODEL", "BridgeTower/bridgetower-large-itm-mlm-itc")
+
+# Redis Connection Information
+REDIS_HOST = os.getenv("REDIS_HOST", "localhost")
+REDIS_PORT = int(os.getenv("REDIS_PORT", 6379))
+
+
+def format_redis_conn_from_env():
+    redis_url = os.getenv("REDIS_URL", None)
+    if redis_url:
+        return redis_url
+    else:
+        using_ssl = get_boolean_env_var("REDIS_SSL", False)
+        start = "rediss://" if using_ssl else "redis://"
+
+        # if using RBAC
+        password = os.getenv("REDIS_PASSWORD", None)
+        username = os.getenv("REDIS_USERNAME", "default")
+        if password is not None:
+            start += f"{username}:{password}@"
+
+        return start + f"{REDIS_HOST}:{REDIS_PORT}"
+
+
+REDIS_URL = format_redis_conn_from_env()
+
+# Vector Index Configuration
+INDEX_NAME = os.getenv("INDEX_NAME", "test-index")
+
+REDIS_SCHEMA = os.getenv("REDIS_SCHEMA", "redis_schema.yml")
+schema_path = os.path.join(parent_dir, REDIS_SCHEMA)
+INDEX_SCHEMA = schema_path
+NUM_RETRIEVED_RESULTS = int(os.getenv("NUM_RETRIEVED_RESULTS", 1))

comps/retrievers/multimodal_langchain/redis/redis_schema.yml

@@ -0,0 +1,19 @@
+# Copyright (C) 2024 Intel Corporation
+# SPDX-License-Identifier: Apache-2.0
+
+text:
+  - name: content
+  - name: b64_img_str
+  - name: video_id
+  - name: source_video
+  - name: embedding_type
+  - name: title
+  - name: transcript_for_inference
+numeric:
+  - name: time_of_frame_ms
+vector:
+  - name: content_vector
+    algorithm: HNSW
+    datatype: FLOAT32
+    dims: 512
+    distance_metric: COSINE

comps/retrievers/multimodal_langchain/redis/requirements.txt

@@ -0,0 +1,11 @@
+docarray[full]
+fastapi
+langchain_community
+opentelemetry-api
+opentelemetry-exporter-otlp
+opentelemetry-sdk
+prometheus-fastapi-instrumentator
+redis
+shortuuid
+uvicorn
+transformers

comps/retrievers/multimodal_langchain/redis/retriever_redis.py

@@ -0,0 +1,93 @@
+# Copyright (C) 2024 Intel Corporation
+# SPDX-License-Identifier: Apache-2.0
+
+import time
+from typing import Union
+
+from comps.embeddings.multimodal_embeddings.bridgetower import BridgeTowerEmbedding
+from langchain_community.vectorstores import Redis
+from multimodal_config import INDEX_NAME, REDIS_URL, REDIS_SCHEMA
+
+from comps import (
+    EmbedMultimodalDoc,
+    SearchedMultimodalDoc,
+    ServiceType,
+    TextDoc,
+    opea_microservices,
+    register_microservice,
+    register_statistics,
+    statistics_dict,
+)
+from comps.cores.proto.api_protocol import (
+    ChatCompletionRequest,
+    RetrievalRequest,
+    RetrievalResponse,
+    RetrievalResponseData,
+)
+
+@register_microservice(
+    name="opea_service@multimodal_retriever_redis",
+    service_type=ServiceType.RETRIEVER,
+    endpoint="/v1/multimodal_retrieval",
+    host="0.0.0.0",
+    port=7000,
+)
+
+@register_statistics(names=["opea_service@multimodal_retriever_redis"])
+def retrieve(
+    input: Union[EmbedMultimodalDoc, RetrievalRequest, ChatCompletionRequest]
+) -> Union[SearchedMultimodalDoc, RetrievalResponse, ChatCompletionRequest]:
+
+    start = time.time()
+    # check if the Redis index has data
+    if vector_db.client.keys() == []:
+        search_res = []
+    else:
+        # if the Redis index has data, perform the search
+        if input.search_type == "similarity":
+            search_res = vector_db.similarity_search_by_vector(embedding=input.embedding, k=input.k)
+        elif input.search_type == "similarity_distance_threshold":
+            if input.distance_threshold is None:
+                raise ValueError("distance_threshold must be provided for " + "similarity_distance_threshold retriever")
+            search_res = vector_db.similarity_search_by_vector(
+                embedding=input.embedding, k=input.k, distance_threshold=input.distance_threshold
+            )
+        elif input.search_type == "similarity_score_threshold":
+            docs_and_similarities = vector_db.similarity_search_with_relevance_scores(
+                query=input.text, k=input.k, score_threshold=input.score_threshold
+            )
+            search_res = [doc for doc, _ in docs_and_similarities]
+        elif input.search_type == "mmr":
+            search_res = vector_db.max_marginal_relevance_search(
+                query=input.text, k=input.k, fetch_k=input.fetch_k, lambda_mult=input.lambda_mult
+            )
+        else:
+            raise ValueError(f"{input.search_type} not valid")
+
+    # return different response format
+    retrieved_docs = []
+    if isinstance(input, EmbedMultimodalDoc):
+        metadata_list = []
+        for r in search_res:
+            metadata_list.append(r.metadata)
+            retrieved_docs.append(TextDoc(text=r.page_content))
+        result = SearchedMultimodalDoc(retrieved_docs=retrieved_docs, initial_query=input.text, metadata=metadata_list)
+    else:
+        for r in search_res:
+            retrieved_docs.append(RetrievalResponseData(text=r.page_content, metadata=r.metadata))
+        if isinstance(input, RetrievalRequest):
+            result = RetrievalResponse(retrieved_docs=retrieved_docs)
+        elif isinstance(input, ChatCompletionRequest):
+            input.retrieved_docs = retrieved_docs
+            input.documents = [doc.text for doc in retrieved_docs]
+            result = input
+
+    statistics_dict["opea_service@multimodal_retriever_redis"].append_latency(time.time() - start, None)
+    return result
+
+
+if __name__ == "__main__":
+
+    embeddings = BridgeTowerEmbedding()
+    vector_db = Redis.from_existing_index(embedding=embeddings, schema=REDIS_SCHEMA, index_name=INDEX_NAME, redis_url=REDIS_URL)
+    opea_microservices["opea_service@multimodal_retriever_redis"].start()