Merge pull request #150 from davidatbraintrust/patch-1

Add braintrust integration tutorial

Author: jeffchuber

docs/integrations/braintrust.md

@@ -0,0 +1,61 @@
+---
+slug: /integrations/braintrust
+title: Braintrust
+---
+
+[Braintrust](braintrustdata.com) is an enterprise-grade stack for building AI products including: evaluations, prompt playground, dataset management, tracing, etc.
+
+Braintrust provides a Typescript and Python library to run and log evaluations and integrates well with Chroma.
+
+- [Tutorial: Evaluate Chroma Retrieval app w/ Braintrust](https://www.braintrustdata.com/docs/examples/rag)
+
+Example evaluation script in Python:
+(refer to the tutorial above to get the full implementation)
+```python
+from autoevals.llm import *
+from braintrust import Eval
+
+PROJECT_NAME="Chroma_Eval"
+
+from openai import OpenAI
+
+client = OpenAI()
+leven_evaluator = LevenshteinScorer()
+
+async def pipeline_a(input, hooks=None):
+    # Get a relevant fact from Chroma
+    relevant = collection.query(
+        query_texts=[input],
+        n_results=1,
+    )
+    relevant_text = ','.join(relevant["documents"][0])
+    prompt = """
+        You are an assistant called BT. Help the user.
+        Relevant information: {relevant}
+        Question: {question}
+        Answer:
+        """.format(question=input, relevant=relevant_text)
+    messages = [{"role": "system", "content": prompt}]
+    response = client.chat.completions.create(
+        model="gpt-3.5-turbo",
+        messages=messages,
+        temperature=0,
+        max_tokens=100,
+    )
+
+    result = response.choices[0].message.content
+    return result
+
+# Run an evaluation and log to Braintrust
+await Eval(
+    PROJECT_NAME,
+    # define your test cases
+    data = lambda:[{"input": "What is my eye color?", "expected": "Brown"}], 
+    # define your retrieval pipeline w/ Chroma above
+    task = pipeline_a, 
+    # use a prebuilt scoring function or define your own :)
+    scores=[leven_evaluator], 
+)
+```
+
+Learn more: [docs](https://www.braintrustdata.com/docs).

docs/integrations/index.md

@@ -16,9 +16,10 @@ We welcome pull requests to add new Integrations to the community.
 |--------------|-----------|---------------|
 | [🦜️🔗 Langchain](/integrations/langchain) | ✅  | ✅ |
 | [🦙 LlamaIndex](/integrations/llama-index) | ✅  | :soon: |
+| [Braintrust](/integrations/braintrust) | ✅  | ✅ |
 | [🔭 OpenLLMetry](/integrations/openllmetry) | ✅     | :soon: |
 
-*Coming soon* - integrations with LangSmith, JinaAI, Braintrust and more.
+*Coming soon* - integrations with LangSmith, JinaAI, and more.
 
 *** 
 

docs/integrationsold.md

@@ -41,6 +41,14 @@ import TabItem from '@theme/TabItem';
 </TabItem>
 <TabItem value="js" label="JavaScript">
 
+## Braintrust
+
+Braintrust is the enterprise-grade stack for building AI products. From evaluations, to prompt playground, to data management,
+we take uncertainty and tedium out of incorporating AI into
+your business.
+
+- [Evaluate a Chroma RAG app using Braintrust](https://www.braintrustdata.com/docs/examples/rag)
+
 ## 🦜️🔗 LangchainJS
 
 Here is an [example in LangChainJS](https://github.com/hwchase17/langchainjs/blob/main/examples/src/chains/chat_vector_db_chroma.ts)

sidebars.js

@@ -65,6 +65,7 @@ const sidebars = {
       items: [
         'integrations/langchain',
         'integrations/llama-index',
+        'integrations/braintrust',
         'integrations/openllmetry',
       ],
     },