v0.2.2 (#48)

* OpenAI args updated

'Model_name' deprecated in favour 'model', making whole a more uniform experience

* Unlocked OpenAI parameter

Now, use can pass any parameter to openai

* Update OpenAI.md

* Typo fixes

* Create Example OpenAI Response Format.py

* Utitlity added - Sanitize prompts

A utility that make is simple to sanitize prompts

* Utility added - Unpack Json Response

* Refactor a bit

* Index doc

* Create Sanitize Prompt.md

* Update utilities.py

* Update Sanitize Prompt.md

* Create Unpack Json Response.md

* Updated documentation

* New Utilities - Sanitize prompts, unpack json response

* Ruffed

* bump version

Author: aryadhruv

Examples/Example Arrays.py

@@ -29,7 +29,7 @@ def main():
         provider="openai",
         system_prompt="Process these Data rows as per the provided prompt",
         options={
-            "model_name": "gpt-4o-mini",
+            "model": "gpt-4o-mini",
             "temperature": 1,
             "max_tokens": 1024,
         },

Examples/Example Batch Processing.py

@@ -25,7 +25,7 @@ def main():
         provider="openai",
         system_prompt="Process these Data rows as per the provided prompt",
         options={
-            "model_name": "gpt-4o-mini",
+            "model": "gpt-4o-mini",
             "temperature": 1,
             "max_tokens": 2048,  # Ensure token limit is set
         },

Examples/Example DataFrames.py

@@ -23,7 +23,7 @@ def main():
         provider="openai",
         system_prompt="Process these Data rows as per the provided prompt",
         options={
-            "model_name": "gpt-4o-mini",
+            "model": "gpt-4o-mini",
             "temperature": 1,
             "max_tokens": 1024,
         },

Examples/Example Excel.py

@@ -25,7 +25,7 @@ def main():
         provider="openai",
         system_prompt="Process these Data rows as per the provided prompt",
         options={
-            "model_name": "gpt-4o-mini",
+            "model": "gpt-4o-mini",
             "temperature": 1,
             "max_tokens": 1024,
         },

Examples/Example OpenAI JSON Response Unpacked.py

@@ -0,0 +1,60 @@
+"""
+Example usage script for OpenAI's response format with unpacking feature.
+The utility unpack_json_responses allows user to unpack the json response into various columns.
+
+This utility support - Dataframe, Array, List.
+"""
+
+import pandas as pd
+from llmworkbook import LLMConfig, LLMRunner, LLMDataFrameIntegrator, unpack_json_responses
+from dotenv import load_dotenv
+
+load_dotenv()
+
+
+def main():
+    # 1. Create a sample dataframe
+    data = {
+        "id": [1, 2, 3, 4, 5],
+        "prompt_text": [
+            "Extract key entities (persons, places, organizations) from this text: 'OpenAI, based in San Francisco, is a leading AI research lab founded by Sam Altman and Greg Brockman.'",
+            "Convert this product description into structured data: 'The iPhone 15 Pro features a 6.1-inch display, A17 Bionic chip, and a titanium body.'",
+            "Provide a breakdown of this sentence into subject, verb, and object: 'The cat chased the mouse across the room.'",
+            "Generate a JSON object with three random trivia questions along with their answers.",
+            "Summarize the given customer review into structured categories like 'sentiment', 'key topics', and 'rating' from this text: 'The camera quality of this phone is fantastic, but the battery life could be better. I would rate it 4 out of 5.'",
+        ],
+    }
+
+    df = pd.DataFrame(data)
+
+    # 2. Create an LLM configuration
+    config = LLMConfig(
+        provider="openai",
+        system_prompt="Process these data rows as per the provided prompt. Ensure the response is strictly in JSON format.",
+        options={
+            "model": "gpt-4o-mini",
+            "temperature": 1,
+            "max_tokens": 1024,
+            "response_format" : { "type": "json_object" },
+        },
+    )
+
+    # 3. Instantiate the runner and the integrator
+    runner = LLMRunner(config)
+    integrator = LLMDataFrameIntegrator(runner=runner, df=df)
+
+    # 4. Add LLM responses to the df
+    updated_df = integrator.add_llm_responses(
+        prompt_column="prompt_text", response_column="llm_response", async_mode=True
+    )
+
+    # 5. Unpack JSON responses
+    updated_df = unpack_json_responses(updated_df)
+
+    print("DataFrame with Unpacked LLM responses:\n", updated_df)
+
+    updated_df.to_excel("testdf.xlsx")
+
+
+if __name__ == "__main__":
+    main()

Examples/Example OpenAI Response Format.py

@@ -0,0 +1,58 @@
+"""
+Example usage script for OpenAI's response format.
+"""
+
+import pandas as pd
+from llmworkbook import LLMConfig, LLMRunner, LLMDataFrameIntegrator
+from dotenv import load_dotenv
+
+load_dotenv()
+
+
+def main():
+    # 1. Create a sample dataframe
+    data = {
+        "id": [1, 2, 3, 4, 5],
+        "prompt_text": [
+            "Extract key entities (persons, places, organizations) from this text: 'OpenAI, based in San Francisco, is a leading AI research lab founded by Sam Altman and Greg Brockman.'",
+            "Convert this product description into structured data: 'The iPhone 15 Pro features a 6.1-inch display, A17 Bionic chip, and a titanium body.'",
+            "Provide a breakdown of this sentence into subject, verb, and object: 'The cat chased the mouse across the room.'",
+            "Generate a JSON object with three random trivia questions along with their answers.",
+            "Summarize the given customer review into structured categories like 'sentiment', 'key topics', and 'rating' from this text: 'The camera quality of this phone is fantastic, but the battery life could be better. I would rate it 4 out of 5.'",
+        ],
+    }
+
+    df = pd.DataFrame(data)
+
+    # 2. Create an LLM configuration
+    config = LLMConfig(
+        provider="openai",
+        system_prompt="Process these data rows as per the provided prompt. Ensure the response is strictly in JSON format.",
+        options={
+            "model": "gpt-4o-mini",
+            "temperature": 1,
+            "max_tokens": 1024,
+            "response_format" : { "type": "json_object" },
+        },
+    )
+
+    # 3. Instantiate the runner and the integrator
+    runner = LLMRunner(config)
+    integrator = LLMDataFrameIntegrator(runner=runner, df=df)
+
+    # 4. Add LLM responses to the df
+    updated_df = integrator.add_llm_responses(
+        prompt_column="prompt_text", response_column="llm_response", async_mode=True
+    )
+
+    print("DataFrame with LLM responses:\n", updated_df)
+
+    #Print Expected output - 
+    # {
+    # "subject": "The cat",
+    # "verb": "chased",
+    # "object": "the mouse"
+    # }
+
+if __name__ == "__main__":
+    main()

Examples/Example OpenAI.py

@@ -26,7 +26,7 @@ def main():
         provider="openai",
         system_prompt="Process these Data rows as per the provided prompt",
         options={
-            "model_name": "gpt-4o-mini",
+            "model": "gpt-4o-mini",
             "temperature": 1,
             "max_tokens": 1024,
         },

Examples/Example PromptSeries.py

@@ -26,7 +26,7 @@ def main():
         provider="openai",
         system_prompt="Process these prompts",
         options={
-            "model_name": "gpt-4o-mini",
+            "model": "gpt-4o-mini",
             "temperature": 1,
             "max_tokens": 1024,
         },

Examples/Example Sanitize Prompts.py

@@ -0,0 +1,62 @@
+"""
+LLMWORKBOOK provides a easy to use utility function that allow you to quickly clean the prompt inputs.
+This utility allows developer to ensure that the prompt passed to wrappers or integrators are secure and clean.
+"""
+
+from llmworkbook import sanitize_prompt
+
+prompt : str = "Some placefolder prompt"
+
+#Example 1: Sanitizing a single string prompt
+prompt = "   Tell me about AI <script>alert('XSS')</script> with [markdown] formatting!    "
+clean_prompt = sanitize_prompt(prompt)
+print(clean_prompt)
+# Output: "Tell me about AI alert('XSS') with markdown formatting!"
+
+#Example 2: Sanitizing a list of prompts
+prompt_list = [
+    "Tell me about {Python}",
+    "  <script>alert('XSS')</script>  ",
+    "Analyze the following data: [1, 2, 3, 4, 5]"
+]
+clean_list = sanitize_prompt(prompt_list)
+print(clean_list)
+# Output: ['Tell me about Python', "alert('XSS')", 'Analyze the following data: 1, 2, 3, 4, 5']
+
+# Example 3: Sanitizing a pandas DataFrame column
+import pandas as pd # noqa: E402
+
+# Create a sample DataFrame with prompts
+df = pd.DataFrame({
+    'user_id': [1, 2, 3],
+    'prompt': [
+        "Generate a *summary* of this article",
+        "  <script>malicious code</script>  ",
+        "What is the answer to [4+5]?"
+    ]
+})
+
+# Sanitize the 'prompt' column
+df['clean_prompt'] = sanitize_prompt(df['prompt'])
+print(df)
+# Output:
+#    user_id                            prompt                     clean_prompt
+# 0        1  Generate a *summary* of this article  Generate a summary of this article
+# 1        2    <script>malicious code</script>             malicious code
+# 2        3               What is the answer to [4+5]?     What is the answer to 45?
+
+#Example 4: Sanitizing a numpy array
+import numpy as np  # noqa: E402
+
+# Create an array of prompts
+prompt_array = np.array([
+    "Calculate 2+2=?",
+    "   What is the ```result```?  ",
+    "<script>alert('XSS')</script>"
+])
+
+clean_array = sanitize_prompt(prompt_array)
+print(clean_array)
+
+
+#Above clean prompts can be pass through the wrapper or integrator as needed. 

README.md

@@ -21,14 +21,10 @@
 
 ---
 
-🚀 New Feature: Batch & Row-wise Processing
-LLMWorkbook v0.2.1 now shows progress bars 🦦:
-
+## LLMWorkbook v0.2.2 new utilities 🦦:
+✔ New Utilities - Sanitize prompts, unpack json response
 ✔ Rich Console Progress bar
-✔ Row-wise Processing (Default) – Each row is sent individually to the LLM.
-✔ Row-wise Processing (Default) – Each row is sent individually to the LLM.
-✔ Batch Processing – Multiple rows are grouped together and sent as one request for efficiency.
-✔ Automatic Token Limit Handling – Ensures batch prompts stay within max_tokens limits.
+✔ Row-wise or batch-wise Processing – Choose what meets your need.
 
 ---
 

docs/CLI Usage.md

@@ -38,7 +38,7 @@ llmworkbook version
 
 - **Test LLM Connectivity:**
   ```bash
-  llmworkbook test YOUR_API_KEY --model_name gpt-4
+  llmworkbook test YOUR_API_KEY --model gpt-4
   ```
 
 - **See Version of package:**

docs/Providers/OpenAI.md

@@ -8,7 +8,7 @@ Each provider function uses specific keys from the configuration’s `options` d
 
 **Configuration Keys in `options`:**
 
-- **`model`**  
+- **`model` (earlier 'model_name')**  
   - **Type:** `str`  
   - **Description:** Specifies the model to use for generating responses (e.g., `"gpt-4o-mini"`).  
   - **Default Behavior:** If not provided, the code defaults to `"gpt-4o-mini"`.
@@ -17,7 +17,23 @@ Each provider function uses specific keys from the configuration’s `options` d
   - **Type:** `float` or `int`  
   - **Description:** Controls the randomness of the output. A higher temperature produces more varied results.
 
-**Additional Configurations (Outside `options`):**
+**Additional OpenAI API Parameters**
+Description: Any valid OpenAI API parameter (e.g., max_tokens, top_p, frequency_penalty, etc.) can be provided via options. This ensures full control over the API request without modifying the function.
+
+Example -
+```
+config = {
+    "options": {
+        "model": "gpt-4o-mini",
+        "temperature": 0.7,
+        #Additional Parameter as needed
+        "max_tokens": 500,
+        "top_p": 0.9,
+        "frequency_penalty": 0.5
+        #Output format
+        'response_format' : ...,
+    },
+```
 
 - **`api_key`**  
   - **Description:** Your OpenAI API key. If not provided in the config, the code will attempt to read it from the environment variable `OPENAI_API_KEY`.

docs/README.md

@@ -0,0 +1,19 @@
+# Documentation Index
+
+Welcome to the documentation for **LLMWorkbook**. Below is the index of all available documents:
+
+Topics - 
+- [Batch and Row Processing](Batch%20and%20Row%20Processing.md)
+- [CLI Usage](CLI%20Usage.md)
+- [Wrapping](wrapping.md)
+
+- [Providers](Providers)
+- - [GPTALL](Providers/Gpt4All.md)
+- - [Ollama](Providers/Ollama.md)
+- - [OpenAI](Providers/OpenAI.md)
+
+- [Utilities](Utilities)
+- - [Sanitize Prompt](Utilities/Sanitize%20Prompt.md)
+- - [Unpack Json Response](Utilities/Unpack%20Json%20Response.md)
+
+