release notes

Author: finitearth

docs/release-notes/v2.1.0.md

@@ -0,0 +1,20 @@
+## Release vX.X.X
+### What's changed
+
+#### Added features:
+* We added Reward and LLM-as-a-Judge to our task family
+    * Reward allows you to write a custom function that scores the prediction, without requiring groundtruth
+    * LLM-as-a-Judge allows you to deligate the task of scoring a prediction to a Judge-LLM, optionally accepting groundtruth
+
+* Changes to CAPO, to make it applicable to the new tasks:
+    * CAPO now accepts input parameter "check_fs_accuracy" (default True) - in case of reward tasks the accuracy cannot be evaluated, so we will take the prediction of the downstream_llm as target of fs.
+    * CAPO also accepts "create_fs_reasoning" (default is True): if set to false, just use input-output pairs from df_few_shots
+
+* introduces tag-extraction function, to centralize repeated code for extractions like "<final_answer>5</final_answer>"
+
+#### Further changes:
+* We now utilize mypy for automated type checking
+* core functionalities of classification task has been moved to base task to prevent code duplication for other tasks
+* test coverage is now boosted to >90%
+
+**Full Changelog**: [here](https://github.com/finitearth/promptolution/compare/W.W.W...vX.X.X)

mkdocs.yml

@@ -47,6 +47,7 @@ nav:
   - Home: index.md
   - Release Notes:
     - Overview: release-notes.md
+    - v2.1.0: release-notes/v2.1.0.md
     - v2.0.1: release-notes/v2.0.1.md
     - v2.0.0: release-notes/v2.0.0.md
     - v1.4.0: release-notes/v1.4.0.md

tests/llms/test_vllm.py

@@ -2,7 +2,6 @@
 
 import pytest
 
-# Import the module to test
 from promptolution.llms.vllm import VLLM
 
 vllm = pytest.importorskip("vllm")