Refactor/hf df functions

.Rbuildignore

@@ -2,7 +2,7 @@
 ^EndpointR\.Rproj$
 ^\.Rproj\.user$
 todos.md
-dev_docs/
+^dev_docs/
 README\.Rmd
 CONTRIBUTORS\.md
 todos\.qmd
@@ -11,3 +11,4 @@ todos\.qmd
 ^docs$
 ^pkgdown$
 ^\.github$
+^test_dir/

.gitignore

@@ -36,12 +36,11 @@ rsconnect/
 .Rproj.user
 inst/doc
 EndpointR.Rproj
-
 *.html
 *_dev_files*
-
 dev_docs/project_test_run.qmd
 docs
-
 # testing /dev_docs/ artifacts
 *.csv
+test_dir
+metadata_test_dir

DESCRIPTION

@@ -1,6 +1,6 @@
 Package: EndpointR
 Title: Connects to various Machine Learning inference providers
-Version: 0.1.1
+Version: 0.1.2
 Authors@R: 
     person("Jack", "Penzer", , "Jack.penzer@sharecreative.com", role = c("aut", "cre"))
 Description: EndpointR is a 'batteries included', open-source R package for connecting to various APIs for Machine Learning model predictions. EndpointR is built for company-specific use cases, so may not be useful to a wide audience.
@@ -32,7 +32,8 @@ Imports:
     tibble,
     S7,
     jsonvalidate,
-    readr
+    readr,
+    arrow
 VignetteBuilder: knitr
 Depends: 
     R (>= 3.5)

NAMESPACE

@@ -6,12 +6,15 @@ export(hf_build_request)
 export(hf_build_request_batch)
 export(hf_build_request_df)
 export(hf_classify_batch)
+export(hf_classify_chunks)
 export(hf_classify_df)
 export(hf_classify_text)
 export(hf_embed_batch)
 export(hf_embed_chunks)
 export(hf_embed_df)
 export(hf_embed_text)
+export(hf_get_endpoint_info)
+export(hf_get_model_max_length)
 export(hf_perform_request)
 export(json_dump)
 export(json_schema)

NEWS.md

@@ -1,6 +1,27 @@
-# Endpointr 0.1.2
+# EndpointR 0.1.2
 
--   [ ] `hf_embed_df()`, `hf_classify_df()` improved to write to files similarly to the upgrades applied in 0.1qq.1
+-   **File writing improvements**: `hf_embed_df()` and `hf_classify_df()` now write intermediate results as `.parquet` files to `output_dir` directories, similar to improvements in 0.1.1 for OpenAI functions
+
+-   **Parameter changes**: Moved from `batch_size` to `chunk_size` argument across `hf_embed_df()`, `hf_classify_df()`, and `oai_complete_df()` for consistency
+
+-   **New chunking functions**: Introduced `hf_embed_chunks()` and `hf_classify_chunks()` for more efficient batch processing with better error handling
+
+-   **Dependency update**: Package now depends on `arrow` for faster `.parquet` file writing and reading
+
+-   **Metadata tracking**: Hugging Face functions that write to files (`hf_embed_df()`, `hf_classify_df()`, `hf_embed_chunks()`, `hf_classify_chunks()`) now write `metadata.json` to output directories containing:
+    -   Endpoint URL and API key name used
+    -   Processing parameters (chunk_size, concurrent_requests, timeout, max_retries)
+    -   Inference parameters (truncate, max_length)
+    -   Timestamp and row counts
+    -   Useful for debugging, reproducibility, and tracking which models/endpoints were used
+
+-   **max_length parameter**: Added `max_length` parameter to `hf_classify_df()` and `hf_classify_chunks()` for text truncation control. Note: `hf_embed_df()` handles truncation automatically via endpoint configuration (set `AUTO_TRUNCATE` in endpoint settings)
+
+-   **New utility functions**:
+    -   `hf_get_model_max_length()` - Retrieve maximum token length for a Hugging Face model
+    -   `hf_get_endpoint_info()` - Retrieve detailed information about a Hugging Face Inference Endpoint
+
+-   **Improved reporting**: Chunked/batch processing functions now report total successes and failures at completion
 
 # EndpointR 0.1.1
 
@@ -15,3 +36,4 @@ Initial BETA release, ships with:
 -   Support for text completion using OpenAI models via the Chat Completions API
 -   Support for embeddings with the OpenAI Embeddings API
 -   Structured outputs via JSON schemas and validators
+