Elusion v8.3.0

🚀 Elusion v8.3.0 Release Notes

📦 Installation

[dependencies]
elusion = "8.3.0"
tokio = { version = "1.50.0", features = ["rt-multi-thread"] }

[8.3.0] - 2026-04-01

Added - Elusion Project (Medallion Architecture Pipeline Framework)

• New ElusionProject struct — declarative, config-driven data pipeline framework inspired by medallion architecture (Bronze/Silver/Gold) used in Microsoft Fabric, Databricks, and Azure Data Lake
• DAG-based execution engine with topological sort — correct execution order guaranteed regardless of registration order
• Parallel execution via level-based DAG grouping — independent nodes run concurrently with tokio, dependent nodes wait automatically
• Config-driven setup via elusion.toml (project config) and connections.toml (source declarations)
• Fail-fast source validation at startup — missing files, paths, or unresolved env vars caught before execution begins
• Env var resolution via dotenvy — supports both .env files (local dev) and system environment variables (Docker/CI/CD)
• Materialization per layer — automatic write to Parquet or Delta after each node completes, before next layer starts
• Local and Microsoft Fabric/OneLake output destinations supported
• Per-model materialization overrides in elusion.toml — override layer defaults for specific models
• Execution plan printed before run — shows all nodes, layers, and materialization type per node
• Execution summary table printed after completion — model name, layer, row count, and timing per node

Medallion Layer Naming (Bronze / Silver / Gold)

• NodeLayer enum uses Bronze, Silver, Gold naming aligned with Microsoft Fabric and Databricks medallion architecture
• Bronze — raw ingestion layer (replaces Stage)
• Silver — cleaned and enriched layer (replaces Intermediate)
• Gold — aggregated business-ready layer (replaces Marts)
• NodeRegistry exposes ref_source(), ref_bronze(), ref_silver(), ref_gold() for accessing resolved DataFrames between layers

Source connectivity

• connections.toml source declarations supporting: CSV, Parquet, Delta, Fabric Service Principal, Fabric SAS token
• .source("name") — loads source automatically from connections.toml
• .source_fn("name", closure) — manual source loading without config files for quick prototyping

Layer registration — three API styles supported

• DataFrame API: .bronze(), .silver(), .gold() — chainable Elusion DataFrame operations inside async closures
• Raw SQL API: .bronze_sql(), .silver_sql(), .gold_sql() — write SQL directly against registered table names from the registry
• Slice deps API: .bronze_slice(), .silver_slice(), .gold_slice(), .bronze_sql_slice(), .silver_sql_slice(), .gold_sql_slice() — for use with DEPS: &[&str] constants declared in separate model files

Const generics support

• All layer registration methods accept [&str; N] const generic arrays instead of Vec — no vec![] required
• Slice variants accept &[&str] for use with pub const DEPS: &[&str] in model files

Separated model files (medallion-style)

• Models organized in separate files per layer: src/bronze/, src/silver/, src/gold/
• Each model file exports pub const DEPS: &[&str] and either a pub async fn model(ctx: NodeRegistry) or a pub const SQL: &str
• main.rs becomes minimal wiring only — one line per model
• Adding a new model means creating one file and adding one line to main.rs

elusion.toml configuration

[project]
name = "my_pipeline"
version = "1.0"

[materialization]
bronze = "parquet"
silver = "parquet"
gold = "parquet" -- or delta

[output]
destination = "local"

[output.local]
bronze_path = "C:\\Data\\output\\bronze"
silver_path = "C:\\Data\\output\\silver"
gold_path = "C:\\Data\\output\\gold"

connections.toml configuration

[sources.raw_sales]
type = "csv"
path = "C:\\Data\\SalesData2022.csv"

[sources.raw_fabric_orders]
type = "fabric"
abfss_path = "abfss://container@account.dfs.core.windows.net"
file_path = "bronze/orders.parquet"
tenant_id = "TENANT_ID"
client_id = "CLIENT_ID"
client_secret = "CLIENT_SECRET"

Fixed

• drop_null() now correctly handles non-string columns — previously applied TRIM() to Int64/Float64 columns causing DataFusion btrim function errors. Non-string columns now use IS NOT NULL only, string columns retain full null string detection ('', 'NULL', 'NA', 'N/A', 'NONE', '-', '?', 'NaN')

Elusion v8.2.0

🚀 Elusion v8.2.0 Release Notes

📦 Installation

[dependencies]
elusion = "8.2.0"
tokio = { version = "1.50.0", features = ["rt-multi-thread"] }

Updated

• Major dependency upgrades for long-term compatibility:
  • tokio from 1.48.0 to 1.50.0
  • datafusion from 43.0.0 to 46.0.1
  • deltalake from 0.23.0 to 0.31.0
  • arrow from 53.2.0 to 57.x (now sourced via datafusion::arrow to avoid dual-version conflicts)
  • chrono from =0.4.38 (pinned) to 0.4.44 (unpinned)
  • reqwest from 0.12.x to 0.13.2 (feature rustls-tls renamed to rustls)
  • parquet from 53.x to 58.1.0
  • calamine updated with chrono feature enabled for proper ExcelDateTime handling
  • url promoted from optional to required dependency

Fixed

• Resolved arrow-arith trait clash with chrono >= 0.4.40 (quarter() function ambiguity)
• Removed dual arrow crate conflict between deltalake (arrow 57) and datafusion (arrow 54) by routing all arrow imports through datafusion::arrow
• Fixed deltalake 0.31 breaking API changes:
  • from_uri() → from_url() (now requires Url instead of &str)
  • try_from_uri() → try_from_url()
  • StructType::new() → StructType::try_new()
  • Metadata fields made private, now accessed via methods (id(), name(), description(), etc.)
  • Protocol::new() → Protocol::default()
  • LocalFileSystem now sourced from deltalake::logstore::object_store::local
  • table.version() now returns Option<i64>
  • open_table() now requires Url instead of &str
• Fixed calamine breaking API changes:
  • DataType trait renamed to Data enum
  • Duration variant removed, mapped to Null
  • ExcelDateTime serial value conversion updated
• Fixed object_store dual-version conflict (0.12 vs 0.13)
• Fixed RecordBatch and Schema type mismatches across delta, redis, calendar, xml, excel, and with_schema modules via IPC round-trip conversion

Elusion v8.1.0

🚀 Elusion v8.1.0 Release Notes

📦 Installation

[dependencies]
elusion = "8.1.0"
tokio = { version = "1.45.0", features = ["rt-multi-thread"] }

Added

• Schema Loading Feature - Load files with predefined schemas using new_with_schema() method
• Support for custom schema definition via JSON specification with schema_from_json()
• SchemaBuilder for ergonomic, programmatic schema construction without requiring manual vec![] declarations
• FileSchema::schema_builder() convenience method for fluent schema building
• Schema support for all file formats: CSV, Parquet, JSON, Excel, XML, and Delta Lake
• FileSchema struct for schema representation and reuse
• Exposed schema-loading APIs through CustomDataFrame wrappers (not direct imports in prelude)

Elusion v8.0.0

🚀 Elusion v8.0.0 Release Notes

📦 Installation

[dependencies]
elusion = "8.0.0"
tokio = { version = "1.45.0", features = ["rt-multi-thread"] }

Added

• COPY DATA feature for continuous reading and writing for files bigger than RAM. Currenty source can be JSON, CSV and PARQUET, and destinations: CSV and PARQUET.
• RAW SQL Execution with Macro implementation for better user experience.

Elusion v7.9.0

🚀 Elusion v7.9.0 Release Notes

📦 Installation

[dependencies]
elusion = "7.9.0"
tokio = { version = "1.45.0", features = ["rt-multi-thread"] }

Added

• Dashboard Filtering Features (requires dashboard feature flag)
• Category filtering for bar, pie and donut plots with category picker UI
• Waterfall plot
• NEW Color schema for all plots

Elusion v7.7.0

🚀 Elusion v7.7.0 Release Notes

📦 Installation

[dependencies]
elusion = "7.7.0"
tokio = { version = "1.45.0", features = ["rt-multi-thread"] }

ADDED

• New features for DASHBOARD: export_report_to_pdf(), export_plot_to_png()

Elusion v7.6.0

🚀 Elusion v7.6.0 Release Notes

📦 Installation

[dependencies]
elusion = "7.6.0"
tokio = { version = "1.45.0", features = ["rt-multi-thread"] }

ENHANCED

• Schema Type conversion for Excel file reading

Elusion v7.5.0

🚀 Elusion v7.5.0 Release Notes

📦 Installation

[dependencies]
elusion = "7.5.0"
tokio = { version = "1.45.0", features = ["rt-multi-thread"] }

ADDED

• drop_duplicates_by_column() function that creates new dataframe with removed duplicate rows by keeping first row occurence

Elusion v7.4.0

🚀 Elusion v7.4.0 Release Notes

📦 Installation

[dependencies]
elusion = "7.4.0"
tokio = { version = "1.45.0", features = ["rt-multi-thread"] }

IMPROVED

• Error message for select() and group_by_all()

ADDED

• drop_duplicates() function that creates new dataframe with removed duplicate rows by keeping first row occurence

Elusion v7.3.0

🚀 Elusion v7.3.0 Release Notes

📦 Installation

[dependencies]
elusion = "7.3.0"
tokio = { version = "1.45.0", features = ["rt-multi-thread"] }

ADDED

• FTP/FTPS Connector for reading and writing files with numerous functions: from_ftp(), from_ftp_folder(), write_csv_to_ftp(), write_excel_to_ftp(), write_parquet_to_ftp()...