Summary
SQLConf.getConf for certain configuration keys now returns an enum type instead of a string in Spark 4.1.0.
Details
- Spark Version: 4.1.0
- Change Type: Return type change (String → Enum)
Impact
Code that expects SQLConf.getConf(...) to return a String and performs string comparisons will fail to compile or behave incorrectly against Spark 4.1.0.
Example pattern that breaks:
// Before (Spark ≤4.0.x): returns String
val mode = conf.getConf(SOME_CONFIG)
if (mode == "LEGACY") { ... }
// After (Spark 4.1.0): returns Enum
val mode = conf.getConf(SOME_CONFIG)
// mode is now an enum, not a stringProposed Solution
Create version-specific handling or use .toString on the returned value where appropriate, or match on enum values directly for Spark 4.1.0+.
Related
- Part of Spark 4.1.0 API compatibility work
Summary
SQLConf.getConffor certain configuration keys now returns an enum type instead of a string in Spark 4.1.0.Details
Impact
Code that expects
SQLConf.getConf(...)to return aStringand performs string comparisons will fail to compile or behave incorrectly against Spark 4.1.0.Example pattern that breaks:
Proposed Solution
Create version-specific handling or use
.toStringon the returned value where appropriate, or match on enum values directly for Spark 4.1.0+.Related