`datasketches-memory` 3.0.2 does not work on Java 25

[pan3793]

Background

Apache Spark currently supports Java 17 and 21, and I'm exploring making it support Java 25. As a result, it must use datasketches-java 6.2.0 for compatibility, which stops work on Java 25

Cause: java.lang.NoClassDefFoundError: Could not initialize class org.apache.datasketches.memory.internal.BaseWritableMemoryImpl
  at org.apache.datasketches.memory.WritableMemory.writableWrap(WritableMemory.java:333)
  at org.apache.datasketches.memory.WritableMemory.writableWrap(WritableMemory.java:275)
  at org.apache.datasketches.frequencies.ItemsSketch.toByteArray(ItemsSketch.java:509)

Expectation

I understand that Java 17/21/25 is not supported by datasketches-memory 3.0.2, but I'd like to run it in limited mode with Java 25, as it does on Java 17/21.

[leerho]

@pan3793,
Sorry, that won't work. DS-memory 3.0.2 requires access to several JVM internal classes (Unsafe, VM, Bits, etc) and since Java 11, Oracle has gradually removed the ability to access those internal classes at all. The Panama project at Oracle eventually created a replacement for what we did with Memory (12 years ago), but it wasn't fully available as an LTS java release until Java 25. As of Java 25 (our DS-java 9.0.0 release) we completely rewrote the whole library, removing DS-Memory and using the new FFM capabilities native to Java.

We had no choice, and as a result the APIs had to change as well. The good news is that our library is still backward compatible with the binary serialization formats created from our very first releases. So you can still read the old sketch images in binary form.

My advice is for you to rewrite your applications to adapt to the API in our DS-Java 9.0.0. It is 100% native Java code with no runtime dependencies and It will be around for quite a while. It has a number of other improvements as well.

As for Spark, I'm not sure what the solution is. You may have to stick with Java 8/11 until Spark is fully compatible with Java 25.

[leerho]

I am closing this issue.

[pan3793]

You may have to stick with Java 8/11 until Spark is fully compatible with Java 25.

@leerho This is actually a blocker for Spark to be fully compatible with Java 25.

Spark currently uses Java 17 as a baseline, and is trying its best to be compatible with new Java versions. Before moving the baseline to Java 25, it must at least be compatible with Java 25, so it's a deadlock ...

As a fundamental library, I wonder if datasketches could leverage modern Java techniques like JEP 238: Multi-Release JAR Files to have one unified version to support Java 17/21/25?

[leerho]

We tried MR-JARs and Java Modules with ds-memory release 3.02, which supports Java 8, 11. But it was an absolute nightmare. At the time the issue wasn't with Java, per se, it was the lack of tools support: Eclipse, Intellij, etc. Also, the MR-JAR, from a design point of view, is not very flexible.

But there is another problem. In order to release ds-memory on Java 17 and still leverage off-heap memory, I had to use JVM flags "--add-modules=jdk.incubator.foreign" and on Java 21 I had to use "--enable-preview" for access to the early versions of FFM -- and those incubators and previews are no longer supported (or even available?).

So the only thing that might be possible would be a MR-JAR for Java 8 or 11 and 25. And if you don't actually use the off-heap memory functions, the Java 8/11 version will work on Java 17. It might work on 21, but I haven't tried. (There is a java version check that might have to be fixed, and you'd have to remove the JVM flags, but otherwise it might work).

Perhaps the tools support has gotten better, but I am the only one doing this kind of work, and I don't have the time or the energy to go back and do this. If someone was willing to contribute this it would be great. But, trust me, it is a lot of work.

[pan3793]

Have a look more closely, I found the root cause actually is hidden by NoClassDefFoundError

Cause: java.lang.NoClassDefFoundError: Could not initialize class org.apache.datasketches.memory.internal.BaseWritableMemoryImpl
  at org.apache.datasketches.memory.WritableMemory.writableWrap(WritableMemory.java:333)
  at org.apache.datasketches.memory.WritableMemory.writableWrap(WritableMemory.java:275)
  at org.apache.datasketches.frequencies.ItemsSketch.toByteArray(ItemsSketch.java:509)

Root cause

scala> org.apache.datasketches.memory.WritableMemory.allocate(3)
java.lang.IllegalArgumentException: Unsupported JDK Major Version. It must be one of 1.8, 8, 11, 17, 21: 25.0.1
  at org.apache.datasketches.memory.internal.ResourceImpl.checkJavaVersion(ResourceImpl.java:155)
  at org.apache.datasketches.memory.internal.ResourceImpl.parseJavaVersion(ResourceImpl.java:391)
  ... 42 elided

I opened #271 to add Java 25 to the allow list, and have some smoke tests with the latest Spark, it seems to work the same as Java 17 and 21.

With #271 patched

scala> org.apache.datasketches.memory.WritableMemory.allocate(3)
val res0: org.apache.datasketches.memory.WritableMemory =
### HeapWritableMemoryImpl SUMMARY ###
Type Info           : Writable +                      Heap +         NativeOrder +    Memory +

Header Comment      :
Call Parameters     : .toHexString(..., 0, 3), hashCode: 383679057
UnsafeObj, hashCode : byte[], 843299175
UnsafeObjHeader     : 16
ByteBuf, hashCode   : null
RegionOffset        : 0
Capacity            : 3
CumBaseOffset       : 16
MemReqSvr, hashCode : null
is Alive            : true
Read Only           : false
Type Byte Order     : LITTLE_ENDIAN
Native Byte Order   : LITTLE_ENDIAN
JDK Runtime Version : 25.0
### END SUMMARY

scala> spark.sql("""
     | SELECT hll_sketch_estimate(hll_sketch_agg(col))
     | FROM VALUES (1), (1), (2), (2), (3) tab(col);
     | """).show
warning: 1 deprecation (since 2.13.3); for details, enable `:setting -deprecation` or `:replay -deprecation`
+--------------------------------------------+
|hll_sketch_estimate(hll_sketch_agg(col, 12))|
+--------------------------------------------+
|                                           3|
+--------------------------------------------+

@leerho, do you think we can process it in this direction before Spark moves to Java 25+? And if so, is ds-memory able to deploy a SNAPSHOT or RC version to the Maven repo so I can run full integration tests with Apache Spark?