Question on Spark 2.4.x support

[groodt]

Looking at the pom.xml and the release notes, it appears that version 0.2.1 supports Spark 2.4.x+ and Spark 3.x? With Scala 2.11. Is that correct?

This section of the README is misleading if so:

Version 0.1.x is based on Spark 2.3.
Version 0.2.x is based on Spark 3.0.

Given that Spark 3.0 also runs on Scala 2.12, is there likely to be a version released to bintray or maven central for 2.12? I realise that I can also build my own jar, I am just curious. Could we also not modify the pom.xml to publish for both 2.11 and 2.12?

[junshi15]

I believe v0.2.1 supports Spark 2.4 as well, but we have not tested it thoroughly. Let me know if you see issues with Spark 2.4. I will change the README accordingly.

We can try to build a jar for Scala 2.12 if it does not involve a lot of code changes. But I don't know how to do this without changing the pom.xml file. Currently I am thinking to have a dedicated branch for Scala 2.12. If you know an easy way to do so, please let us know. Thanks.

[groodt]

Thanks @junshi15 We're not seeing any issues in our tests so far on 2.4.5, but I'll let you know after any further testing.

In terms of Scala cross-compilation, should we try something like this?
https://github.com/kelnos/scala-cross-maven-plugin

It seems to handle the variable interpolation and pom rewriting. Do you think it will work? Would you accept a PR that enables this?

[junshi15]

Thanks for your suggestion. I have no experience with this plugin.
I am using IntelliJ, the limitation below is a bit annoying. But it is worth a try. Send in a PR whenever you are ready. I can test it out.

You do not put defaults for the above properties in the POM's main properties section. (Note: Some IDEs, like IntelliJ IDEA, won't work properly without a default for scala.binary.version, even if you properly select the correct profile to use. Adding a default for that property seems to work ok.)

[groodt]

I've created a PR here:
#12

[junshi15]

PR merged. Artifacts published. Thank you @groodt !