Can't connect to HA HDFS cluster

[ottomata]

I've successfully launched ammonite-spark in local mode with hive support enabled.

I've done so with the following script and predef files:
https://gist.github.com/ottomata/f63bd89ca2acbe9a9d26d3226eaf3405
(I can't remember where I got this predef code...I did not write it).

My spark2 jars are installed via a debian package into /usr/lib/spark2 and configured in /etc/spark2/conf, so I want to use those to load and run spark.

It seems to mostly work. I can do spark SQL actions that talk to Hive (e.g. spark.table("hive_db.table")). But, when I cause spark to actually have to read underlying data from HDFS via a query, I get:

java.lang.IllegalArgumentException: java.net.UnknownHostException: analytics-hadoop

Where 'analytics-hadoop' is the HA name of my HDFS cluster. In hdfs-site.xml I have:

  <property>
    <name>dfs.nameservices</name>
    <value>analytics-hadoop</value>
  </property>

I guess my predef code is missing something to properly set up the SparkSession with Hadoop confs?

Anyway, I want to get ammonite-spark to work just like the spark-shell REPL does, so users can swap them out transparently without having to manually instantiate the SparkSession themselves.

Any tips to do this?

Thanks!
-Andrew Otto - Wikimedia Foundation