how to set spark worker/executor/driver memory in R environment?

[ljzzju]

and How to set these memories when run ./bin/ddf-shell on spark standalone with master=local

I found that the spark params are restricted in the code io.spark.ddf.SparkDDFManager

to be :

private static final String[][] SPARK_ENV_VARS = new String[][] {
// @Formatter:off
{ "SPARK_APPNAME", "spark.appname" },
{ "SPARK_MASTER", "spark.master" },
{ "SPARK_HOME", "spark.home" },
{ "SPARK_SERIALIZER", "spark.kryo.registrator" },
{ "HIVE_HOME", "hive.home" },
{ "HADOOP_HOME", "hadoop.home" },
{ "DDFSPARK_JAR", "ddfspark.jar" }
// @Formatter:on
};

Does that mean I cannot set the other spark parameters in ddf?

terrible if it is true!!

[khangich]

@ljzzju I think you can set it here: https://github.com/ddf-project/DDF/blob/master/spark/conf/local/ddf-env.sh

[binhmop]

Hi @ljzzju those are default environmental values if they are not set.
You can always set spark parameters, e.g., export SPARK_EXECUTOR_MEMORY=4G
@khangich I think we need to update that script since some options are deprecated in Spark 1.0+

[ljzzju]

Thanks, all. I have got the way to set spark parameters in ddf.

However, still I have no idea how to set these parameters when using ddf in R. in case I donot want to use the spark default parameters.

Thanks!

[ctn]

@binhmop @khangich @ljzzju is this still an issue?