How to light your 'Spark on a stick'

You do not need to make any changes to the file! This section is just FYI.

Spark properties set in the file control most Spark parameters.

The file is located in the following directory: USB:\spark\conf\

Note, the actual file in the directory will be named spark-defaults.conf.template. In order to set configurations in this file, first copy the file to a new file named and make the changes in the new file.

Example file:

spark.master            spark://
spark.executor.memory   512m
spark.eventLog.enabled  true
spark.serializer        org.apache.spark.serializer.KryoSerializer
spark.driver.memory              5g
spark.executor.extraJavaOptions  -XX:+PrintGCDetails -Dkey=value -Dnumbers="one two three"

Instead of setting Spark properties in this file, it is possible to also set properties during run time in the SparkConf or via the spark-submit command. For details, see:

Properties set directly on the SparkConf take highest precedence, then flags passed to spark-submit or spark-shell, then options in the spark-defaults.conf file.