How to light your 'Spark on a stick'

spark-env.sh

You do not need to make any changes to the spark-env.sh file! This section is also just FYI.

Certain Spark settings can be configured through environment variables, which are read from the USB:\spark\conf\spark-env.sh script.

Example file:

# - HADOOP_CONF_DIR, to point Spark towards Hadoop configuration files
# - SPARK_LOCAL_IP, to set the IP address Spark binds to on this node
# - SPARK_PUBLIC_DNS, to set the public dns name of the driver program
# - SPARK_CLASSPATH, default classpath entries to append

# Options read by executors and drivers running inside the cluster
# - SPARK_LOCAL_IP, to set the IP address Spark binds to on this node
# - SPARK_PUBLIC_DNS, to set the public DNS name of the driver program
# - SPARK_CLASSPATH, default classpath entries to append
# - SPARK_LOCAL_DIRS, storage directories to use on this node for shuffle and RDD data
# - MESOS_NATIVE_LIBRARY, to point to your libmesos.so if you use Mesos

More details about the environments file can be found at: https://spark.apache.org/docs/latest/configuration.html#environment-variables