How to light your 'Spark on a stick'

Help and Quit

Run :help to display the help menu:

scala> :help
All commands can be abbreviated, e.g. :he instead of :help.
Those marked with a * have more detailed help, e.g. :help imports.

:cp <path>                 add a jar or directory to the classpath
:help [command]            print this summary or command-specific help
:history [num]             show the history (optional num is commands to show)
:h? <string>               search the history
:imports [name name ...]   show import history, identifying sources of names
:implicits [-v]            show the implicits in scope
:javap <path|class>        disassemble a file or class name
:load <path>               load and interpret a Scala file
:paste                     enter paste mode: all input up to ctrl-D compiled tog
:quit                      exit the repl
:replay                    reset execution and replay all previous commands
:reset                     reset the repl to its initial state, forgetting all s
ession entries
:sh <command line>         run a shell command (result is implicitly => List[Str
:silent                    disable/enable automatic printing of results
disable/enable advanced repl changes, these fix some issues but may introduce ot
This mode will be removed once these fixes stablize
:type [-v] <expr>          display the type of an expression without evaluating
:warnings                  show the suppressed warnings from the most recent lin
e which had any

We will explore some of the above commands throughout this lab.

Try quiting out of the Scala shell:

scala> :q
Stopping spark context.

You should now be back at the CMD-line.

You may notice that the Spark applications UI webpage will not fail or load or refresh as we just quit the application:

Let's open a new Spark Scala shell. Type in spark-shell and hit enter to launch the shell:

Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 1.1.0

Using Scala version 2.10.4 (Java HotSpot(TM) 64-Bit Server VM, Java 1.7.0_67)
Type in expressions to have them evaluated.
Type :help for more information.
Spark context available as sc.


Now that the Spark application is running again, the corresponding UI will also be exposed at http://localhost:4040:

There will be no active or completed stages listed as of yet, but after the next section, you will see activitity on this page...