Spark Master UI

There are 4 was to run spark:

• local
• Standalone
• YARN
• Mesos

When Spark is run in local mode (as you're doing on your laptop) a separate Spark Master and separate Spark Workers + Exectuors are not launched.

Instead a single Java Virtual Machine (JVM) is launched with one Executor, whose ID is <driver>. This special Executor runs the Driver (which is the "Spark shell" application in this instance) and this special Executor also runs our Scala code. By default, this single Executor will be started with X threads, where X is equal to the # of cores on your machine.

Local mode is used when you want to run Spark locally and not in a distributed cluster. Spark local mode is different than Standalone mode (which is still designed for a cluster setup).

To summarize, in local mode, the Spark shell application (aka the Driver) and the Spark Executor is run within the same JVM. More precisely, the single Executor that is launched is named <driver> and this Executor runs both the driver code and the executes our Spark Scala transformations and actions.

So, although there is no Master UI in local mode, if you are curious, here is what a Master UI looks like here is a screenshot: