What is the role of SparkSession?

Updated May 5, 2026

Short answer

SparkSession is the main entry point for programming with Apache Spark and is used to create and manage Spark applications. It provides access to Spark functionality such as creating DataFrames, running SQL queries, reading data, and interacting with Spark configurations. In Spark 2.0 and later, SparkSession replaced older entry points like SQLContext and HiveContext by combining their capabilities into one unified interface.

Deep explanation

SparkSession is an object that acts as a gateway between an application and the Spark execution engine. Every Spark application typically starts by creating a SparkSession, which is then used throughout the application to perform data processing tasks.

A typical Spark application flow looks like:

TEXT

1Application Code
2       |
3       v
4SparkSession
5       |
6       v
7Spark Engine
8       |
9       v
10Cluster Resources
11       |
12       v
13Data Processing Jobs

The SparkSession manages the connection between the user's code and Spark's internal components.

---

Why SparkSession Is Important

Before Spark 2.0, developers commonly used multiple entry points:

SparkContext for core Spark functionality.
SQLContext for SQL and DataFrame operations.
HiveContext for Hive integration.

Example older approach:

Python

1sc = SparkContext()
2
3sqlContext = SQLContext(sc)
4
5hiveContext = HiveContext(sc)

Spark 2.0 introduced SparkSession to simplify this:

Python

1from pyspark.sql import SparkSession
2
3spark = SparkSession.builder \
4    .appName("MyApplication") \
5    .getOrCreate()

Now a single object provides access to most Spark features.

---

Main Responsibilities of SparkSession

1. Creating DataFrames

A major responsibility of SparkSession is creating DataFrames from different data sources.

Example:

Python

df = spark.read.csv("customers.csv", header=True)

The SparkSession provides the read API that allows applications to load data from:

CSV files.
JSON files.
Parquet files.
Hive tables.
JDBC databases.
Cloud storage systems.

Example:

Python

employees = spark.read.json("employees.json")

employees.show()

---

2. Running SQL Queries

SparkSession allows developers to execute SQL queries directly.

Example:

Python

1df.createOrReplaceTempView("employees")
2
3result = spark.sql(
4    "SELECT * FROM employees WHERE salary > 50000"
5)
6
7result.show()

The SparkSession manages the SQL execution environment.

---

3. Managing Spark Configuration

SparkSession provides access to Spark configuration settings.

Example:

Python

1spark.conf.set(
2    "spark.sql.shuffle.partitions",
3    "100"
4)

Configurations control things such as:

Memory usage.
Number of partitions.
Execution behavior.
Optimization settings.

---

4. Accessing the SparkContext

Although SparkSession is the preferred entry point, it internally uses a SparkContext.

Example:

Python

sc = spark.sparkContext

SparkContext is responsible for:

Connecting to the cluster manager.
Creating RDDs.
Coordinating distributed execution.

Relationship:

TEXT

1SparkSession
2      |
3      v
4SparkContext
5      |
6      v
7Cluster Manager
8      |
9      v
10Executors

---

5. Supporting DataFrame and Dataset APIs

SparkSession enables Spark's higher-level APIs:

DataFrames.
Datasets.
Spark SQL.

These APIs allow Spark to optimize queries using the Catalyst optimizer.

Example:

Python

1df = spark.read.parquet("sales.parquet")
2
3df.filter(
4    df.amount > 1000
5).groupBy(
6    "region"
7).count()

SparkSession helps Spark analyze and optimize this operation before execution.

---

Creating a SparkSession

The standard way to create a SparkSession is with the builder pattern.

Example:

Python

1from pyspark.sql import SparkSession
2
3spark = SparkSession.builder \
4    .appName("SalesAnalysis") \
5    .master("local[*]") \
6    .getOrCreate()

Explanation:

appName() gives the Spark application a name.
master() defines where Spark runs.
getOrCreate() creates a new session or returns an existing one.

---

getOrCreate() Behavior

A common pattern is:

Python

spark = SparkSession.builder.getOrCreate()

getOrCreate() checks whether an active SparkSession already exists.

If one exists:

TEXT

Return existing SparkSession

If one does not exist:

TEXT

Create a new SparkSession

This prevents accidentally creating multiple Spark sessions in the same application.

---

SparkSession and Cluster Execution

SparkSession itself does not process data. It coordinates requests that are executed by Spark's distributed engine.

Example:

TEXT

1User Code
2    |
3    v
4SparkSession
5    |
6    v
7Logical Plan
8    |
9    v
10Catalyst Optimizer
11    |
12    v
13Physical Plan
14    |
15    v
16Executors Process Data

The SparkSession helps create execution plans, but the actual computation happens on worker nodes.

---

SparkSession in Different Languages

SparkSession is available in multiple Spark APIs.

PySpark

Python

1from pyspark.sql import SparkSession
2
3spark = SparkSession.builder.appName(
4    "Example"
5).getOrCreate()

Scala

SCALA

val spark = SparkSession.builder()
  .appName("Example")
  .getOrCreate()

Java

1SparkSession spark =
2    SparkSession.builder()
3    .appName("Example")
4    .getOrCreate();

---

Lifecycle of SparkSession

A typical Spark application follows this lifecycle:

TEXT

11. Create SparkSession
2          |
3          v
42. Load data
5          |
6          v
73. Transform data
8          |
9          v
104. Execute actions
11          |
12          v
135. Stop SparkSession

At the end of an application:

Python

spark.stop()

This releases resources used by Spark.

---

SparkSession vs SparkContext

Feature	SparkSession	SparkContext
Introduced	Spark 2.0	Earlier Spark versions
Main purpose	Unified Spark entry point	Core Spark connection
DataFrames	Yes	No direct support
SQL	Yes	No
RDD operations	Through SparkContext	Yes
Recommended for new applications	Yes	Only when low-level APIs are needed

---

Real-world example

A company collects sales data from multiple stores and wants to calculate total revenue by region.

The Spark application starts by creating a SparkSession:

Python

1from pyspark.sql import SparkSession
2
3spark = SparkSession.builder \
4    .appName("SalesReport") \
5    .getOrCreate()
6
7sales = spark.read.csv(
8    "sales.csv",
9    header=True,
10    inferSchema=True
11)
12
13report = sales.groupBy(
14    "region"
15).sum(
16    "amount"
17)
18
19report.show()
20
21spark.stop()

Execution flow:

TEXT

1sales.csv
2    |
3    v
4SparkSession
5    |
6    v
7DataFrame Created
8    |
9    v
10Transformations Applied
11    |
12    v
13Spark Optimizes Query
14    |
15    v
16Executors Process Data
17    |
18    v
19Revenue Report Generated

The developer interacts mainly through SparkSession, while Spark handles distributed processing behind the scenes.

Common mistakes

* Thinking SparkSession performs the actual data processing. It only provides access to Spark functionality and coordinates execution.
* Creating multiple SparkSessions unnecessarily in the same application.
* Using old `SQLContext` and `HiveContext` patterns in new Spark applications without a specific reason.
* Confusing SparkSession with SparkContext. SparkSession is a higher-level entry point that internally uses SparkContext.
* Forgetting to call `spark.stop()` in long-running scripts when resources need to be released.
* Assuming SparkSession makes operations run immediately. Most DataFrame operations are lazy and execute only when an action is called.
* Using SparkSession configuration changes after jobs have already started and expecting existing executions to change.
* Believing SparkSession stores all application data. Data is distributed across Spark executors, not stored inside the session.

Follow-up questions

What is the difference between SparkSession and SparkContext?
Why was SparkSession introduced in Spark 2.0?
Does SparkSession execute Spark jobs immediately?
How does SparkSession help optimize Spark queries?
How do you create a SparkSession in PySpark?
When would you use SparkContext directly instead of SparkSession?

Short answer

Deep explanation

Why SparkSession Is Important

Main Responsibilities of SparkSession

1. Creating DataFrames

2. Running SQL Queries

3. Managing Spark Configuration

4. Accessing the SparkContext

5. Supporting DataFrame and Dataset APIs

Creating a SparkSession

getOrCreate() Behavior

SparkSession and Cluster Execution

SparkSession in Different Languages

PySpark

Scala

Java

Lifecycle of SparkSession

SparkSession vs SparkContext

Real-world example

Common mistakes

Follow-up questions

More Apache Spark interview questions