← Back to Categories
PySpark Interview Questions
Comprehensive list of 32 questions to test your knowledge.
Theory & Concepts
1
Spark Architecture?
2
RDD vs DataFrame?
3
Transformations vs actions?
4
Lazy evaluation?
5
Narrow vs wide transformation?
6
repartition vs coalesce?
7
Broadcast Join?
8
Data skew?
9
Catalyst Optimizer?
10
OOM errors?
11
Caching/persisting?
12
Lineage?
13
UDFs?
14
Pandas UDFs?
15
Spark UI metrics?
16
Checkpointing?
17
AQE?
18
Data serialization?
19
Job vs Stage vs Task?
20
JDBC connections?
21
DAG?
22
Spark shuffle?
23
Controlling partitions?
24
spark-submit?
25
Structured Streaming?
Hands-on & Coding Scenarios
1
Clickstream Analysis
Context / Sample Data
DataFrame: clickstream_df
| user_id | session_id | page_url | timestamp |
|---------|------------|------------|---------------------|
| u1 | s1 | /home | 2023-10-01 14:00:00 |
| u1 | s1 | /products | 2023-10-01 14:05:00 |
| u1 | s1 | /checkout | 2023-10-01 14:10:00 |
| u2 | s2 | /home | 2023-10-01 14:01:00 |
| u2 | s2 | /home | 2023-10-01 14:03:00 |
| u3 | s3 | /checkout | 2023-10-01 15:00:00 |Tasks:
- Find the top 10 most visited pages.
- Calculate session duration (max - min timestamp) per session.
- Count unique users who visited /checkout.
Follow-up / Optimization:
- Handle user_id skew (e.g., bot traffic)?
- show() vs write.parquet()?
2
Streaming + Batch Join
Context / Sample Data
Batch DataFrame: customers
| customer_id | name | tier |
|-------------|-------|--------|
| c1 | Alice | Gold |
| c2 | Bob | Silver |
| c3 | Carol | Bronze |
Streaming DataFrame: transactions
| tx_id | customer_id | amount | timestamp |
|-------|-------------|--------|---------------------|
| t1 | c1 | 100.0 | 2024-01-01 10:00:00 |
| t2 | c2 | 1500.0 | 2024-01-01 10:05:00 |
| t3 | c1 | 250.0 | 2024-01-01 10:10:00 |Tasks:
- Join transactions stream with customers batch.
- Flag transactions where amount > 1000 as 'High Value'.
- Group by tier and calculate sum of amounts.
Follow-up / Optimization:
- Optimize if customers table is small (50MB)?
- Watermarks for stateful aggregations?
3
IoT Sensor Data Parsing
Context / Sample Data
DataFrame: iot_raw
| device_id | payload |
|-----------|----------------------------------------------------|\n| sensor_01 | {"temp": 22.5, "humidity": 45, "status": "ok"} |
| sensor_02 | {"temp": 35.1, "humidity": 80, "status": "warning"} |
| sensor_03 | {"temp": 18.0, "humidity": 50, "status": "ok"} |
| sensor_04 | {"temp": 42.0, "humidity": 90, "status": "critical"} |
| sensor_05 | MALFORMED_JSON_STRING |Tasks:
- Parse the JSON 'payload' column into a Struct type.
- Extract 'temp' and 'status' into their own top-level columns.
- Filter for records where status is not 'ok' and temp > 30.
Follow-up / Optimization:
- Handle malformed JSON (sensor_05)?
- Write parsed data to Delta Lake?
4
Fraud Detection Feature Engineering
Context / Sample Data
DataFrame: transactions_df
| account_id | tx_date | amount |
|------------|------------|---------|
| ACC001 | 2024-01-01 | 50.00 |
| ACC001 | 2024-01-02 | 75.00 |
| ACC001 | 2024-01-03 | 60.00 |
| ACC001 | 2024-01-04 | 5000.00 |
| ACC002 | 2024-01-01 | 200.00 |
| ACC002 | 2024-01-02 | 180.00 |Tasks:
- Calculate a 3-day rolling average of transaction amounts per account.
- Create a lag column showing the previous transaction amount.
- Flag transactions where amount exceeds 3x the rolling average.
Follow-up / Optimization:
- Mitigate OOM for accounts with millions of transactions?
- Prepare features for MLlib using VectorAssembler?
5
Sales Data Pivoting
Context / Sample Data
DataFrame: sales_df
| store_id | product_category | revenue |
|----------|------------------|---------|
| S1 | Electronics | 1000.0 |
| S1 | Clothing | 500.0 |
| S1 | Food | 300.0 |
| S2 | Electronics | 800.0 |
| S2 | Food | 450.0 |
| S3 | Clothing | 600.0 |Tasks:
- Pivot the DataFrame so product_category values become columns.
- Fill any null values resulting from the pivot with 0.0.
- Calculate a total_revenue column summing all pivoted categories.
Follow-up / Optimization:
- Why is pivot computationally expensive?
- Optimize if you know the categories upfront?
6
Window Functions — Running Totals, LAG & LEAD
Context / Sample Data
DataFrame: daily_orders
| order_date | store_id | category | revenue |
|------------|----------|-------------|---------|
| 2024-01-01 | S1 | Electronics | 1200.0 |
| 2024-01-01 | S1 | Clothing | 400.0 |
| 2024-01-02 | S1 | Electronics | 1500.0 |
| 2024-01-02 | S2 | Electronics | 800.0 |
| 2024-01-03 | S1 | Clothing | 350.0 |
| 2024-01-03 | S2 | Clothing | 600.0 |Tasks:
- Calculate a cumulative running total of revenue per store, ordered by date.
- Add a column showing the previous day's revenue for the same store using lag().
- Calculate day-over-day revenue change using lead().
- Calculate each row's percentage contribution to its store's total revenue.
Follow-up / Optimization:
- What is the difference between rowsBetween and rangeBetween?
- How to calculate a 7-day rolling average with rangeBetween using dates?
7
Window Functions — Ranking, Top-N & Deduplication
Context / Sample Data
DataFrame: employee_salaries
| emp_id | name | department | salary | hire_date |
|--------|-------|------------|--------|------------|
| E1 | Alice | Eng | 95000 | 2020-01-15 |
| E2 | Bob | Eng | 95000 | 2021-03-10 |
| E3 | Carol | Eng | 80000 | 2022-06-01 |
| E4 | Dave | Sales | 70000 | 2019-09-20 |
| E5 | Eve | Sales | 85000 | 2020-11-05 |
| E6 | Frank | Sales | 85000 | 2023-01-12 |
| E7 | Grace | HR | 72000 | 2021-07-01 |Tasks:
- Add rank, dense_rank, and row_number columns partitioned by department, ordered by salary descending.
- Find the top 2 highest-paid employees per department (include all ties).
- Deduplicate: keep only the latest hired employee per department using row_number().
- Calculate each employee's salary gap from their department's top earner using first_value().
Follow-up / Optimization:
- When to use row_number vs dense_rank for top-N?
- How to use PARTITION BY with multiple columns?