Databricks Fundamentals for Beginners: What Actually Happens When You Execute a Spark DataFrame? Coming from SQL Server, understanding how Spark handles data is an important first step in learning Databricks. Consider this simple PySpark example. . 1. Read a Parquet dataset df = spark.read.parquet("/sales/sales.parquet") This creates a Spark DataFrame representing the dataset. Spark does not normally load every row into memory when this statement runs. 2. Apply a transformation filtered_df = df.filter("Region = 'Kerala'") This defines a filter operation. Spark generally builds a logical execution plan rather than immediately processing and storing the filtered rows. 3. Trigger execution filtered_df.show() "show()" is an action. It triggers execution of the required computation, reads the necessary data, applies the filter, and displays a limited number of rows. Does "show()" store the...
1.Clustered Index Scan ✅ What it is: A Clustered Index Scan means SQL Server is scanning every row of the entire table, essentially performing a "table scan" because the clustered index is the table itself. 📌 When it happens: No WHERE clause or very broad filter (low selectivity). Join/filter columns aren’t indexed selectively. Query needs most or all columns (covering queries). 🧠Performance Insight: Can be expensive for large tables. Usually appears when SQL Server can’t seek and needs to read all data. Example: Table: CREATE TABLE dbo.Sales ( SaleID INT NOT NULL IDENTITY, SaleDate DATE NOT NULL, Quantity INT NOT NULL, PRIMARY KEY (SaleID) ) Query: SELECT * FROM dbo.Sales WHERE SaleDate BETWEEN '2025-02-04' AND '2025-02-10' 2. Index Scan (Non-Clustered) ✅ What it is: A Non-Clustered Index Scan scans all rows in a specific non-clustered index( not the full table) 📌 When it happens: The query filters on a ...