Skip to main content

Databricks Fundamentals for Beginners:

 

 

What Actually Happens When You Execute a Spark DataFrame?



 

Coming from SQL Server, understanding how Spark handles data is an important first step in learning Databricks.

 

Consider this simple PySpark example. 

.
1. Read a Parquet dataset

 


df = spark.read.parquet("/sales/sales.parquet")

 

This creates a Spark DataFrame representing the dataset. Spark does not normally load every row into memory when this statement runs.

 

2. Apply a transformation

 

filtered_df = df.filter("Region = 'Kerala'")

 

This defines a filter operation. Spark generally builds a logical execution plan rather than immediately processing and storing the filtered rows.

 

3. Trigger execution

 

filtered_df.show()

 

"show()" is an action.

 

It triggers execution of the required computation, reads the necessary data, applies the filter, and displays a limited number of rows.

 

Does "show()" store the result inside the DataFrame?

 

No. A DataFrame is not a container holding a complete copy of the dataset. It represents a distributed dataset and its computation plan.

 

During execution, Spark processes data across its executors. Intermediate data may occupy memory or disk, but that does not mean the entire result is permanently stored inside the DataFrame.
This behavior is part of Spark's lazy evaluation model: transformations are generally evaluated when an action requires a result.



 

When is data physically stored?

 

To persist the filtered result as a Delta dataset, we can write it to storage:

 

filtered_df.write \
    .format("delta") \
    .mode("overwrite") \
    .save("/sales_delta")


 

This executes the computation and writes the result to the specified location. A Delta dataset includes Parquet data files and a transaction log.

 

Key concepts to remember

 

- DataFrame: Represents a distributed dataset and its computation plan.
- Transformation: Defines an operation, such as filtering or selecting columns.
- Action: Triggers computation, such as "show()" or "count()".
- Caching: Can retain computed data for reuse.
- Write operation: Persists the result to physical storage.



 

For SQL Server developers transitioning to Databricks, understanding the distinction between a query plan, execution, and physical storage provides a foundation for learning Spark optimization, caching, and distributed processing.
x

Comments

Post a Comment

Popular posts from this blog

SQL Server script Error: Insufficient result space to convert uniqueidentifier value to char.

Error:    G etting below error while trying to copy data from one table to another: Msg 8170, Level 16, State 2, Line 1004  Insufficient result space to convert uniqueidentifier value to char.   Fix: UserID column which would be having Unique identifier data require more space (36 character). Size is specified as 15 here. To fix it just specify 36 as varchar length or specify Uniqueidentifier as data type for UserID  column.
Post a Comment
Read more
TRUNCATE  Vs  DELETE       Q1. What are the difference between TRUNCATE command and DELETE command? Q2. Can TRUNCATE be rolled back like the DELETE command? Q3. What are the advantages of TRUNCATE  command compare to DELETE?    Q1.What are the difference between TRUNCATE command and DELETE command ? TRUNCATE : 1.Remove all pages from the table. You can’t use WHERE clause with TRUNCATE . 2.Deallocate the data pages in the table. 3.Faster than DELETE. 4.Reset the identity column. TRUNCATE  is a DDL statement. So you need ALTER permissions for it. 5.Doesn’t fire trigger. 6.Acquire only page or table lock.. 7. TRUNCATE  cannot be used with indexed views. 8. Drop all object’s statistics and leave the table empty. Zero pages are left in the table. DELETE: 1.Remove all pages from file if used without a WHERE clause. You can use WHERE clause with DELETE. ...
Post a Comment
Read more
Top 10 essential SQL skills for data analysts, data engineers, and data scientists Photo by  Carlos Muza  on  Unsplash It doesn’t matter how good you are at creating visualizations, extracting insights, or building complicated models; you need data in order to do those things. SQL enters the picture when it comes to extracting data from the database in order to work with it. Mastering the commonly used SQL concepts is essential to work effectively as a data analyst, data engineer, and data scientist in today's world.  The following are the top 10 skills for mastering SQL: 1. Knowing SQL Fundamentals:  Understanding the following keywords is essential to write common DML queries: SELECT, FROM, WHERE, GROUP BY, HAVING, ORDER BY, WINDOW\RANK FUNCTIONS. 2. Understanding different real-world scenarios:  Analyzing and understanding multiple domains and how to use SQL, like writing complex queries, manipulating data, pulling reports, and identifying patterns and t...
Post a Comment
Read more