Where and Filter in Spark Dataframes

Updated On February 11, 2021 | By Mahesh Mogal

Filtering rows from dataframe is one of the basic tasks performed when analyzing data with Spark. Spark provides two ways to filter data. Where and Filter function. Both of these functions work in the same way, but mostly we will be using "where" due to its familiarity with SQL.

Using Where / Filter in Spark Dataframe

We can easily filter rows with some conditions as we do in SQL using "Where" function. Say we need to find all rows where the number of flights is more than 50 between the two countries.

We can also use column expressions. This time we will use "Filter" function to get desired rows from dataframe.

Chaining Multiple Conditions

Though it is possible to write multiple where conditions in one statement, it is not necessary. Even when we chain multiple conditions one after another while creating a physical plan for execution spark will optimize these operations in one single step.

That is why it is always a better idea to write multiple where conditions separately which will be easier to understand while reading code.

Multiple Where Clauses Chained Together
Multiple Where Clauses Chained Together

I hope you found this useful :). See you in next blog.

Where and Filter in Spark Dataframes

Mahesh Mogal

I am passionate about Cloud, Data Analytics, Machine Learning, and Artificial Intelligence. I like to learn and try out new things. I have started blogging about my experience while learning these exciting technologies.

Stay Updated with Latest Blogs

Get latest blogs delivered to your mail directly.

Recent Posts

Where and Filter in Spark Dataframes

In this blog, we will learn how to filter rows from spark dataframe using Where and Filter functions.

Where and Filter in Spark Dataframes
Read More
Distinct Rows and Distinct Count from Spark Dataframe

Getting distinct values from columns or rows is one of most used operations. We will learn how to get distinct values as well as count of distinct values.

Distinct Rows and Distinct Count from Spark Dataframe
Read More
Sorting in Spark Dataframe

In this blog, we will learn how to sort rows in spark dataframe based on some column values.

Sorting in Spark Dataframe
Read More

Leave a Reply

Your email address will not be published. Required fields are marked *

linkedin facebook pinterest youtube rss twitter instagram facebook-blank rss-blank linkedin-blank pinterest youtube twitter instagram
Share via
Copy link
Powered by Social Snap