Distinct Rows and Distinct Count from Spark Dataframe

Updated On February 11, 2021 | By Mahesh Mogal

In this blog, we will learn how to get distinct values from columns or rows in the Spark dataframe. We will also learn how we can count distinct values. We will be using our same flight data for example.

Distinct Values from Dataframe

Consider that we want to get all combinations of source and destination countries from our data. We can easily do this using the following code.

In spark, we can chain multiple operations one after another. Here we are using where clause with distinct values.

Counting Distinct Values

We can also easily count distinct values by chaining count function after we distinct function.

Using dropDuplicates function

If we want to drop all duplicate rows from the dataframe we can also use "dropDuplicates" function.

I hope this helps. See you soon 🙂

Distinct Rows and Distinct Count from Spark Dataframe

Mahesh Mogal

I am passionate about Cloud, Data Analytics, Machine Learning, and Artificial Intelligence. I like to learn and try out new things. I have started blogging about my experience while learning these exciting technologies.

Stay Updated with Latest Blogs

Get latest blogs delivered to your mail directly.

Recent Posts

Spark Join Types With Examples

In this blog, we are going to learn different spark join types. We will also write code and validate data output for each join type to better understand them.

Spark Join Types With Examples
Read More
Integrate Spark with Jupyter Notebook and Visual Studio Code

In this blog, we are going to integrate spark with jupyter notebook and visual studio code to create easy-to-use development environment.

Integrate Spark with Jupyter Notebook and Visual Studio Code
Read More
Reading Data From SQL Tables in Spark

In this blog, we are going to learn about reading data from SQL tables in Spark. We will create Spark data frames from tables and query results as well.

Reading Data From SQL Tables in Spark
Read More

Leave a Reply

Your email address will not be published. Required fields are marked *

linkedin facebook pinterest youtube rss twitter instagram facebook-blank rss-blank linkedin-blank pinterest youtube twitter instagram
Copy link
Powered by Social Snap