Formatting Dates in Spark

Updated On February 11, 2021 | By Mahesh Mogal

In this blog, we will be going over spark functions used to format dates, change their format, and also convert strings to date. So let us get started.

to_date - Convert String to Date

First we will see how we can convert normal date to string. We can use to_Date function for this.

It is simple right? But what will happen if our date is not in same format as "yyyy-MM-dd"? Let us see output of such code.

Oh, we are getting null. This is because spark is not able to understand our date string. To overcome this, we can specify the format for our date. to_date function accepts optional parameter as a format for the date.

format_with_to_date
Format with to_date function

Spark supported simple date format used in Java language

Spark Facts

So we are able to let spark know the format of our date and spark picked our date correctly this time. hurry!!

Changing Format of Date in Spark

We now our date is correct but we do not want this "yyyy-MM-dd" format. We want in "dd/MM/yyyy" format for some reason. We can do that as well. We can convert our date format easily.

As we can see, using date_format function we can change the format of date too as per our requirement.

I hope you found this useful. See you in next blog.

Formatting Dates in Spark

Mahesh Mogal

I am passionate about Cloud, Data Analytics, Machine Learning, and Artificial Intelligence. I like to learn and try out new things. I have started blogging about my experience while learning these exciting technologies.

Stay Updated with Latest Blogs

Get latest blogs delivered to your mail directly.

Recent Posts

Where and Filter in Spark Dataframes

In this blog, we will learn how to filter rows from spark dataframe using Where and Filter functions.

Where and Filter in Spark Dataframes
Read More
Distinct Rows and Distinct Count from Spark Dataframe

Getting distinct values from columns or rows is one of most used operations. We will learn how to get distinct values as well as count of distinct values.

Distinct Rows and Distinct Count from Spark Dataframe
Read More
Sorting in Spark Dataframe

In this blog, we will learn how to sort rows in spark dataframe based on some column values.

Sorting in Spark Dataframe
Read More

Leave a Reply

Your email address will not be published. Required fields are marked *

linkedin facebook pinterest youtube rss twitter instagram facebook-blank rss-blank linkedin-blank pinterest youtube twitter instagram
Share via
Copy link
Powered by Social Snap