Date & Timestamp Functions in Spark

Updated On February 12, 2021 | By Mahesh Mogal

Spark has multiple date and timestamp functions to make our data processing easier. handling date type data can become difficult if we do not know easy functions that we can use. Below is a list of multiple useful functions with examples from the spark. So let us get started.

current_date

Using this function, we can get current date.

current_timestamp

Similarly, we can use current_timestamp to get the exact time with date. You can check the above code for it. We can see its output as below.

dayofmonth

This will return day of month

Day of Week

We can get day of week using aptly name function"dayofweek"

Day of Year

There is also a function to get the day of the year for any date. In the below example, we are getting a day of the year for today's date which is the 19th of Aug.

day_week_year
Day of Week and Year Example

Week of Year

We can get week of year from date using code below.

Year, Month & Quarter from Date

We can also extract the year, month, and quarter from date using inbuilt functions in spark.

Last Day of Month

There if function in spark "last_day" which returns the last day of the month from the date provided.

Format Date

We can use below code to format date as well change date format.

If you want to know more about formatting date you can read this blog.

Date Difference

We can find a date after or before "x days" using functions below. We can also find the difference between dates and months.

  • add_months - adding months to current date
  • datediff - difference between dates in days
  • months_between- difference between dates in months

You can get more details about these functions in this blog.

Current Timestamp to Unix timestamp and Vice Versa

Below is an example, which shows how to convert the current timestamp to the Unix timestamp.

And we can get the current timestamp from Unix timestamp using function from_unixtime.

If you want to know more about working with timestamps in Spark, please check out this blog.

Conclusion

We have listed many date and timestamp functions from the spark. I hope you will find it useful. if you know more useful function please let me know, i will add it to this list. See you soon.

Date & Timestamp Functions in Spark

Mahesh Mogal

I am passionate about Cloud, Data Analytics, Machine Learning, and Artificial Intelligence. I like to learn and try out new things. I have started blogging about my experience while learning these exciting technologies.

Stay Updated with Latest Blogs

Get latest blogs delivered to your mail directly.

Recent Posts

Where and Filter in Spark Dataframes

In this blog, we will learn how to filter rows from spark dataframe using Where and Filter functions.

Where and Filter in Spark Dataframes
Read More
Distinct Rows and Distinct Count from Spark Dataframe

Getting distinct values from columns or rows is one of most used operations. We will learn how to get distinct values as well as count of distinct values.

Distinct Rows and Distinct Count from Spark Dataframe
Read More
Sorting in Spark Dataframe

In this blog, we will learn how to sort rows in spark dataframe based on some column values.

Sorting in Spark Dataframe
Read More

Leave a Reply

Your email address will not be published. Required fields are marked *

linkedin facebook pinterest youtube rss twitter instagram facebook-blank rss-blank linkedin-blank pinterest youtube twitter instagram
Share via
Copy link
Powered by Social Snap