Date & Timestamp Functions in Spark

Updated On February 12, 2021 | By Mahesh Mogal

Spark has multiple date and timestamp functions to make our data processing easier. handling date type data can become difficult if we do not know easy functions that we can use. Below is a list of multiple useful functions with examples from the spark. So let us get started.

current_date

Using this function, we can get current date.

current_timestamp

Similarly, we can use current_timestamp to get the exact time with date. You can check the above code for it. We can see its output as below.

dayofmonth

This will return day of month

Day of Week

We can get day of week using aptly name function"dayofweek"

Day of Year

There is also a function to get the day of the year for any date. In the below example, we are getting a day of the year for today's date which is the 19th of Aug.

day_week_year
Day of Week and Year Example

Week of Year

We can get week of year from date using code below.

Year, Month & Quarter from Date

We can also extract the year, month, and quarter from date using inbuilt functions in spark.

Last Day of Month

There if function in spark "last_day" which returns the last day of the month from the date provided.

Format Date

We can use below code to format date as well change date format.

If you want to know more about formatting date you can read this blog.

Date Difference

We can find a date after or before "x days" using functions below. We can also find the difference between dates and months.

  • add_months - adding months to current date
  • datediff - difference between dates in days
  • months_between- difference between dates in months

You can get more details about these functions in this blog.

Current Timestamp to Unix timestamp and Vice Versa

Below is an example, which shows how to convert the current timestamp to the Unix timestamp.

And we can get the current timestamp from Unix timestamp using function from_unixtime.

If you want to know more about working with timestamps in Spark, please check out this blog.

Conclusion

We have listed many date and timestamp functions from the spark. I hope you will find it useful. if you know more useful function please let me know, i will add it to this list. See you soon.

Mahesh Mogal

I am passionate about Cloud, Data Analytics, Machine Learning, and Artificial Intelligence. I like to learn and try out new things. I have started blogging about my experience while learning these exciting technologies.

Stay Updated with Latest Blogs

Get latest blogs delivered to your mail directly.

Recent Posts

Spark Join Types With Examples

In this blog, we are going to learn different spark join types. We will also write code and validate data output for each join type to better understand them.

Read More
Integrate Spark with Jupyter Notebook and Visual Studio Code

In this blog, we are going to integrate spark with jupyter notebook and visual studio code to create easy-to-use development environment.

Read More
Reading Data From SQL Tables in Spark

In this blog, we are going to learn about reading data from SQL tables in Spark. We will create Spark data frames from tables and query results as well.

Read More

Leave a Reply

Your email address will not be published. Required fields are marked *

linkedin facebook pinterest youtube rss twitter instagram facebook-blank rss-blank linkedin-blank pinterest youtube twitter instagram