Date & Timestamp Functions in Spark

Spark has multiple date and timestamp functions to make our data processing easier. handling date type data can become difficult if we do not know easy functions that we can use. Below is a list of multiple useful functions with examples from the spark. So let us get started.

current_date

Using this function, we can get current date.

current_timestamp

Similarly, we can use current_timestamp to get the exact time with date. You can check the above code for it. We can see its output as below.

dayofmonth

This will return day of month

Day of Week

We can get day of week using aptly name function”dayofweek”

Day of Year

There is also a function to get the day of the year for any date. In the below example, we are getting a day of the year for today’s date which is the 19th of Aug.

day_week_year
Day of Week and Year Example

Week of Year

We can get week of year from date using code below.

Year, Month & Quarter from Date

We can also extract the year, month, and quarter from date using inbuilt functions in spark.

Last Day of Month

There if function in spark “last_day” which returns the last day of the month from the date provided.

Format Date

We can use below code to format date as well change date format.

If you want to know more about formatting date you can read this blog.

Date Difference

We can find a date after or before “x days” using functions below. We can also find the difference between dates and months.

  • add_months – adding months to current date
  • datediff – difference between dates in days
  • months_between- difference between dates in months

You can get more details about these functions in this blog.

Current Timestamp to Unix timestamp and Vice Versa

Below is an example, which shows how to convert the current timestamp to the Unix timestamp.

And we can get the current timestamp from Unix timestamp using function from_unixtime.

If you want to know more about working with timestamps in Spark, please check out this blog.

Conclusion

We have listed many date and timestamp functions from the spark. I hope you will find it useful. if you know more useful function please let me know, i will add it to this list. See you soon.

Date & Timestamp Functions in Spark

Similar Posts

Leave a Reply

Your email address will not be published. Required fields are marked *