Working With Timestamps in Spark
Working with timestamps while processing data can be a headache sometimes. Luckily Spark has some in-built functions to make our life easier when working with timestamps. Let us go over these functions.
current_timestamp – Getting Current Timestamp
We can get current timestamp using current_timestamp function.
|
1
2
3
4
5
6
7
8
9
10
11
12
|
from pyspark.sql.functions import current_date,current_timestamp
>>>
>>> df = spark.range(2) \
... .withColumn("current_date", current_date()) \
... .withColumn("current_timestamp", current_timestamp())
>>> df.show()
+---+------------+--------------------+
| id|current_date| current_timestamp|
+---+------------+--------------------+
| 0| 2020-08-19|2020-08-19 11:21:...|
| 1| 2020-08-19|2020-08-19 11:21:...|
+---+------------+--------------------+
|
date_trunc – Truncated Date From Timestamp
We can get truncated date from timestamp using code below.
|
1
2
3
4
5
6
|
from pyspark.sql.functions import date_trunc
df.select("current_timestamp", \
date_trunc("year", col("current_timestamp")), \
date_trunc("yyyy", col("current_timestamp")), \
date_trunc("month", col("current_timestamp")), \
).show()
|
Converting Timestamp to Unix Timestamp
Spark has provided function to convert timestamp to Unix Timestamp.
|
1
2
3
4
5
6
7
8
9
|
>>> from pyspark.sql.functions import unix_timestamp
>>> df.select("current_timestamp", \
... unix_timestamp(col("current_timestamp")) \
... ).show()
+--------------------+------------------------------------------------------+
| current_timestamp|unix_timestamp(current_timestamp, yyyy-MM-dd HH:mm:ss)|
+--------------------+------------------------------------------------------+
|2020-08-19 11:27:...| 1597850861|
|2020-08-19 11:27:...| 1597850861|
|
Get Timestamp from Unix Timestamp
|
1
2
3
4
5
6
7
8
9
10
11
12
|
>>> df_unix = df.withColumn("unix_timestamp", unix_timestamp(col("current_timestamp")))
>>>
>>> from pyspark.sql.functions import from_unixtime
>>> df_unix.select("unix_timestamp", \
... from_unixtime(col("unix_timestamp")) \
... ).show()
+--------------+--------------------------------------------------+
|unix_timestamp|from_unixtime(unix_timestamp, yyyy-MM-dd HH:mm:ss)|
+--------------+--------------------------------------------------+
| 1597850920| 2020-08-19 11:28:40|
| 1597850920| 2020-08-19 11:28:40|
+--------------+--------------------------------------------------+
|
Get Timestamp from Date
We can also get from timestamp using to_timestamp function
|
1
2
3
4
5
6
7
8
9
10
|
>>> from pyspark.sql.functions import to_timestamp,col
>>> df.select("current_date", \
... to_timestamp(col("current_date"), "yyyy-MM-dd") \
... ).show()
+------------+------------------------------------------+
|current_date|to_timestamp(`current_date`, 'yyyy-MM-dd')|
+------------+------------------------------------------+
| 2020-08-19| 2020-08-19 00:00:00|
| 2020-08-19| 2020-08-19 00:00:00|
+------------+------------------------------------------+
|
I hope you found this helpful. See you in in next blog.
