Sorting in Spark Dataframe
In this blog, we will learn how to sort rows in spark dataframe based on some column values.
Big Data
Removing White Spaces From Data in Spark
White spaces can be a headache if not removed before processing data. We will learn how to remove spaces from data in spark using inbuilt functions.
Adding White Spaces to Data in Spark Dataframe
In this blog, we will learn how to use rpad and lpad functions to add padding to data in spark dataframe.
String Functions in Spark
This blog is intended to be a quick reference for the most commonly used string functions in Spark. It will cover all of the core string processing operations that are supported by Spark. In addition, it should serve as a useful guide for users who wish to easily integrate these into their own applications.
Converting Strings to Dates in Spark
In this blog, we are going to learn how to format dates in spark along with, changing date format and converting strings to dates with proper format.
Date Difference functions in Spark
we need to find a difference between dates or find a date after or before “n” days from a given date.
We are going to use spark function to solve such problems.
Working With Timestamps in Spark
Working with timestamps while processing data can be a headache sometimes. Luckily Spark has some in-built functions to make our life easier when working with timestamps. Let us go over these functions.
Date & Timestamp Functions in Spark
Spark provides multiple Date and Timestamp functions to make processing dates easier. In this blog, we will see the date and timestamp functions with examples.
Select Expr in Spark Dataframe
In this blog, we will learn how to use select and expr in the Spark data frame. We will learn multiple use cases along with selectExpr.
Add, Rename, Drop Columns in Spark Dataframe
We will go through common column operations like add, rename, list, select, and dropping a column from spark dataframe.
MSCK Repair – Fixing Partitions in Hive Table
We will learn how to add multiple partitions to hive table using msck repair table command in hive.
Inserting Data In Hive Partitioned tables
In this blog, we will learn how to insert data in partitions in hive table. We will write queries to insert data in static as well as dynamic partitions.