Where and Filter in Spark Dataframes
In this blog, we will learn how to filter rows from spark dataframe using Where and Filter functions.
Removing White Spaces From Data in Spark
White spaces can be a headache if not removed before processing data. We will learn how to remove spaces from data in spark using inbuilt functions.
Adding White Spaces to Data in Spark Dataframe
In this blog, we will learn how to use rpad and lpad functions to add padding to data in spark dataframe.
String Functions in Spark
This blog is intended to be a quick reference for the most commonly used string functions in Spark. It will cover all of the core string processing operations that are supported by Spark. In addition, it should serve as a useful guide for users who wish to easily integrate these into their own applications.
Select Expr in Spark Dataframe
In this blog, we will learn how to use select and expr in the Spark data frame. We will learn multiple use cases along with selectExpr.
Inserting Data In Hive Partitioned tables
In this blog, we will learn how to insert data in partitions in hive table. We will write queries to insert data in static as well as dynamic partitions.
Adding Custom Schema to Spark Dataframe
We will learn how to specify our custom schema with column names and data types for Spark data frames.
Reading data from a file in Spark
We will learn how to load data from JSON, CSV, TSV, Pipe Delimited or any other type for delimited file to spark Dataframe.
Hive Data Manipulation – Loading Data to Hive Tables
We will learn how to load and populate data to hive table. We will also learn how to copy data to hive tables from local system.