Category: Big Data

Select Expr in Spark Dataframe

In this blog, we will learn how to use select and expr in the Spark data frame. We will learn multiple use cases along with selectExpr.

Read More
Add, Rename, Drop Columns in Spark Dataframe

We will go through common column operations like add, rename, list, select, and dropping a column from spark dataframe.

Read More
MSCK Repair - Fixing Partitions in Hive Table

We will learn how to add multiple partitions to hive table using msck repair table command in hive.

msck repair hive
Read More
Inserting Data In Hive Partitioned tables

In this blog, we will learn how to insert data in partitions in hive table. We will write queries to insert data in static as well as dynamic partitions.

inserting data to hive partition
Read More
Bucketing in Hive

With Bucketing in Hive, we can group similar kinds of data and write it to one single file. This allows better performance while reading data & when joining two tables.

bucketing in hive
Read More
Altering Table Partitions in Hive

We have created partitioned tables, inserted data into them. Now, we will learn how to drop some partition or add a new partition to the table in hive.

altering hive table partition
Read More
Static vs Dynamic Partitioning in Hive

Hive supports Static and Dynamic Partitions. Let us understand what is difference between them and their use cases.

static vs dynamic
Read More
Partitioning in Hive

Using Partitioning, We can increase hive query performance. But if we do not choose partitioning column correctly it can create small file issue.

Partitioning in Hive
Read More
Adding Custom Schema to Spark Dataframe

We will learn how to specify our custom schema with column names and data types for Spark data frames.

Read More
Reading data from a file in Spark

We will learn how to load data from JSON, CSV, TSV, Pipe Delimited or any other type for delimited file to spark Dataframe.

Read More
Hive Data Manipulation - Loading Data to Hive Tables

We will learn how to load and populate data to hive table. We will also learn how to copy data to hive tables from local system.

Loading Data to Hive Tables
Read More
Create, Alter, Delete Tables in Hive

We will learn how to create Hive tables, also altering table columns, adding comments and table properties and deleting Hive tables.

manage tables in hive -2
Read More
linkedin facebook pinterest youtube rss twitter instagram facebook-blank rss-blank linkedin-blank pinterest youtube twitter instagram