In this blog, we will learn how to use select and expr in the Spark data frame. We will learn multiple use cases along with selectExpr.
In this blog, we will learn how to use select and expr in the Spark data frame. We will learn multiple use cases along with selectExpr.
We will go through common column operations like add, rename, list, select, and dropping a column from spark dataframe.
We will learn how to add multiple partitions to hive table using msck repair table command in hive.
In this blog, we will learn how to insert data in partitions in hive table. We will write queries to insert data in static as well as dynamic partitions.
With Bucketing in Hive, we can group similar kinds of data and write it to one single file. This allows better performance while reading data & when joining two tables.
We have created partitioned tables, inserted data into them. Now, we will learn how to drop some partition or add a new partition to the table in hive.
Hive supports Static and Dynamic Partitions. Let us understand what is difference between them and their use cases.
Using Partitioning, We can increase hive query performance. But if we do not choose partitioning column correctly it can create small file issue.
We will learn how to specify our custom schema with column names and data types for Spark data frames.
We will learn how to load data from JSON, CSV, TSV, Pipe Delimited or any other type for delimited file to spark Dataframe.
We will learn how to load and populate data to hive table. We will also learn how to copy data to hive tables from local system.
We will learn how to create Hive tables, also altering table columns, adding comments and table properties and deleting Hive tables.
