Renaming DataFrame Columns in Spark
Many times we have to change column names in our data. Either the existing column name is too long or too short or not descriptive enough to understand what data we are accessing. In this blog, we are going to learn about renaming data frame columns in spark. There are many ways you can do this and you can choose whatever best fits for your needs.
Using withColumnRenamed
Before we start, let us create data frame for to work with.
|
1 2 3 4 5 6 7 8 9 10 11 |
df = spark.read\ .option("inferSchema", "true")\ .option("header","true")\ .csv("D:\\code\\spark\\spark-basics\\data\\flight-data\\csv\\2010-summary.csv") df.printSchema() root |-- DEST_COUNTRY_NAME: string (nullable = true) |-- ORIGIN_COUNTRY_NAME: string (nullable = true) |-- count: integer (nullable = true) |
Spark has provided us in built function when we want rename columns. Let us see how we can use this.
|
1 2 3 4 5 6 7 8 |
df = df.withColumnRenamed("DEST_COUNTRY_NAME","DESTINATION") df.printSchema() root |-- DESTINATION: string (nullable = true) |-- ORIGIN_COUNTRY_NAME: string (nullable = true) |-- count: integer (nullable = true) |
We can also rename multiple columns at once by chaining withColumnRenamed calls.
|
1 2 3 4 5 6 7 8 9 |
df = df.withColumnRenamed("DESTINATION","dest")\ .withColumnRenamed("ORIGIN_COUNTRY_NAME","src") df.printSchema() root |-- dest: string (nullable = true) |-- src: string (nullable = true) |-- count: integer (nullable = true) |
Using withColumn
Though “withColumn” function is used to add new columns to Spark data frame, we can also use it to rename columns as well. We need to pass expression to select old column value when using “withColumn”. Let us check how we can do it.
|
1 2 3 4 5 6 7 8 9 10 11 |
# Coming back to initial stage df = spark.read\ .option("inferSchema", "true")\ .option("header","true")\ .csv("D:\\code\\spark\\spark-basics\\data\\flight-data\\csv\\2010-summary.csv") df.printSchema() root |-- DEST_COUNTRY_NAME: string (nullable = true) |-- ORIGIN_COUNTRY_NAME: string (nullable = true) |-- count: integer (nullable = true) |
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 |
from pyspark.sql.functions import expr,col df = df.withColumn("dest",expr("DEST_COUNTRY_NAME")) df.printSchema() root |-- DEST_COUNTRY_NAME: string (nullable = true) |-- ORIGIN_COUNTRY_NAME: string (nullable = true) |-- count: integer (nullable = true) |-- dest: string (nullable = true) df = df.withColumn("src",col("ORIGIN_COUNTRY_NAME")) df.printSchema() root |-- DEST_COUNTRY_NAME: string (nullable = true) |-- ORIGIN_COUNTRY_NAME: string (nullable = true) |-- count: integer (nullable = true) |-- dest: string (nullable = true) |-- src: string (nullable = true) |
We can either use “col” or “expr” function to pass whichever column we want to rename with this function. But there is one issue when using “withColumn”. Our exiting column is still there in the data frame. To handle this condition, we can drop that column while renaming it.
|
1 2 3 4 5 6 7 8 9 10 11 |
df = df\ .withColumn("dest",expr("DEST_COUNTRY_NAME"))\ .withColumn("src",col("ORIGIN_COUNTRY_NAME"))\ .drop("DEST_COUNTRY_NAME") \ .drop("ORIGIN_COUNTRY_NAME") df.printSchema() root |-- count: integer (nullable = true) |-- dest: string (nullable = true) |-- src: string (nullable = true) |
Now we are getting the desired result. We have deleted old columns in the data frame while renaming them.
Using Select Expression to Rename Columns
Spark data frames act much like SQL statements in most cases. Like SQL, we can also rename columns using “SELECT” or “SELECTEXPR” functions in Spark. This is really simple to understand if you are familiar with SQL queries.
|
1 2 3 4 5 6 7 |
df = df.selectExpr("dest as dest_country", "src as src_country", "count") df.printSchema() root |-- dest_country: string (nullable = true) |-- src_country: string (nullable = true) |-- count: integer (nullable = true) |
We can also use select statement to rename columns.
|
1 2 3 4 5 6 7 8 9 |
from pyspark.sql.functions import expr df = df.select(expr("dest_country as DEST_COUNTRY_NAME"), expr("src_country as ORIGIN_COUNTRY_NAME"), "count") df.printSchema() root |-- DEST_COUNTRY_NAME: string (nullable = true) |-- ORIGIN_COUNTRY_NAME: string (nullable = true) |-- count: integer (nullable = true) |
Conclusion
Renaming columns is one of frequent things we need to do while processing data with Spark. It is a good thing that we have so many easy options to achieve this. You can find code written in this blog at GitHub. I hope you will find this useful. Keep Learning.
