Different ways to configure credentials with boto3

Updated On April 12, 2020 | By Mahesh Mogal

Boto3 is python's library to interact with AWS services. When we want to use AWS services we need to provide security credentials of our user to boto3. Like most things in life, we can configure or use user credentials with boto3 in multiple ways. Some are worst and never to be used and others are recommended ways. In this blog, let us take a look at how to configure credentials with boto 3.

Using as method parameters

This is the easiest way to use user credentials with boto3. And in my opinion, this is the worst way to configure boto3. Here we can simply pass our access key id and secret access to boto3 as a parameter while creating service client or resource.

import boto3

# Hard coded strings as credentials, not recommended.
client = boto3.client(
    's3',
    aws_access_key_id=ACCESS_KEY,
    aws_secret_access_key=SECRET_KEY,
    aws_session_token=SESSION_TOKEN,
)

With this approach user, keys are visible to everyone. If you commit such code to GitHub, anyone who has access to your repository can use these user credentials and have access to your AWS account. I do not need to tell you what can happen next. That is why I will recommend not use this way of setting AWS credentials with boto3.

Use a common configuration file

One simple way to abstract access key and secret access key is the starting session in another file.

# via the Session
session = boto3.Session(
    aws_access_key_id=ACCESS_KEY,
    aws_secret_access_key=SECRET_KEY,
    aws_session_token=SESSION_TOKEN,
)

Then you can import this session file in another python file and use it to start AWS sessions to connect with services.

import aws_session

sqs = aws_session.client('sqs')

Though this method prevents direct visible access to AWS credentials, there is still an issue when sharing your code to someone or adding it to GitHub. You have to make sure not to commit that file to GitHub.

Environment variables

Boto3 automatically checks for environment variables. If it finds these variables it will use them for connecting to AWS.

  • AWS_ACCESS_KEY_ID - The access key for your AWS account.
  • AWS_SECRET_ACCESS_KEY - The secret key for your AWS account.

Once you set these environment variables, you can directly create boto3 client or session for service. In the backend, boto3 will use these keys to communicate with AWS

import boto3

# uses credentials from environment
s3 = boto3.client('s3')

With this approach, you can be sure that your access key is only used on your machine It is not easily visible to someone watching. You can also share your code on GitHub or to some person without any worries about exposing your user credentials.

One drawback of this method is when you have multiple AWS users (from different AWS accounts or for different AWS roles) then switching between them becomes difficult. You have to change environment variables each time when you want to use different users. And what happens when you want to use multiple users' credentials (like copying files from s3 bucket in one account to s3 bucket in another account) in one single session?

Using AWS CLI profile

This is similar to setting up Environment variables on your machine. In this case, Boto3 uses credentials that you have used when setting up a default profile while configuring AWS CLI. You can learn more about how to configure AWS CLI here.

Once you have configured AWS CLI, you can directly use boto3 to create a service client or resource.

import boto3

# uses credentials from default profile of AWS CLI
s3 = boto3.client('s3')

But this approach has the same drawback, what if when we have multiple user profiles? Can we tell boto3 which profile to use when connecting to AWS? Well, of course, we can.

Using multiple AWS CLI profiles

There is a simple way to state profile name while initiating client in Boto3.

import boto3
#
# setting up configured profile on your machine.
# You can ignore this step if you want use default AWS CLI profile.
#
boto3.setup_default_session(profile_name='admin-analyticshut')


s3 = boto3.client('s3')
# This will use user keys set up for admin-analyticshut profile.

Conclusion

We have seen different ways to configure credentials with Boto3. Configuring AWS CLI profiles and using different profiles depending on our need is way to go!!! Hope this helps. If you have any questions please let me know.

.

Mahesh Mogal

I am passionate about Cloud, Data Analytics, Machine Learning, and Artificial Intelligence. I like to learn and try out new things. I have started blogging about my experience while learning these exciting technologies.

Stay Updated with Latest Blogs

Get latest blogs delivered to your mail directly.

Recent Posts

Sorting in Spark Dataframe

In this blog, we will learn how to sort rows in spark dataframe based on some column values.

Read More
Removing White Spaces From Data in Spark

White spaces can be a headache if not removed before processing data. We will learn how to remove spaces from data in spark using inbuilt functions.

Read More
Padding Data in Spark Dataframe

In this blog, we will learn how to use rpad and lpad functions to add padding to data in spark dataframe.

Read More

Leave a Reply

Your email address will not be published. Required fields are marked *

linkedin facebook pinterest youtube rss twitter instagram facebook-blank rss-blank linkedin-blank pinterest youtube twitter instagram
Share via
Copy link
Powered by Social Snap