Set, Get and Delete AWS S3 bucket policies

Updated On - August 13, 2020  |  By Mahesh Mogal

In the last blog post, we have learned how to create S3 buckets. By default, all S3 buckets are private and there is no policy attached to them. S3 policies can define which user can perform which kind of actions on this bucket. If you want to know how S3 policies are different from IAM policies you can read this post.

In this tutorial, let us learn how we can manage S3 bucket policies. We will learn how to check existing bucket polices, attach new ones and delete policies from S3 console as well as pragmatically using AWS CLI and Python.

Using S3 Console

Reading Existing Bucket Polices

First, we will understand, how to check existing bucket policies from the S3 console. As mentioned before all S3 buckets have no policy attached by default. You can validate that, when you select any bucket then click on permissions -> and then bucket policy.

Attaching Bucket Policy

We can generate AWS policy using a simple tool provided by AWS. Before we attach policy, let us try to access S3 bucket using "testuser". This user currently does not have any access to S3. So when we try to list files in the S3 bucket we will see the following output.

listing S3 bucket without permission
Listing S3 bucket without permission

We can generate the following policy using the AWS Policy Generator. This policy grants all access to S3 bucket "testbucket-frompython-1" to IAM user "testuser".

{
  "Id": "Policy1586690842642",
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "Stmt1586690839614",
      "Action": "s3:*",
      "Effect": "Allow",
      "Resource": "arn:aws:s3:::testbucket-frompython-1",
      "Principal": {
        "AWS": [
          "arn:aws:iam::195556345987:user/testuser"
        ]
      }
    }
  ]
}

Now if we have attached this policy correctly and try to list S3 files from "testbucket-frompython-1" bucket, we should see some output now.

Listing S3 Bucket after attaching policy to S3
Listing S3 Bucket after attaching policy to S3

Deleting bucket policy

Deleting Bucket's policy is easy. You can again open the S3 bucket, go to the permissions tab and then to Bucket Policy and click on the Delete button. This will delete all polices attached to this bucket.

Managing Bucket Polices With AWS CLI

Listing Bucket Polices

You can easily list bucket policies using the following AWS CLI command. If you have policies attached to the bucket you will see output otherwise you will get "The bucket policy does not exist" message as shown in the below image. ( We do not have any policy attached to this bucket as we have deleted all attached policies in the last step.)

aws s3api get-bucket-policy --bucket testbucket-frompython-1 --profile admin-analyticshut
Get S3 bucket polices using AWS CLI
Get S3 bucket polices using AWS CLI

Attaching Policy to S3 Bucket

For attaching policy to S3 bucket using CLI, we need to create JSON document with policy that we want to attach. We will be using same policy attached mentioend above, which will grant "testuser" all access to S3 bucket. Then we can run following command to attach policy to bucket. (please use proper file path when using command below.)

aws s3api put-bucket-policy \
  --profile admin-analyticshut \
  --bucket testbucket-frompython-1 \
  --policy file://D:\\Projects\\python\\Working-with-AWS-S3\\S3-Buckets\\AWS-CLI\\bucket_policy.json
Attach S3 Bucket Policy using AWS CLI
Attach S3 Bucket Policy using AWS CLI

Deleting S3 Bucket Policy

We can also delete S3 bucket policy using simple CLI command.

 aws s3api delete-bucket-policy \
 --profile admin-analyticshut \
 --bucket testbucket-frompython-1
Delete S3 Bucket Policies using AWS CLI
Delete S3 Bucket Policies using AWS CLI

Managing Bucket Policies With Python

At last we will write python scripts to get, put and delete S3 bucket policies. let us get started.

Get Bucket Policies

We can get S3 bucket policies using following code. If there are no policies attached to the bucket, then get_bucket_policy() function throws error. We have to manage that in code.

import boto3
import pprint
from botocore.exceptions import ClientError

#
# setting up configured profile on your machine.
# You can ignore this step if you want use default AWS CLI profile.
#
boto3.setup_default_session(profile_name='admin-analyticshut')


s3 = boto3.client('s3')

try:
    response = s3.get_bucket_policy(Bucket='testbucket-frompython-1')
    print(pprint.pprint(response))
except ClientError as e:
    # if you do not have any policy attached to bucket it will throw error
    # An error occurred (NoSuchBucketPolicy) when calling the GetBucketPolicyStatus operation:
    # The bucket policy does not exist
    print(e)

As usual, We can use S3 resource as well to list bucket policies.

import boto3
import pprint
from botocore.exceptions import ClientError

#
# setting up configured profile on your machine.
# You can ignore this step if you want use default AWS CLI profile.
#
boto3.setup_default_session(profile_name='admin-analyticshut')
# using s3 resource
s3_resource = boto3.resource('s3')
try:
    bucket_policy = s3_resource.BucketPolicy('testbucket-frompython-1')
    # bucket policy resource has policy attribute which returns policy as JSON string
    print(bucket_policy.policy)
except ClientError as e:
    print(e)

Put S3 Bucket Policy

We will be using same above policy. When using Python we do not need to store policy in separate document.

import boto3
import pprint
from botocore.exceptions import ClientError
import json

#
# setting up configured profile on your machine.
# You can ignore this step if you want use default AWS CLI profile.
#
boto3.setup_default_session(profile_name='admin-analyticshut')

s3 = boto3.client('s3')
policy = """{
"Id": "Policy1586690842642",
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "Stmt1586690839614",
      "Action": "s3:*",
      "Effect": "Allow",
      "Resource": "arn:aws:s3:::testbucket-frompython-1",
      "Principal": {
        "AWS": [
          "arn:aws:iam::195556345987:user/testuser"
        ]
      }
    }
  ]
}"""
try:
    response = s3.put_bucket_policy(Bucket='testbucket-frompython-1', Policy=policy)
    print(pprint.pprint(response))
except ClientError as e:
    # if you do not have any policy attached to bucket it will throw error
    # An error occurred (NoSuchBucketPolicy) when calling the GetBucketPolicyStatus operation:
    # The bucket policy does not exist
    print(e)

We can achieve same effect using bucket resource as well.

s3_resource = boto3.resource('s3')
bucket_policy = s3_resource.BucketPolicy('testbucket-frompython-1')
bucket_policy.put(Policy=policy)

Deleting S3 Bucket Policy

Following python code snippet can be used to delete attached bucket policy.

import boto3
import pprint
from botocore.exceptions import ClientError

#
# setting up configured profile on your machine.
# You can ignore this step if you want use default AWS CLI profile.
#
boto3.setup_default_session(profile_name='admin-analyticshut')

s3 = boto3.client('s3')

# return None
response = s3.delete_bucket_policy(Bucket='testbucket-frompython-1')

# if bucket does not have any policy attached, it will not throw any error

# using s3 resource
s3_resource = boto3.resource('s3')
bucket_policy = s3_resource.BucketPolicy('testbucket-frompython-1')

# returns None
bucket_policy.delete()

Conclusion

I hope this article helped you in understanding different ways in which you can manage S3 bucket policies. You can try performing operations at each step to validate if the policy is attached or deleted correctly. You can get code created in this blog from this git repo. If you have any questions please let me know. See you in the next blog.

Manage S3 Bucket Polices
Mahesh Mogal
I am passionate about Cloud, Data Analytics, Machine Learning, and Artificial Intelligence. I like to learn and try out new things. I have started blogging about my experience while learning these exciting technologies.

Stay Updated with Latest Blogs

Get latest blogs delivered to your mail directly.

Recent Posts

Partitioning in Hive

Using Partitioning, We can increase hive query performance. But if we do not choose partitioning column correctly it can create small file issue.

Partitioning in Hive
Read More
Hive Data Manipulation - Loading Data to Hive Tables

We will learn how to load and populate data to hive table. We will also learn how to copy data to hive tables from local system.

Loading Data to Hive Tables
Read More
Create, Alter, Delete Tables in Hive

We will learn how to create Hive tables, also altering table columns, adding comments and table properties and deleting Hive tables.

manage tables in hive -2
Read More

Leave a Reply

Your email address will not be published. Required fields are marked *

2 comments on “Set, Get and Delete AWS S3 bucket policies”

  1. is it possible to list bucket from console once you attach the bucket policy to the bucket? lets go with your example above. i know it works from CLI perspective but what about console. can i list this bucket with attached policy in console as the allowed user?
    food for thought....

    1. Hello Santosh,
      Thanks for taking an interest. I am not sure I understand you correctly but if you are saying will you be able to list all buckets on the console to which you have access to then yes you can.

      If you have any more questions do reach out to me.

      Thanks
      Mahesh

linkedin facebook pinterest youtube rss twitter instagram facebook-blank rss-blank linkedin-blank pinterest youtube twitter instagram
Share via
Copy link