Amazon S3

Follow the below steps to set up a bucket, IAM policy, AWS user, and a Kinesis Data Firehose role (if you plan to use Firehose):

1

Create an S3 bucket

Please follow AWS's instructions on how to create an S3 bucket:

Creating a general purpose bucket - Amazon Simple Storage ServiceAmazon Simple Storage Service

We suggest configuring your bucket to expire objects. When setting the expiration period, consider how long you want to access any stored data before it expires.

2

Create an IAM policy

You'll need to create an IAM policy to later attach to a user and Kinesis Firehose role (if applicable).

1. Go to the IAM dashboard in your AWS account

2. Go to Policies in the left-hand menu and click to create a new policy

Depending on how you plan to use S3, you'll configure your policy differently. Refer to the JSON permission sets below.

Wherever you see {s3 bucket name}, {region}, or {account id}, please enter your bucket name, region, or account ID and remove the brackets.

To post and read data

If you are using Firehose, see below

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:AbortMultipartUpload",
        "s3:GetBucketLocation",
        "s3:GetObject",
        "s3:ListBucket",
        "s3:ListBucketMultipartUploads",
        "s3:PutObject"
      ],
      "Resource": [
        "arn:aws:s3:::{s3 bucket name}",
        "arn:aws:s3:::{s3 bucket name}/*"
      ]
    },
    {
      "Effect": "Allow",
      "Action": [
        "iam:GetRole",
        "iam:PassRole"
      ],
      "Resource": [
        "arn:aws:iam::{account id}:role/FirehoseToS3Role"
      ]
    }
  ]
}

If using Firehose

{
   "Version": "2012-10-17",
   "Statement": [
       {
           "Effect": "Allow",
           "Action": [
               "s3:AbortMultipartUpload",
               "s3:GetBucketLocation",
               "s3:GetObject",
               "s3:ListBucket",
               "s3:ListBucketMultipartUploads",
               "s3:PutObject"
           ],
           "Resource": [
               "arn:aws:s3:::{s3 bucket name}",
		  "arn:aws:s3:::{s3 bucket name}/*"
           ]
       },
       {
           "Effect": "Allow",
           "Action": [
               "iam:GetRole",
               "iam:PassRole"
           ],
           "Resource": [
               "arn:aws:iam::{account id}:role/FirehoseToS3Role"
           ]
       },
       {
           "Effect": "Allow",
           "Action": [
               "firehose:PutRecord",
               "firehose:CreateDeliveryStream",
               "firehose:PutRecordBatch"
           ],
           "Resource": "arn:aws:firehose:{region}:{account id}:deliverystream/*"
       }
   ]
}

To read data

To only read data in S3 you'll need to set up a policy that has S3 as a service and the following actions: GetObject and ListBucket.

You will then want to set up the source bucket as a specific resource. Most likely, you'll want all objects to be accessible.

{
   "Version": "2012-10-17",
   "Statement": [
       {
           "Effect": "Allow",
           "Action": [
               "s3:GetObject",
               "s3:ListBucket"
           ],
           "Resource": [
               "arn:aws:s3:::{s3 bucket name}",
		  "arn:aws:s3:::{s3 bucket name}/*"
           ]
       }
    ]
}

Review the policy

Adding tags is optional and dependent on how you manage your AWS installation. Go ahead and name the policy, add a description, and make sure the summary matches what we intended on setting up.

3

Create a role for Firehose (optional)

Skip this section if you are not streaming or loading data into Amazon S3 via Firehose

If you plan to stream data into Amazon S3 using Firehose, you will need to create a "FirehoseToS3Role" to connect to the IAM policy you created.

1. The Roles tab should be found in the left-hand menu of your IAM dashboard

2. Click to create a new one.

3. You will want to select AWS service for the trusted entity type

4. From the User cases for other AWS services dropdown, select Kinesis and then Kinesis Firehose

5. Click Next

6. On the next page, you'll want to connect the IAM policy you made earlier to this new role

7. Click Next

8. You should then be able to name (name it "FirehoseToS3Role") and review the role

9. Click, Create role

Select Kinesis and then Kinesis Firehose for the AWS service you'd like to create the policy for

4

Create a user

You will need to create a user to attach to your newly created IAM policy. This is the user you'll authenticate with when adding your app connection.

1. Click Users in the IAM side menu

2. Click Add users

3. Name the user

4. You will likely not want to provide this user AWS Management Console access: I.E. giving the user a login. This user will be used for API access.

5. Click Next

6. Click to Attach existing policies directly

7. Search for the policy you created at the beginning of this process and select this policy

8. Click Next

9. Review and add tags if you wish (optional)

10. Click Create user

Once your user is created you will need the user's access key ID and secret access key. These are going to be used in the Amazon S3 app authentication or the authentication for your app that pulls data from S3.

5

Get the user's secret access key

1. Click on the newly created user

2. Go to the Security credentials tab

3. Scroll down to Access keys

4. Click Create access key

5. Select the Other option

6. Create a tag if you wish (optional)

7. Click Create access key

8. Save your access key ID and secret access in a secure location for future reference.

After initially creating your access keys you won't be able to access them again. If you misplace or forget your access keys you'll need to delete your old ones and create new ones.