# Amazon S3 Follow the below steps to set up a bucket, IAM policy, AWS user, and a Kinesis Data Firehose role (if you plan to use Firehose): {% stepper %} {% step %} ### Create an S3 bucket Please follow AWS's instructions on how to create an S3 bucket: {% embed url="" %} {% hint style="warning" %} We suggest configuring your bucket to expire objects. When setting the expiration period, consider how long you want to access any stored data before it expires. {% endhint %} {% endstep %} {% step %} ### Create an IAM policy You'll need to create an IAM policy to later attach to a user and Kinesis Firehose role (if applicable). 1. Go to the IAM dashboard in your AWS account 2. Go to **Policies** in the left-hand menu and click to create a new policy Depending on how you plan to use S3, you'll configure your policy differently. Refer to the JSON permission sets below. {% hint style="warning" %} Wherever you see {s3 bucket name}, {region}, or {account id}, please enter your bucket name, region, or account ID and remove the brackets. {% endhint %} ### To post and read data {% hint style="info" %} If you are using Firehose, see below {% endhint %} ```json { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "s3:AbortMultipartUpload", "s3:GetBucketLocation", "s3:GetObject", "s3:ListBucket", "s3:ListBucketMultipartUploads", "s3:PutObject" ], "Resource": [ "arn:aws:s3:::{s3 bucket name}", "arn:aws:s3:::{s3 bucket name}/*" ] }, { "Effect": "Allow", "Action": [ "iam:GetRole", "iam:PassRole" ], "Resource": [ "arn:aws:iam::{account id}:role/FirehoseToS3Role" ] } ] } ``` #### If using Firehose ```json { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "s3:AbortMultipartUpload", "s3:GetBucketLocation", "s3:GetObject", "s3:ListBucket", "s3:ListBucketMultipartUploads", "s3:PutObject" ], "Resource": [ "arn:aws:s3:::{s3 bucket name}", "arn:aws:s3:::{s3 bucket name}/*" ] }, { "Effect": "Allow", "Action": [ "iam:GetRole", "iam:PassRole" ], "Resource": [ "arn:aws:iam::{account id}:role/FirehoseToS3Role" ] }, { "Effect": "Allow", "Action": [ "firehose:PutRecord", "firehose:CreateDeliveryStream", "firehose:PutRecordBatch" ], "Resource": "arn:aws:firehose:{region}:{account id}:deliverystream/*" } ] } ``` ### To read data To only read data in S3 you'll need to set up a policy that has S3 as a service and the following actions: **GetObject** and **ListBucket**. You will then want to set up the source bucket as a specific resource. Most likely, you'll want all objects to be accessible. ```json { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "s3:GetObject", "s3:ListBucket" ], "Resource": [ "arn:aws:s3:::{s3 bucket name}", "arn:aws:s3:::{s3 bucket name}/*" ] } ] } ``` #### Review the policy Adding tags is optional and dependent on how you manage your AWS installation. Go ahead and name the policy, add a description, and make sure the summary matches what we intended on setting up. {% endstep %} {% step %} ### Create a role for Firehose (optional) {% hint style="success" %} Skip this section if you are not streaming or loading data into Amazon S3 via Firehose {% endhint %} If you plan to stream data into Amazon S3 using Firehose, you will need to create a "FirehoseToS3Role" to connect to the IAM policy you created. 1. The **Roles** tab should be found in the left-hand menu of your IAM dashboard 2. Click to create a new one. 3. You will want to select **AWS service** for the trusted entity type 4. From the **User cases for other AWS services** dropdown, select **Kinesis** and then **Kinesis Firehose** 5. Click **Next** 6. On the next page, you'll want to connect the IAM policy you made earlier to this new role 7. Click **Next** 8. You should then be able to name (name it "FirehoseToS3Role") and review the role 9. Click, **Create role**

Select Kinesis and then Kinesis Firehose for the AWS service you'd like to create the policy for

{% endstep %} {% step %} ### Create a user You will need to create a user to attach to your newly created IAM policy. This is the user you'll authenticate with when adding your app connection. 1. Click **Users** in the IAM side menu 2. Click **Add users** 3. Name the user 4. You will likely not want to provide this user AWS Management Console access: I.E. giving the user a login. This user will be used for API access. 5. Click **Next** 6. Click to **Attach existing policies directly** 7. Search for the policy you created at the beginning of this process and select this policy 8. Click **Next** 9. Review and add tags if you wish (optional) 10. Click **Create user** Once your user is created you will need the user's access key ID and secret access key. These are going to be used in the Amazon S3 [app authentication](/middle-docs/building-workflows/connect-or-edit-an-app-connection.md) or the [authentication](/middle-docs/app-development/authentication.md) for your app that pulls data from S3. {% endstep %} {% step %} ### Get the user's secret access key 1. Click on the newly created user 2. Go to the **Security credentials** tab 3. Scroll down to **Access keys** 4. Click **Create access key** 5. Select the **Other** option 6. Create a tag if you wish (optional) 7. Click **Create access key** 8. Save your access key ID and secret access in a secure location for future reference. {% hint style="danger" %} After initially creating your access keys you won't be able to access them again. If you misplace or forget your access keys you'll need to delete your old ones and create new ones. {% endhint %} {% endstep %} {% endstepper %} --- # Agent Instructions: Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter: ``` GET https://docs.middle.app/middle-docs/guides/app-reference/amazon-s3.md?ask= ``` The question should be specific, self-contained, and written in natural language. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.