> For the complete documentation index, see [llms.txt](https://docs.middle.app/middle-docs/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.middle.app/middle-docs/guides/app-reference/amazon-s3.md). # Amazon S3 Follow the below steps to set up a bucket, IAM policy, AWS user, and a Kinesis Data Firehose role (if you plan to use Firehose): {% stepper %} {% step %} ### Create an S3 bucket Please follow AWS's instructions on how to create an S3 bucket: {% embed url="" %} {% hint style="warning" %} We suggest configuring your bucket to expire objects. When setting the expiration period, consider how long you want to access any stored data before it expires. {% endhint %} {% endstep %} {% step %} ### Create an IAM policy You'll need to create an IAM policy to later attach to a user and Kinesis Firehose role (if applicable). 1. Go to the IAM dashboard in your AWS account 2. Go to **Policies** in the left-hand menu and click to create a new policy Depending on how you plan to use S3, you'll configure your policy differently. Refer to the JSON permission sets below. {% hint style="warning" %} Wherever you see {s3 bucket name}, {region}, or {account id}, please enter your bucket name, region, or account ID and remove the brackets. {% endhint %} ### To post and read data {% hint style="info" %} If you are using Firehose, see below {% endhint %} ```json { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "s3:AbortMultipartUpload", "s3:GetBucketLocation", "s3:GetObject", "s3:ListBucket", "s3:ListBucketMultipartUploads", "s3:PutObject" ], "Resource": [ "arn:aws:s3:::{s3 bucket name}", "arn:aws:s3:::{s3 bucket name}/*" ] }, { "Effect": "Allow", "Action": [ "iam:GetRole", "iam:PassRole" ], "Resource": [ "arn:aws:iam::{account id}:role/FirehoseToS3Role" ] } ] } ``` #### If using Firehose ```json { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "s3:AbortMultipartUpload", "s3:GetBucketLocation", "s3:GetObject", "s3:ListBucket", "s3:ListBucketMultipartUploads", "s3:PutObject" ], "Resource": [ "arn:aws:s3:::{s3 bucket name}", "arn:aws:s3:::{s3 bucket name}/*" ] }, { "Effect": "Allow", "Action": [ "iam:GetRole", "iam:PassRole" ], "Resource": [ "arn:aws:iam::{account id}:role/FirehoseToS3Role" ] }, { "Effect": "Allow", "Action": [ "firehose:PutRecord", "firehose:CreateDeliveryStream", "firehose:PutRecordBatch" ], "Resource": "arn:aws:firehose:{region}:{account id}:deliverystream/*" } ] } ``` ### To read data To only read data in S3 you'll need to set up a policy that has S3 as a service and the following actions: **GetObject** and **ListBucket**. You will then want to set up the source bucket as a specific resource. Most likely, you'll want all objects to be accessible. ```json { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "s3:GetObject", "s3:ListBucket" ], "Resource": [ "arn:aws:s3:::{s3 bucket name}", "arn:aws:s3:::{s3 bucket name}/*" ] } ] } ``` #### Review the policy Adding tags is optional and dependent on how you manage your AWS installation. Go ahead and name the policy, add a description, and make sure the summary matches what we intended on setting up. {% endstep %} {% step %} ### Create a role for Firehose (optional) {% hint style="success" %} Skip this section if you are not streaming or loading data into Amazon S3 via Firehose {% endhint %} If you plan to stream data into Amazon S3 using Firehose, you will need to create a "FirehoseToS3Role" to connect to the IAM policy you created. 1. The **Roles** tab should be found in the left-hand menu of your IAM dashboard 2. Click to create a new one. 3. You will want to select **AWS service** for the trusted entity type 4. From the **User cases for other AWS services** dropdown, select **Kinesis** and then **Kinesis Firehose** 5. Click **Next** 6. On the next page, you'll want to connect the IAM policy you made earlier to this new role 7. Click **Next** 8. You should then be able to name (name it "FirehoseToS3Role") and review the role 9. Click, **Create role**

Select Kinesis and then Kinesis Firehose for the AWS service you'd like to create the policy for

{% endstep %} {% step %} ### Create a user You will need to create a user to attach to your newly created IAM policy. This is the user you'll authenticate with when adding your app connection. 1. Click **Users** in the IAM side menu 2. Click **Add users** 3. Name the user 4. You will likely not want to provide this user AWS Management Console access: I.E. giving the user a login. This user will be used for API access. 5. Click **Next** 6. Click to **Attach existing policies directly** 7. Search for the policy you created at the beginning of this process and select this policy 8. Click **Next** 9. Review and add tags if you wish (optional) 10. Click **Create user** Once your user is created you will need the user's access key ID and secret access key. These are going to be used in the Amazon S3 [app authentication](/middle-docs/product/accounts/connect-or-edit-an-app-connection.md) or the [authentication](/middle-docs/product/apps/authentication.md) for your app that pulls data from S3. {% endstep %} {% step %} ### Get the user's secret access key 1. Click on the newly created user 2. Go to the **Security credentials** tab 3. Scroll down to **Access keys** 4. Click **Create access key** 5. Select the **Other** option 6. Create a tag if you wish (optional) 7. Click **Create access key** 8. Save your access key ID and secret access key in a secure location for future reference. {% hint style="danger" %} After initially creating an access key you won't be able to view its secret again. If you misplace or forget the secret access key you'll need to create a new key pair. {% endhint %} {% endstep %} {% endstepper %} --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation by asking a question. Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters: ``` GET https://docs.middle.app/middle-docs/guides/app-reference/amazon-s3.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.