Amazon S3
Learn what's needed to connect Amazon S3 with Middle
Last updated
Was this helpful?
Learn what's needed to connect Amazon S3 with Middle
Last updated
Was this helpful?
Middle's integration with allows you to stream data into sub-files via and then concatenate those sub-files into a single file in your S3 bucket. Using Middle's , you can also build apps that pull data objects from S3 into Middle and store that data relationally record-by-record for use by . Reading data from S3 enables Middle to sync data via file transfer protocols.
Whether you're using Middle's Amazon S3 app or syncing data from an S3 bucket, some IAM (Identity and Access Management) configuration will be necessary. Follow the below steps to set up a bucket, IAM policy, AWS user, and if you are using Kinesis Data Firehose, a role:
An S3 bucket is a place for objects to live. As it relates to Middle, an S3 bucket is a place where data, often in the form CSVs, can be loaded or extracted.
Please refer to .
We suggest configuring your bucket to expire objects after a certain number of days. This helps eliminate unnecessary clutter.
After you have an S3 bucket created, you'll need to create an IAM policy to later attach to a user and Kinesis Firehose role (if applicable).
Go to the IAM dashboard in your AWS account. This can be done by searching for IAM in the AWS search bar.
Go to Policies in the left-hand menu and click to create a new policy
Depending on how you plan to use S3, you'll configure your policy differently. Refer to one of the two sections below. Scroll down to see the JSON version of your configuration.
To set up a policy that will work with Middle's Amazon S3 app, the services should be S3, Firehose, and IAM. One permission per service will need to be set up if you are using the visual editor setup. We suggest using the JSON editor:
Wherever you see {s3 bucket name}, {region}, or {account id}, please enter your bucket name, region, or account ID and remove the brackets.
AbortMultipartUpload
GetBucketLocation
GetObject
ListBucket
ListBucketMultipartUploads
PutObject
GetRole
PassRole
PutRecord
Create
DeliveryStream
PutRecordBatch
For pulling data into Middle via S3, you'll need to set up a policy that has S3 as a service and the following actions: GetObject and ListBucket.
You will then want to set up the source bucket as a specific resource. Most likely, you'll want all objects to be accessible.
Wherever you see {s3 bucket name} please enter your bucket name, region, or account ID and remove the brackets.
Adding tags is optional and dependent on how you manage your AWS installation. Go ahead and name the policy, add a description, and make sure the summary matches what we intended on setting up.
If you plan to stream data into Amazon S3 using Firehose, you will need to create a "FirehoseToS3Role" to connect to the IAM policy you created.
The Roles tab should be found in the left-hand menu of your IAM dashboard
Click to create a new one.
You will want to select AWS service for the trusted entity type
From the User cases for other AWS services dropdown, select Kinesis and then Kinesis Firehose
Click Next
On the next page, you'll want to connect the IAM policy you made earlier to this new role
Click Next
You should then be able to name (name it "FirehoseToS3Role") and review the role
Click, Create role
You will need to create a user to attach to your newly created IAM policy. This is the user you'll authenticate through when adding your app connection.
Click Users in the IAM side menu
Click Add users
Name the user
You will likely not want to provide this user AWS Management Console access: I.E. giving the user a login. This user will be used for API access.
Click Next
Click to Attach existing policies directly
Search for the policy you created at the beginning of this process and select this policy
Click Next
Review and add tags if you wish (optional)
Click Create user
Click on the newly created user
Go to the Security credentials tab
Scroll down to Access keys
Click Create access key
Select the Other option
Create a tag if you wish (optional)
Click Create access key
Save your access key ID and secret access in a secure location for future reference.
After initially creating your access keys you won't be able to access them again. If you misplace or forget your access keys you'll need to delete your old ones and create new ones.
Once your user is created you will need the user's access key ID and secret access key. These are going to be used in the Amazon S3 or the for your app that pulls data from S3.
