Amazon S3

Learn what's needed to connect Amazon S3 with Middle

Middle's integration with Amazon S3 allows you to stream data into sub-files via Amazon Kinesis Data Firehose and then concatenate those sub-files into a single file in your S3 bucket. Using Middle's developer portal, you can also build apps that pull data objects from S3 into Middle and store that data relationally record-by-record for use by Middle's workflows. Reading data from S3 enables Middle to sync data via file transfer protocols.

To set up Amazon S3 to be used by Middle, you will need an AWS account. If you have not created an AWS account, please go ahead and do so and then return to these instructions.

Whether you're using Middle's Amazon S3 app or syncing data from an S3 bucket, some IAM (Identity and Access Management) configuration will be necessary. Follow the below steps to set up a bucket, IAM policy, AWS user, and if you are using Kinesis Data Firehose, a role:

• Create an S3 bucket (if you have not already)

• Create an IAM policy to attach to the user and the role

• Create a Kinesis Firehose role (if you intend on loading data into S3 using Firehose)

• Create a user

• Save the user's credentials in a secure place (these cannot be accessed after creation)

S3 stands for simple storage solution

Create an S3 bucket

An S3 bucket is a place for objects to live. As it relates to Middle, an S3 bucket is a place where data, often in the form CSVs, can be loaded or extracted.

Please refer to AWS's instructions on how to create an S3 bucket.

We suggest configuring your bucket to expire objects after a certain number of days. This helps eliminate unnecessary clutter.

Create an IAM policy

After you have an S3 bucket created, you'll need to create an IAM policy to later attach to a user and Kinesis Firehose role (if applicable).

1. Go to the IAM dashboard in your AWS account. This can be done by searching for IAM in the AWS search bar.

2. Go to Policies in the left-hand menu and click to create a new policy

Depending on how you plan to use S3, you'll configure your policy differently. Refer to one of the two sections below. Scroll down to see the JSON version of your configuration.

To send and sync data

To set up a policy that will work with Middle's Amazon S3 app, the services should be S3, Firehose, and IAM. One permission per service will need to be set up if you are using the visual editor setup. We suggest using the JSON editor:

Wherever you see {s3 bucket name}, {region}, or {account id}, please enter your bucket name, region, or account ID and remove the brackets.

{
   "Version": "2012-10-17",
   "Statement": [
       {
           "Effect": "Allow",
           "Action": [
               "s3:AbortMultipartUpload",
               "s3:GetBucketLocation",
               "s3:GetObject",
               "s3:ListBucket",
               "s3:ListBucketMultipartUploads",
               "s3:PutObject"
           ],
           "Resource": [
               "arn:aws:s3:::{s3 bucket name}",
		  "arn:aws:s3:::{s3 bucket name}/*"
           ]
       },
       {
           "Effect": "Allow",
           "Action": [
               "iam:GetRole",
               "iam:PassRole"
           ],
           "Resource": [
               "arn:aws:iam::{account id}:role/FirehoseToS3Role"
           ]
       },
       {
           "Effect": "Allow",
           "Action": [
               "firehose:PutRecord",
               "firehose:CreateDeliveryStream",
               "firehose:PutRecordBatch"
           ],
           "Resource": "arn:aws:firehose:{region}:{account id}:deliverystream/*"
       }
   ]
}

S3 actions

• AbortMultipartUpload

• GetBucketLocation

• GetObject

• ListBucket

• ListBucketMultipartUploads

• PutObject

IAM actions

• GetRole

• PassRole

Firehose actions

• PutRecord

• Create

• DeliveryStream

• PutRecordBatch

Just to read data

For pulling data into Middle via S3, you'll need to set up a policy that has S3 as a service and the following actions: GetObject and ListBucket.

You will then want to set up the source bucket as a specific resource. Most likely, you'll want all objects to be accessible.

Wherever you see {s3 bucket name} please enter your bucket name, region, or account ID and remove the brackets.

{
   "Version": "2012-10-17",
   "Statement": [
       {
           "Effect": "Allow",
           "Action": [
               "s3:GetObject",
               "s3:ListBucket"
           ],
           "Resource": [
               "arn:aws:s3:::{s3 bucket name}",
		  "arn:aws:s3:::{s3 bucket name}/*"
           ]
       }
    ]
}

Review the policy

Adding tags is optional and dependent on how you manage your AWS installation. Go ahead and name the policy, add a description, and make sure the summary matches what we intended on setting up.

Create a role for Firehose

Skip this section if you are not streaming or loading data into Amazon S3

If you plan to stream data into Amazon S3 using Firehose, you will need to create a "FirehoseToS3Role" to connect to the IAM policy you created.

1. The Roles tab should be found in the left-hand menu of your IAM dashboard

2. Click to create a new one.

3. You will want to select AWS service for the trusted entity type

4. From the User cases for other AWS services dropdown, select Kinesis and then Kinesis Firehose

5. Click Next

6. On the next page, you'll want to connect the IAM policy you made earlier to this new role

7. Click Next

8. You should then be able to name (name it "FirehoseToS3Role") and review the role

9. Click, Create role

Select Kinesis and then Kinesis Firehose for the AWS service you'd like to create the policy for

Create a user

You will need to create a user to attach to your newly created IAM policy. This is the user you'll authenticate through when adding your app connection.

1. Click Users in the IAM side menu

2. Click Add users

3. Name the user

4. You will likely not want to provide this user AWS Management Console access: I.E. giving the user a login. This user will be used for API access.

5. Click Next

6. Click to Attach existing policies directly

7. Search for the policy you created at the beginning of this process and select this policy

8. Click Next

9. Review and add tags if you wish (optional)

10. Click Create user

Once your user is created you will need the user's access key ID and secret access key. These are going to be used in the Amazon S3 app authentication or the authentication for your app that pulls data from S3.

Get the user's secret access key

1. Click on the newly created user

2. Go to the Security credentials tab

3. Scroll down to Access keys

4. Click Create access key

5. Select the Other option

6. Create a tag if you wish (optional)

7. Click Create access key

8. Save your access key ID and secret access in a secure location for future reference.

After initially creating your access keys you won't be able to access them again. If you misplace or forget your access keys you'll need to delete your old ones and create new ones.