Diamond 2 will use AWS’s recommended Cross-Account IAM Role Assumption method for secure file uploads. This is combined with a dedicated S3 bucket for each broadcaster. This document sets out how broadcasters will gain access to their Diamond 2 XML upload bucket. Diamond 2 will run on the platform provided by The Everyone Project — abbreviated as TEP below.
When TEP sets up your access, you’ll receive the following details:
| Item | Description | Example |
|---|---|---|
| Role ARN | The IAM role you’ll assume | arn:aws:iam::060011828759:role/tep-xml-ingestion-{broadcaster}-upload |
| External ID | Security token for role assumption | tep-xml-upload-{broadcaster} |
| Bucket Name | Your dedicated S3 bucket | tep-xml-ingestion-staging-{broadcaster} |
| Upload Path | Where to place files | incoming/ |
| Region | AWS region | eu-west-2 (London) |
| SNS Topic ARN | For processing notifications (optional) | arn:aws:sns:eu-west-2:060011828759:tep-xml-ingestion-staging-{broadcaster}-notifications |
Note: The External ID is a required security feature to prevent the “confused deputy problem”. You must provide this External ID when assuming the role.
If your upload process runs on an AWS service (EC2, ECS, Lambda, etc.), you can attach an IAM role to that service which has permission to assume TEP’s role. This means your application can assume the role automatically without needing to manually export credentials — the AWS SDK handles it all for you.
For testing or one-off uploads, you can manually assume the role from the command line:
aws sts assume-role \
--role-arn arn:aws:iam::060011828759:role/tep-xml-ingestion-{broadcaster}-upload \
--role-session-name xml-upload-session \
--external-id tep-xml-upload-{broadcaster}
This returns a JSON response containing temporary credentials.
Parse the JSON response and export the credentials using jq:
# Capture the credentials in a variable
CREDENTIALS=$(aws sts assume-role \
--role-arn arn:aws:iam::060011828759:role/tep-xml-ingestion-{broadcaster}-upload \
--role-session-name xml-upload-session \
--external-id tep-xml-upload-{broadcaster})
# Extract and export the credentials
export AWS_ACCESS_KEY_ID=$(echo $CREDENTIALS | jq -r '.Credentials.AccessKeyId')
export AWS_SECRET_ACCESS_KEY=$(echo $CREDENTIALS | jq -r '.Credentials.SecretAccessKey')
export AWS_SESSION_TOKEN=$(echo $CREDENTIALS | jq -r '.Credentials.SessionToken')
aws s3 cp --sse AES256 your-file.xml s3://tep-xml-ingestion-staging-{broadcaster}/incoming/
You can now use any AWS S3 commands with these temporary credentials.
Cross-account IAM role assumption is the officially supported approach. Where it is not possible for a broadcaster, TEP will instead issue an AWS access key (an access key ID and secret access key) for the broadcaster’s bucket. The key is subject to the following conditions:
Each key is issued with permission to manage its own rotation, so you can rotate it whenever you choose without any input from TEP.
The simplest way to rotate a key is the key rotation utility at:
https://lite.creativediversitynetwork.com/key-rotation.html
It needs only the existing access key ID and secret access key, and walks you through the rotation step by step, making all of the necessary AWS API calls for you:
The utility runs entirely in your browser: nothing needs to be installed, and your keys are not sent to any system other than AWS.
You can equally build rotation into your own systems using the AWS IAM API (CreateAccessKey, then DeleteAccessKey once the new key is confirmed working).
With an access key there is no role to assume. Store the key in a named profile, then upload:
# Enter the access key ID, secret access key and eu-west-2 when prompted
aws configure --profile tep
# Upload a file (server-side encryption is required)
aws s3 cp --sse AES256 your-file.xml s3://tep-xml-ingestion-staging-{broadcaster}/incoming/ --profile tep
TEP will send notifications via SNS when your uploaded files have been processed and moved to either the complete/ or errors/ directory. You can subscribe to these notifications in several ways:
aws sns subscribe \
--topic-arn arn:aws:sns:eu-west-2:060011828759:tep-xml-ingestion-staging-{broadcaster}-notifications \
--protocol email \
--notification-endpoint your-email@example.com
AWS will send you a confirmation email with a link you must click. To unsubscribe later, click the unsubscribe link in any notification email.
aws sns subscribe \
--topic-arn arn:aws:sns:eu-west-2:060011828759:tep-xml-ingestion-staging-{broadcaster}-notifications \
--protocol email-json \
--notification-endpoint your-email@example.com
aws sns subscribe \
--topic-arn arn:aws:sns:eu-west-2:060011828759:tep-xml-ingestion-staging-{broadcaster}-notifications \
--protocol sqs \
--notification-endpoint arn:aws:sqs:eu-west-2:{YOUR_ACCOUNT_ID}:{YOUR_QUEUE_NAME}
aws sns subscribe \
--topic-arn arn:aws:sns:eu-west-2:060011828759:tep-xml-ingestion-staging-{broadcaster}-notifications \
--protocol lambda \
--notification-endpoint arn:aws:lambda:eu-west-2:{YOUR_ACCOUNT_ID}:function:{YOUR_FUNCTION_NAME}
Broadcasters can opt to batch multiple projects / batches / episodes into a single XML file, but can equally well opt to split each individual project, or even episode, into its own XML file. The ingestion process will handle either, however notifications will be issued per file, not per project or per episode. For this reason, integrators may choose to opt for more granular XML files (e.g. one per project) to facilitate logging, debugging and interpreting SNS notifications.
The role will be given the following per-directory permissions:
| Directory | Permissions |
|---|---|
/incoming/ |
Read, Write, Delete |
/complete/ |
Read only (the files will automatically be deleted no sooner than 7 days after ingestion) |
/errors/ |
Read only (the files will automatically be deleted no sooner than 7 days after ingestion failure) |
| Directory | Permissions |
|---|---|
/incoming/ |
Read, Write, Delete |
/complete/ |
Read, Write, Delete |
/errors/ |
Read, Write, Delete |
The additional permissions for the complete and errors directories are intended to facilitate testing of SNS integration i.e. you can create your own files in /complete/ or /errors/ and check that this triggers the expected workflow.