S3 connector API guide

Last updated on October 24, 2024

Overview

This guide provides detailed instructions on how to create, configure, and manage an S3 connector using the API. With the provided endpoints, you can seamlessly integrate the S3 connector into your data streaming workflows.

Create an S3 connector using the API

1. Create an S3 bucket on your AWS account. This is where the streamed data will be stored.

2. Generate the S3 bucket policy script via API endpoint [GET] /analytics-connector/v1/admin/tools/s3/policies/{bucketName}. Replace {bucketName} with your created S3 bucket name. The API will return the S3 bucket policy script.

   {
     "Version": "2012-10-17",
     "Statement": [
       {
         "Sid": "AccelByteAnalyticsConnector",
         "Effect": "Allow",
         "Principal": {
           "AWS": "arn:aws:iam::<accelbyte-aws-account-id>:role/analytics-connector"
       },
         "Action": [
           "s3:ListBucket",
           "s3:GetBucketLocation",
           "s3:GetObject",
           "s3:GetObjectVersion",
           "s3:PutObject"
       ],
         "Resource": [
           "arn:aws:s3:::<your-s3-bucket>/*",
           "arn:aws:s3:::<your-s3-bucket>"
         ]
       }
     ]
   }

3. Implement the S3 bucket policy script on the S3 bucket you created. Please refer to our Bucket Policy Implementation.

4. Create the S3 connector configuration via API endpoint [POST] /analytics-connector/v1/admin/connectors. Provide the following configuration structure. Retrieve a list of analytics data to fill out the filter configuration.

   {
     "connectorName": "<s3-connector-name>",
     "connectorType": "SINK",
     "storageType": "KAFKA_S3",
     "config": {
       "bucketName": "<s3-bucket-name>",
       "region": "<s3-bucket-region>",
       "customPath": "<s3-path>",
       "customPartitionDuration": "<s3-partition-duration>",
       "partSize": "5242880",
       "flushInterval": "<flush-interval>",
       "flushSize": "<flush-size>",
       "eventType": "<event-type>"
     },
     "filter": {
       "<namespace>": [
         "<topic>"
       ]
     }
   }

5. Activate the S3 connector configuration via API endpoint [PUT] /analytics-connector/v1/admin/connectors/{id}/activate. Replace {id} with the S3 connector ID from the API response.

Connector configuration description

Config

• s3-connector-name: The name of the S3 connector, please note that the connector name will have an additional random number after the S3 connector is created.

• s3-bucket-name: The name of the S3 bucket.

• s3-bucket-region: The AWS region where the S3 bucket has been created.

• s3-path: Determines the path to store the data in the S3 bucket. Please refer to our S3 Path Format Configuration Guide.

• s3-partition-duration: Determines how the data are partitioned into S3 objects based on the minutes. Please refer to our S3 Partition Duration Guide.

• event-type: The type of event. There are two types justice_event and game_telemetry.

  • justice_event: System-generated events from AccelByte services (Service Telemetry).

  • game_telemetry: Custom telemetry events that are sent from game clients (Custom Telemetry).

• flush-interval: The maximum time interval in minutes that the data should be periodically written into S3. The flush interval range is between 1 and 15 minutes.

• flush-size: The maximum number of events that should be written into S3. The flush size range is 100 and 1000.

  note

  Note that data will be sent depending on which condition is reached first between flush-interval or flush-size.

Filter

• namespace: Properties to filter specific namespace, using an asterisk (*) for all namespaces.

• topic: Properties to filter specific analytics topics, using an asterisk (*) for all topics.

Example S3 connector configuration

Create an S3 connector configuration with a five-minute batch interval and custom S3 path.

Templated path:

accelbyte/{realm}/{topic}/{namespace}/year={year}/month={month}/day={day}/hour={hour}/minute={mm}/{topic}-{realm}-{namespace}.json

Expected S3 path result:

accelbyte/dev/analytics_game_telemetry.dev.lightfantastic.gameEnded/lightfantastic/year=2023/month=03/day=21/hour=17/minute=15/analytics_game_telemetry.dev.lightfantastic.gameEnded-dev-lightfantastic-000000000.json

S3 connector configuration:

{
  "connectorName": "s3-connector-game-telemetry",
  "connectorType": "SINK",
  "storageType": "KAFKA_S3",
  "config": {
    "bucketName": "game-telemetry-bucket-01",
    "region": "us-west-2",
    "customPath": "accelbyte/{realm}/{topic}/{namespace}/year={yyyy}/month={MM}/day={dd}/hour={hh}/minute={mm}/{topic}-{realm}-{namespace}.json",
    "customPartitionDuration": "15",
    "partSize": "5242880",
    "flushInterval": "5",
    "flushSize": "1000",
    "eventType": "game_telemetry"
  },
  "filter": {
    "*": [
      "*"
    ]
  }
}