Configure the S3 path format

Last updated on October 23, 2024

Overview

The S3 path format gives the flexibility to define the S3 path, where the data will be stored.

S3 path format

The S3 path format will have a static and dynamic format. The static format will be written as it is in the S3 path and the dynamic format will be written as the data value. The dynamic format has the following ten fields:

• {eventType}: property of event type would be game_telemetry or justice_event.

• {namespace}: property of namespace would follow the data namespace.

• {realm}: property of realm would be the environment name.

• {topic}: property of the topic would follow the source of Kafka topics.

• {eventName}: property of the event name would follow the data event name.

• {yyyy}: a time format year property. Example result: 2023

• {MM}: a time format month property. Example result: 06

• {dd}: a time format day property. Example result: 19

• {hh}: a time format hour property. Example result: 24

• {mm}: a time format minute property. Example result: 00,15,30,45. The default is a 15-minute rotation, which is the S3 partition duration.

Default value

The following is the default S3 path format value.

{eventType}/realm/{realm}/topics/{topic}/year={yyyy}/month={MM}/day={dd}/hour={hh}/minute={mm}/{topic}.json

The default format would result as follows.

game_telemetry/realm/dev/topics/analytics_game_telemetry.dev.accelbyte.gameEvent/year=2023/month=06/day=15/hour=09/minute=45/analytics_game_telemetry.dev.accelbyte.gameEvent-000001.json

Sample configuration

Example value:

accelbyte/event={eventType}/environment={realm}/namespace={namespace}/topic={topic}/event_name={eventName}/year={yyyy}/month={MM}/day={dd}/hour={hh}/file-{topic}-{namespace}.json

Example result: These sample values would produce the following result.

accelbyte/event=game_telemetry/environment=dev/namespace=accelbyte/event=analytics_game_telemetry.dev.accelbyte.gameEvent/event_name=gameEvent/year=2023/month=06/day=15/hour=09/file-analytics_game_telemetry.dev.accelbyte.gameEvent-accelbyte-000001.json

In this case, the values for accelbyte/event=, environment=, namespace=, topic=, event_name=, year=, month=, day=, and hour= value are in static format, while the values for {eventType}, {realm}, {namespace}, {topic}, {yyyy}, {MM}, {dd}, and {hh} are in dynamic format.

S3 partition duration

The S3 partition duration determines how data are partitioned into S3 objects based on the minutes. The minute partition is set to 15 minutes by default. It can be configured from one up to 60 minutes.

注記

If the partition doesn't reach the minute path, set it to the default value of 15.

Example value:

• S3 path format

  accelbyte/event={eventType}/environment={realm}/namespace={namespace}/topic={topic}/event_name={eventName}/year={yyyy}/month={MM}/day={dd}/hour={hh}/minute={mm}/file-{topic}-{namespace}.json

• S3 partition duration: 15

Example result:

accelbyte/event=game_telemetry/environment=dev/namespace=accelbyte/event=analytics_game_telemetry.dev.accelbyte.gameEvent/event_name=gameEvent/year=2023/month=06/day=15/hour=09/minute=15/file-analytics_game_telemetry.dev.accelbyte.gameEvent-accelbyte-000001.json

accelbyte/event=game_telemetry/environment=dev/namespace=accelbyte/event=analytics_game_telemetry.dev.accelbyte.gameEvent/event_name=gameEvent/year=2023/month=06/day=15/hour=09/minute=30/file-analytics_game_telemetry.dev.accelbyte.gameEvent-accelbyte-000002.json