Skip to main content

Manage message processing and idempotency

Last updated on November 4, 2026

Overview​

The AccelByte Gaming Services (AGS) Extend Event Handler uses at-least-once message delivery. Your handler can receive the same event more than once, including when a batch is redelivered after one message fails. Design each side effect to tolerate redelivery.

An acknowledgement records message processing progress; it does not make a reward grant and the message acknowledgement one atomic operation. Keep a durable record of each logical reward and its fulfillment transaction ID. A record of completed work does not undo side effects after a failure.

Exceptions and error handling​

The only method to guarantee the messages are delivered and fully processed by the Event Handler is to receive acknowledgement from it. Without acknowledgements from the Event Handler, Kafka Connect will attempt the redelivery of the entire batch depending on the gRPC code returned by the Event Handler or the gRPC framework on a set of deemed "Retriable" status codes.

Kafka Connect will not send a new batch of messages until all messages in the current batch are acknowledged with a non-retryable gRPC status code.

To signal successes or failures and retry attempts (the entire batch of messages will be retried as a whole), it's important to catch and handle all possible exceptions and only throw with a "Retriable" gRPC status code if the message cannot be processed and message retry is desired.

The retriable gRPC status codes from the gRPC Event Handler are as follows:

  • UNAVAILABLE: For example, the Event Handler gRPC Server isn't up.
  • CANCELLED: For example, the call was canceled before the Event Handler finished processing it.
  • DEADLINE_EXCEEDED: For example, the Event Handler didn't respond before the call deadline.
  • RESOURCE_EXHAUSTED: For example, the server is temporarily out of resources.
  • INTERNAL: For example, an internal server error due to custom logic issues.
  • UNKNOWN: For example, the Event Handler app throws an exception or executes an action other than returning a status code when ending the gRPC.

To reduce Kafka Connect retry attempts, only return retryable gRPC status codes if there is a transient issue that can be retried in future calls.

Grant rewards with Fulfillment V3​

Use Implement Fulfillment on a Game Server for the published Go, Java, C#, and Python SDK methods and request examples. Those examples use V3 and check the transaction state rather than requiring new entitlement summaries.

Existing Event Handler template revisions can still call the deprecated fulfillItem endpoint. Update the template's grant function and its SDK dependency before applying the workflow below. Preserve the SDK configuration and authentication used by your application.

Choose the logical reward identity​

  • For one reward per event, derive or persist a transaction ID from the event's stable ID and the reward operation. Include a reward identifier when one event triggers multiple grants.
  • For a once-per-player reward, such as a welcome item, use a stable reward identity scoped to the namespace, player, and reward campaign. Using each login event's ID would grant the reward again on subsequent logins.
  • Keep the original item list, quantities, and other request fields with that identity. Do not change the payload when retrying or reuse its transaction ID for another reward.
  • Do not use a query of the last 10 or 20 fulfillment histories as a deduplication guarantee. Older grants can fall outside the page, and concurrent handlers can both pass a read-before-write check.

Process a message​

  1. Validate the event and decide which logical reward it represents.
  2. Atomically create or load a durable reward record containing the transaction ID and original payload. Concurrent deliveries must use the same record. Serialize processing for that reward where needed.
  3. Call Fulfillment V3 with that transaction ID and payload.
  4. Inspect the result:
    • FULFILLED: record completion, then acknowledge the event. Empty summaries are valid when an earlier attempt already completed.
    • FULFILL_FAILED: inspect stateInfo.failedList, including nested bundle errors. Some components may already be granted. Fix the cause before retrying the saved transaction.
    • Transport error or interrupted execution: reconcile the transaction and affected assets before deciding to retry. A failed HTTP request does not prove that no grant occurred.
  5. Return a retryable gRPC status only when processing should be retried. Route permanent failures to your application's failure-handling process instead of blocking the batch indefinitely.
warning

V3 skips successful components whose state has been persisted, but asset writes and fulfillment state persistence are not atomic. A crash between these steps can leave an ambiguous result. Keep a reconciliation path; neither V3 nor a separate event-processing record provides an unconditional exactly-once guarantee.

For transaction queries and retries, see Manage item fulfillments with third-party stores.

The message batch retry duration is indefinite, with each batch retry interval set at 30 seconds. When all gRPC calls receive non-retriable return codes, the entire batch offset will be committed to Kafka.

Reference​

For more information about processing guarantee and idempotency using the "at least once" semantic, see Kafka Connect's official documentation on message delivery guarantees.