Extend Event Handler Observability

Last updated on December 18, 2024

Overview

You can use Grafana Cloud with AccelByte Gaming Services (AGS) to:

• Use Grafana's query editor that allows you to specify the exact logs and metrics data you want to retrieve. You can pick the data source for logs or metrics.
• Use Grafana's logging and metrics capabilities to enable effective debugging of the Extend Event Handler service.

AGS Shared Cloud

AGS Shared Cloud customers are eligible for a free trial of Grafana Cloud. The trial period runs from March 18, 2024 at 7:00 AM PDT until April 26, 2024 at 11:59 PM PDT. Please note that this offer is exclusive to the first user (within your AGS account) who logs in to Grafana Cloud during the time frame. Only this user will have access to Grafana Cloud throughout the trial period. For more information on how to start your free trial, see Grafana Cloud free trial.

Prerequisite

• The steps in this guide use the sample described in Getting started with Event Handler as a practical example. Ensure that the Extend Event Handler app has been deployed.

Dashboard overview

You can access the Grafana Cloud dashboard for your Extend app from the AGS Admin Portal. For more information, see the Access Grafana Cloud dashboards guide.

The Grafana Cloud dashboards for Extend Event Handler apps have various tabs and panels. An important feature is the hamburger button (three horizontal lines) located in the top-left corner of the screen, which allows you to access the following important components:

Home: displays a quick overview of various dashboards and settings.

Dashboard: displays pre-configured panels containing data for logs and metrics. Accessible through the hamburger button.

Explore: allows you to manually query logs and metrics. To access this, click on the hamburger button and select Explore. In the Explore tab's Data Sources dropdown, you can select specific data sources, such as Loki for logs, Tempo for traces, and others for metrics. You can use these data sources for debugging purposes.

Working with logs

You can use Grafana's Loki feature to:

• Confirm the successful deployment and startup of the services.
• Confirm successful data transmission between these services.

Navigate to Logs

To view logs, click the hamburger button on the top-left corner to access the Explore tab.

Query logs

To execute a basic query log, follow these steps:

1. Select data source. Select the Data Sources dropdown, then choose the logs option. In this example, the logs option is logs.

   

   tip

   Follow the number annotations in this image to locate the described actions in each step.

2. Set query type. Switch between Builder and Code to set the query type. Ensure that you select the type that supports your query.

3. Input the log query. If you selected the Builder input, you can follow the example shown in the image in step 1. If you selected the Code input, use the following query and change the values for environment_name and game_namespace accordingly.

   {environment_name="(your-environment-name e.g. dev, prod, etc)", game_namespace="(your-namespace)", app_name="(your-app-name)"} |= ``

4. Adjust the time range. Set the time frame of the query to the time frame you need.

5. Execute the query. Click Run query to display the logs that match your query.

Confirm successful deployment

To verify the successful startup of Event Handler Service, follow the following instruction.

In Code query input mode, run the following query. To narrow down the results, you can add more queries as needed. For example, you can use environment_name to query for a specific deployment environment.

{environment_name="(your-environment-name e.g. dev, prod, etc)", game_namespace="(you-namespace)", app_name="(your-app-name)", container="service"} 

You'll see logs similar to the image below.

info

The query in the image above uses line_format to make the logs more readable.

Confirm successful data transmission

To confirm that data is being successfully transmitted to the Event Handler service, follow these steps:

1. Trigger the transmission to the Event Handler service. Follow the steps in Getting started with Event Handler to trigger the test that produces the "user logged in" event.

2. Query for Event Handler Transmission Logs. In Code query input mode, run the following query. To narrow down the results, you can add more queries as needed. For example, you can set environment_name to query for a specific deployment environment.

   {environment_name="(your-environment-name e.g. dev, prod, etc)", game_namespace="(you-namespace)", app_name="(your-app-name)", container="service"} |= "(your-namespace)" |= "userLoggedIn" | logfmt | json |  line_format "{{.body}}"

   This query filters logs for the app and specifically looks for logs containing the phrases "(your-namespace)" , "service" container, and "userLoggedIn". It then processes these logs through logfmt and json parsers. For more details, see the Message userLoggedIn section in IAM Client.

   

Monitor metrics in Grafana

Grafana's metrics provide quantitative data that helps you monitor the performance and health of the Event Handler Service. Understanding how to navigate and interpret these metrics is crucial for ensuring optimal system performance.

Navigate to metrics

To view metrics, click the hamburger button on the top-left corner to access the Explore tab. Open the Data Sources dropdown, then select the metrics option. In this example, the metrics option is metrics-extend.

Query metrics

To query metrics in Grafana, follow these steps:

1. Select data source. Open the Data Sources dropdown, then select the metrics option. In this example, the metrics option is metrics-extend.

   

   tip

   Follow the number annotations in this image to locate the described actions in each step.

2. Set query type. Switch between Builder and Code to set the query type. Ensure that you select the type that supports your query.

3. Input the log query. If you selected the Builder input, you can follow the example shown in the image in step 1. If you selected the Code input, use the following query and change the values for game_namespace and environment_name accordingly.

   metrics_grpc_bridge:t0:accelbyte_dev_userAuthentication:total_messages_received_Count{game_namespace="(your-namespace)", environment_name="(your-environment-name e.g. dev, prod, etc)"}

4. Adjust the time range. Set the time frame of the query to the time frame you need.

5. Execute the query. Click Run query to display the logs that match your query.

Confirm successful message processing

The metric grpc_server_invocation_successful_Count represents the total number of messages that have been processed successfully by the Event Handler Service. This is a counter metric, which means that its value increments over time, and can be used to gauge the performance and reliability of the services' business logic.

To ensure that messages are successfully processed by the Event Handler Service, you can monitor the grpc_server_invocation_successful_Count metric. Run this query:

metrics_grpc_bridge:t0:accelbyte_dev_userAuthentication:userLoggedIn:grpc_server_invocation_successful_Count{game_namespace="your-namespace"}

After running the query, you'll see results similar to the image below:

An incrementing metric count for successfully_processed_messages_total confirms that messages are being successfully processed.