Extend app custom metrics

Last updated on March 3, 2025

Overview

Extend app custom metrics are additional metrics other than the default metrics exposed by Extend apps.

This article guides you through the process of adding a custom metric to your Extend app, using the Extend Service Extension app template as an example. However, the steps outlined apply to all types of Extend apps.

Prerequisites

Before following this walkthrough, you should be familiar with how to set up an Extend Service Extension app.

Clone the app template

Go

git clone https://github.com/AccelByte/extend-service-extension-go

C#

git clone https://github.com/AccelByte/extend-service-extension-csharp

Java

git clone https://github.com/AccelByte/extend-service-extension-java

Python

git clone https://github.com/AccelByte/extend-service-extension-python

Add a custom metric

In this example, we will add a counter metric that tracks the number of times the GetGuildProgress function is called.

Go

1. Define the new metric by creating a Prometheus CounterVec and storing it in counterGetGuildProgress variable in main.go. The metric will include one label: method.

   var (
       serviceName             = common.GetEnv("OTEL_SERVICE_NAME", "ExtendCustomServiceGoDocker")
       logLevelStr             = common.GetEnv("LOG_LEVEL", logrus.InfoLevel.String())
       // The new variable
       counterGetGuildProgress = prometheus.NewCounterVec(
           prometheus.CounterOpts{
               Name: "ab_count_get_guild_progress",
               Help: "Count number of GetGuildProgress called",
           },
           []string{"method"},
       )
   )

2. Register the counterGetGuildProgress variable with the Prometheus registry in main.go.

   // Register Prometheus metrics
   prometheusRegistry := prometheus.NewRegistry()
   prometheusRegistry.MustRegister(
       prometheusCollectors.NewGoCollector(),
       prometheusCollectors.NewProcessCollector(prometheusCollectors.ProcessCollectorOpts{}),
       prometheusGrpc.DefaultServerMetrics,
       // The new custom metric
       counterGetGuildProgress,
   )

3. Pass the counterGetGuildProgress variable to the GetGuildProgress function. Add a new field to the MyServiceServerImpl struct in pkg/service/myService.go to hold the custom metric. Additionally, update the constructor to initialize counterGetGuildProgress when creating a new instance of MyServiceServerImpl.

   type MyServiceServerImpl struct {
       pb.UnimplementedServiceServer
       tokenRepo               repository.TokenRepository
       configRepo              repository.ConfigRepository
       refreshRepo             repository.RefreshTokenRepository
       storage                 storage.Storage
       counterGetGuildProgress prometheus.CounterVec // New field
   }
   func NewMyServiceServer(
       tokenRepo repository.TokenRepository,
       configRepo repository.ConfigRepository,
       refreshRepo repository.RefreshTokenRepository,
       storage storage.Storage,
       counterGetGuildProgress prometheus.CounterVec, // Newly added
   ) *MyServiceServerImpl {
       return &MyServiceServerImpl{
           tokenRepo:               tokenRepo,
           configRepo:              configRepo,
           refreshRepo:             refreshRepo,
           storage:                 storage,
           counterGetGuildProgress: counterGetGuildProgress, // Newly added
       }
   }

4. In the GetGuildProgress function, increment the metrics counter each time the function is called.

   func (g MyServiceServerImpl) GetGuildProgress(
       ctx context.Context, req *pb.GetGuildProgressRequest,
   ) (*pb.GetGuildProgressResponse, error) {
       namespace := req.Namespace
       guildProgressKey := fmt.Sprintf("guildProgress_%s", req.GuildId)
       // called the custom metric
       g.counterGetGuildProgress.WithLabelValues("GET").Inc()
       guildProgress, err := g.storage.GetGuildProgress(namespace, guildProgressKey)
       if err != nil {
           return nil, status.Errorf(codes.Internal, "Error getting guild progress: %v", err)
       }
       return &pb.GetGuildProgressResponse{
           GuildProgress: guildProgress,
       }, nil
   }

C#

1. Define the new metric by creating a metric provider class. Save it inside Classes directory or any other directory.

   using System;
   using Prometheus;
   
   namespace AccelByte.Extend.ServiceExtension.Server
   {
       public class GetGuildCountMetric
       {
           private Counter _Counter;
   
           public GetGuildCountMetric()
           {
               _Counter = Metrics.CreateCounter(
                   "ab_count_get_guild_progress",
                   "Count number of GetGuildProgress called", labelNames: new[] { "method" });
           }
   
           public void Increment(string method)
           {
               _Counter.WithLabels(method).Inc();
           }
       }
   }

2. Register the new metric in service provider inside Program.cs file. Put code below before registering any grpc services.

   ...
   
   builder.Services
       .AddSingleton(new GetGuildCountMetric());
   
   ...

3. Add the new metric inside grpc service constructor in Services/MyService.cs.

   ...
   
   public class MyService : Service.ServiceBase
   {
       ...
   
       private readonly GetGuildCountMetric _GetGuildCountMetric;
   
       ...
   
       public MyService(
           ILogger<MyService> logger,
           IAccelByteServiceProvider abProvider,
           Tracer tracer,
           GetGuildCountMetric getGuildCountMetric)
       {
           ...
   
           _GetGuildCountMetric = getGuildCountMetric;
       }
   
       ...
   }

4. In the GetGuildProgress function, increment the metrics counter each time the function is called.

   _GetGuildCountMetric.Increment("GET", context.Method);

Java

1. Add the spring-aspects dependency to build.gradle file.

   dependencies {
       ...
       implementation 'org.springframework:spring-aspects:6.1.4'
   
       ...
       testImplementation 'org.springframework:spring-aspects:6.1.4'
   }

2. Add CountedAspect to src/main/java/net/accelbyte/extend/serviceextension/config/AppConfig.java file.

   ...
   import io.micrometer.core.aop.CountedAspect;
   import io.micrometer.core.instrument.MeterRegistry;
   
   @Configuration
   public class AppConfig {
   
       ...
       @Bean
       public CountedAspect provideCounterAspect(MeterRegistry registry) {
           return new CountedAspect(registry);
       }
   }

3. Add @Counted annotation to getGuildProgress method in src/main/java/net/accelbyte/extend/serviceextension/service/MyService.java file.

   ...
   import io.micrometer.core.annotation.Counted;
   
   @GRpcService
   @Slf4j
   public class MyService extends ServiceGrpc.ServiceImplBase {
       
       @Counted(value = "ab_count_get_guild_progress", 
               description = "Count number of GetGuildProgress called", 
               extraTags = { "method", "GET" })
       @Override
       public void getGuildProgress(
           GetGuildProgressRequest request, StreamObserver<GetGuildProgressResponse> responseObserver
       ) {
           ...
       }
   }

Python

1. Define the new metric by creating a Prometheus Counter and storing it in counter_get_guild_progress variable in your service's .py file. The metric will include one label: method.

   from prometheus_client import Counter
   
   class AsyncMyService(MyServiceServices):
       def __init__(self, *args, **kwargs) -> None:
           self.counter_get_guild_progress = Counter(
               name="ab_count_get_guild_progress",
               documentation="Count number of GetGuildProgress called",
               labelnames=["method"],
               unit="count",
           )

2. In the GetGuildProgress function, increment the metrics counter each time the function is called.

   async def GetGuildProgress(
       self, request: GetGuildProgressRequest, context: Any
   ) -> GetGuildProgressResponse:
       # ...
   
       self.counter_get_guild_progress.labels(method="GET").inc(amount=1)
   
       # ...

Explore metrics in Grafana Cloud

1. Access the Grafana Cloud by clicking the Open Grafana Cloud button on your Extend app details page in the Admin Portal..

   

2. Click the Explore menu in the Grafana Cloud.

   

3. Select a data source to load the metrics data.

   There are two Prometheus data sources that you can select depending on your account permissions in Admin Portal. If you have access to all namespaces (super admin), you can select metrics data source (cross-namespace data). If you only have access to a specific namespace, you can only select prom-<namespace>.* data source (per-namespace data).

   Cross-namespace data

   Per-namespace data

4. Add or remove multiple queries as needed. Once the source is selected, click the Add query button. Additionally, Grafana allows you to view query history by clicking the Query history button.

   

5. To select a custom metric, use the Select metric dropdown and search for your custom metric. You can also click the Operations button to apply a specific operation.

   

   If you have a complex query, click the Code tab to use Prometheus queries for advanced calculations.

   

   Follow the steps in the image below: First, select the source (1). Next, choose the time frame for the query (2). Then, select values from the Metric drop down box and Label filters (3). Optionally, apply a specific operation to your metrics (4). Finally, click the Run query button to view the metric.

   

Important notes

1. A series is defined as a unique combination of a metric name and a set of label key-value pairs. Each distinct metric name, along with its corresponding label key-value pairs, is treated as a separate series. For example:

   ab_count_get_guild_progress{method="GET"} 1 -> series_count=1
   ab_count_get_guild_progress{method="GET"} 5 -> series_count=1
   ab_count_get_guild_progress{method="POST"} 5 -> series_count=2
   ab_sum_get_guild_progress{method="GET", service="guild"} 1 -> series_count=3

2. Custom metrics are limited to a maximum of 1,000 active series. Once this limit is reached, no new series can be pushed. a. A metric is considered an active series if it is consistently scraped and received by Grafana Cloud. b. A metric is considered a non-active series if it is not scraped or received by Grafana Cloud for 20 minutes.

3. Currently, it is not possible to create custom dashboard to display metrics.

4. Extend app metrics, including custom metrics, are published at the /metrics endpoint on port 8080. This configuration is set up by default when using the Extend app templates to create your Extend app.

5. The table below shows the reserved labels that cannot be used in custom metrics. If any of these are used, they will be replaced with the predefined values.

   Label Key	Value	Description
   environment_name	${env_name}	Environment identifier name
   product	extend	Product identifier name: extend
   scenario	${extend_app_type}	Extend app type: function-override, event-handler, or service-extension
   union_namespace	${publisher_name}	Publisher name
   game_namespace	${game_namespace}	Extend app namespace
   app_name	${app_name}	Extend app name
   job	metric-services/extend-metrics	Predefined job name
   instance	extend-metric-instance	Predefined instance value

Tips

Add a unique prefix to your custom metric to easily recognize it and differentiate it from default metrics. For example, ab_count_get_guild_progress, where ab_ serves as the unique identifier prefix.