Out of memory issues with Go Extend apps

Last updated on October 25, 2024

Overview

This article provides best practices for efficient memory management and steps on using [pprof](https://pkg.go.dev/net/http/pprof) to streamline the memory and CPU profiling processes.

Memory management best practices

Memory leaks in Go can occur due to improper usage of memory. We recommend the following best practices for efficient memory management and proper resource cleanup:

1. Avoid global variables.
2. Use the defer statement.
3. Close Channels properly.
4. Handle Goroutines correctly.
5. Monitor memory usage.
6. Use contexts to manage Goroutines.

Use pprof for memory and CPU profiling

In Go, the most common used profiling is pprof and runtime package. Using pprof for memory and CPU profiling in Go applications can significantly aid in identifying and resolving out of memory (OOM) issues, especially when working within a Kubernetes environment.

pprof installation

This pprof package is implemented by default in the main.go file of the Extend's Go application:

import (
    _ "net/http/pprof"
    "runtime"
       ...
)

func main() {
   go func () {
      runtime.SetBlockProfileRate(1)
      runtime.SetMutexProfileFraction(10)
   }()
   
   logrus.Infof("starting app server..")
   
   // The rest of the program's logic
}

Then, the UI in http://localhost:8080/debug/pprof/ can be seen in the web browser.

Memory profiling

Memory profiling helps in understanding and optimizing the memory consumption of Go programs.

Use the following commands to generate a memory profile:

go tool pprof "http://localhost:8080/debug/pprof/heap?seconds=30"

Or

go tool pprof -png http://localhost:8080/debug/pprof/heap > heap.png

The file heap.png will show a node graph of inuse_space profiling in the application.

CPU profiling

CPU profiling primarily focuses in identifying performance bottlenecks in terms of execution time, which can indirectly help manage memory usage by optimizing code.

Use the following commands to generate a CPU profile:

go tool pprof "http://localhost:8080/debug/pprof/profile?seconds=30"

Or

   go tool pprof -png http://localhost:8080/debug/pprof/profile > profile.png

The file profile.png will show a node graph of CPU profiling in the application.

Run pprof commands in Kubernetes pods

To be able to run the the CPU and memory profiling pprof commands in Kubernetes pods, the pods must be port forwarded first.

To port forward a Kubernetes pod, follow these steps:

1. On your Grafana dashboard, find the pod with the highest memory or CPU usage.

2. Run this command to port forward the pod.

   kubectl port-forward <pods-name> --address 0.0.0.0 <local-port>:<pod-port> -n <namespace>

   Replace <pods-name>, <local-port>, <pod-port>, and <namespace> with appropriate values.

3. Run the pprof commands for CPU and memory profiling.