The observability blind spot for non-HTTP traffic within a service mesh is a recurring production challenge. Our platform, running on Kubernetes and instrumented with Linkerd, provided exceptional L7 metrics for gRPC and HTTP/2 services. However, a critical component, our distributed Memcached fleet, remained an opaque black box. Linkerd dutifully reported TCP-level metrics—bytes transferred, connection duration—but offered zero insight into the actual cache performance. When a service’s P99 latency spiked, the dashboard would show increased latency on the connection to Memcached, but the root cause remained elusive. Was it a network issue? A sudden flood of cache misses causing expensive database fallbacks? High eviction rates within Memcached itself? Answering these questions involved cross-referencing disparate dashboards and manually correlating timestamps, a process far too slow for a production incident.
The initial thought was to deploy the standard memcached_exporter. This approach quickly proved inadequate. The standard exporter provides instance-level metrics: total gets, sets, evictions, memory usage. It tells you what is happening to the Memcached server as a whole, but not who is causing it. In a multi-tenant cache used by dozens of microservices, global statistics are insufficient for pinpointing a misbehaving client. We needed to correlate the application-level cache behavior (hit/miss ratio, command rates) directly with the specific client pod that Linkerd identified as having high latency.
The architectural goal became clear: create a single Grafana dashboard that visualizes Linkerd’s client-side golden metrics for a given pod alongside the granular Memcached protocol metrics generated by that same pod’s traffic. This required a solution that could parse the Memcached protocol on-the-fly and enrich the resulting metrics with Kubernetes pod metadata, enabling a direct join with Linkerd’s telemetry in Prometheus. Modifying dozens of legacy applications to add this instrumentation was a non-starter due to the immense engineering effort and risk involved. The solution had to be non-invasive, deployed at the infrastructure layer.
This led to the design of a custom Memcached protocol-aware proxy, implemented in Go. This proxy would be deployed as a sidecar container in any application pod that communicates with Memcached. The application connects to localhost, talking to our proxy. The proxy parses the Memcached binary protocol, generates detailed Prometheus metrics tagged with the client’s identity, and then forwards the raw traffic to the actual Memcached service. Linkerd’s proxy, already present, would simply see a TCP connection to this local proxy sidecar. This architecture creates a chain: Application -> Linkerd Proxy -> Memcached Metrics Proxy -> Memcached Server.
graph TD
subgraph "Application Pod"
A[Application Container] -->|Connects to localhost:11211| LP[Linkerd Proxy]
LP -->|Forwards to localhost:11212| PMP[Memcached Metrics Proxy]
PMP -->|Parses protocol, emits metrics| M((Metrics Endpoint :9151))
end
PMP -->|Forwards raw TCP| MS[Memcached Service]
PRO[Prometheus] -->|Scrapes :4191| LP
PRO -->|Scrapes :9151| PMP
subgraph "Memcached Pod"
MI[Memcached Instance]
end
MS --> MI
G[Grafana] -->|Queries| PRO
This design is a trade-off. It introduces an extra network hop (albeit on the local loopback interface), which adds a small amount of latency. However, the value of obtaining correlated, client-specific cache metrics without any application code changes far outweighs the minimal performance cost for our use case.
Building the Protocol-Aware Metrics Proxy
The core of the solution is a Go application that performs three functions: listen for incoming TCP connections, parse Memcached binary protocol packets from the client stream, and forward those packets to the upstream Memcached server. While doing this, it inspects the requests and responses to generate Prometheus metrics.
The Memcached binary protocol is well-defined. A request packet consists of a 24-byte header followed by an optional key and value. The response packet has a similar structure. We only need to parse enough of the header to identify the command (Opcode) and the response status to determine a hit or miss.
Here is the core implementation of the proxy. This is not a toy example; it includes graceful shutdown, concurrent connection handling, and integration with the official Prometheus Go client.
main.go:
package main
import (
"context"
"fmt"
"io"
"log"
"net"
"os"
"os/signal"
"sync"
"syscall"
"time"
"github.com/prometheus/client_golang/prometheus"
"github.com/prometheus/client_golang/prometheus/promhttp"
"net/http"
)
var (
// Configuration from environment variables
listenAddr = getEnv("LISTEN_ADDR", ":11212")
memcachedUpstream = getEnv("MEMCACHED_UPSTREAM", "memcached-service:11211")
metricsAddr = getEnv("METRICS_ADDR", ":9151")
clientPodName = getEnv("CLIENT_POD_NAME", "unknown")
clientNamespace = getEnv("CLIENT_NAMESPACE", "unknown")
// Prometheus Metrics
opsCounter = prometheus.NewCounterVec(
prometheus.CounterOpts{
Name: "memcached_proxy_ops_total",
Help: "Total number of Memcached operations by command.",
},
[]string{"command", "client_pod", "client_namespace"},
)
hitsCounter = prometheus.NewCounterVec(
prometheus.CounterOpts{
Name: "memcached_proxy_hits_total",
Help: "Total number of cache hits.",
},
[]string{"command", "client_pod", "client_namespace"},
)
missesCounter = prometheus.NewCounterVec(
prometheus.CounterOpts{
Name: "memcached_proxy_misses_total",
Help: "Total number of cache misses.",
},
[]string{"command", "client_pod", "client_namespace"},
)
connectionsGauge = prometheus.NewGauge(
prometheus.GaugeOpts{
Name: "memcached_proxy_active_connections",
Help: "Number of active connections.",
},
)
)
// Memcached binary protocol constants
const (
requestMagic byte = 0x80
responseMagic byte = 0x81
opCodeGet byte = 0x00
opCodeSet byte = 0x01
opCodeDelete byte = 0x04
// ... other opcodes can be added here
)
var commandMap = map[byte]string{
opCodeGet: "get",
opCodeSet: "set",
opCodeDelete: "delete",
}
func init() {
prometheus.MustRegister(opsCounter)
prometheus.MustRegister(hitsCounter)
prometheus.MustRegister(missesCounter)
prometheus.MustRegister(connectionsGauge)
}
func main() {
log.Printf("Starting memcached-metrics-proxy...")
log.Printf("Listening on: %s", listenAddr)
log.Printf("Upstream Memcached: %s", memcachedUpstream)
log.Printf("Metrics endpoint: %s", metricsAddr)
log.Printf("Client identity: pod=%s, namespace=%s", clientPodName, clientNamespace)
listener, err := net.Listen("tcp", listenAddr)
if err != nil {
log.Fatalf("Failed to listen on %s: %v", listenAddr, err)
}
defer listener.Close()
// Start Prometheus metrics server
http.Handle("/metrics", promhttp.Handler())
go func() {
if err := http.ListenAndServe(metricsAddr, nil); err != nil {
log.Fatalf("Metrics server failed: %v", err)
}
}()
// Graceful shutdown setup
ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGINT, syscall.SIGTERM)
defer stop()
var wg sync.WaitGroup
go func() {
for {
conn, err := listener.Accept()
if err != nil {
select {
case <-ctx.Done():
log.Println("Listener shutting down.")
return
default:
log.Printf("Failed to accept connection: %v", err)
}
continue
}
wg.Add(1)
go func() {
defer wg.Done()
handleConnection(ctx, conn)
}()
}
}()
<-ctx.Done()
log.Println("Shutting down proxy...")
listener.Close() // This will unblock the Accept loop
// A simple wait with timeout to allow active connections to finish
shutdownComplete := make(chan struct{})
go func() {
wg.Wait()
close(shutdownComplete)
}()
select {
case <-shutdownComplete:
log.Println("All connections closed gracefully.")
case <-time.After(5 * time.Second):
log.Println("Shutdown timeout reached. Forcing exit.")
}
}
func handleConnection(ctx context.Context, clientConn net.Conn) {
defer clientConn.Close()
connectionsGauge.Inc()
defer connectionsGauge.Dec()
log.Printf("Accepted connection from %s", clientConn.RemoteAddr())
serverConn, err := net.DialTimeout("tcp", memcachedUpstream, 5*time.Second)
if err != nil {
log.Printf("Failed to connect to upstream %s: %v", memcachedUpstream, err)
return
}
defer serverConn.Close()
var wg sync.WaitGroup
wg.Add(2)
// Goroutine to proxy client -> server and parse requests
go func() {
defer wg.Done()
defer serverConn.Close()
proxyAndParse(clientConn, serverConn, true)
}()
// Goroutine to proxy server -> client and parse responses
go func() {
defer wg.Done()
defer clientConn.Close()
proxyAndParse(serverConn, clientConn, false)
}()
wg.Wait()
log.Printf("Closing connection from %s", clientConn.RemoteAddr())
}
func proxyAndParse(src, dst net.Conn, isRequest bool) {
buffer := make([]byte, 4096)
for {
n, err := src.Read(buffer)
if err != nil {
if err != io.EOF {
log.Printf("Read error: %v", err)
}
return
}
if n > 0 {
chunk := buffer[:n]
if isRequest {
parseRequest(chunk)
} else {
parseResponse(chunk)
}
_, writeErr := dst.Write(chunk)
if writeErr != nil {
log.Printf("Write error: %v", writeErr)
return
}
}
}
}
// A simplified parser for Memcached binary protocol. A production version would be more robust.
func parseRequest(data []byte) {
// A request must be at least 24 bytes for the header
if len(data) < 24 {
return
}
if data[0] != requestMagic {
return // Not a binary protocol request
}
opCode := data[1]
command, ok := commandMap[opCode]
if !ok {
command = "unknown"
}
opsCounter.WithLabelValues(command, clientPodName, clientNamespace).Inc()
}
func parseResponse(data []byte) {
if len(data) < 24 {
return
}
if data[0] != responseMagic {
return // Not a binary protocol response
}
opCode := data[1]
command, ok := commandMap[opCode]
if !ok {
command = "unknown"
}
// In the binary protocol, the status is in bytes 6 and 7.
// 0x0000 means success. 0x0001 means key not found (a miss).
status := uint16(data[6])<<8 | uint16(data[7])
if command == "get" {
if status == 0x0000 {
hitsCounter.WithLabelValues(command, clientPodName, clientNamespace).Inc()
} else if status == 0x0001 {
missesCounter.WithLabelValues(command, clientPodName, clientNamespace).Inc()
}
}
}
func getEnv(key, fallback string) string {
if value, ok := os.LookupEnv(key); ok {
return value
}
return fallback
}This code uses environment variables to configure its listeners and upstream target. The crucial CLIENT_POD_NAME and CLIENT_NAMESPACE variables will be injected into the container using the Kubernetes Downward API. This is how we stamp each metric with the identity of its source pod.
Deployment on Kubernetes
To deploy this, we need a Dockerfile and a Kubernetes Deployment manifest.
Dockerfile:
FROM golang:1.21-alpine AS builder
WORKDIR /app
COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN CGO_ENABLED=0 GOOS=linux go build -a -installsuffix cgo -o memcached-proxy .
FROM alpine:latest
RUN apk --no-cache add ca-certificates
WORKDIR /root/
COPY /app/memcached-proxy .
CMD ["./memcached-proxy"]Now, we modify the application’s Deployment to include our proxy as a sidecar. The key changes are:
- Adding the
memcached-proxycontainer to the pod spec. - Using the Downward API to pass pod metadata as environment variables to the proxy.
- Changing the application’s environment variable for the Memcached host to
localhost:11212, redirecting its traffic through our proxy. - Adding annotations for Prometheus to discover and scrape both the Linkerd (
:4191) and our custom proxy (:9151) metrics endpoints.
app-deployment.yaml:
apiVersion: apps/v1
kind: Deployment
metadata:
name: sample-app
namespace: default
spec:
replicas: 1
selector:
matchLabels:
app: sample-app
template:
metadata:
labels:
app: sample-app
annotations:
prometheus.io/scrape: "true"
prometheus.io/path: "/metrics"
prometheus.io/port: "9151" # Scrape our proxy
# Linkerd's metrics are scraped via its own config, but this is an alternative
# prometheus.io/scrape_port_linkerd: "4191"
spec:
containers:
- name: sample-app
image: your-repo/sample-app:latest # Replace with your application image
ports:
- containerPort: 8080
env:
- name: MEMCACHED_HOST
# The app now talks to our local proxy
value: "localhost:11212"
- name: memcached-proxy
image: your-repo/memcached-proxy:latest # Replace with your proxy image
ports:
- name: mc-proxy
containerPort: 11212
- name: mc-metrics
containerPort: 9151
env:
- name: LISTEN_ADDR
value: ":11212"
- name: METRICS_ADDR
value: ":9151"
- name: MEMCACHED_UPSTREAM
# The proxy talks to the actual Memcached service
value: "memcached-service.default.svc.cluster.local:11211"
- name: CLIENT_POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: CLIENT_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
resources:
requests:
cpu: 50m
memory: 64Mi
limits:
cpu: 100m
memory: 128MiA pitfall here is networking. Ensure the MEMCACHED_UPSTREAM service name is resolvable from the pod. Using the full FQDN (service.namespace.svc.cluster.local) is a robust practice.
Fusing the Data in Grafana
With Prometheus scraping both Linkerd’s metrics and our new proxy’s metrics, we can now build the unified dashboard. Linkerd provides metrics like tcp_open_total and tcp_close_total which can be used to derive connection behavior. More importantly, if Linkerd’s protocol detection were to support Memcached, it would emit L7 metrics we could use. For now, we focus on what we have: TCP stats from Linkerd and protocol stats from our proxy.
The magic happens in PromQL. We can construct queries that leverage the client_pod and client_namespace labels, which are now present in both datasets.
Panel 1: Client-Side Latency (from Linkerd)
This is hypothetical if Linkerd provided latency histograms for opaque TCP, which it doesn’t directly. A proxy for this could be connection duration, but let’s assume for a more advanced scenario Linkerd is providing some form of latency measurement. A more realistic metric is TCP write bytes, indicating traffic volume.
// Shows the rate of bytes written to the Memcached service from each client pod
sum(rate(tcp_write_bytes_total{namespace="$namespace", pod="$pod", direction="outbound", target_addr=~"memcached-service.*"}[5m])) by (pod)In Grafana, $namespace and $pod would be template variables.
Panel 2: Cache Hit/Miss Ratio (from our Proxy)
This query calculates the hit rate per pod. A sharp drop here is a clear indicator of a problem.
sum(rate(memcached_proxy_hits_total{command="get", client_namespace="$namespace", client_pod="$pod"}[5m]))
/
(
sum(rate(memcached_proxy_hits_total{command="get", client_namespace="$namespace", client_pod="$pod"}[5m]))
+
sum(rate(memcached_proxy_misses_total{command="get", client_namespace="$namespace", client_pod="$pod"}[5m]))
)
* 100Panel 3: Memcached Operations Rate (from our Proxy)
This shows the request pressure from a specific client. A sudden spike in get operations could explain a performance degradation.
sum(rate(memcached_proxy_ops_total{client_namespace="$namespace", client_pod="$pod"}[5m])) by (command)The Unified View
By placing these panels on the same dashboard, filtered by the same pod template variable, the correlation becomes instant. When an on-call engineer sees an alert for high application latency, they can select the affected pod from a dropdown. They will immediately see Linkerd’s view of the network behavior next to our proxy’s view of the application-level cache behavior. A latency spike that correlates with a plummeting cache hit rate and a surge in get requests tells a complete story: the application is suddenly requesting keys that are not in the cache, forcing it into a slower fallback path, which manifests as high end-to-end latency. The problem is not the network or Memcached itself, but the application’s query pattern. We’ve achieved root cause analysis in seconds, not hours.
This solution, while effective, is not without its own set of considerations for a real-world project. The Go proxy introduces a new component to maintain, secure, and monitor. Its resource footprint, while small, is not zero and must be accounted for in capacity planning. The protocol parser in this example is rudimentary; a production version would need to handle pipelined requests and more complex protocol edge cases with greater care.
The logical next step in this evolution would be to replace the userspace proxy with an eBPF-based solution. An eBPF program attached to the kernel’s network stack could sniff Memcached traffic non-invasively, eliminating the extra network hop and the need to reconfigure the application. This would offer the same deep visibility with near-zero performance overhead, but comes with a significantly higher development and operational complexity, requiring deep kernel-level expertise. The proxy sidecar remains a pragmatic and powerful intermediate step, delivering the majority of the value with a fraction of the implementation cost.
