The core constraint was latency. Our real-time bidding platform responds to ad auctions in under 50 milliseconds, and every microsecond counts. The introduction of a conventional user-space WAF, whether a sidecar proxy or a cloud appliance, imposed a 5-15ms penalty per request. This was commercially unacceptable, directly impacting our win rate. Simultaneously, we were facing increasingly sophisticated L7 attacks—not just volumetric floods, but low-and-slow credential stuffing and endpoint scraping from distributed botnets. Our static IP blocklists and simple rate-limiting rules were proving ineffective. The problem was clear: we needed a security layer with sub-millisecond overhead that could react intelligently to threats in seconds, not minutes.
Our initial concept gravitated towards moving security enforcement from user space into the Linux kernel. The natural choice for this was eBPF, specifically attaching a program to the Traffic Control (TC) ingress hook. This allows packet inspection and filtering before the kernel’s network stack even fully processes the packet, offering unparalleled performance. The pitfall here, however, is that eBPF programs are intentionally simple and cannot, for example, make external API calls or access complex state. A purely eBPF-based solution would be fast but unintelligent, bringing us back to the problem of static rules.
The solution was to architect a closed-loop system. The eBPF program would act as the “muscle”—the high-speed sensor and enforcer. A user-space monitoring and decision-making system would act as the “brain.” For the brain, Prometheus was the obvious choice. In a real-world project dealing with network traffic, you’re dealing with extremely high-cardinality data: every source IP, every destination port, every API endpoint is a potential label dimension. Prometheus’s TSDB is explicitly designed to handle this. Our architecture crystallized:
- eBPF Layer: An eBPF program at the TC hook inspects incoming packets, collects basic metrics (packet/byte counts per source IP), and consults an eBPF map for a blocklist.
- Data Pipeline: A user-space Go agent loads the eBPF program and efficiently reads the metrics from the eBPF map using a perf buffer or ring buffer, exposing them to Prometheus via a standard
/metricsendpoint. - Analytics & Alerting: Prometheus scrapes these metrics, and we define Alertmanager rules using PromQL to detect anomalous patterns (e.g., a sudden spike in requests from an IP to a login endpoint).
- Dynamic Actuation: Alertmanager fires a webhook to a small “Control Plane” API. This API receives the malicious IP and updates the eBPF blocklist map in the kernel, closing the loop and blocking the attacker within seconds.
- Human Interface: During an attack, SecOps needs a real-time command center. A conventional web dashboard with slow build times is a liability. We chose to build a Next.js dashboard accelerated by Turbopack, where near-instantaneous hot module replacement (HMR) is not a developer convenience but a critical operational requirement for rapidly visualizing data and iterating on incident response views.
The entire system flow can be visualized as a continuous feedback loop:
graph TD
subgraph Kernel Space
A[Incoming Packet] --> B{eBPF Program @ TC Ingress};
B -- Reads --> I[eBPF Blocklist Map];
B -- Updates --> D[eBPF Metrics Map];
B -- Action: Drop --> X[Dropped];
B -- Action: Pass --> C[Application Network Stack];
end
subgraph User Space
E[Go Collector Agent] -- Reads --> D;
E -- Exposes --> F["/metrics Endpoint"];
G[Prometheus] -- Scrapes --> F;
H[Alertmanager] -- Evaluates Rules --> G;
J[Control Plane API] -- Receives Webhook --> H;
J -- Writes --> I;
K[Turbopack Dashboard] -- Real-time Data via API --> J;
K -- Visualizes --> G;
end
subgraph Operator
L[SecOps Analyst] -- Interacts --> K;
L -- Manual Block/Unblock --> J;
end
The eBPF Enforcer
The core of the enforcement logic resides in a C program compiled by Clang into eBPF bytecode. We attach it to the clsact qdisc (queueing discipline) on our public-facing network interface, which provides both ingress and egress hooks. For a WAF, the ingress hook is primary.
The program relies on two key eBPF maps:
-
ip_blocklist: A hash map where the key is the source IPv4 address and the value is a simple flag. It’s used to make near-instantaneous block decisions. -
ip_stats: A per-CPU array hash map to collect metrics without lock contention. The key is the source IP, and the value is a struct containing packet and byte counts.
Here’s a production-grade snippet of the eBPF C code. A common mistake is to perform complex logic inside the eBPF program itself; the goal here is to keep it minimal, offloading all complex analysis to user space.
// File: bpf/tc_waf.c
#include <linux/bpf.h>
#include <linux/if_ether.h>
#include <linux/ip.h>
#include <linux/tcp.h>
#include <bpf/bpf_helpers.h>
#include <bpf/bpf_endian.h>
#define MAX_MAP_ENTRIES 10240
// A struct to hold traffic statistics for a given IP.
struct ip_stats_t {
__u64 packets;
__u64 bytes;
};
// Map to store the blocklist. Key is source IPv4 address.
struct {
__uint(type, BPF_MAP_TYPE_HASH);
__uint(max_entries, MAX_MAP_ENTRIES);
__type(key, __u32);
__type(value, __u8);
} ip_blocklist SEC(".maps");
// Per-CPU map for contention-free statistics collection.
struct {
__uint(type, BPF_MAP_TYPE_PERCPU_HASH);
__uint(max_entries, MAX_MAP_ENTRIES);
__type(key, __u32);
__type(value, struct ip_stats_t);
} ip_stats SEC(".maps");
SEC("tc")
int tc_ingress_filter(struct __sk_buff *skb) {
void *data_end = (void *)(long)skb->data_end;
void *data = (void *)(long)skb->data;
struct ethhdr *eth = data;
if ((void *)eth + sizeof(*eth) > data_end) {
return TC_ACT_OK; // Not a valid Ethernet frame
}
// We only care about IPv4 for this implementation.
if (eth->h_proto != bpf_htons(ETH_P_IP)) {
return TC_ACT_OK;
}
struct iphdr *iph = data + sizeof(*eth);
if ((void *)iph + sizeof(*iph) > data_end) {
return TC_ACT_OK; // Not a valid IP packet
}
__u32 src_ip = iph->saddr;
// Phase 1: Enforcement. Check if the source IP is in our blocklist.
// This lookup is extremely fast.
if (bpf_map_lookup_elem(&ip_blocklist, &src_ip)) {
bpf_printk("TC WAF: Dropping packet from blocked IP: %x\n", src_ip);
return TC_ACT_SHOT; // Drop the packet
}
// Phase 2: Statistics Collection.
struct ip_stats_t *stats = bpf_map_lookup_elem(&ip_stats, &src_ip);
if (stats) {
// Atomically update packet and byte counts for this CPU.
__sync_fetch_and_add(&stats->packets, 1);
__sync_fetch_and_add(&stats->bytes, skb->len);
} else {
// Initialize stats for a new IP address.
struct ip_stats_t new_stats = { .packets = 1, .bytes = skb->len };
bpf_map_update_elem(&ip_stats, &src_ip, &new_stats, BPF_ANY);
}
return TC_ACT_OK; // Pass the packet to the network stack
}
char __license[] SEC("license") = "GPL";This code is compiled using clang -O2 -g -target bpf -c bpf/tc_waf.c -o bpf/tc_waf.o.
The Go Collector and Prometheus Bridge
The user-space agent is written in Go for its performance and robust ecosystem for eBPF and networking. We use the cilium/ebpf library, which provides excellent abstractions for loading eBPF objects and interacting with maps.
This agent’s responsibilities are:
- Load the compiled eBPF object file and attach the
tc_ingress_filterprogram to the specified network interface. - Set up a Prometheus HTTP server to expose metrics.
- Periodically iterate through the
ip_statseBPF map, aggregate the per-CPU values, and update the Prometheus metrics.
// File: cmd/agent/main.go
package main
import (
"log"
"net"
"net/http"
"os"
"os/signal"
"syscall"
"time"
"github.com/cilium/ebpf"
"github.com/cilium/ebpf/link"
"github.com/prometheus/client_golang/prometheus"
"github.com/prometheus/client_golang/prometheus/promhttp"
)
//go:generate go run github.com/cilium/ebpf/cmd/bpf2go bpf ../../bpf/tc_waf.c -- -I../../headers
const (
ifaceName = "eth0" // The interface to attach to
metricsPort = ":9898"
scanInterval = 5 * time.Second
)
var (
packetCount = prometheus.NewGaugeVec(
prometheus.GaugeOpts{
Name: "waf_ingress_packets_total",
Help: "Total number of ingress packets per source IP.",
},
[]string{"source_ip"},
)
byteCount = prometheus.NewGaugeVec(
prometheus.GaugeOpts{
Name: "waf_ingress_bytes_total",
Help: "Total number of ingress bytes per source IP.",
},
[]string{"source_ip"},
)
)
func main() {
// Subscribe to signals for graceful shutdown
stop := make(chan os.Signal, 1)
signal.Notify(stop, syscall.SIGTERM, syscall.SIGINT)
if err := prometheus.Register(packetCount); err != nil {
log.Fatalf("Failed to register packetCount metric: %v", err)
}
if err := prometheus.Register(byteCount); err != nil {
log.Fatalf("Failed to register byteCount metric: %v", err)
}
// Load pre-compiled programs and maps into the kernel.
objs := bpfObjects{}
if err := loadBpfObjects(&objs, nil); err != nil {
log.Fatalf("Loading eBPF objects failed: %v", err)
}
defer objs.Close()
iface, err := net.InterfaceByName(ifaceName)
if err != nil {
log.Fatalf("Getting interface %s failed: %v", ifaceName, err)
}
// Attach the TC program.
l, err := link.AttachTC(link.TCOptions{
Program: objs.TcIngressFilter,
Interface: iface.Index,
Attach: ebpf.AttachTCІngress,
})
if err != nil {
log.Fatalf("Attaching TC program failed: %v", err)
}
defer l.Close()
log.Printf("eBPF WAF attached to %s. Awaiting metrics...", ifaceName)
// Start Prometheus metrics server
http.Handle("/metrics", promhttp.Handler())
go func() {
if err := http.ListenAndServe(metricsPort, nil); err != nil {
log.Fatalf("Failed to start metrics server: %v", err)
}
}()
// Periodically scan the eBPF map for metrics
ticker := time.NewTicker(scanInterval)
defer ticker.Stop()
for {
select {
case <-ticker.C:
if err := collectMetrics(&objs); err != nil {
log.Printf("Error collecting metrics: %v", err)
}
case <-stop:
log.Println("Shutting down...")
return
}
}
}
// A pitfall in real-world projects is not correctly handling per-CPU map aggregation.
// The values for each key are an array, one element per CPU core. These must be summed.
func collectMetrics(objs *bpfObjects) error {
var (
key uint32
stats []bpfIpStatsT
)
iter := objs.IpStats.Iterate()
for iter.Next(&key, &stats) {
var totalPackets, totalBytes uint64
for _, stat := range stats {
totalPackets += stat.Packets
totalBytes += stat.Bytes
}
// Convert uint32 IP to string format for Prometheus label
ipStr := net.IP{byte(key >> 24), byte(key >> 16), byte(key >> 8), byte(key)}.String()
packetCount.WithLabelValues(ipStr).Set(float64(totalPackets))
byteCount.WithLabelValues(ipStr).Set(float64(totalBytes))
}
return iter.Err()
}This agent is built and run as a systemd service on our edge nodes.
Prometheus Configuration for Threat Detection
With the agent exporting metrics, we configure Prometheus to scrape them. The critical piece is the alerting rule. A simple but effective rule can be based on the rate of increase of packets from a previously unseen or low-traffic IP.
prometheus.yml:
global:
scrape_interval: 15s
scrape_configs:
- job_name: 'ebpf_waf_agents'
static_configs:
- targets: ['node1.example.com:9898', 'node2.example.com:9898']alert.rules.yml:
groups:
- name: WAF_Alerts
rules:
- alert: HighPacketRateFromNewIP
# This PromQL expression is key. It calculates the 5-minute rate of packets.
# It alerts if an IP's packet rate exceeds 100 packets/sec AND that IP was not
# seen (had 0 packets) 10 minutes ago. This helps filter out legitimate, consistently high-traffic sources.
expr: |
rate(waf_ingress_packets_total[5m]) > 100
and
waf_ingress_packets_total offset 10m == 0
for: 1m
labels:
severity: critical
annotations:
summary: "High packet rate detected from new IP {{ $labels.source_ip }}"
description: "Source IP {{ $labels.source_ip }} is sending packets at a rate of {{ $value }} pps."When this alert fires, Alertmanager is configured to send a POST request to our control plane API.
The Control Plane Actuator
This is another small Go service. Its purpose is to receive alerts and translate them into actions on the eBPF maps of the relevant nodes.
// File: cmd/controller/main.go
package main
import (
"encoding/json"
"log"
"net"
"net/http"
"unsafe"
"github.com/cilium/ebpf"
)
const (
// In a real system, this would be a list of agents or discovered via service discovery.
// We also need a way to find the correct map path in bpffs.
bpfMapPath = "/sys/fs/bpf/tc/globals/ip_blocklist"
)
// Simplified Alertmanager webhook payload structure
type AlertPayload struct {
Alerts []struct {
Labels map[string]string `json:"labels"`
} `json:"alerts"`
}
var blocklistMap *ebpf.Map
func main() {
var err error
blocklistMap, err = ebpf.LoadPinnedMap(bpfMapPath, nil)
if err != nil {
log.Fatalf("Failed to load pinned map: %v. Ensure the agent is running and has pinned the map.", err)
}
defer blocklistMap.Close()
http.HandleFunc("/webhook", handleWebhook)
log.Println("Control plane listening on :8080...")
if err := http.ListenAndServe(":8080", nil); err != nil {
log.Fatalf("Server failed: %v", err)
}
}
func handleWebhook(w http.ResponseWriter, r *http.Request) {
if r.Method != http.MethodPost {
http.Error(w, "Invalid request method", http.StatusMethodNotAllowed)
return
}
var payload AlertPayload
if err := json.NewDecoder(r.Body).Decode(&payload); err != nil {
http.Error(w, "Failed to decode payload", http.StatusBadRequest)
return
}
for _, alert := range payload.Alerts {
if ipStr, ok := alert.Labels["source_ip"]; ok {
log.Printf("Received alert for IP: %s. Adding to blocklist.", ipStr)
if err := addIPToBlocklist(ipStr); err != nil {
log.Printf("Error adding IP %s to blocklist: %v", ipStr, err)
http.Error(w, "Internal server error", http.StatusInternalServerError)
return
}
}
}
w.WriteHeader(http.StatusOK)
}
func addIPToBlocklist(ipStr string) error {
ip := net.ParseIP(ipStr).To4()
if ip == nil {
return log.Output(1, "Invalid IPv4 address")
}
// Convert Go's []byte IP representation to a uint32, respecting network byte order.
ipU32 := *(*uint32)(unsafe.Pointer(&ip[0]))
// The value can be anything, we just check for key existence in eBPF.
value := uint8(1)
// This is the actuation step: updating the kernel map.
return blocklistMap.Put(ipU32, value)
}This closes the automation loop. An attack begins, metrics spike, Prometheus detects it, Alertmanager fires, and the control plane blocks the IP at the kernel level on all nodes—all within about 20-30 seconds.
The Turbopack-Powered Command Center
During an incident, automation isn’t always enough. Analysts need to see what’s happening, correlate data, and sometimes manually intervene. The dashboard we built for this had to be incredibly fast—not just in rendering data, but in its development cycle. An analyst might say, “I need a new panel that shows the rate of traffic to the /v2/auth endpoint from IPs in ASN 12345.” A traditional webpack-based dev server might take 15-30 seconds to rebuild and refresh. In a crisis, that’s an eternity.
This is where Turbopack became a non-negotiable requirement. We built the dashboard with Next.js.
In package.json:
{
"scripts": {
"dev": "next dev --turbo",
"build": "next build",
"start": "next start"
}
}The --turbo flag engages Turbopack. The difference was night and day. HMR for component changes was consistently under 50ms. An engineer could add a new visualization component, wire it up to a new API endpoint on our control plane, and see it live with data in seconds. This agility was a force multiplier for our SecOps team.
The dashboard itself is a React application that:
- Uses WebSockets to receive a real-time stream of events from the control plane (e.g., “IP 1.2.3.4 was automatically blocked”).
- Queries a backend API (which in turn queries Prometheus) to render historical graphs and charts.
- Provides forms for analysts to manually add or remove IPs from the blocklist via the control plane API.
A core component might look like this:
// components/LiveBlockedFeed.js
import React, { useState, useEffect } from 'react';
// A mock WebSocket hook for demonstration
const useWebSocket = (url) => {
const [messages, setMessages] = useState([]);
useEffect(() => {
// In production, use a robust WebSocket client library
const ws = new WebSocket(url);
ws.onmessage = (event) => {
const message = JSON.parse(event.data);
if (message.type === 'ip_blocked') {
setMessages(prev => [message.payload, ...prev].slice(0, 50)); // Keep last 50
}
};
return () => ws.close();
}, [url]);
return messages;
};
export default function LiveBlockedFeed() {
// The control plane would expose a WebSocket endpoint for real-time events.
const blockedEvents = useWebSocket('ws://controller.example.com/ws/events');
return (
<div className="feed-container">
<h3>Live Block Events</h3>
<ul>
{blockedEvents.map((event, index) => (
<li key={index}>
<span>{new Date(event.timestamp).toLocaleTimeString()}</span>
<strong>{event.ip}</strong>
<span>blocked due to rule: {event.rule}</span>
</li>
))}
</ul>
</div>
);
}The true value of Turbopack wasn’t in the final built asset, but in the velocity it enabled during the high-stress, iterative process of responding to a security incident.
The current system isn’t without its limitations. The PromQL rules are effective for volumetric attacks but less so for detecting nuanced business logic abuse. A future iteration will involve piping the metrics from our Go agent into a stream processing system like Flink to build a stateful model for more complex anomaly detection. Furthermore, the eBPF blocklist map has a finite size; a distributed denial-of-service attack with millions of source IPs could exhaust it. We are investigating more advanced data structures like counting Bloom filters within eBPF to handle this scale more gracefully without storing every single IP. Finally, the control plane is a centralized component that requires its own high-availability architecture to prevent it from becoming a single point of failure.
