Server-Side Matplotlib Rendering for High-Frequency Data Visualization with Client-Side Zustand State Hydration

The initial requirement seemed straightforward: display high-resolution time-series data from our monitoring database in a web interface. The reality was a cascade of performance failures. Our database, a TimescaleDB instance, held billions of data points. Any attempt to send a raw dataset for a meaningful time window (e.g., a few days of second-level granularity) to the client resulted in multi-megabyte JSON payloads that would lock up the browser tab during parsing, let alone rendering with a library like Chart.js or D3. The DOM simply cannot handle rendering and managing event listeners for millions of individual elements. This client-side rendering approach was a dead end.

Our second attempt involved server-side aggregation. While this reduced the data volume, it destroyed the fidelity of the visualization. Users needed to zoom in and see the raw, high-frequency signal, not a pre-canned, 100-point summary. The core problem remained: how to deliver the visual fidelity of a massive dataset without overwhelming the browser. This led to a counter-intuitive proposal: what if the server renders the entire visualization, not as a static image, but as a structured, interactive vector graphic (SVG)? The browser’s job would then shift from rendering data points to simply displaying a pre-rendered SVG and managing interactions. This is the build log of that system.

The Architectural Concept: SVG as the API Contract

The finalized architecture hinges on a Python backend and a React frontend communicating over WebSockets.

1. Backend (Python/FastAPI):

   • Receives a request from the client specifying a time range and a set of metrics.
   • Queries the TimescaleDB, performing intelligent, resolution-aware downsampling using functions like time_bucket. This is critical; we don’t query millions of raw points if the final image is only 800 pixels wide.
   • Uses Matplotlib’s object-oriented API to programmatically construct a plot from the downsampled data.
   • During plot construction, it injects unique identifiers (gid) into the core SVG elements (e.g., line segments, markers). This is the key to enabling frontend interactivity.
   • Serializes the final Matplotlib figure into an in-memory SVG string.
   • Pushes this SVG string over a WebSocket to the specific client that requested it.

2. Frontend (React/Zustand):

   • Manages the WebSocket connection.
   • Stores the incoming SVG string in a Zustand store.
   • A React component renders this SVG using dangerouslySetInnerHTML.
   • A useEffect hook attaches delegated event listeners (e.g., click, mouseover) to the SVG’s container element. Event delegation is used to avoid attaching thousands of individual listeners.
   • When an event occurs, the handler inspects the event.target to retrieve the unique gid we embedded on the backend.
   • This gid is used to update the application state via Zustand, for example, to display details about the selected data point. The key benefit is that only components subscribed to this specific part of the state (e.g., a tooltip) will re-render, not the entire SVG.

This model treats the SVG not as an image, but as a serialized, interactive scene graph.

sequenceDiagram
    participant Client (React/Zustand)
    participant Server (FastAPI/Matplotlib)
    participant DB (TimescaleDB)

    Client->>Server: open WebSocket connection
    Client->>Server: send({ timeRange: ['t0', 't1'], resolution: 800px })
    Server->>DB: SELECT time_bucket('15s', ts) AS bucket, avg(value) FROM metrics WHERE ts BETWEEN 't0' AND 't1' GROUP BY bucket ORDER BY bucket;
    DB-->>Server: return downsampled data
    Server->>Server: Matplotlib generates SVG with unique IDs for data points
    Server-->>Client: push(svg_string)
    
    Client->>Client: Zustand store receives SVG string
    Client->>Client: React component renders the SVG
    
    Note right of Client: User clicks on a point in the SVG
    
    Client->>Client: Event handler extracts point ID from SVG element
    Client->>Client: Zustand action updates selectedPoint state
    Client->>Client: Tooltip component, subscribed to selectedPoint, re-renders

Backend Implementation: The Data-to-SVG Pipeline

The backend is built with FastAPI for its excellent asynchronous and WebSocket support.

1. Configuration and Database Connection

A production-grade service starts with solid configuration. We’ll use Pydantic for settings management and establish a robust connection pool.

# src/config.py
import os
from pydantic_settings import BaseSettings

class Settings(BaseSettings):
    """
    Application settings loaded from environment variables.
    """
    DB_HOST: str = "localhost"
    DB_PORT: int = 5432
    DB_USER: str = "user"
    DB_PASSWORD: str = "password"
    DB_NAME: str = "metrics_db"
    
    # Connection pool settings
    DB_POOL_MIN_SIZE: int = 2
    DB_POOL_MAX_SIZE: int = 10

    @property
    def database_url(self) -> str:
        return f"postgresql://{self.DB_USER}:{self.DB_PASSWORD}@{self.DB_HOST}:{self.DB_PORT}/{self.DB_NAME}"

settings = Settings()

The database connection manager handles the lifecycle of the connection pool, ensuring we don’t exhaust connections under load.

# src/db.py
import asyncpg
import logging
from contextlib import asynccontextmanager
from .config import settings

logger = logging.getLogger(__name__)

class DatabaseManager:
    """
    Manages the asyncpg connection pool.
    """
    _pool: asyncpg.Pool = None

    @classmethod
    async def get_pool(cls) -> asyncpg.Pool:
        if cls._pool is None:
            logger.info("Creating new database connection pool.")
            try:
                cls._pool = await asyncpg.create_pool(
                    dsn=settings.database_url,
                    min_size=settings.DB_POOL_MIN_SIZE,
                    max_size=settings.DB_POOL_MAX_SIZE
                )
            except Exception as e:
                logger.critical(f"Failed to create database pool: {e}")
                raise
        return cls._pool

    @classmethod
    async def close_pool(cls):
        if cls._pool:
            logger.info("Closing database connection pool.")
            await cls._pool.close()
            cls._pool = None

@asynccontextmanager
async def get_db_connection():
    pool = await DatabaseManager.get_pool()
    conn = None
    try:
        conn = await pool.acquire()
        yield conn
    except Exception as e:
        logger.error(f"Error acquiring DB connection: {e}")
        raise
    finally:
        if conn:
            await pool.release(conn)

2. Data Retrieval and Downsampling

The data access layer is where the core optimization happens. Instead of fetching raw data, we bucket it. The bucket size is dynamically calculated based on the requested time range and the target resolution (e.g., the pixel width of the chart on the client). A common mistake is to hardcode the bucket interval.

# src/data_service.py
import datetime
import logging
from typing import List, Dict, Any
from .db import get_db_connection

logger = logging.getLogger(__name__)

class TimeSeriesService:
    @staticmethod
    def _calculate_bucket_interval(start_time: datetime.datetime, end_time: datetime.datetime, resolution_pixels: int) -> str:
        """
        Dynamically calculates an appropriate time_bucket interval.
        A real-world project would have more sophisticated logic here.
        """
        duration_seconds = (end_time - start_time).total_seconds()
        seconds_per_pixel = duration_seconds / resolution_pixels
        
        if seconds_per_pixel <= 10:
            return "1 second"
        elif seconds_per_pixel <= 60:
            return "10 seconds"
        elif seconds_per_pixel <= 300:
            return "1 minute"
        elif seconds_per_pixel <= 3600:
            return "5 minutes"
        else:
            return "1 hour"

    async def get_downsampled_metric(
        self,
        metric_name: str,
        start_time: datetime.datetime,
        end_time: datetime.datetime,
        resolution: int = 800
    ) -> List[Dict[str, Any]]:
        """
        Fetches downsampled time-series data from TimescaleDB.
        """
        interval = self._calculate_bucket_interval(start_time, end_time, resolution)
        query = """
            SELECT
                time_bucket($1::interval, ts) AS bucket,
                AVG(value) AS avg_value,
                MAX(value) AS max_value,
                MIN(value) AS min_value
            FROM metrics
            WHERE metric_name = $2 AND ts BETWEEN $3 AND $4
            GROUP BY bucket
            ORDER BY bucket ASC;
        """
        try:
            async with get_db_connection() as conn:
                records = await conn.fetch(query, interval, metric_name, start_time, end_time)
                # Convert asyncpg.Record to a list of dicts for easier processing
                return [dict(record) for record in records]
        except Exception as e:
            logger.error(f"Database query failed for metric '{metric_name}': {e}")
            return []

3. Matplotlib SVG Generation

This is the heart of the backend. We use Matplotlib’s object-oriented API for precise control. The critical step is setting the gid (group identifier) on plotted elements. This gid will be our hook for frontend interaction.

# src/plot_service.py
import io
import logging
import datetime
from typing import List, Dict, Any

# Configure Matplotlib for a headless environment
import matplotlib
matplotlib.use('Agg')
import matplotlib.pyplot as plt
import matplotlib.dates as mdates

logger = logging.getLogger(__name__)

class PlottingService:
    
    @staticmethod
    def generate_timeseries_svg(data: List[Dict[str, Any]], metric_name: str) -> str:
        """
        Generates an SVG string from time-series data using Matplotlib.
        Each point is given a unique group ID (gid) for frontend interactivity.
        """
        if not data:
            logger.warning(f"No data provided for metric '{metric_name}' to plot.")
            return "<svg width='800' height='400'><text x='50%' y='50%' dominant-baseline='middle' text-anchor='middle'>No data available for the selected range.</text></svg>"

        timestamps = [d['bucket'] for d in data]
        avg_values = [d['avg_value'] for d in data]

        fig, ax = plt.subplots(figsize=(10, 5), dpi=96)
        fig.patch.set_facecolor('#1a1a1a')
        ax.set_facecolor('#1a1a1a')

        # Plot the main line
        ax.plot(timestamps, avg_values, color='#00aaff', linewidth=1.5, alpha=0.9)

        # Plot individual markers with unique GIDs
        # In a real-world project, you might only plot markers at a certain density
        # to avoid overly large SVG files.
        for i, (ts, val) in enumerate(zip(timestamps, avg_values)):
            point_gid = f"{metric_name}_{i}"
            # The 'gid' attribute is directly translated to the SVG 'id' attribute
            ax.plot(ts, val, 'o', markersize=4, color='#ffaa00', gid=point_gid)

        # Formatting
        ax.tick_params(axis='x', colors='white')
        ax.tick_params(axis='y', colors='white')
        ax.spines['bottom'].set_color('gray')
        ax.spines['left'].set_color('gray')
        ax.spines['top'].set_visible(False)
        ax.spines['right'].set_visible(False)
        
        ax.set_title(f"Time Series for {metric_name}", color='white')
        ax.set_ylabel("Value", color='white')
        
        fig.autofmt_xdate()
        ax.xaxis.set_major_formatter(mdates.DateFormatter('%H:%M:%S\n%Y-%m-%d'))

        plt.tight_layout()

        # Serialize to an in-memory string buffer
        svg_buffer = io.StringIO()
        fig.savefig(svg_buffer, format='svg', facecolor=fig.get_facecolor(), edgecolor='none')
        
        # VERY IMPORTANT: Close the figure to free up memory.
        # A common mistake in long-running services is leaking Matplotlib figures.
        plt.close(fig)

        svg_buffer.seek(0)
        return svg_buffer.getvalue()

4. WebSocket Endpoint

The FastAPI endpoint orchestrates the flow. It handles client connections, processes incoming requests, and pushes the generated SVG.

# src/main.py
import asyncio
import logging
import json
import datetime
from fastapi import FastAPI, WebSocket, WebSocketDisconnect
from .db import DatabaseManager
from .data_service import TimeSeriesService
from .plot_service import PlottingService

# --- Basic Logging Setup ---
logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(name)s - %(levelname)s - %(message)s')
logger = logging.getLogger(__name__)

app = FastAPI(
    on_startup=[DatabaseManager.get_pool],
    on_shutdown=[DatabaseManager.close_pool]
)

@app.websocket("/ws/metrics")
async def websocket_endpoint(websocket: WebSocket):
    await websocket.accept()
    logger.info(f"WebSocket connection accepted from {websocket.client.host}")
    try:
        while True:
            # A production system would use a more robust message format (e.g., Pydantic model)
            message = await websocket.receive_text()
            try:
                payload = json.loads(message)
                metric_name = payload.get("metric")
                # ISO 8601 format is expected from the client
                start_time_str = payload.get("start")
                end_time_str = payload.get("end")
                resolution = payload.get("resolution", 800)

                if not all([metric_name, start_time_str, end_time_str]):
                    await websocket.send_text(json.dumps({"error": "Missing required parameters."}))
                    continue
                
                start_time = datetime.datetime.fromisoformat(start_time_str)
                end_time = datetime.datetime.fromisoformat(end_time_str)

                # 1. Fetch data
                data = await TimeSeriesService().get_downsampled_metric(metric_name, start_time, end_time, resolution)
                
                # 2. Generate plot
                svg_content = PlottingService.generate_timeseries_svg(data, metric_name)

                # 3. Push to client
                await websocket.send_text(json.dumps({"type": "chart_update", "svg": svg_content}))

            except json.JSONDecodeError:
                logger.warning("Received invalid JSON message.")
                await websocket.send_text(json.dumps({"error": "Invalid JSON format."}))
            except Exception as e:
                logger.error(f"An error occurred while processing WebSocket message: {e}", exc_info=True)
                await websocket.send_text(json.dumps({"error": "An internal server error occurred."}))

    except WebSocketDisconnect:
        logger.info(f"WebSocket connection closed for {websocket.client.host}")
    except Exception as e:
        logger.error(f"Unhandled exception in WebSocket handler: {e}", exc_info=True)

Frontend Implementation: State Management with Zustand

The frontend’s primary responsibility is efficient state management and interaction.

1. Zustand Store

The store is minimal. It holds the SVG content and information about the user’s current selection. We use Zustand because its selector-based model prevents unnecessary re-renders. When a user hovers over a point, we don’t want the entire SVG component to re-render; we only want the tooltip component to update.

// src/stores/chartStore.ts
import { create } from 'zustand';

interface ChartState {
  svgContent: string | null;
  selectedPointId: string | null;
  isLoading: boolean;
  error: string | null;
}

interface ChartActions {
  setSvgContent: (svg: string) => void;
  setSelectedPointId: (id: string | null) => void;
  setLoading: (loading: boolean) => void;
  setError: (error: string | null) => void;
}

export const useChartStore = create<ChartState & ChartActions>((set) => ({
  svgContent: null,
  selectedPointId: null,
  isLoading: false,
  error: null,
  setSvgContent: (svg) => set({ svgContent: svg, isLoading: false, error: null }),
  setSelectedPointId: (id) => set({ selectedPointId: id }),
  setLoading: (loading) => set({ isLoading: loading }),
  setError: (error) => set({ error: error, isLoading: false }),
}));

2. WebSocket Service Hook

A custom hook encapsulates the WebSocket logic, connecting it to our Zustand store.

// src/hooks/useChartWebSocket.ts
import { useEffect, useRef } from 'react';
import { useChartStore } from '../stores/chartStore';

const WEBSOCKET_URL = 'ws://localhost:8000/ws/metrics';

export const useChartWebSocket = () => {
  const ws = useRef<WebSocket | null>(null);
  const { setSvgContent, setLoading, setError } = useChartStore();

  useEffect(() => {
    ws.current = new WebSocket(WEBSOCKET_URL);

    ws.current.onopen = () => {
      console.log('WebSocket connected');
    };

    ws.current.onmessage = (event) => {
      try {
        const data = JSON.parse(event.data);
        if (data.error) {
          setError(data.error);
        } else if (data.type === 'chart_update' && data.svg) {
          setSvgContent(data.svg);
        }
      } catch (e) {
        setError('Failed to parse message from server.');
      }
    };

    ws.current.onerror = () => {
      setError('WebSocket connection error.');
    };

    ws.current.onclose = () => {
      console.log('WebSocket disconnected');
    };

    return () => {
      ws.current?.close();
    };
  }, [setSvgContent, setError]);

  const requestChartUpdate = (metric: string, start: string, end: string, resolution: number) => {
    if (ws.current?.readyState === WebSocket.OPEN) {
      setLoading(true);
      const payload = JSON.stringify({ metric, start, end, resolution });
      ws.current.send(payload);
    } else {
      setError('WebSocket is not connected.');
    }
  };

  return { requestChartUpdate };
};

3. React Components

The SVG rendering component is the most interesting part. It uses dangerouslySetInnerHTML which is often discouraged, but it’s the correct tool for this job since we are injecting a full, self-contained SVG document generated by our trusted backend. We attach a single mouseover event listener to the container.

// src/components/InteractiveChart.tsx
import React, { useEffect, useRef } from 'react';
import { useChartStore } from '../stores/chartStore';

export const InteractiveChart: React.FC = () => {
  const svgContent = useChartStore((state) => state.svgContent);
  const isLoading = useChartStore((state) => state.isLoading);
  const setSelectedPointId = useChartStore((state) => state.actions.setSelectedPointId);
  const containerRef = useRef<HTMLDivElement>(null);

  useEffect(() => {
    const container = containerRef.current;
    if (!container) return;

    const handleMouseOver = (event: MouseEvent) => {
      const target = event.target as SVGElement;
      // We look for the 'id' attribute, which Matplotlib populates from the 'gid'
      const pointId = target.id;
      
      if (pointId && pointId.startsWith('cpu_usage_')) { // Filter for our specific metric points
        setSelectedPointId(pointId);
      }
    };
    
    const handleMouseOut = () => {
        setSelectedPointId(null);
    }

    // Event delegation: attach one listener to the parent
    container.addEventListener('mouseover', handleMouseOver);
    container.addEventListener('mouseout', handleMouseOut);

    return () => {
      container.removeEventListener('mouseover', handleMouseOver);
      container.removeEventListener('mouseout', handleMouseOut);
    };
  }, [setSelectedPointId]);

  if (isLoading) {
    return <div>Loading chart...</div>;
  }

  return (
    <div
      ref={containerRef}
      dangerouslySetInnerHTML={{ __html: svgContent || '' }}
      style={{ width: '100%', userSelect: 'none' }}
    />
  );
};

The tooltip component subscribes to a very small slice of the state. This is extremely efficient.

// src/components/ChartTooltip.tsx
import React from 'react';
import { useChartStore } from '../stores/chartStore';

export const ChartTooltip: React.FC = () => {
  // This component ONLY re-renders when `selectedPointId` changes.
  const selectedPointId = useChartStore((state) => state.selectedPointId);

  if (!selectedPointId) {
    return null;
  }

  return (
    <div style={{ position: 'fixed', top: 20, right: 20, background: 'rgba(0,0,0,0.7)', color: 'white', padding: '10px', borderRadius: '5px' }}>
      Selected Point ID: {selectedPointId}
    </div>
  );
};

Limitations and Future Considerations

This architecture, while effective, is not a silver bullet. The primary limitation is the size of the SVG payload. For extremely dense or complex plots, the SVG string itself can become large, reintroducing network latency as a bottleneck. A future iteration could explore sending a more compact data representation (e.g., binary packed coordinates) and a separate, one-time payload for the chart’s chrome (axes, labels), reconstructing the plot client-side with a more performant renderer like WebGL.

Furthermore, the interactivity model is currently limited to identifying elements. More complex interactions like zoom-and-pan would require a bidirectional communication flow where the client sends viewport changes back to the server, which then generates a new, appropriately downsampled SVG for that specific window. This introduces state synchronization challenges and requires a more sophisticated WebSocket protocol. Finally, the backend’s memory management for Matplotlib figures is critical; while plt.close(fig) is used, under extreme concurrent load, a more robust object pooling or process-based isolation strategy for the plotting service might be necessary to ensure long-term stability.