The technical pain point emerged from a seemingly simple requirement in our multi-tenant data processing platform. The platform, running entirely within a secure VPC, needed to allow individual tenants to apply custom logic to their data streams. The transformations were varied and specific: PII redaction based on tenant-specific rules, data enrichment from proprietary lookups, or reformatting payloads for legacy downstream systems. Our initial approach of having engineering teams deploy bespoke microservices for each new transformation rule was not scaling. The lead time was weeks, and the operational overhead of maintaining dozens of near-identical services was becoming a significant drain on resources.
An alternative was to let tenants run their own code directly on our processing nodes. This idea was dismissed almost immediately. The security implications of executing untrusted, third-party code on shared infrastructure are unacceptable in any serious production environment. The potential for resource exhaustion, malicious system calls, or data exfiltration was a non-starter. We briefly considered containerization—packaging each tenant’s logic into a Docker container. While this provides strong isolation, the resource footprint and cold-start latency for what were often sub-second transformation tasks felt disproportionately heavy. We needed something with the isolation of a container but the startup speed and lightweight footprint of a function call. This led us to WebAssembly.
WASM’s sandboxed execution model, near-native performance, and language-agnostic compilation target made it a compelling candidate. We could build a single, robust host service in Go that would securely load and execute tenant-provided WASM modules. Go was a natural choice for its performance, strong concurrency primitives, and the availability of mature, production-ready WASM runtimes like wazero. To expose the management and execution capabilities, a lightweight API was necessary, and Gin provided the performance and simplicity we needed without unnecessary boilerplate. The entire architecture would remain confined within our existing VPC, leveraging its network isolation as a foundational security layer. This post-mortem documents the build process, the critical design decisions, and the code that formed the core of this secure WASM plugin service.
Defining the Host-Guest Contract (ABI)
The first and most critical step in a host-guest system is defining a rigid, unambiguous contract, or Application Binary Interface (ABI). A common mistake is to create a chatty interface with many exported functions, which increases the surface area for bugs and complicates maintenance. For our data transformation use case, the contract could be simplified to its essence: the host provides a chunk of data (a byte slice), and the guest returns a new, transformed chunk of data.
This requires two core functions to be implemented by the guest WASM module and called by the Go host:
-
allocate(size): The host calls this to request a memory buffer of a specific size within the WASM module’s linear memory. It returns a pointer to the allocated buffer. -
transform(ptr, size): The host calls this after writing data to the allocated buffer. The function performs the transformation and returns a combined 64-bit integer, where the upper 32 bits are the pointer to the result buffer and the lower 32 bits are its size.
To provide a concrete example, we’ll use Rust to build the WASM plugin. Rust’s safety guarantees and excellent WASM toolchain make it an ideal choice.
Here is the Rust implementation of a simple JSON redaction plugin. It accepts a JSON string, redacts a specific field (e.g., “ssn”), and returns the modified string.
plugin/src/lib.rs:
use std::mem;
use serde_json::{Value, from_slice, to_vec};
// A global buffer to hold the result of the transformation.
// In a real-world project, memory management would need to be more robust,
// but for this example, a static buffer simplifies the ABI.
static mut RESULT_BUFFER: Vec<u8> = Vec::new();
/// Allocates a buffer of a given size in the WASM module's linear memory
/// and returns a pointer to it. The host will use this to pass data into the module.
#[no_mangle]
pub extern "C" fn allocate(size: usize) -> *mut u8 {
let mut buffer = Vec::with_capacity(size);
let ptr = buffer.as_mut_ptr();
// Prevent Rust from deallocating the buffer when `buffer` goes out of scope.
// The host is now responsible for this memory region.
mem::forget(buffer);
ptr
}
/// The core transformation logic. This function is the entry point for the plugin.
/// It takes a pointer and size, reconstructs the input data, performs the transformation,
// and returns a pointer/size pair for the result.
#[no_mangle]
pub extern "C" fn transform(ptr: *mut u8, size: usize) -> u64 {
// Reconstruct the input byte slice from the pointer and size provided by the host.
// This is unsafe because we are trusting the host to provide a valid pointer and size.
// The WASM sandbox itself prevents memory access outside the module's linear memory.
let input_data = unsafe { Vec::from_raw_parts(ptr, size, size) };
// Perform the actual business logic: parse JSON and redact a field.
let output_data = match process_json(&input_data) {
Ok(data) => data,
Err(e) => {
// In a production system, you'd want a more sophisticated error reporting mechanism.
// For now, we return an empty buffer on error.
let error_msg = format!("Error: {}", e);
eprintln!("{}", error_msg);
Vec::new()
}
};
let result_ptr;
let result_size = output_data.len();
// Store the result in our static buffer and get a pointer to it.
unsafe {
RESULT_BUFFER = output_data;
result_ptr = RESULT_BUFFER.as_mut_ptr();
}
// Combine pointer and size into a single u64 for the return value.
// Pointer is in the upper 32 bits, size is in the lower 32 bits.
// This is a common ABI pattern to return two values from a WASM function.
(result_ptr as u64) << 32 | (result_size as u64)
}
/// The actual data processing logic. This is kept separate from the ABI functions.
fn process_json(data: &[u8]) -> Result<Vec<u8>, serde_json::Error> {
let mut v: Value = from_slice(data)?;
if let Some(obj) = v.as_object_mut() {
if obj.contains_key("ssn") {
obj["ssn"] = Value::String("REDACTED".to_string());
}
}
to_vec(&v)
}To compile this, the Cargo.toml must specify a crate-type of cdylib, which is necessary for creating a dynamic library suitable for WASM.
plugin/Cargo.toml:
[package]
name = "plugin"
version = "0.1.0"
edition = "2021"
[dependencies]
serde_json = "1.0"
[lib]
crate-type = ["cdylib"]Compile with the command: cargo build --target wasm32-unknown-unknown --release. The output will be target/wasm32-unknown-unknown/release/plugin.wasm.
The Go Host Service Implementation
The host service orchestrates the entire process. It manages plugin lifecycles, handles API requests, and interfaces with the wazero runtime to execute the WASM code.
graph TD
A[API Client] -- HTTP POST /transform --> B(Gin Router);
B -- Request --> C{Plugin Service};
C -- Get Module --> D[Wazero Runtime Cache];
D -- Compiled Module --> C;
C -- Instantiate Module --> E[Wazero Instance];
subgraph "WASM Sandbox"
E -- Calls allocate() --> F[WASM Guest Memory];
C -- Writes Input Data --> F;
E -- Calls transform() --> G[WASM Guest Logic];
G -- Writes Output Data --> F;
E -- Reads Output Data --> F;
end
C -- Transformation Result --> B;
B -- HTTP 200 OK --> A;
1. Project Structure and Configuration
A pragmatic project structure separates concerns.
/wasm-service
|-- /cmd/server/main.go # Application entrypoint
|-- /internal/api # Gin router and handlers
|-- /internal/config # Configuration loading (Viper)
|-- /internal/plugin # Core plugin management logic
| |-- manager.go
| |-- manager_test.go
|-- go.mod
|-- go.sum
|-- config.yaml # Service configurationConfiguration is externalized to a config.yaml file to avoid hardcoding values. In a real-world project, this would be managed by a configuration service or Kubernetes ConfigMaps.
config.yaml:
server:
host: "0.0.0.0"
port: "8080"
logging:
level: "info"
plugins:
# In a production environment, this path would point to a volume
# where uploaded plugins are stored securely.
directory: "./plugins"
# Resource constraints for WASM modules
max_memory_pages: 64 # 64 pages * 64 KiB/page = 4MB2. The Plugin Manager
The PluginManager is the heart of the host. It’s responsible for discovering, compiling, and managing the lifecycle of WASM modules. Using wazero‘s compilation cache is critical for performance; it avoids recompiling the WASM bytecode on every request.
internal/plugin/manager.go:
package plugin
import (
"context"
"crypto/sha256"
"fmt"
"log/slog"
"os"
"path/filepath"
"sync"
"github.com/tetratelabs/wazero"
"github.com/tetratelabs/wazero/api"
"github.com/tetratelabs/wazero/imports/wasi_snapshot_preview1"
)
// Plugin represents a compiled and loaded WASM module.
type Plugin struct {
Module api.Module
Runtime wazero.Runtime
checksum [32]byte
}
// Manager handles the discovery, loading, and execution of WASM plugins.
type Manager struct {
sync.RWMutex
runtime wazero.Runtime
plugins map[string]*Plugin
pluginDir string
maxMemPages uint32
logger *slog.Logger
}
// NewManager creates and initializes a new plugin manager.
// It pre-compiles all valid WASM modules found in the plugin directory.
func NewManager(ctx context.Context, logger *slog.Logger, pluginDir string, maxMemoryPages uint32) (*Manager, error) {
// A common pitfall is to create a new runtime for each request.
// Caching configuration should be shared across the application.
cache := wazero.NewCompilationCache()
runtimeConfig := wazero.NewRuntimeConfig().
WithCompilationCache(cache).
// Enforce a hard memory limit for all modules instantiated from this runtime.
WithMemoryLimitPages(maxMemoryPages)
runtime := wazero.NewRuntimeWithConfig(ctx, runtimeConfig)
// Instantiating WASI is necessary even if not used directly, as many toolchains
// (like Rust's) implicitly link against it. A mistake is to forget this and
// face mysterious instantiation failures.
wasi_snapshot_preview1.MustInstantiate(ctx, runtime)
m := &Manager{
runtime: runtime,
plugins: make(map[string]*Plugin),
pluginDir: pluginDir,
maxMemPages: maxMemoryPages,
logger: logger,
}
if err := m.loadPlugins(ctx); err != nil {
return nil, fmt.Errorf("failed to load plugins: %w", err)
}
return m, nil
}
// loadPlugins scans the plugin directory and compiles all .wasm files.
func (m *Manager) loadPlugins(ctx context.Context) error {
files, err := os.ReadDir(m.pluginDir)
if err != nil {
return fmt.Errorf("could not read plugin directory %s: %w", m.pluginDir, err)
}
for _, file := range files {
if !file.IsDir() && filepath.Ext(file.Name()) == ".wasm" {
pluginID := file.Name()
path := filepath.Join(m.pluginDir, pluginID)
if err := m.loadAndCompilePlugin(ctx, path, pluginID); err != nil {
m.logger.Error("failed to load plugin", "path", path, "error", err)
// Continue loading other plugins
}
}
}
return nil
}
func (m *Manager) loadAndCompilePlugin(ctx context.Context, path, pluginID string) error {
m.logger.Info("Loading plugin", "id", pluginID, "path", path)
wasmBytes, err := os.ReadFile(path)
if err != nil {
return fmt.Errorf("failed to read wasm file %s: %w", path, err)
}
// Compile the module. This is a one-time cost. The result is cached.
compiledModule, err := m.runtime.CompileModule(ctx, wasmBytes)
if err != nil {
return fmt.Errorf("failed to compile wasm module %s: %w", path, err)
}
// A real-world project would use a more robust versioning and identification scheme.
// For now, a SHA256 checksum serves to identify the module's content.
checksum := sha256.Sum256(wasmBytes)
m.Lock()
defer m.Unlock()
m.plugins[pluginID] = &Plugin{
Module: wazero.NewModuleBuilder(m.runtime).Instantiate(ctx, compiledModule),
checksum: checksum,
}
return nil
}
// GetPlugin retrieves a loaded plugin by its ID.
func (m *Manager) GetPlugin(pluginID string) (*Plugin, bool) {
m.RLock()
defer m.RUnlock()
plugin, ok := m.plugins[pluginID]
return plugin, ok
}
// Close gracefully closes the wazero runtime.
func (m *Manager) Close(ctx context.Context) error {
m.Lock()
defer m.Unlock()
return m.runtime.Close(ctx)
}
// Transform executes a plugin's transformation logic on the given data.
func (p *Plugin) Transform(ctx context.Context, data []byte) ([]byte, error) {
// Get references to exported functions. A common mistake is to look these up
// on every call, which adds overhead. Caching them is better if possible.
allocateFunc := p.Module.ExportedFunction("allocate")
transformFunc := p.Module.ExportedFunction("transform")
// 1. Allocate memory in the WASM module for the input data.
inputSize := uint64(len(data))
results, err := allocateFunc.Call(ctx, inputSize)
if err != nil {
return nil, fmt.Errorf("allocate call failed: %w", err)
}
inputPtr := results[0]
// 2. Write the input data to the allocated memory.
if !p.Module.Memory().Write(uint32(inputPtr), data) {
return nil, fmt.Errorf("memory write failed for offset %d size %d", inputPtr, inputSize)
}
// 3. Call the transform function.
results, err = transformFunc.Call(ctx, inputPtr, inputSize)
if err != nil {
return nil, fmt.Errorf("transform call failed: %w", err)
}
outputPacked := results[0]
// Deconstruct the packed u64 result into pointer and size.
outputPtr := uint32(outputPacked >> 32)
outputSize := uint32(outputPacked)
// 4. Read the transformed data from the WASM module's memory.
output, ok := p.Module.Memory().Read(outputPtr, outputSize)
if !ok {
return nil, fmt.Errorf("memory read failed for offset %d size %d", outputPtr, outputSize)
}
// Create a copy of the output buffer. The WASM module's memory might be invalidated
// on subsequent calls, so we must not hold onto a slice pointing into it.
result := make([]byte, outputSize)
copy(result, output)
return result, nil
}3. The Gin API Layer
The API layer is intentionally thin. Its job is to handle HTTP requests, validate inputs, and delegate to the PluginManager.
internal/api/handlers.go:
package api
import (
"io"
"log/slog"
"net/http"
"github.com/gin-gonic/gin"
"wasm-service/internal/plugin"
)
type TransformHandler struct {
manager *plugin.Manager
logger *slog.Logger
}
func NewTransformHandler(manager *plugin.Manager, logger *slog.Logger) *TransformHandler {
return &TransformHandler{manager: manager, logger: logger}
}
func (h *TransformHandler) RegisterRoutes(router *gin.Engine) {
router.POST("/transform/:pluginID", h.handleTransform)
}
func (h *TransformHandler) handleTransform(c *gin.Context) {
pluginID := c.Param("pluginID")
p, ok := h.manager.GetPlugin(pluginID)
if !ok {
h.logger.Warn("Plugin not found", "pluginID", pluginID)
c.JSON(http.StatusNotFound, gin.H{"error": "plugin not found"})
return
}
inputData, err := io.ReadAll(c.Request.Body)
if err != nil {
h.logger.Error("Failed to read request body", "error", err)
c.JSON(http.StatusBadRequest, gin.H{"error": "invalid request body"})
return
}
// The context from Gin should be passed down to the WASM execution.
// This allows for cancellation or timeouts to propagate.
ctx := c.Request.Context()
outputData, err := p.Transform(ctx, inputData)
if err != nil {
h.logger.Error("Plugin transformation failed", "pluginID", pluginID, "error", err)
c.JSON(http.StatusInternalServerError, gin.H{"error": "plugin execution failed"})
return
}
c.Data(http.StatusOK, "application/json", outputData)
}
The main.go ties everything together.
cmd/server/main.go:
package main
import (
"context"
"log/slog"
"os"
"github.com/gin-gonic/gin"
"github.com/spf13/viper"
"wasm-service/internal/api"
"wasm-service/internal/config"
"wasm-service/internal/plugin"
)
func main() {
// Setup structured logging
logger := slog.New(slog.NewJSONHandler(os.Stdout, nil))
// Load configuration
cfg, err := config.LoadConfig(".") // Assumes config.yaml is in the root
if err != nil {
logger.Error("Cannot load config", "error", err)
os.Exit(1)
}
ctx := context.Background()
// Initialize the plugin manager
pluginManager, err := plugin.NewManager(ctx, logger, cfg.Plugins.Directory, cfg.Plugins.MaxMemoryPages)
if err != nil {
logger.Error("Failed to initialize plugin manager", "error", err)
os.Exit(1)
}
defer pluginManager.Close(ctx)
// Setup Gin router
router := gin.Default()
transformHandler := api.NewTransformHandler(pluginManager, logger)
transformHandler.RegisterRoutes(router)
addr := cfg.Server.Host + ":" + cfg.Server.Port
logger.Info("Starting server", "address", addr)
if err := router.Run(addr); err != nil {
logger.Error("Failed to start server", "error", err)
os.Exit(1)
}
}Testing the Implementation
In a real-world project, testing is non-negotiable. Untested code is broken code. We need both unit tests for isolated components and integration tests for the end-to-end flow.
Here’s an example of an integration test for the transform handler using Go’s httptest package. This test compiles the Rust plugin, starts an in-memory version of our Gin server, and makes a real HTTP request to it.
internal/api/handlers_test.go:
package api
import (
"bytes"
"context"
"log/slog"
"net/http"
"net/http/httptest"
"os"
"os/exec"
"path/filepath"
"strings"
"testing"
"github.com/gin-gonic/gin"
"github.com/stretchr/testify/require"
"wasm-service/internal/plugin"
)
// compileTestPlugin is a helper function to compile the Rust plugin for testing.
func compileTestPlugin(t *testing.T) string {
t.Helper()
pluginDir := "../../plugin" // Relative path to the Rust project
cmd := exec.Command("cargo", "build", "--target", "wasm32-unknown-unknown", "--release")
cmd.Dir = pluginDir
output, err := cmd.CombinedOutput()
require.NoError(t, err, "failed to compile wasm plugin: %s", string(output))
return filepath.Join(pluginDir, "target/wasm32-unknown-unknown/release/plugin.wasm")
}
func TestTransformHandler_Integration(t *testing.T) {
// 1. Setup: Compile plugin and create a temporary directory for it
wasmPath := compileTestPlugin(t)
tmpDir := t.TempDir()
// The plugin file name must match the ID used in the URL
destPath := filepath.Join(tmpDir, "redact.wasm")
data, err := os.ReadFile(wasmPath)
require.NoError(t, err)
err = os.WriteFile(destPath, data, 0644)
require.NoError(t, err)
// 2. Initialize the server components
logger := slog.New(slog.NewTextHandler(os.Stdout, &slog.HandlerOptions{Level: slog.LevelDebug}))
manager, err := plugin.NewManager(context.Background(), logger, tmpDir, 64)
require.NoError(t, err)
defer manager.Close(context.Background())
gin.SetMode(gin.TestMode)
router := gin.New()
handler := NewTransformHandler(manager, logger)
handler.RegisterRoutes(router)
// 3. Execute the test case
t.Run("successful transformation", func(t *testing.T) {
inputJSON := `{"user": "test", "ssn": "000-00-0000", "data": [1,2,3]}`
expectedJSON := `{"data":[1,2,3],"ssn":"REDACTED","user":"test"}` // Note: key order can change
req, _ := http.NewRequest(http.MethodPost, "/transform/redact.wasm", bytes.NewBufferString(inputJSON))
req.Header.Set("Content-Type", "application/json")
rr := httptest.NewRecorder()
router.ServeHTTP(rr, req)
require.Equal(t, http.StatusOK, rr.Code)
// We unmarshal and remarshal to ignore key order differences in the JSON
require.JSONEq(t, expectedJSON, rr.Body.String())
})
t.Run("plugin not found", func(t *testing.T) {
req, _ := http.NewRequest(http.MethodPost, "/transform/nonexistent.wasm", strings.NewReader("{}"))
rr := httptest.NewRecorder()
router.ServeHTTP(rr, req)
require.Equal(t, http.StatusNotFound, rr.Code)
})
}Final Result and VPC Context
With the service built and tested, let’s place it back into its intended environment: the VPC. The Go service runs on an EC2 instance or in an ECS/EKS container. A critical aspect of the design is that this service should not be exposed to the public internet. Its API is internal.
- Security Groups: The instance/container running the service would be in a Security Group that only allows inbound traffic on port 8080 from other specific Security Groups within the VPC (e.g., the application servers that need to perform transformations). All other inbound traffic is denied.
- Network ACLs: At the subnet level, NACLs can provide a stateless firewall layer, further restricting traffic to known internal IP ranges.
- IAM Roles: The instance should have a minimal IAM Role attached, granting it only the permissions it needs (e.g., to read plugins from an S3 bucket, if we were to extend it).
To use the service, an upstream application would make a simple HTTP call:
curl -X POST http://<internal-service-ip>:8080/transform/redact.wasm \
-H "Content-Type: application/json" \
--data '{"user": "alice", "ssn": "123-456-7890", "payload": "sensitive-info"}'The response would be the transformed JSON payload, computed securely and efficiently inside the WASM sandbox.
The primary limitation of this implementation is its simplistic plugin management. In a production system, plugins wouldn’t be loaded from a local directory. They would be stored in a versioned artifact repository or an object store like S3, and the service would need a secure mechanism to fetch and update them dynamically. Furthermore, the ABI is rudimentary; it forces all data through byte slices, which requires serialization and deserialization on both sides. For more complex data structures, interface-type specifications like WIT (Wasm Interface Types) could provide a more efficient and type-safe contract. Finally, while we’ve set a memory limit, true defense against denial-of-service attacks would require “fueling” or metering execution—terminating modules that run for too long—a feature available in some WASM runtimes that adds complexity but is crucial for multi-tenant stability.
