Constructing a Meta-Programmable UI Engine with Domain-Driven Design, Dgraph, etcd, and Dynamic Rollup Bundling

The mandate was to build a low-code platform, but with a critical constraint: the platform itself had to be evolvable without full-scale redeployments. Standard low-code tools offer a fixed palette of widgets and layouts. Our requirement was to allow platform engineers to introduce entirely new component types—a sophisticated data grid, a new charting library—and have them become available to application builders in real-time. This meant the definition of the platform’s capabilities couldn’t be baked into the application binary. It had to be live, distributed, and consistent. This was our primary technical pain point.

Our initial concept broke the system into three distinct logical layers. First, a Meta-Model that defines the “building blocks” of the platform: the available ComponentDefinitions, their configurable properties, and validation rules. Second, an Application Model representing the user-built applications: a graph of WidgetInstances connected to DataSources and arranged in Layouts. Finally, a Composition Engine that could take an Application Model, resolve its components against the live Meta-Model, and produce a runnable, optimized front-end artifact. This separation was the key, but it introduced significant technical challenges for state management and asset generation.

The technology selection process was rigorous and driven by these layers. For modeling the complex domains of both the platform’s structure and the user’s applications, Domain-Driven Design (DDD) was a natural fit. It forced us to define clear Bounded Contexts: a PlatformContext for the Meta-Model and a TenantApplicationContext for the user-generated graphs.

For the Application Model, the choice was Dgraph. User interfaces are inherently graphs. A page contains layouts, which contain widgets, which are bound to data sources. Representing this in a relational database would lead to a nightmare of join tables and recursive queries. A graph database lets us model these relationships natively. We could ask questions like “Find all widgets dependent on this specific API endpoint” with a simple, performant query.

The most critical decision was how to manage the Meta-Model. We needed a source of truth that was highly available, strictly consistent, and offered a mechanism to push updates to all services instantly. A database with a caching layer and a message queue for invalidation felt clumsy. This is where etcd came in. It’s designed for exactly this use case: managing distributed configuration and service discovery. We decided to store our ComponentDefinitions directly in etcd. Any service that needed to know about available components could place a watch on a key prefix, receiving immediate notifications of any changes.

Finally, the Composition Engine posed a unique problem. If new front-end components could be introduced at any time, we couldn’t pre-bundle all possible combinations. We needed to perform bundling on-demand, or just-in-time. This led us to the unconventional choice of using Rollup’s JavaScript API on the server side. A dedicated service would be responsible for taking an application graph, generating a dynamic entry point, and programmatically invoking Rollup to create a lean, code-split bundle for that specific application view.

Domain Modeling and Core Structures

Before writing a line of service code, we used DDD to outline our core aggregates and entities in Go. This clarity was essential to prevent the two contexts from bleeding into each other.

In the PlatformContext, the central aggregate is ComponentDefinition.

// platform/domain/component_definition.go
package domain

import (
	"encoding/json"
	"errors"
	"time"
)

// PropertyType defines the type of a configurable property for a component.
type PropertyType string

const (
	StringType  PropertyType = "string"
	NumberType  PropertyType = "number"
	BooleanType PropertyType = "boolean"
	ObjectType  PropertyType = "object"
)

// ComponentProperty defines a single configurable attribute of a component.
type ComponentProperty struct {
	Name         string       `json:"name"`
	Type         PropertyType `json:"type"`
	Required     bool         `json:"required"`
	DefaultValue interface{}  `json:"defaultValue,omitempty"`
}

// ComponentDefinition is the aggregate root for a platform-level component.
// It defines the schema and metadata for a type of widget that can be used.
type ComponentDefinition struct {
	ID                string              `json:"id"` // e.g., "material-ui-button"
	Version           string              `json:"version"`
	DisplayName       string              `json:"displayName"`
	Description       string              `json:"description"`
	SourceURL         string              `json:"sourceUrl"` // URL to the component's JS module
	Properties        []ComponentProperty `json:"properties"`
	CreatedAt         time.Time           `json:"createdAt"`
	UpdatedAt         time.Time           `json:"updatedAt"`
}

// NewComponentDefinition creates a new, valid component definition.
func NewComponentDefinition(id, version, displayName, sourceURL string) (*ComponentDefinition, error) {
	if id == "" || version == "" || displayName == "" || sourceURL == "" {
		return nil, errors.New("id, version, displayName, and sourceURL are required")
	}
	return &ComponentDefinition{
		ID:          id,
		Version:     version,
		DisplayName: displayName,
		SourceURL:   sourceURL,
		Properties:  make([]ComponentProperty, 0),
		CreatedAt:   time.Now().UTC(),
		UpdatedAt:   time.Now().UTC(),
	}, nil
}

// ToJSON serializes the definition to its JSON representation for storage in etcd.
func (cd *ComponentDefinition) ToJSON() ([]byte, error) {
	return json.Marshal(cd)
}

// FromJSON deserializes a definition from JSON.
func ComponentDefinitionFromJSON(data []byte) (*ComponentDefinition, error) {
	var cd ComponentDefinition
	if err := json.Unmarshal(data, &cd); err != nil {
		return nil, err
	}
	return &cd, nil
}

In the TenantApplicationContext, the aggregate is the ApplicationGraph, which contains entities like WidgetInstance.

// application/domain/application_graph.go
package domain

import (
	"errors"
)

// WidgetInstance represents a specific usage of a ComponentDefinition within an application.
// It's an entity within the ApplicationGraph aggregate.
type WidgetInstance struct {
	UID              string                 `json:"uid,omitempty"` // UID from Dgraph
	InstanceID       string                 `json:"instanceId"`
	ComponentID      string                 `json:"componentId"` // Foreign key to ComponentDefinition.ID
	ComponentVersion string                 `json:"componentVersion"`
	Properties       map[string]interface{} `json:"properties"` // User-configured values
	Children         []*WidgetInstance      `json:"children,omitempty"`
}

// ApplicationGraph is the aggregate root for a user-created application.
type ApplicationGraph struct {
	UID         string            `json:"uid,omitempty"`
	ApplicationID string          `json:"applicationId"`
	Name        string            `json:"name"`
	Root        *WidgetInstance   `json:"root"`
}

func NewApplicationGraph(appID, name string, root *WidgetInstance) (*ApplicationGraph, error) {
	if appID == "" || name == "" || root == nil {
		return nil, errors.New("applicationId, name, and a root widget are required")
	}
	return &ApplicationGraph{
		ApplicationID: appID,
		Name:          name,
		Root:          root,
	}, nil
}

This clear separation allowed us to reason about the system’s two distinct functions: defining capabilities vs. using them.

The etcd-Backed Component Registry

The heart of the live update mechanism is a service that maintains an in-memory registry of all available ComponentDefinitions. This service connects to etcd and places a watch on the prefix /platform/components/.

Here is a simplified implementation of this registry service in Go. A real-world project would require more robust error handling and reconnection logic.

// infrastructure/etcd_registry.go
package infrastructure

import (
	"context"
	"log"
	"sync"
	"time"

	"github.com/you/project/platform/domain"
	clientv3 "go.etcd.io/etcd/client/v3"
)

const componentPrefix = "/platform/components/"

// ComponentRegistry provides a thread-safe, in-memory cache of ComponentDefinitions.
// It is kept in sync with etcd via a background watch.
type ComponentRegistry struct {
	client *clientv3.Client
	log    *log.Logger

	mu         sync.RWMutex
	components map[string]*domain.ComponentDefinition
}

// NewComponentRegistry creates and initializes the registry.
func NewComponentRegistry(endpoints []string, logger *log.Logger) (*ComponentRegistry, error) {
	cli, err := clientv3.New(clientv3.Config{
		Endpoints:   endpoints,
		DialTimeout: 5 * time.Second,
	})
	if err != nil {
		return nil, err
	}

	registry := &ComponentRegistry{
		client:     cli,
		log:        logger,
		components: make(map[string]*domain.ComponentDefinition),
	}

	// Initial population of the registry
	if err := registry.populate(); err != nil {
		cli.Close()
		return nil, err
	}

	// Start the background watch process
	go registry.watch()

	return registry, nil
}

// populate performs an initial load of all components from etcd.
func (r *ComponentRegistry) populate() error {
	ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
	defer cancel()

	resp, err := r.client.Get(ctx, componentPrefix, clientv3.WithPrefix())
	if err != nil {
		return err
	}

	r.mu.Lock()
	defer r.mu.Unlock()

	for _, kv := range resp.Kvs {
		def, err := domain.ComponentDefinitionFromJSON(kv.Value)
		if err != nil {
			r.log.Printf("Failed to deserialize component %s: %v", string(kv.Key), err)
			continue
		}
		r.components[def.ID] = def
	}
	r.log.Printf("Initial registry population complete. Loaded %d components.", len(r.components))
	return nil
}

// watch listens for changes on the component prefix and updates the in-memory map.
func (r *ComponentRegistry) watch() {
	watchChan := r.client.Watch(context.Background(), componentPrefix, clientv3.WithPrefix())

	for watchResp := range watchChan {
		for _, event := range watchResp.Events {
			r.mu.Lock()
			switch event.Type {
			case clientv3.EventTypePut:
				def, err := domain.ComponentDefinitionFromJSON(event.Kv.Value)
				if err != nil {
					r.log.Printf("Watch Error: Failed to deserialize updated component %s: %v", string(event.Kv.Key), err)
					continue
				}
				r.log.Printf("Component updated: %s (version: %s)", def.ID, def.Version)
				r.components[def.ID] = def

			case clientv3.EventTypeDelete:
				// The key is like "/platform/components/component-id"
				// We need to extract the component-id to delete it from the map.
				// A real implementation would need a more robust way to parse this.
				id := string(event.Kv.Key)[len(componentPrefix):]
				if _, ok := r.components[id]; ok {
					r.log.Printf("Component deleted: %s", id)
					delete(r.components, id)
				}
			}
			r.mu.Unlock()
		}
	}
}

// GetDefinition retrieves a component definition by its ID.
func (r *ComponentRegistry) GetDefinition(id string) (*domain.ComponentDefinition, bool) {
	r.mu.RLock()
	defer r.mu.RUnlock()
	def, ok := r.components[id]
	return def, ok
}

The pitfall here is that the watch can be interrupted. A production-grade implementation needs a robust loop that handles etcd connection errors, re-establishes the watch, and potentially re-runs populate() to ensure no events were missed during the disconnect.

Modeling and Persisting Application Graphs in Dgraph

With the platform’s capabilities managed by etcd, we turned to storing the user-generated application data. Dgraph’s schema-first approach using GraphQL was a clean fit.

Our Dgraph schema (schema.graphqls):

type ApplicationGraph {
    applicationId: String! @id
    name: String!
    root: WidgetInstance!
}

type WidgetInstance {
    instanceId: String! @id
    componentId: String!
    componentVersion: String!
    properties: String! # Storing properties as a JSON string for flexibility
    children: [WidgetInstance] @hasInverse(field: "parent")
    parent: WidgetInstance
}

The repository layer in our Go service would then handle the mutation and query logic. The key is translating our domain models into Dgraph’s JSON mutation format and parsing the results back.

// infrastructure/dgraph_repository.go
package infrastructure

import (
	"context"
	"encoding/json"
	"errors"
	"fmt"
	
	"github.com/dgraph-io/dgo/v210"
	"github.com/dgraph-io/dgo/v210/protos/api"
	"github.com/you/project/application/domain"
)

type DgraphRepository struct {
	client *dgo.Dgraph
}

// ... Dgraph client initialization ...

// dgraphWidgetInstance is a private struct for Dgraph serialization.
// We add dgraph.type for Dgraph to recognize the node type.
type dgraphWidgetInstance struct {
	UID              string                  `json:"uid,omitempty"`
	DgraphType       string                  `json:"dgraph.type,omitempty"`
	InstanceID       string                  `json:"instanceId"`
	ComponentID      string                  `json:"componentId"`
	ComponentVersion string                  `json:"componentVersion"`
	Properties       string                  `json:"properties"` // Stored as JSON string
	Children         []*dgraphWidgetInstance `json:"children,omitempty"`
}

// Save persists the entire ApplicationGraph.
func (r *DgraphRepository) Save(ctx context.Context, graph *domain.ApplicationGraph) error {
	txn := r.client.NewTxn()
	defer txn.Discard(ctx)

	// Recursively convert domain model to Dgraph model
	dgraphRoot := convertToDgraphWidget(graph.Root)

	dgraphGraph := map[string]interface{}{
		"dgraph.type":   "ApplicationGraph",
		"applicationId": graph.ApplicationID,
		"name":          graph.Name,
		"root":          dgraphRoot,
	}

	mu := &api.Mutation{
		SetJson:   mustMarshal(dgraphGraph),
		CommitNow: true,
	}
	
	_, err := txn.Mutate(ctx, mu)
	if err != nil {
		return fmt.Errorf("failed to save application graph: %w", err)
	}

	return nil
}

// FindByApplicationID retrieves an application graph.
func (r *DgraphRepository) FindByApplicationID(ctx context.Context, appID string) (*domain.ApplicationGraph, error) {
	txn := r.client.NewTxn().ReadOnly()
	defer txn.Discard(ctx)
	
	query := `
		query getApp($appId: string) {
			app(func: eq(applicationId, $appId)) {
				uid
				applicationId
				name
				root @recurse(loop: false) {
					uid
					instanceId
					componentId
					componentVersion
					properties
					children
				}
			}
		}`

	vars := map[string]string{"$appId": appID}
	resp, err := txn.QueryWithVars(ctx, query, vars)
	if err != nil {
		return nil, fmt.Errorf("dgraph query failed: %w", err)
	}

	var result struct {
		App []struct {
			UID           string               `json:"uid"`
			ApplicationID string               `json:"applicationId"`
			Name          string               `json:"name"`
			Root          dgraphWidgetInstance `json:"root"`
		} `json:"app"`
	}

	if err := json.Unmarshal(resp.Json, &result); err != nil {
		return nil, fmt.Errorf("failed to unmarshal dgraph response: %w", err)
	}
	
	if len(result.App) == 0 {
		return nil, errors.New("application not found")
	}

	appData := result.App[0]
	domainRoot := convertToDomainWidget(&appData.Root)
	
	return &domain.ApplicationGraph{
		UID: appData.UID,
		ApplicationID: appData.ApplicationID,
		Name: appData.Name,
		Root: domainRoot,
	}, nil
}

// Helper functions for conversion and marshaling would be defined here
// e.g., convertToDgraphWidget, convertToDomainWidget, mustMarshal

The recursive query (@recurse) is a powerful feature of Dgraph that makes fetching the entire UI tree trivial. A common mistake is to store complex nested objects like properties as native map types in Dgraph. This can complicate schema management. For maximum flexibility, serializing it to a JSON string is often a more pragmatic approach in a low-code context where the property schema can change frequently per component.

The Dynamic Bundler: Server-Side Rollup

This was the most experimental part of the architecture. We created a Node.js service that exposed an endpoint to generate a JavaScript bundle for a given applicationId.

The core logic flow is:

1. Receive a request for an applicationId.
2. Call the Go service (or query Dgraph directly) to fetch the ApplicationGraph.
3. Traverse the graph, and for each WidgetInstance, look up its ComponentDefinition in the etcd-backed registry (via another API call).
4. Dynamically generate an entrypoint file (e.g., index.js) in memory that imports the required components and wires them up.
5. Use Rollup’s JavaScript API to bundle this virtual file.
6. Upload the resulting bundle to a CDN/object store and return the URL.

// bundler-service/src/bundler.js
import { rollup } from 'rollup';
import virtual from '@rollup/plugin-virtual';
import { nodeResolve } from '@rollup/plugin-node-resolve';
import commonjs from '@rollup/plugin-commonjs';
import replace from '@rollup/plugin-replace';
import { terser } from 'rollup-plugin-terser';
import crypto from 'crypto';

// In a real app, this would come from an API call to the Go services
async function getApplicationData(appId) {
    // Mock data for demonstration
    return {
        graph: {
            applicationId: appId,
            name: 'My Dashboard',
            root: {
                instanceId: 'root-layout-1',
                componentId: 'grid-layout',
                componentVersion: '1.0.0',
                properties: { columns: 12 },
                children: [
                    {
                        instanceId: 'button-1',
                        componentId: 'material-ui-button',
                        componentVersion: '5.2.0',
                        properties: { text: 'Submit', color: 'primary' }
                    }
                ]
            }
        },
        componentRegistry: {
            'grid-layout': { id: 'grid-layout', sourceUrl: '@myapp/grid-layout' },
            'material-ui-button': { id: 'material-ui-button', sourceUrl: '@mui/material/Button' }
        }
    };
}

function generateEntryPoint(graph, registry) {
    const imports = [];
    const componentMap = {};
    let componentCounter = 0;

    function traverse(node) {
        if (!node) return null;

        const def = registry[node.componentId];
        if (!def) {
            throw new Error(`Component definition not found for: ${node.componentId}`);
        }

        const componentVarName = `Component${componentCounter++}`;
        imports.push(`import ${componentVarName} from '${def.sourceUrl}';`);
        componentMap[node.instanceId] = {
            varName: componentVarName,
            props: node.properties
        };

        if (node.children) {
            node.children.forEach(traverse);
        }
    }

    traverse(graph.root);

    // This is a simplified rendering logic. A real app would use React.createElement
    // or a similar mechanism to construct the tree.
    const renderLogic = `
        // This is pseudo-code for building the component tree.
        // A real implementation would use React's JSX runtime or similar.
        console.log('Rendering application tree...');
        console.log(${JSON.stringify(componentMap, null, 2)});
    `;

    return `${imports.join('\n')}\n\n${renderLogic}`;
}

export async function createBundle(appId) {
    const { graph, componentRegistry } = await getApplicationData(appId);
    
    // Create a hash of the application structure for caching
    const appHash = crypto.createHash('sha256').update(JSON.stringify(graph)).digest('hex');
    const entryPointCode = generateEntryPoint(graph, componentRegistry);

    const bundle = await rollup({
        input: 'entry.js',
        plugins: [
            virtual({ 'entry.js': entryPointCode }),
            nodeResolve({ browser: true }),
            commonjs(),
            replace({
                'process.env.NODE_ENV': JSON.stringify('production'),
                preventAssignment: true
            }),
            terser()
        ],
        // Mark shared libraries like 'react' as external to avoid duplicating them in every bundle
        external: ['react', 'react-dom']
    });

    const { output } = await bundle.generate({
        format: 'es',
        sourcemap: false
    });

    const code = output[0].code;

    // In a real system, you'd upload this 'code' to S3/GCS and return the URL.
    // The key would be something like `bundles/${appId}/${appHash}.js`.
    console.log(`Bundle created for ${appId}. Hash: ${appHash}. Size: ${code.length} bytes.`);
    return { code, hash: appHash };
}

The workflow is visualized below:

sequenceDiagram
    participant Client
    participant APIGateway as API Gateway
    participant BundlingService as Dynamic Bundler (Node.js)
    participant CoreService as Core Service (Go)
    participant Dgraph
    participant etcd
    participant ObjectStorage as S3/GCS
    participant Cache as Redis

    Client->>APIGateway: GET /render/app-dashboard
    APIGateway->>BundlingService: Request to render app-dashboard

    Note over BundlingService, Cache: Calculate structural hash of app request
    BundlingService->>Cache: GET bundle_url:hash_of_app
    alt Cache Hit
        Cache-->>BundlingService: Return S3 URL
        BundlingService-->>Client: 302 Redirect to S3 URL
    else Cache Miss
        BundlingService->>CoreService: Fetch full application data for 'app-dashboard'
        CoreService->>Dgraph: Query for ApplicationGraph
        Dgraph-->>CoreService: Return graph
        Note right of CoreService: CoreService has an in-memory
component registry powered by an etcd watch.
        CoreService-->>BundlingService: Return aggregated graph and component metadata
        
        BundlingService->>BundlingService: Generate virtual entrypoint.js
        BundlingService->>BundlingService: Invoke Rollup API in-process
        
        Note over BundlingService, ObjectStorage: Bundling completes...
        BundlingService->>ObjectStorage: Upload generated bundle.js
        ObjectStorage-->>BundlingService: Return S3 URL
        
        BundlingService->>Cache: SET bundle_url:hash_of_app = S3 URL
        BundlingService-->>Client: 302 Redirect to S3 URL
    end

A significant problem we encountered was cold start latency. The first time an un-cached application was requested, the sequence of data fetching and bundling could take several seconds. The solution was to move the bundling step from being purely on-demand to being triggered when an application is saved. The on-demand endpoint now only serves as a fallback. This pre-compilation (or “ahead-of-time”) approach dramatically improved perceived performance.

The current design still has limitations. The server-side bundling process is a potential security risk if we ever allow users to provide raw JavaScript for custom components; it would require a secure sandboxing environment. Furthermore, managing dependencies for user-provided components is unsolved; a component might require a specific version of a library, which could conflict with another. A proper package resolution and import map generation step would be necessary to handle that complexity. The system also assumes component source code is accessible via package names (@mui/material/Button), which requires a carefully configured Node.js environment on the bundling server.