ADR: Real-Time Data Change Notification System¶
Status¶
Implemented - October 2025
Implementation Status¶
- Phase 1-2: Core Foundation (Complete)
- Phase 3: Observability (Complete)
- Phase 4: Client SDK & Production (Pending)
See Implementation Progress section for details.
Context¶
LBS Foundry currently requires clients to poll the /api/readmodel endpoint to detect data changes, resulting in unnecessary server load, increased latency, and poor user experience during live sporting events. This ADR defines a real-time notification system to eliminate polling and enable sub-100ms data updates.
Problem Statement¶
Current State: - React frontend polls for updates every N seconds - High API load during live matches (thousands of concurrent users polling) - Poor user experience: delayed updates, wasted bandwidth - No support for collaborative editing scenarios - Cache invalidation requires manual polling or time-based expiry
Desired State: - Zero API polling - push-based notifications - Sub-100ms latency for live match updates - Automatic cache invalidation on data changes - Real-time collaborative editing support - Scale from 10K to 100K+ concurrent connections
Current Architecture¶
- Database Layer: PostgreSQL with Marten for event sourcing and projections
- Multi-Database Support: Main (Foundry), Analytics, Reporting, Archive, Ballr databases
- Authentication: Clerk JWT (human users) + Basic Auth (service accounts)
- Authorization: Role-based with
[RequiresRoles]attributes, query-level permissions - API Patterns: CQRS with
/api/readmodel,/api/command,/api/import - Deployment: Azure Container Apps + .NET Aspire orchestration
- Infrastructure: Redis cache already deployed
Requirements Questionnaire¶
Please answer the following questions inline to finalize the design.
Database Architecture¶
1. Database Scope¶
Current databases: Main (event store), Analytics, Reporting, Archive
Q1.1: Do you need notifications from all databases, or just specific ones? - [ ] Main event store only - [X] All databases (Main, Analytics, Reporting, Archive) - [ ] Configurable per-feature - [X] Other: Currently Ballr and Founry databases.
A1.1: [X] All databases (Main, Analytics, Reporting, Archive) [X] Other: Currently Ballr and Founry databases.
Q1.2: Should each database have its own notification channel, or unified notifications?
- [ ] Separate channels per database (e.g., main-notifications, analytics-notifications)
- [X] Single unified channel (all databases send to same channel)
- [ ] Database name included in message payload for routing
A1.2: [X] Single unified channel (all databases send to same channel) Though this is based on the notion that an application is going to know its list of databases.
2. Tables to Monitor¶
Q2.1: What level of change do you want to monitor?
- [ ] Marten event tables (mt_events, mt_streams) - raw event appends
- [ ] Marten projection tables (mt_doc_*) - aggregate document changes
- [X] Both event and projection tables
- [ ] Custom business tables only
- [ ] Other: _______
A2.1: - [X] Both event and projection tables though the focus is on projections
Q2.2: Any specific aggregates/tables you know you'll need to monitor immediately? (e.g., Team, Fixture, RugbyGameState, User, etc.)
A2.2: no all of the tables based on what the queries and readmodels..
Subscription Model¶
3. Subscription Granularity¶
Q3.1: Choose your primary subscription model (select one or multiple):
Option A - Entity/Aggregate Level
Client notified when Team aggregate with ID "team-123" changes.Option B - Table Level
Client notified when specific document row changes.Option C - Query/View Level
{
"subscribe": {
"queryType": "TeamsByCompetition",
"parameters": { "competitionId": "comp-789" }
}
}
Option D - Stream Level
Client notified of all events in a specific Marten stream.Option E - Event Type Level
Client notified when any event of this type is raised.- [X] Option A - Entity/Aggregate Level
- [ ] Option B - Table Level
- [ ] Option C - Query/View Level
- [ ] Option D - Stream Level
- [ ] Option E - Event Type Level
- [ ] Multiple options (specify which): _______
A3.1: - [X] Option A - Entity/Aggregate Level in our queries we send the primary type and we should use a contract/table
Q3.2: Explain your reasoning for the choice above:
A3.2: The primary reason for this is because we want to get the end users to have updates to the site based on the read model changing. we may need to do more in the furture to change the way we deal with events but for now this is fine.
4. Subscription Scope¶
Q4.1: Should a single subscription support multiple entities? - [X] Single entity per subscription only - [ ] Bulk subscriptions (e.g., "all teams in competition X") - [ ] Both single and bulk
A4.1: - [X] Single entity per subscription only
Q4.2: Do you need wildcard/broadcast subscriptions? - [ ] Yes - subscribe to "all changes of type X" (e.g., any Team change) - [ ] No - explicit entity IDs only - [X] Maybe later, not for initial version
A4.2: - [X] Maybe later, not for initial version
Message Payload¶
5. Notification Content¶
Q5.1: What should the notification message contain? (choose one)
Option A - Minimal (Refetch Pattern) Client receives notification, then makes API call to fetch latest data.
Pros: Small message size, always fresh data, simple Cons: Extra API call requiredOption B - Change Hints Includes hints about what changed so client can decide whether to refetch.
{
"type": "Team",
"id": "team-123",
"changeType": "updated",
"changedFields": ["name", "logo"],
"version": 42,
"timestamp": "2025-10-07T10:30:00Z"
}
Option C - Full Payload Complete updated data included in notification.
{
"type": "Team",
"id": "team-123",
"changeType": "updated",
"data": {
"id": "team-123",
"name": "Updated Team Name",
"logo": "https://...",
"founded": 1908
},
"previousVersion": 41,
"newVersion": 42,
"timestamp": "2025-10-07T10:30:00Z"
}
Option D - Event Payload Include the actual domain event(s) that caused the change.
{
"type": "Team",
"id": "team-123",
"changeType": "updated",
"events": [
{
"eventType": "TeamNameUpdatedEvent",
"data": { "newName": "Updated Team Name" },
"timestamp": "2025-10-07T10:30:00Z",
"version": 42
}
]
}
- [ ] Option A - Minimal (Refetch Pattern)
- [ ] Option B - Change Hints
- [ ] Option C - Full Payload
- [ ] Option D - Event Payload
- [X] Custom/Hybrid: _______
A5.1: - [ ] Custom/Hybrid: - Option A - Minimal (Refetch Pattern) & Option C - Full Payload . there may be a need stream that we feed that pushes out the changes on a different channel for match update or other things as a later iteration. this is for high frequency of changes that happen during a live udpate.
Q5.2: Should notifications include user/audit information? (e.g., who made the change, transaction ID, etc.) - [ ] Yes, include audit metadata - [X] No, just data changes - [ ] Optional based on subscription settings
A5.2: - [X] No, just data changes
Client Connections¶
6. Connection Protocol¶
Q6.1: Which protocol should we support?
WebSocket - Bidirectional communication - Client can send subscribe/unsubscribe messages - More complex to implement - Better for interactive applications
Server-Sent Events (SSE) - One-way server → client - Simpler implementation - Subscriptions managed via query parameters or initial request - Better for read-only notifications
- [X] WebSocket only
- [ ] SSE only
- [ ] Both (let clients choose)
- [ ] Start with SSE, add WebSocket later
A6.1: - [X] WebSocket only
Q6.2: Expected connection lifetime? - [ ] Long-lived (hours) - users keep browser tab open - [ ] Short-lived (minutes) - for specific operations only - [ ] Mixed - depends on use case
A6.2:
"Long-lived (hours)" - users watching matches or editing content will keep browser tabs open for extended periods.¶
7. Client Types¶
Q7.1: Who/what will be connecting? - [X] React frontend (ballr.live) only - [ ] Multiple frontend applications - [ ] Backend services (C# service-to-service) - [ ] Mobile applications - [ ] All of the above
A7.1: - [X] React frontend (ballr.live) only for now. in the future this will change
Q7.2: Expected concurrent connections (current and 12-month projection)?
A7.2: 10K today and 100K in 12 months
Security & Authorization¶
8. Access Control¶
Q8.1: Should subscription requests be authenticated? - [ ] Always authenticated (Clerk JWT or Basic Auth required) - [ ] Public access allowed (anonymous users can subscribe) - [ ] Mixed - some notifications public, some require auth
A8.1: - [X] Mixed - some notifications public, some require auth
Q8.2: Authorization model for subscriptions? - [ ] Users can subscribe to any entity - [X] Users can only subscribe to entities they have permission to view - [ ] Role-based (e.g., only Admins can subscribe to User changes) - [ ] Custom authorization per entity type
A8.2: - [X] Users can only subscribe to entities they have permission to view - Authorization is entity-type based, not query-based - Three categories: Public entities (anyone), Private entities (authenticated only), Admin-only entities (Admin role required)
Q8.3: Example scenario: Can a regular Member role subscribe to: - [ ] Yes/No: Admin-only aggregate changes (e.g., User management)? - [ ] Yes/No: Their own user data changes? - [ ] Yes/No: Public data (e.g., Team, Fixture)?
A8.3:
- NO : Admin-only aggregate changes (e.g., User management)? - they would/should not be able to see this anyway
- YES : Their own user data changes? - yes
- YES : Public data (e.g., Team, Fixture)? - the game will be changing in the background so they should be told of the change.
Use Cases & Priorities¶
9. Primary Use Case¶
Q9.1: What's the #1 scenario you're solving? (rank 1-5, 1 = highest priority)
- [X] Real-time dashboard updates (live match scores, statistics)
- [X] Collaborative editing (multiple users editing same entity)
- [X] Cache invalidation (client knows when to refetch)
- [X] Live activity feed (recent changes across system)
- [ ] Other: _______
A9.1: - [X] Real-time dashboard updates (live match scores, statistics) - [X] Cache invalidation (client knows when to refetch) - [X] Live activity feed (recent changes across system) - [X] Collaborative editing same or multiple users editing same entity in the one or more browsers
Q9.2: Describe your highest-priority use case in detail:
A9.2: Realtime updates of scores dashbords
10. Specific Examples¶
Q10.1: Provide 2-3 concrete examples of notifications you need:
Example 1: - User action: ____ - What triggers notification: ___ - Who receives notification: ____ - What they do with it: ___
Example 2: - User action: ____ - What triggers notification: ___ - Who receives notification: ____ - What they do with it: ___
Example 3 (optional): - User action: ____ - What triggers notification: ___ - Who receives notification: ____ - What they do with it: ___
A10.1: User visits a fixtures or match page and then the system subsribes to changes of the match center that user when the system updates the score that user along with anyone else who have actively subsribed should get told the live scores via a the full content push channel. The site then updates with the latest scores.
- User action: User navigates to fixture/match page
- What triggers notification: Match score update or game state change in the background
- Who receives notification: All users subscribed to that specific match/fixture
- What they do with it: Receive minimal notification, refetch latest match data via API, update UI, though i would like to determine what is better for perfomance before locking in. I think we need to have a full match stats push feed so there is no need for everyone to go and get the whole match by swarming the server when notifications happen. can you verify this thinking.
User visits a page with their team in play then the system will subsribe for changes to player stats for all players when stats change we can run the query to get the updated stats for that users team. - User action: User visits page showing their team roster - What triggers notification: Player stats change (e.g., during live match) - Who receives notification: Users subscribed to players on that team - What they do with it: Receive minimal notification, refetch latest match data via Query API, update UI
User visits a team selection page on two different browsers and connects and subsribes for changes to both browser sessions then they add a player on one browser. i would like the result to reflect on the other browers
User action: User opens team selection page in two browsers - What triggers notification: Player added to team in browser A - Who receives notification: Browser B & A (same user, different session) - What they do with it: Receive minimal notification, refetch latest match data via Query API, update UI
Infrastructure Preferences¶
11. Pub/Sub Technology¶
Q11.1: For the fan-out layer (PostgreSQL → Pub/Sub → Clients), which technology?
- [X] Redis Pub/Sub (already have Redis? simple, in-memory)
- [ ] Azure Service Bus (cloud-native, managed)
- [ ] SignalR with Redis backplane (ASP.NET native, good for WebSocket)
- [ ] RabbitMQ (robust, self-hosted)
- [ ] NATS (lightweight, high-performance)
- [ ] Other: _______
- [ ] Not sure yet / need recommendation
A11.1: [X] Redis Pub/Sub (already have Redis? simple, in-memory) for today. and [X] NATS (lightweight, high-performance) for future. we may still use this for micro service commuications. I would hope that we design it in a way that would allow us to swap implemenations.
Q11.2: Do you already have any of these technologies deployed?
A11.2: Redis
12. Deployment & Operations¶
Q12.1: Hosting environment? - [X] Azure Container Apps - [ ] Azure App Service - [ ] Kubernetes - [ ] Docker Compose (local/dev) - [X] Other: Aspire_______
A12.1: [X] Azure Container Apps [X] Other: Aspire
Q12.2: Should the notification service be: - [ ] Part of Ballr.WebApi (same process) - [X] Separate standalone service - [X] Either, design for both
A12.2: - [X] Separate standalone service - [X] Either, design for both
Performance & Scale¶
13. Performance Requirements¶
Q13.1: Latency requirements? - [ ] Best effort (no specific SLA) - [ ] Under 1 second - [ ] Under 500ms - [X] Under 100ms - [ ] Other: _______
A13.1: - [X] Under 100ms
Q13.2: Acceptable message loss? - [ ] Zero loss (guaranteed delivery) - [ ] Best effort (occasional loss acceptable) - [ ] Explain: _______
A13.2: ideally zero loss however when dealing with the web loss is ok. however when notifying in the future between Machines we would want zero loss
Decision Criteria¶
14. Success Metrics¶
Q14.1: How will you measure success of this system? (e.g., reduced API polling, user engagement, performance metrics)
A14.1: we need zero api polling. we want faster enduser data delivery
Additional Notes¶
Q15.1: Any other requirements, constraints, or context we should consider?
A15.1: we need to be able to monitor using otel and other industry standard mechanics. this should also help us with undertanding when the system is under load and needs to scale.
Decision¶
Based on the requirements questionnaire above, we will implement a WebSocket-based real-time notification system with the following architecture:
Architecture Overview¶
PostgreSQL (All DBs) → PostgreSQL LISTEN/NOTIFY → Notification Service
↓
Redis Pub/Sub
↓
WebSocket Server(s)
↓
React Clients
Key Decisions¶
1. Subscription Model: Entity/Aggregate Level¶
- Clients subscribe to specific entity instances:
{ type: "Team", id: "team-123" } - Single entity per subscription (no bulk/wildcard in v1)
- Aligns perfectly with existing CQRS query model
- Authorization enforced at subscription time (users can only subscribe to entities they can view)
2. Message Payload: Hybrid Strategy¶
To avoid "thundering herd" problems during live matches while maintaining efficiency:
Full Payload (Option C) for:
- Live match updates (RugbyGameState, Fixture during live games)
- High-frequency scenarios where many users watch the same entity
- Prevents thousands of simultaneous API requests when scores change
Minimal Refetch Pattern (Option A) for: - Collaborative editing (team selection, roster management) - Administrative changes (user management, configuration) - Low-frequency updates
Example messages:
// Full payload (live match)
{
"type": "RugbyGameState",
"id": "match-123",
"changeType": "updated",
"payloadType": "full",
"data": { /* complete match state */ },
"timestamp": "2025-10-08T10:30:00Z"
}
// Refetch pattern (team edit)
{
"type": "Team",
"id": "team-456",
"changeType": "updated",
"payloadType": "minimal",
"timestamp": "2025-10-08T10:30:00Z"
}
3. Database Triggers: Universal Pattern¶
- Single PostgreSQL trigger function for all Marten projection tables (
mt_doc_*) - Triggers send NOTIFY with:
{ table, id, operation, timestamp } - Notification service enriches with entity type and determines payload strategy
- Supports all databases: Main, Analytics, Reporting, Archive, Ballr
4. Pub/Sub Layer: Redis (v1) → NATS (future)¶
- V1: Redis Pub/Sub (already deployed, simple, sufficient for 100K connections)
- Future: NATS for microservice communication and higher scale
- Design: Swappable implementation via
IPubSubProviderabstraction
5. Client Protocol: WebSocket Only¶
- Bidirectional: clients send subscribe/unsubscribe commands
- Long-lived connections (hours) for match viewing and editing sessions
- Authentication: Clerk JWT or Basic Auth in initial handshake
- Authorization: Validate subscription requests against existing query permissions
6. Service Deployment: Standalone (with embedded option)¶
- Primary: Separate
LBS.NotificationServicefor independent scaling - Alternative: Embeddable in
Ballr.WebApifor simpler deployments - Horizontal scaling: Multiple instances backed by Redis Pub/Sub
- Aspire integration for local development
7. Security & Authorization¶
- Authentication: Clerk JWT (human users) + Basic Auth (service accounts) via ASP.NET Core authentication middleware
- Authorization Model: Entity-type based (NOT query-based)
- Public Entities: Anyone can subscribe (Team, Fixture, Competition, Player, RugbyGameState)
- Private Entities: Authentication required, user can only subscribe to their own data (User, UserSettings)
- Admin-Only Entities: Admin role required (ServiceAccount, AdminReport)
- Subscription Validation: Check entity type authorization at subscribe time
- WebSocket Authentication: Token passed via query parameter (
?token=xxx) due to browser WebSocket API limitations - No audit metadata in notifications (just data changes)
Important: Subscription authorization is based on entity types (what data is being subscribed to), not query authorization patterns. Do not conflate subscription permissions with ISecureQuery/IPublicQuery which govern query execution.
8. Performance & Scale¶
- Target Latency: < 100ms (PostgreSQL NOTIFY → Client receives message)
- Scale: 10K concurrent connections today → 100K in 12 months
- Message Loss: Best effort for web clients (acceptable), zero loss for future M2M
- Observability: OpenTelemetry integration for metrics, traces, and scaling signals
Technical Components¶
New Projects:
1. LBS.NotificationService - Standalone WebSocket notification server
2. LBS.Notification.Abstractions - Contracts and interfaces
3. LBS.Notification.Redis - Redis Pub/Sub implementation
4. LBS.Notification.Client - TypeScript/React SDK for ballr.live
Database:
- PostgreSQL trigger function: notify_document_change()
- Applied to all mt_doc_* tables via Marten module configuration
Infrastructure:
- Redis Pub/Sub channels: foundry:notifications
- WebSocket endpoint: wss://api.ballr.live/ws/notifications
Consequences¶
Positive¶
- Eliminates API Polling: Zero polling = reduced server load and bandwidth
- Sub-100ms Latency: Near-instant updates for live matches and collaborative editing
- Better User Experience: Real-time score updates, live collaboration without page refresh
- Scales Horizontally: Stateless WebSocket servers behind Redis Pub/Sub
- Future-Proof: Swappable pub/sub (Redis → NATS), supports M2M in future
- Security: Reuses existing authentication and authorization infrastructure
- Observability: OpenTelemetry metrics for monitoring and auto-scaling decisions
Negative¶
- Complexity: Adds new service, database triggers, WebSocket management
- Connection Management: Need to handle reconnections, heartbeats, timeouts
- Redis Dependency: Single point of failure (mitigated by Redis persistence/replication)
- Message Size: Full payload for live matches increases bandwidth (acceptable trade-off)
- Authorization Overhead: Must validate subscriptions against entity type permissions (not query-based)
- Testing Complexity: WebSocket integration tests, trigger testing, authorization testing
Risks & Mitigations¶
| Risk | Mitigation |
|---|---|
| Thundering herd on refetch | Use full payload for high-traffic entities (live matches) |
| Redis unavailability | Redis replication, graceful degradation (clients fall back to polling) |
| Unauthorized subscriptions | Validate at subscribe time using entity-type authorization (public/private/admin-only) |
| Connection leak/abuse | Subscription limits per connection (50-100), auto-cleanup on disconnect |
| Stale data in full payload | Include version/timestamp, clients can refetch if suspicious |
| Scaling beyond 100K | Migrate to NATS when needed (abstraction already in place) |
Implementation Notes¶
Phase 1: Foundation (Weeks 1-2)¶
- Create
LBS.Notification.Abstractionswith core contracts - Implement
IPubSubProviderabstraction + Redis implementation - Create PostgreSQL trigger function for Marten projection tables
- Build basic WebSocket server with connection management
Phase 2: Core Features (Weeks 3-4)¶
- Implement subscription management and authorization
- Add entity-type mapping (table name → aggregate type)
- Build payload strategy logic (minimal vs full)
- Integrate with existing authentication (Clerk + Basic Auth)
Phase 3: Client & Observability (Weeks 5-6)¶
- Create TypeScript/React SDK with reconnection logic
- Add OpenTelemetry metrics (connections, latency, throughput)
- Aspire integration for local dev
- Integration tests (WebSocket, triggers, authorization)
Phase 4: Production Readiness (Weeks 7-8)¶
- Azure Container Apps deployment configuration
- Load testing (10K → 100K connections)
- Documentation and runbooks
- Gradual rollout strategy
Additional Considerations Captured¶
Connection State: - Stateless (fire-and-forget): no missed message replay - Clients refetch full state on reconnect
Subscription Lifecycle: - Auto-unsubscribe on connection drop - Max 50-100 subscriptions per connection (prevent abuse)
Authorization: - Entity-type based: Public entities (Team, Fixture), Private entities (User), Admin-only entities - Checked at subscription time only (not per message) - Trusted after initial authorization - NOT based on query permissions - authorization is about entity types, not query execution
Deduplication: - One notification per aggregate per transaction (not per event)
Client Library: - TypeScript/React SDK with automatic reconnection, subscription management, typed messages
Implementation Progress¶
Phase 1-2: Core Foundation (Complete)¶
Infrastructure (Complete)
- LBS.NotificationService - Standalone WebSocket service
- LBS.Notification.Abstractions - Core interfaces and contracts
- LBS.Notification.Redis - Redis Pub/Sub provider with distributed subscription store
- LBS.Notification.PostgreSQL - PostgreSQL LISTEN/NOTIFY integration
- PostgreSQL trigger function (notify_document_change())
- Applied to all mt_doc_* tables automatically
Core Features (Complete) - WebSocket connection management with distributed state (Redis-backed) - Subscription management with authorization validation - Entity type auto-discovery from Marten document stores (Ballr + Foundry) - Notification routing (PostgreSQL → Redis → WebSocket clients) - Payload strategy service (minimal vs full payload) - Horizontal scaling support (multi-node deployment with Redis)
Authentication & Authorization (Complete)
- Clerk JWT authentication for human users
- Basic Auth for service accounts
- WebSocket token authentication via query parameter (?token=xxx)
- Entity-type based authorization (public/private/admin-only)
- Configuration-driven entity type lists
WebSocket Protocol (Complete) - Subscribe/Unsubscribe commands - Ping/Pong heartbeat - Error handling and client feedback - Subscription success/failure responses
Phase 3: Observability (Complete)¶
OpenTelemetry Metrics (Complete) - Connection metrics (opened, closed, active) - Subscription metrics (created, removed, active, per-connection distribution) - Notification metrics (sent, failed, latency) - Authentication/Authorization metrics (attempts, failures) - End-to-end latency tracking (database → client) - Per-entity-type metric tags - Integration with ServiceDefaults OpenTelemetry configuration
Phase 4: Client SDK & Production (Pending)¶
Remaining Tasks - TypeScript/React client SDK - WebSocket connection wrapper with reconnection logic - Subscription management API - Typed message contracts - Integration examples for ballr.live - Configuration limits - Max subscriptions per connection enforcement - Connection timeout/heartbeat configuration - Rate limiting for subscription requests - Integration tests - WebSocket connection/disconnection flows - Subscribe/unsubscribe scenarios - Authorization validation tests - Multi-database notification tests - Reconnection and error handling tests - Production deployment - Azure Container Apps configuration - Load testing (10K → 100K connections) - Documentation and runbooks - Gradual rollout strategy - Health check enhancements
Architecture Decisions Implemented¶
Distributed Subscription Manager - Redis-backed subscription state sharing across multiple service instances - Each instance tracks its own connections via instance ID - Bidirectional lookups: connection→subscriptions, entity→connections - TTL strategy: 1 hour for connection/instance keys, no expiry for entity keys
Entity Type Auto-Discovery
- Uses reflection to access Marten's internal Storage.AllDocumentMappings
- Extracts DataContract names for entity type mapping
- Registers from both Ballr (IDocumentStore) and Foundry (IFoundryStore) databases
- Manual registration interface available for custom mappings
Authentication Flow
Client → /ws?token={jwt}
↓
WebSocket endpoint extracts token from query parameter
↓
Sets Authorization header and triggers authentication
↓
ClerkAuthenticationHandler or BasicAuthenticationHandler validates
↓
context.User populated with authenticated principal
↓
WebSocketHandler receives authenticated user
↓
EntityAuthorizationValidator checks subscription permissions
Next Steps¶
- Review and approve this ADR
- Set up project structure (
LBS.NotificationService, abstractions, Redis impl) - Define PostgreSQL trigger SQL and Marten integration
- Design WebSocket protocol (subscription commands, message formats)
- Implement authentication and authorization
- Add OpenTelemetry metrics
- Build TypeScript client SDK
- Integration tests
- Production deployment and load testing