At a Glance
Executive Summary
CloudGeometry's internal AI platform team operates LangBuilder, a Langflow-based AI orchestration platform running production flows for client engagements, internal tooling, and R&D experimentation. As usage scaled across dozens of active flows consuming tokens from Anthropic, OpenAI, and Google, a critical gap emerged: no visibility into LLM spending. Budget conversations relied on monthly invoices and guesswork.
The team prepared a comprehensive PRD, then onboarded the LangBuilder codebase into AI-MSL to deliver a full-featured Cost Tracking Dashboard — moving from requirements to production-ready feature in a single development cycle. The result: real-time cost visibility, budget alerting, period-over-period comparison, and exportable reporting — delivered in a timeline that compressed what would typically be weeks of engineering into days.
The Challenge: Scaling LLM Usage Without Cost Visibility
LangBuilder processes production AI workflows across multiple LLM providers, with cost distributed across dozens of flows, multiple models, and several user teams. Three critical problems emerged:
- No Cost Attribution. Platform admins could not answer "which flow is driving spend?" without manually cross-referencing LangWatch traces with provider invoices. The VP of Technology could not produce cost trend data for board presentations without hours of spreadsheet work.
- No Budget Awareness. Monthly LLM spend occasionally spiked due to high-volume flows or model upgrades, with no early warning system. Finance discovered overruns only after the billing cycle closed.
- No Self-Service for Non-Technical Stakeholders. FP&A analysts needed exportable cost data for monthly allocation reports. Flow builders wanted to see their own usage. Neither had access without admin intervention.
The opportunity: build a native cost tracking dashboard serving four user personas — platform admins, executive leadership, finance analysts, and individual flow builders — each with appropriate access controls and data scoping.
The Solution: AI-MSL Framework — PRD to Production
Phase 1: PRD Preparation — Structured Requirements
The team prepared a comprehensive PRD covering 20 sections, 4 user personas, 17 features, and 80+ acceptance criteria. Research into LangWatch's trace API, cost computation model, and data aggregation patterns informed every design decision.
- Identified LangWatch's /api/traces/search endpoint as the primary data source with scroll pagination (up to 10,000 traces per request)
- Mapped model name normalization challenges (vendor prefixes, date suffixes, casing inconsistencies) that would erode dashboard trust
- Validated that client-side CSV generation eliminated the need for a dedicated export API endpoint
- Designed demo mode architecture using deterministic seeded RNG for investor and stakeholder demonstrations
Phase 2: Implementation — Parallel Full-Stack Delivery
The team onboarded the LangBuilder codebase into AI-MSL and executed implementation across backend and frontend workstreams with TDD cycles, parallel execution, and automated verification.
Backend (FastAPI + Python):
- LangWatch integration service with scroll-paginated trace ingestion and ownership-scoped filtering
- Single-pass aggregation engine: per-flow, per-day, per-model cost breakdowns with model name normalization
- Redis caching (5-min TTL, org-scoped keys) + Fernet-encrypted API key storage
- Budget management API with threshold computation and top-spender identification
- Demo mode data generator using real flow names with Pareto-distributed popularity
Frontend (React + TypeScript):
- 4-card summary metrics with locale formatting and 4-decimal-place precision
- Interactive area chart (Recharts) with gap-filled daily cost trend and truncation warnings
- Per-model breakdown table with normalized names and percentage progress bars
- Paginated per-flow breakdown with expandable run-level detail (lazy-loaded)
- Date range picker with presets, period comparison toggle, and debounced API calls
- Budget alert banner: yellow at 75%, red at 100%, inline top-spender identification
- Period-over-period comparison with per-day normalization and color-coded delta badges
- RFC 4180-compliant CSV export with role-based column visibility
- Full state management: loading skeletons, empty states, error states with retry
Phase 3: Validation — Demo Mode + Production Deployment
- Demo mode: Stakeholder review and investor showcases without requiring live LangWatch data — deterministic seeded RNG produces realistic, reproducible cost patterns.
- Production deployment: Full feature validated in the live LangBuilder environment with real LLM cost data from LangWatch traces.
Practical Realities: What We Learned
PRD Quality Determines AI-MSL Velocity
With 80+ acceptance criteria defined upfront, AI-MSL operated with minimal ambiguity. Each criterion translated directly into a testable implementation target. The better the input requirements, the faster and more accurate the governed output.
Model Name Normalization Is a Trust Problem
LLM providers return inconsistent model names — vendor prefixes, date suffixes, casing variations. AI-MSL's context engineering surfaced this during PRD preparation, preventing a data quality issue that would have been discovered much later in a traditional cycle.
Parallel Backend/Frontend Execution Works
Backend and frontend workstreams ran concurrently with independent TDD cycles. Detailed API contract definitions from the specification phase enabled parallel execution without blocking — directly replicable for any full-stack delivery.
Demo Mode as a Governance Tool
Deterministic seeded data served dual purposes: stakeholder review (investors see the dashboard without live data) and QA validation (reproducible test scenarios). This pattern should be standard for any dashboard feature.
Deliverables
Business Impact
The Cost Tracking Dashboard transformed LLM cost management from a reactive, invoice-driven process into a proactive, self-service capability embedded directly in the platform:
- Platform admins: catch budget trends before they become overruns
- Executive leadership: produces cost analysis for board presentations without engineering support
- Finance: receives clean, exportable data without ever logging into the platform
- Flow builders: see their own cost data with budget awareness, scoped to their access level
What Comes Next
Key Value Propositions Demonstrated
1. PRD-to-Production in a Single Cycle
AI-MSL took a comprehensive PRD (20 sections, 80+ acceptance criteria) and delivered a production-deployed, full-stack feature with complete TDD coverage, error handling, and role-based access control.
2. Parallel Full-Stack Execution
Backend (FastAPI) and frontend (React) workstreams executed concurrently. Detailed API contracts eliminated blocking dependencies — a pattern directly replicable for any full-stack delivery.
3. Enterprise-Quality Output, Not a Prototype
Budget alerting, period comparison, encrypted key storage, RFC 4180 CSV export, role-based visibility, and full state management. Production software, not a demo.
4. Weeks Compressed to Days
Multiple sprint cycles of planning, implementation, testing, and deployment compressed into a continuous AI-MSL development cycle.
Technologies
- LangBuilder — Langflow-based AI orchestration platform
- LangWatch — AI observability and trace analytics
- AI-MSL — CloudGeometry's AI-assisted structured development framework
- FastAPI + Python — Backend API and aggregation engine
- React + TypeScript — Frontend dashboard
- Recharts — Data visualization
- Redis — Server-side response caching
- React Query — Client-side cache and retry management
- Fernet encryption — API key storage at rest
- SQLite — Flow and user data
Company Overview
Founded in 2014 in Silicon Valley, CloudGeometry has evolved into a strategic technology transformation partner for technology-driven businesses, helping them upgrade their operations and software for the AI era.
- Global Team: Over 100+ full-time employees (FTEs) worldwide, including Solution Architects, AI Engineers, and DevOps experts.
- LangBuilder: CloudGeometry's internal AI orchestration platform — AI-MSL is used on CG's own production systems, not just client engagements.
Ready to Assess Your System?
See how AI-MSL can map your existing codebase, accelerate planning, and bring supervised automation to your development process. Contact us today to schedule a live demo.
.png)
