Getting a generative AI agent to work in a demo is straightforward. Getting it to work reliably in production is a different problem entirely, one that most teams only fully encounter once they are already in it. This article examines how observability tools like Langfuse can be effectively utilized in the transition of agent-based systems from Proof of Concept to a production-ready implementation. To this end, it addresses the underlying challenges of applying systemic methodology in practical implementation.
The Black Box in Production
Throughout the course of an Agentic Workflow, generative AI produces results that, due to their complexity, are influenced by variables that remain hidden from users. Because of that, unlike a deterministic algorithm, responsibility for these results can only be assumed and understood to a limited extent in practice. And yet, the widespread adoption of the technology has outpaced our understanding of it. This results in a failure rate that is the predictable consequence of that gap: systems put into production face challenges to scale and are hard to govern for debugging and safety purposes.
In our own case at Scalefree, we implemented a Sales Agent to generate follow-up emails for leads that had touchpoints with our website, such as registering to download our Data Vault Handbook. This makes it possible to create personalised emails without the need for an employee to do so manually, which offers enormous advantages in terms of turnaround time and costs. This illustration shows the Workflow of the Sales Agent from a high-level perspective.
Defining the flow is easy; the challenge is that we had multiple LLM calls where error logs, latency, and cost per call remained invisible. And without an evaluation pipeline, results could not be surfaced to all stakeholders in any consistent way: what counted as a failure needs to be defined in a shared language. You cannot catch an error no one has agreed to recognize.
The Agentic AI use case at Scalefree: Sales Agent
- What?: An AI agent that, as part of a multi-stage process, generates personalised emails for leads who interact with our company website or are otherwise known to Scalefree.
- Why?:To reduce manual effort and turnaround time for outbound communication, without requiring an employee to write each email individually.
- How?: Multiple chained LLM calls orchestrated via LangGraph, utilizing Vertex AI, with Langfuse handling observability with tracing latency, cost, and output quality across each step.
The business side is less discussed but equally consequential. The most powerful control mechanism in a generative AI system is the prompt, yet in most organizations, prompt development is treated as an afterthought: no versioning, no structured testing, no shared language between domain experts and engineers to diagnose whether a problem lies in the prompt, the model, or the process.
These issues point in a common direction. What lacks measurement lacks definition. What lacks definition defies engineering and relies solely on trust. And trust, as any engineer will tell you, is no production strategy.
Observe. Measure. Define.
Software engineering has always relied on a continuous loop and generative AI is no exception. What changes is neither the structure of that loop, nor the tools required to run it; only how those tools must be applied:
- Build: Before an agent reaches production, its core behavior needs to be defined and tested in a controlled environment. For generative AI, this means developing and validating prompts with the same rigor applied to any other system component.
- Test: Deterministic unit tests fall short with probabilistic systems. Testing an AI agent means running structured evaluations against defined quality criteria. Automated, repeatable and traceable.
- Ship: Deploying a generative AI system is a continuous commitment, rather than a one-time event. It requires a controlled handover where prompt versions, model configurations, and evaluation baselines are documented and locked before go-live.
- Monitor: Once in production, visibility into agent behavior is mandatory. Every request, every decision, every unexpected output needs to be observable, in real time and in retrospect.
- Debug: When something goes wrong, the question extends beyond what failed, reaching into why. That requires a full trace of what the agent did, what it was told, and how it responded: step by step.
Where Principles Meet Implementation
This section illustrates the implementation of observability in the specific use case of the aforementioned Sales Agent. This was implemented using Langgraph as the orchestration framework, Vertex AI as the agent’s reasoning core, and Langfuse as the observability tool. For the goal of this article we will focus on the observability tool: Langfuse.
What Langfuse brings to the table
The illustration beneath encompasses the core feature provided by Langfuse: Tracing.
Specifically, it provides four capabilities, helping both technical and business users throughout each phase of the development loop:
- Tracing: Captured information of input, output, and throughput, as well as latency and cost attributed to one Agent process run. With the help of tracing runtime decisions become visible and reconstructable, which helps within the process of building, monitoring and debugging the application.
- Prompt Management: Version control of the prompts provided to each LLM Call, wrapped within the UI of Langfuse. The Prompt Management treats Prompts as the critical steering variable of LLMs while keeping Code away from those that neither can or are willing to work with it.
- Evaluation Pipeline: Either human-annotated or powered by a separate LLM acting as “LLM-as-a-judge”, it closes the loop between testing, shipping, monitoring.
- Playground: A unified testing environment natively integrated with the Evaluation Pipeline. Playgrounds gives domain experts the one thing they have rarely had, a structured environment to test and validate prompt changes before they reach production.
eature provided by Langfuse: Tracing.
Deployment
Like many other tools, the underlying standard Langfuse utilizes for tracing is OpenTelemetry. However, Langfuse also makes the code available as open source and in Docker images; the focus is thus on rapid implementation, both within the technical infrastructure and the Lang ecosystem, as well as on data autonomy. The way Langfuse was deployed in our infrastructure can be understood as follows:
In our case, Langfuse was deployed as a container on an EC2 instance to ensure that its functionalities and web interface were always available. The Sales Agent, which is itself encapsulated as a separate instance, can be connected to Langfuse directly within the function call. To do this, only the credentials and the domain of the Langfuse instance are passed on, and the information required for tracing is automatically provided.
This simplified connectivity between Langfuse and agents (which works natively with over 100 platforms and frameworks) ensures that the re-engineering effort required for integration is kept to a minimum. Agents can thus be added to Langfuse in a scalable manner, making it the central tool for observability.
Back to Engineering
Given the opaque nature of agents, which most AI-driven projects face, Langfuse’s observability functionality represents a first step towards gradually bridging the knowledge gap in the transition to production. This represents a realistic return to the core values of software and data engineering: scalability, reproducibility and systematic approaches.