Azure
-
How I Would Build Observability for an Autonomous Agent
I have not built a full production observability stack for an autonomous agent.
I’ve built lots of small wrappers around existing coding harnesses. I have a pretty good idea how quickly their output can turn into a wall of model responses, tool calls, and subprocess logs. But I have not run LangChain across a Kubernetes cluster or operated an LLM router at scale.
So this is not a postmortem. It’s a design exercise.
If one of my wrappers became an always-on agent service tomorrow, what would I need to see when it failed? Where would I put that telemetry on Azure, Railway, or Cloudflare? And which parts of the design should stay the same no matter where it runs?
Start With the Trace, Not the Platform
An agent run is a distributed trace hiding inside a loop.
There is a request that starts the work. The agent calls a model, the model requests a tool, the tool talks to another service, and the result goes back into the model. Repeat that enough times and a normal application log becomes hard to follow because the interesting question is not just, “What failed?” It is, “What sequence of decisions got us here?”
I would use OpenTelemetry and give every run one root trace. Each model call and tool call becomes a child span. Structured logs carry the same trace and span IDs, so I can move from a failed run to the exact log event without searching timestamps and hoping I found the right one.
The minimum useful event would look something like this:
{ "event": "agent.tool.completed", "agent_run_id": "run_01K0...", "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736", "span_id": "00f067aa0ba902b7", "step": 7, "tool": "github.get_issue", "duration_ms": 842, "status": "error", "error_type": "rate_limit" }That gives me four things I can debug:
- The run: which request or scheduled job started the work?
- The model: which provider and model answered, how long did it take, and how many tokens did it use?
- The tool: what operation ran, how long did it take, and did it succeed?
- The sequence: what happened immediately before the failure?
I would keep an
agent_run_idbecause it is useful in a UI and support ticket, but I would not use it as a replacement for a trace ID. The trace context needs to travel across HTTP calls, queues, model gateways, and anything else the agent touches.Do Not Log Everything
The tempting version of agent observability records every prompt, response, tool argument, environment variable, and file body. It is also a convenient way to copy credentials and private data into the system with the broadest internal access.
My default would be structural telemetry first. Model name, token counts, latency, tool name, result status, retry count, and safe error categories are useful without storing the full content.
Prompt and tool payload capture would be an explicit policy, not a debug switch someone leaves enabled for six months. I would use an allowlist, redact before export, encrypt whatever remains, and give raw payloads a shorter retention period than ordinary metrics.
This also answers where an AI gateway fits. The gateway is the natural place to record model latency, provider errors, and token usage. It cannot see the agent’s local decisions or tool calls, so it is one part of the trace, not the entire observability system.
The Same Design in Three Places
The trace model stays the same. The deployment choice changes how much infrastructure I have to own.
Azure: The Integrated Option
If the agent already lived in Azure, I would use the Azure Monitor OpenTelemetry distribution and send telemetry to Application Insights, backed by a Log Analytics workspace. That gives the system a managed place for traces, logs, metrics, exceptions, queries, and retention policies.
Microsoft Foundry can also trace model calls, tool calls, intermediate steps, tokens, latency, and errors into Application Insights. I would treat that as an accelerator, not an excuse to skip the application-level design. Some of its agent tracing paths are still in preview, and content tracing can include sensitive prompts and outputs.
Azure is the option I would choose for an organization already paying the Azure complexity tax. It has the most integrated telemetry path of these three and the governance controls a larger company is likely to ask for. For a personal agent prototype, it is probably more platform than I need.
Railway: The Straightforward Container
Railway is where I would start for a small agent that needs a normal process, a Docker image, background work, and the freedom to use ordinary libraries.
I would instrument the application with OpenTelemetry and export traces over OTLP to an external observability backend. Railway’s built-in logs and container metrics are useful for deployment health, CPU, memory, disk, and network usage. They are not a distributed tracing backend for the agent itself.
That split is fine. Railway runs the service, while the application owns its telemetry schema and exports it somewhere designed to query traces. I would also make sure the exporter flushes on
SIGTERM, because a clean deployment is not helpful if the final spans disappear during shutdown.This is the least ceremonial option. It is also the one where I would have to make a separate decision about the telemetry backend and its retention cost.
Cloudflare: The API-Oriented Agent
Cloudflare gets interesting when the agent mostly calls models and HTTP APIs instead of running shell commands against a workspace.
Workers can collect logs and traces automatically, and Cloudflare can export both in OpenTelemetry format to an external destination. I would still add application spans for the model and tool semantics because automatic request tracing cannot know which prompt, tool, or agent step matters to me. The OTLP export is also currently beta and does not export Worker metrics.
For state, I would use a Durable Object when one run needs a single coordination point, D1 for relational records across runs, and R2 for larger artifacts or archived transcripts. If the agent needs retries, sleeps, or a human callback, Cloudflare Workflows is a better fit than trying to keep one HTTP request alive.
I would not put a coding agent that needs arbitrary processes, a writable repository, and long CPU-heavy tool calls into a Worker just because the edge sounds nice. Cloudflare’s runtime has real CPU, memory, connection, and logging limits. For that workload, I would run the agent in a container and consider Cloudflare for the gateway or API edge instead.
What I Would Pick
For my first version, probably Railway. It matches the small wrappers I already build, lets the agent behave like a normal application, and leaves me free to try different tracing backends without moving the workload.
I would pick Azure when the surrounding organization already uses Azure Monitor and needs one managed governance story. I would pick Cloudflare when the agent is an API orchestrator and its tools already live on the network.
A mistake would be choosing a platform or dashboard first and calling that observability solved. I would start small with a trace per run or tool call with structured logging turned on, but no prompt content until there is a redaction or sanitization policy feature in place.
I haven’t built this stack yet. But that is how I would start building it.
Sources & References
- OpenTelemetry Logs specification — trace and span correlation in structured logs.
- Azure Monitor OpenTelemetry — supported telemetry signals and Application Insights setup.
- Microsoft Foundry tracing — agent trace contents, storage, and data-handling considerations.
- Railway third-party observability — application-side OpenTelemetry export and shutdown guidance.
- Railway metrics — native container and service metrics.
- Cloudflare Workers traces — automatic tracing and native retention.
- Cloudflare OpenTelemetry export — OTLP trace and log export capabilities and current limitations.
- Cloudflare Durable Objects — per-instance coordination and transactional storage.
- Cloudflare Workflows — durable multi-step execution, retries, and external events.
- Cloudflare D1 — relational storage for records shared across runs.
- Cloudflare R2 — object storage for larger artifacts and archived transcripts.
- Cloudflare Workers limits — CPU, memory, connection, and log limits.
I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].
-
When You Should Skip Terraform Entirely
The last post in this series made the case that OpenTofu is the no-regrets default for new infrastructure projects. That’s true for the broad case of cloud-agnostic or multi-cloud setups where HCL parity, provider breadth, and a Linux Foundation governance model matter.
It’s also not the whole story. There are at least three common scenarios where the right answer in 2026 isn’t Terraform or OpenTofu. It’s the cloud-native tool the hyperscaler ships with its platform. AWS has CloudFormation and the CDK. Azure has Bicep. GCP has Config Connector. Each one is technically superior to Terraform inside its own ecosystem, and each one removes a category of operational pain that Terraform inflicts.
If you reflexively reach for Terraform every time, you’re probably overpaying in complexity for a multi-cloud option you’ll never exercise.
The Small AWS-Native Startup: Use CDK
If your engineering team is small, you’re shipping a SaaS product, and you’re 100% on AWS, you should probably ignore Terraform entirely. The right tool is the AWS Cloud Development Kit, layered on top of CloudFormation.
The fundamental win is that CloudFormation eliminates state management. There is no
terraform.tfstatefile. No S3 bucket to provision. No DynamoDB lock table. No state-encryption configuration to figure out. The state lives in the AWS control plane, AWS manages locking and consistency, and your CI pipeline doesn’t need to know about any of that. For a small team, that’s a meaningful operational tax you don’t pay.The CDK is the part that makes this pleasant. It lets you define infrastructure in TypeScript, Python, Java, C#, or Go; so the languages your application engineers already know. There’s no HCL learning curve, no Sentinel policy DSL, no jq-in-bash to manipulate plan output. You write code, the CDK synthesizes CloudFormation templates, CloudFormation provisions the infrastructure.
The objection people raise is “what if you go multi-cloud later?” In practice, most SaaS startups don’t. They get acquired, they pivot, or they grow large enough to have a dedicated platform team that does the migration deliberately. Optimizing for a hypothetical multi-cloud future that 90% of teams will never need is the textbook definition of premature abstraction. If you’re an AWS-native startup with fewer than 50 engineers and no concrete plans to leave AWS, the cost of running Terraform-as-multi-cloud-insurance is higher than the cost of a future migration that probably won’t happen.
The Azure Enterprise: Bicep, Unless You Need More
For organizations heavily invested in Microsoft’s stack, so Azure for compute, Azure DevOps for CI/CD and EntraID for identity; Bicep is the technically correct choice for most workloads.
Bicep is Azure’s domain-specific language for infrastructure, designed as a replacement for the verbose ARM JSON templates everyone hated. Like CloudFormation, it’s stateless. You submit a desired-state Bicep file to the ARM control plane and ARM reconciles. No state file, no remote backend, no risk of corruption. Authentication is whatever RBAC permissions the deploying identity already has, with no provider credential configuration required.
Bicep also gets day-zero feature support for new Azure capabilities. When Microsoft ships a new service, you can use it in Bicep the same day. The Terraform AzureRM provider has historically lagged by weeks or months, occasionally longer.
The catch is scope. Bicep manages Azure. That’s the entire surface area. Larger organizations tend to need management of things outside Azure too: GitHub repositories and branch protection, EntraID groups, Datadog monitors, PagerDuty escalation policies, whatever SaaS services your platform touches. Bicep has no answer for any of that.
That leaves two paths. The first is a hybrid: Bicep for Azure, separate tools for everything else, accept the cost of context-switching and the inability to express cross-domain dependencies in a single deployment. The second is Terraform or OpenTofu for everything, accepting the heavier operational tax of stateful IaC, in exchange for one tool that can do all of it. Neither is wrong; they’re different tradeoffs against the same constraint.
The decision rule: if you’re managing only Azure resources, use Bicep. If you have cross-domain provisioning needs and you’d rather not maintain two parallel IaC stacks, Terraform (or OpenTofu) earns its keep.
The GCP/Kubernetes Shop: Hybrid by Design
For organizations heavily committed to Google Cloud and running most workloads on GKE, the right architecture isn’t either/or. It’s a hybrid that uses Terraform for the foundation and Config Connector for the application layer.
Config Connector is a GCP-shipped Kubernetes add-on. It lets you manage GCP resources — Cloud SQL instances, Pub/Sub topics, storage buckets, service accounts — as standard Kubernetes Custom Resources. You write a YAML manifest, you
kubectl apply, and a controller in the cluster reconciles the real-world GCP resource to match.The differentiator is continuous reconciliation. Terraform is episodic: it checks state at
planandapplytime, and the rest of the time your infrastructure is unmonitored. If someone clicks around in the GCP console and manually changes a setting, Terraform won’t notice until the next pipeline run. Config Connector runs a controller loop that polls continuously. Manual drift gets reverted in real time.The right architectural boundary:
- Platform layer (Terraform/OpenTofu): VPCs, subnets, foundational IAM, the GKE clusters themselves. These are slow-moving, security-critical, and you want a deliberate pipeline approval flow for them.
- Application layer (Config Connector): Application-specific buckets, databases, service accounts, Pub/Sub topics. Application teams own these via the same YAML manifests they use for their pods, with the same GitOps workflow they already understand.
This pattern gives platform teams strict guardrails on the foundation while letting application developers self-serve the resources their services need, without filing a Terraform PR every time they want a new bucket.
The Decision Rule
The honest version of all of this: Terraform/OpenTofu is the right answer when you need cross-domain or cross-cloud governance. For everything else, the cloud-native tool is usually less work, more current with the platform, and avoids the operational tax of state management.
A reasonable decision tree:
- Single-cloud, small team, AWS: AWS CDK + CloudFormation.
- Single-cloud, single-domain, Azure: Bicep.
- GCP with heavy Kubernetes use: Hybrid — Terraform/OpenTofu for foundation, Config Connector for application resources.
- Multi-cloud, or cross-domain platform engineering (GitHub + cloud + identity + monitoring): OpenTofu.
The mistake I think most teams are making is to default to Terraform because it’s the tool the senior engineer learned in their last job. The platform-engineering pitch … “we’ll standardize on Terraform so we can move to any cloud later” is correct in theory but almost never exercised in practice. If your team isn’t using the cross-cloud capability today, you’re paying for an insurance policy you’ll never collect on.
Next post in this series digs into the other side of that calculation: what HCP Terraform actually costs in 2026, and why even teams that need cloud-agnostic IaC are looking for the exit from the commercial orchestration platforms.
Sources
- Bicep Vs Terraform: Choosing The Best IaC Tool For Azure — Synextra
- Terraform vs Bicep vs ARM Templates 2026 Compared — Exodata
- Comparing Terraform and Bicep — Microsoft Learn
- Terraform vs Bicep vs ARM: Lessons from the Trenches — Vaibhav Gujral
- How to Use the GCP Config Connector with Terraform — OneUptime
- How Config Connector compares for infrastructure management — Google Cloud Blog
- Are Terraform’s days numbered? — Alistair Grew
I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].