Vibe Coding Agent
Security & Evaluation

Software engineering is undergoing its most significant transformation since the introduction of high-level languages. The deepest shift is the transition from writing code to expressing intent — trusting intelligent systems to translate that intent into working software. This new paradigm spans a spectrum: from casual "vibe coding", where the developer describes what they want in natural language and accepts whatever the AI generates, to disciplined "agentic engineering", where the AI acts as an implementation engine within carefully designed constraints.

To operationalize vibe coding in the enterprise, we need to redefine trust along two distinct axes:

This whitepaper provides the definitive 2026 framework for both: establishing the strict "safety harness" needed to protect non-deterministic agents, and opening the "glass box" to rigorously measure the quality, efficiency, and alignment of internal reasoning.

Intent-driven development drastically accelerates innovation, but introduces unprecedented security vulnerabilities. We are no longer simply protecting web applications against traditional exploits — we are tasked with protecting a non-human workforce that possesses the ambient agency to execute generated code, access sensitive internal APIs, and dynamically modify production environments.

Traditional software testing and security models rely on deterministic logic, where a fixed set of inputs produces a predictable output. However, in an agentic system, an agent may hold a valid access token yet operate autonomously with misaligned intent. A critical realization: a raw AI model is NOT an agent. It only becomes one when wrapped in a "harness" — the scaffolding that gives it state, tool execution, feedback loops, and enforceable constraints. Securing this new paradigm requires shifting focus from protecting code syntax to protecting the harness.

To achieve this continuous Effective Trust and protect the chaotic reality of vibe coding, we developed a layered defence-in-depth architecture: a strict 7-pillar baseline, extended by high-velocity execution controls, and crowned by active agentic defense mechanisms.

In traditional enterprise environments, security is deterministic. Applications rely on predictable code syntax, and access is governed by static Identity-as-a-Perimeter models, such as RBAC. If a user or service account has the correct authorization token, the system implicitly trusts the execution path.

The agentic environment fundamentally breaks this model. Since everyday agent failures often trace back to a missing tool, a vague rule, or a nonexistent guardrail, organizations must migrate to a "Context-as-a-Perimeter" model. Since we must assume the underlying model can fail or be compromised, security cannot reside solely in the AI — we need to apply a strict, external "safety envelope" spanning multiple disciplines.

Pillar 1 — Infrastructure & Networking

Cloud Infrastructure Engineers must protect the foundational environment against upstream poisoning and container escapes. Since the harness must dictate where the agent's code actually runs and what it cannot reach, we isolate runtime execution inside kernel-level ephemeral sandboxes (such as gVisor). In addition, strict network egress governance ensures that agent-generated data travels only through authorized offline caches or explicit internal proxies, preventing inadvertent public exfiltration.

Pillar 2 — Data

Data Architects face the threat of agents leaking sensitive information from their context windows or ingesting poisoned RAG data. The practice of "context engineering" provides agents with rich, structured information about codebases and intent. To protect this sensitive context: data at rest is protected with Customer-Managed Encryption Keys (CMEK), data in transit via mutual TLS (mTLS). Data access must be strictly scoped to enforce the principle of least privilege. Crucially, long-term memory stores — particularly Vector Databases — must enforce strict tenant partitioning to prevent Cross-Tenant Vector Poisoning: a malicious payload ingested by one tenant cannot be retrieved during another tenant's similarity search.

Pillar 3 — Model

AI Engineers must defend the application's core logic against semantic attacks that subvert the model's instructions. In agentic workflows, the prompt and the "Instructions and Rule Files" that define what the agent is forbidden to do serve as the new source code. Protecting this pillar requires treating the model's system instructions and prompt templates as highly sensitive, cryptographically attested artifacts.

Pillar 4 — Application & Runtime

Agents operate interactively — traditional rules-based firewalls are insufficient. We deploy LLM firewalls for dynamic prompt and response filtering, along with deterministic "hooks" that run at specific points in the lifecycle (before a tool call, after a file edit). Centralised Agent Gateways act as the "bouncers of the ecosystem", governing Agent-to-Agent (A2A) orchestration to prevent unauthorized lateral movement.

Pillar 5 — Identity & Access Management

The main risk is the "Confused Deputy": an over-privileged agent is tricked into executing unauthorized commands. We solve this by assigning unique cryptographic identities (SPIFFE IDs) to each agent. Access depends on Attribute-Based Access Control (ABAC) and Just-In-Time (JIT) token downscoping, enforcing a strict permission matrix of Intent × User × Time — agents receive fresh, hyper-restricted credentials that expire immediately after the task completes.

Pillar 6 — Observability & Security Ops

SecOps must combat "invisible failures" where an agent silently cascades into an infinite reasoning loop. Without observability — logs, traces and metering — there is no way to know whether an agent is performing well or silently drifting. We solve this by deploying an autonomous SecOps triad: the Blue Team uses OpenTelemetry and Agent Behavioural Analytics (ABA), the Red Team proactively simulates multi-hop attacks, and the Green Team executes "Stateful Quarantines" if an anomaly is detected.

Pillar 7 — Governance

Governance Officers must ensure that autonomous decisions meet rigorous regulatory standards. Beyond traditional data frameworks, governance must now strictly adhere to the EU AI Act, mandating Algorithmic Impact Assessments for high-risk autonomous agents. Organizations must create an immutable audit trail that strictly attributes every real-world action to a specific agent and to the human who deployed or approved it. We replace simple approval buttons with mandatory "Logic Reviews" (translating complex syntax into plain language) and use Risk-Stratified Attestation to bind digital signatures to the agent's outputs.

The core mechanism of vibe coding relies on dynamically translating human intention into executable logic on the fly. However, vibe-coded agents rarely write perfect code on the first attempt. The reality of intent-driven development is a high-speed iterative cycle: the agent writes a script, executes it, reads the resulting error logs and autonomously rewrites the logic until it aligns with the user's "vibe". Because this generative process introduces high variability, the resulting code cannot be implicitly trusted. Running these dynamically generated scripts directly alongside the root agent or on standard host infrastructure introduces an unacceptable level of risk.

Agentic code generation introduces a highly specific and dangerous supply chain vulnerability: LLMs frequently hallucinate software packages that don't exist. Malicious actors actively monitor developer forums and AI outputs for these hallucinations, proactively publishing malware under those exact, fabricated names. Because autonomous agents can alter dependency graphs without human confirmation, a single hallucination can pull malware directly into the build environment.

While kernel-level isolation protects the host infrastructure, organizations must also secure the network boundary. In traditional software, outbound traffic is highly predictable. In vibe-coded systems, egress is non-deterministic because it is driven by the dynamic use of newly generated tools. A common failure mode: the agent inadvertently attempts to push unverified code to live environments or exfiltrate sensitive data. However, relying on a simple allowlist of approved domains is insufficient — an allowlist cannot protect an agent against indirect prompt injections hidden in third-party web pages.

To mitigate this risk, agents must be restricted to non-interactive internet access. Administrators must force the agent to fetch external information exclusively through offline caches or dedicated, pre-sanitized web-crawling services. By forcing all data to travel strictly through governed paths, organizations prevent the agent from interacting directly with malicious payloads or inadvertently downloading typosquatted packages.

A perfect sandbox does not prevent an agent from writing fundamentally flawed code or connecting to a malicious internal tool. Vibe coding inherently prioritizes immediate functionality over secure design — the generated applications frequently contain severe structural flaws. Users often implicitly trust generated code simply because it compiles and runs without errors, blind to the fact that the application may have completely bypassed standard backend security controls.

To catch these structural flaws, we need to balance developer speed with strict security enforcement. Aggressively trying to block unsafe prompts directly in the IDE is easily bypassed and causes excessive friction. Instead: "shift left" via Developer Advisory Linters in the IDE (real-time guidance), while non-negotiable enforcement is pushed to deterministic checks in the CI/CD pipeline. Integrating SAST (Static Application Security Testing) and SCA (Software Composition Analysis) into the pipeline ensures that all generated application logic is deterministically scanned for vulnerable dependencies and structural flaws before reaching production.

Once the vibe-coded application logic is deployed, security controls must govern how the agent interacts with external systems. Agents increasingly rely on tool coordination frameworks, such as the Model Context Protocol (MCP), which allow them to discover and connect to external or internal enterprise servers at runtime.

To secure Agent-to-Tool and Agent-to-Agent (A2A) orchestration, organizations must deploy a runtime LLM firewall in front of the active agent to dynamically intercept opportunistic prompt injections. In addition, a Centralised Agent Gateway must evaluate Contextual Authorization — acting as the enforcer of the "Association" trust factor, dynamically verifying whether the agent's request to call a tool aligns perfectly with the developer's original intent.

By routing all tool invocations through a Centralised Agent Gateway, we successfully limit the agent's ability to execute unauthorized actions in the runtime pillar. However, the integrity of these decisions ultimately depends on how we verify the actor behind the agent, manage credentials under pressure and establish human control over high-risk actions. This shifts the security boundary from application orchestration to cryptographic identity verification and the mechanics of human authorization.

Because developers often use vague or highly abstracted natural language to generate code (for example, "fix the backend routing"), the resulting agentic workflows are inherently broad. Granting these autonomous agents shared, long-lived service identities creates an uncontrollable insider threat vector. To secure this pillar, organizations must assign unique cryptographic identities (such as SPIFFE IDs) to each individual agent.

The Confused Deputy and Delegated vs. Agentic Identity

Even with a unique identity, a vibe-coded agent remains highly susceptible to the Confused Deputy problem. This occurs when a prompt injection — such as a malicious instruction hidden inside an open-source repository that a developer unknowingly pasted into the IDE's context window — tricks an over-privileged agent into executing an unauthorized command on the attacker's behalf.

To resolve this, an agent must never be the final arbiter of access. A distinct, observable agentic identity ensures that its permissions remain strictly scoped and subject to granular audit logs.

Zero Ambient Authority and JIT Downscoping

Building on this, the architecture must enforce Zero Ambient Authority. An agent executing a "vibe" must never inherit the developer's full administrative privileges and environments. Instead, the system relies on Just-In-Time (JIT) token downscoping.

While automated identity constraints handle most routine tasks, high-risk actions — such as modifying production databases, executing financial transfers or altering IAM configurations — require explicit verification and cannot rely on simple "approve/deny" buttons. Because vibe coders often rely on AI to write complex syntax they may not fully understand (the "It Works, Ship It" fallacy), simple approval gates quickly cause confirmation fatigue, leading developers to blindly authorize code they don't comprehend.

To combat this, the system must implement structured, context-aware elicitation. The agent is forced to actively request confirmation based on the specific context of a high-risk action, which must be accompanied by two distinct security boundaries:

Even with perfect identity verification and granular human authorization, malicious instructions can still slip past the initial defenses. When developers blindly trust open-source repositories or pull massive blocks of unstructured context, they invite sophisticated semantic attacks that bypass standard IAM controls. To proactively detect and neutralize these hidden threats as code is generated, security operations must evolve to match the exact speed of the agentic workflow.

In a vibe-coded environment, application logic is generated, executed and discarded at unprecedented speed. Because the attack surface is non-deterministic and driven by natural language, traditional manual security operations simply do not scale. To secure autonomous systems, security operations themselves must become agentic, requiring the deployment of a continuous triad of Red, Blue and Green teaming running in parallel with the developer's workflow.

Invisible Payloads and Repository Poisoning

Before deploying defensive operations, we need to understand the stealthy nature of agentic threats. Repositories act as a highly effective attack vector. Threat actors can compromise repositories by inserting zero-width Unicode characters or homoglyphs directly into the codebase. Knostic warns that these "invisible payloads hide in plain sight and bypass human review". Because agents manipulate and replicate code far faster than a human developer, a single hidden payload can "spread across hundreds of files in minutes" before anyone notices.

To prevent agents from generating massive, unreviewable code modifications during this process, developers must restrict the agent's output to small batch sizes. This is ideally achieved using a test-driven loop where the system blocks the agent from modifying tests and implementation code simultaneously, ensuring the test remains as an objective baseline.

With these constraints in place, the Red, Blue and Green triad dynamically adapts the primary agent's behavior at runtime in three distinct phases:

To ensure these automated defense mechanisms can successfully intervene, the security triad requires a granular, unimpeded view into the agent's internal reasoning. An agentic security operation is entirely blind if it only looks at the final code output. We need to shift the focus from observing the host infrastructure to observing the agent's "mind", creating an immutable audit trail that maps exactly how a diffuse intent translates into a real-world action.

To effectively secure and evaluate a vibe-coded agent, we need to recognize a fundamental rule: you cannot secure what you cannot see. In traditional microservices, an HTTP 200 OK status indicates a successful operation. However, in an agentic system, a "success" status may merely mask a scenario where the agent's internal logic has silently cascaded into a hallucination loop. This introduces the critical risk of Denial of Wallet (DoW) attacks, where adversaries intentionally trigger infinite, computationally expensive API loops to deliberately bankrupt the organization's cloud and LLM billing accounts.

Observability is no longer merely an operational concern about uptime and latency; it is a strict security requirement to illuminate the "glass box" of non-deterministic logic.

Tracing the "Vibe Trajectory" and Content Scanning

To answer the critical question "Why did an agent do that?", security teams must build a unified chronological lens to see the agent's cognitive steps. Using standard telemetry frameworks like OpenTelemetry, enterprises can aggregate diverse signals — API calls, tool inputs/outputs, RAG retrievals, and token latency — into a complete Vibe Trajectory.

Tracing this trajectory requires logging the massive cognitive leap from the user's initial prompt to the compiled Abstract Syntax Tree (AST). To fortify this trace, organizations must pair traditional logging with Centralised Content Scanning, explicitly designed to inspect all dynamic code snippets or scripts retrieved by the agent at runtime. This trace securely links the agent's internal reasoning loop to its physical actions, supporting rigorous third-party security audits.

Measuring Intent Drift and Trust Decay

As a vibe-coded agent dynamically generates logic and pulls in new tools, its security perimeter constantly fluctuates, rendering static asset inventories (SBOMs) instantly obsolete. Instead, observability platforms must monitor a Runtime Agent Bill of Materials (AgBOM) — a living document that maps the agent's active blast radius every millisecond.

Since trust in an autonomous system is a degradable asset, the architecture continuously monitors for Intent Drift. The principle of Trust Decay dictates that trust is lost when the agent's internal chain of thought pursues sub-goals that diverge from the original human "vibe". For example, a simple prompt to "optimize the database query" can maliciously drift into the agent attempting to download a new unauthorized indexing library.

Checkpoints and Stateful Circuit Breakers

To prevent destructive actions when this drift occurs, the observability pillar must proactively manage state. Before an agent executes any codebase modification, the system must generate a version control checkpoint.

For developers operationalizing these concepts, securing a vibe-coded architecture depends on abandoning implicit trust and implementing the following practical baseline:

Implementing these security controls helps our vibe-coded agents operate safely within a secure, well-governed perimeter. However, a secure agent is not inherently an effective agent. Security guarantees that the agent did nothing malicious or unauthorized — but how do we definitively prove that it actually achieved the user's nuanced intent?

To truly operationalize these agents, we need to go beyond securing the perimeter and open the "glass box" to measure the quality, efficiency, and alignment of internal reasoning. This brings us to the next crucial phase of the pipeline: Agent Evaluation.

The previous sections covered the security controls that restrict what a vibe-coded agent can do. These controls do not answer the question the developer actually has: did the agent build what I asked for, and is it good? A vibe-coded agent can pass every security check and still misread the developer's intent, ignore project conventions, or break an unrelated feature.

The following sections are structured around three questions: why vibe coding evaluation is different from evaluating other software, what to evaluate, and how to evaluate. Two areas are deliberately out of scope for this framework, both deserving dedicated treatment: subjective evaluation of non-verifiable outputs, where quality is defined by user or business preferences rather than ground truth; and the feedback loop of user corrections back into the model, the harness, or the eval suite to drive improvement.

Figure 2 · The vibe coding agent evaluation framework

Evaluating vibe coding agents is not the same problem as evaluating deterministic software, and it is also not the same problem as evaluating a customer-service agent or a research agent. Three things make it unique:

These three constraints shape every dimension, method, and tip that follows.

The evaluation of vibe coding agents is divided into seven dimensions, in two groups. User-facing dimensions are what the developer experiences directly. Internal dimensions describe what the agent does invisibly to the user. In addition, Safety and Responsible AI is transversal — it intersects multiple dimensions (code vulnerabilities, refusal behavior, content safety, IP exposure) and must be evaluated alongside each of them.

👤 User-Facing Dimensions

🧠 Internal Dimensions

Figure 3 · Evaluation dimensions for vibe coding agents

The seven dimensions are not all measurable in the same way. No single method covers everything, so production pipelines combine several. The matrix below summarizes the evaluation methods and the recommended dimensions for each; the rest of the section describes each one in detail.

Figure 4 · Evaluation methods and recommended dimensions

While custom evaluation frameworks handle the open-ended nature of vibe coding, standardized tests isolate specific cognitive capabilities from the noise of custom enterprise environments. They provide the empirical baseline needed to trust a non-deterministic system.

Kaggle Agent Exams (SAE) and Zero-Setup Evaluation

Addressing the historically heavy infrastructure burden required to run these benchmarks, the Kaggle Standardised Agent Exams (SAE) represent a massive shift toward "zero-setup" autonomous evaluation. Deployed as a lightweight API integration via a SKILL.md file, SAE allows an agent to autonomously register on Kaggle, fetch exam questions, execute multi-step logic within its own sandboxed environment, and instantly publish its score on a live public leaderboard. It serves as a rigorous, frictionless test of the agent's multi-hop reasoning and adversarial safety under pressure.

To evaluate an agent's internal reasoning, developers need the ability to see it. Observability is the absolute prerequisite for Glass Box evaluation; without it, agent failures appear as inexplicable monolithic events.

Vibe coding has no spec to test against — user intent is unstated and evolves across turns. The closest thing to a spec is the first one or two user messages. Treat them as a rubric: derive evaluation criteria automatically from the session prefix, then score each subsequent turn against them. This is the only practical way to evaluate dimension 1 (intent satisfaction) at scale.

from google import genai client = genai.Client(vertexai=True, project="...", location="us-central1")# Derive criteria from the user's opening turnsopening = " ".join(session.user_messages[:2]) criteria = client.models.generate_content( model="gemini-3-pro", contents=f"Produce 3-5 acceptance criteria for: {opening}. Return JSON.", ).parsed["criteria"]# Score every agent turn against the derived criteriascore = client.models.generate_content( model="gemini-3-pro", contents=f"Does this output satisfy {criteria}? Score 1-5 with rationale."f"Output: {agent_response}", ).parsed

In vibe coding the user judges the output, not the diff. A multimodal model looking at the rendered page captures problems that code-level evaluation misses completely: broken layout on mobile, contrast too low for accessibility, wrong button states. Combine this with Playwright assertions — the judge captures visual and design issues, the assertions capture broken interactivity.

from google import genaifrom google.genai import types client = genai.Client(vertexai=True, project="...", location="...") result = client.models.generate_content( model="gemini-3-pro", contents=["Score this rendered web app against the spec on layout_match, styling, ""and interactive_correctness (1-5 each). Return JSON.", user_spec, types.Part.from_bytes(data=screenshot_bytes, mime_type="image/png"), ], )

A vibe coding session is multi-turn by construction. The relevant question is not "was turn 4 correct?" but "did the user converge on something they wanted?" Sessions that converge in few turns are the success cases. Sessions abandoned mid-flow are the most informative failures — far more so than per-turn errors. Cloud Trace exposes a vibe coding session, instrumented via ADK and Agent Engine, as a single trace tree.

from google.cloud import trace_v2 trace = trace_v2.TraceServiceClient().get_trace( name=f"projects/{project}/traces/{session_id}")def session_outcome(trace):return {"converged":         trace.last_turn.user_signal == "satisfied","turns_to_converge": trace.user_correction_count,"abandoned":         trace.last_user_action == "close","cost_to_converge":  trace.total_token_cost_usd,}

Every "no, not like that" from the user is a labeled failure example, and vibe coding produces them in volume. Cluster them and the agent's systematic gaps become visible — much faster than building a synthetic failure benchmark.

from google import genaifrom sklearn.cluster import KMeans client = genai.Client(vertexai=True, project="...", location="...") corrections = [t.user_message for trace in tracesfor t in trace.turns if t.is_correction] emb = client.models.embed_content( model="text-embedding-005", contents=corrections, ) vectors = [e.values for e in emb.embeddings] clusters = KMeans(n_clusters=8).fit(vectors)# Clusters.labels_ is the prioritized list of failure modes for the next iteration

The transition from syntax to intent is not a future prediction; it is the immediate reality of software development. In 2026, the bottlenecks in software creation have fundamentally shifted. We are no longer waiting for human hands to type boilerplate; we are waiting for human minds to define the boundaries, evaluate the outputs, and secure the execution environment.

Migrating from casual vibe coding to disciplined agentic engineering requires abandoning implicit trust. A raw AI model is merely an engine; it only becomes an enterprise-ready agent when wrapped in a robust harness. By implementing the 7-Pillar Security Architecture — applying strict sandboxing, contextual ABAC, and Red/Blue/Green agentic teaming — organizations can safely contain the blast radius of autonomous actions.

However, security alone only proves the agent caused no harm. By pairing these security controls with a rigorous Evaluation Framework — measuring everything from intent satisfaction and trajectory quality to visual correctness — engineering leaders can confidently prove that the agent actually delivered value.

The teams that will thrive in this new era are those that embrace AI as a high-velocity implementation engine while maintaining the rigorous discipline needed to produce software the world can truly rely on.

🎯 Test your understanding

✅ Implementation checklists

📋 Cheat sheets

# SECURE VIBE CODING — 5 COMMANDMENTS1. SANDBOX THE VIBE LOOP→ kernel-level, network-isolated sandboxes→ SCA scan for hallucinated/vulnerable deps2. SHIFT THE PERIMETER LEFT→ trusted sources + verified internal registries→ deterministic checks at multiple CI/CD points3. ENFORCE ZERO AMBIENT AUTHORITY→ delegated user identities + JIT hyper-restricted tokens→ mandatory "Vibe Diff" for high-stakes actions4. DEPLOY AGENTIC SECOPS→ Virtual Red-Teaming Agents inject "Adversarial Vibes"→ ABA monitors Runtime AgBOM + Green auto-refactors5. TRACE THE EXECUTION TRAJECTORY→ log API calls, tool inputs, reasoning steps→ version-control checkpoints to revert on drift

# EVALUATION — 7 DIMENSIONS × 8 METHODSDIMENSIONS  1 Intent satisfaction  2 Functional correctness3 Visual & behavioural  4 Cost & efficiency5 Code quality          6 Trajectory quality7 Self-repair           RAI Safety (transversal)METHODS     benchmarks → 2,4    functional testing → 2,5security eval → RAI  LLM/Agent-judge → 1,5,6browser-based → 3    trajectory inspect → 6,7human review → 1,5,RAI  online eval → all

# DAY 4 — KEY TERMSEffective Trust      continuous metric: supply chain + identity + runtime behaviour + contextual associationsslopsquatting        malware published under hallucinated pkg namesConfused Deputy      over-privileged agent tricked by prompt injectionZero Ambient Auth    agent never inherits dev's full privilegesVibe Diff            plain-English translation before critical tool runsAgBOM                Runtime Agent Bill of Materials (live blast radius)Intent Drift         chain-of-thought diverges from original vibeTrust Decay          trust is a degradable assetDoW                  Denial of Wallet — bankrupt via infinite API loopscircuit breaker      auto rollback when Trust Score drops below thresholdunderspecification   the prompt is not a spec — gap must be reconstructedSAE                  Kaggle Standardised Agent Exams (zero-setup eval)

📚 Endnotes