How to make your AI agent reliable on the modern web

Last updated: October 29, 2026. Kernel capabilities checked against the Kernel docs on that date.

TL;DR: Most AI agents that stall on real websites do so for ordinary reasons. The site can't tell who the agent is, the browser doesn't behave like a person's browser, the agent signs in from scratch on every run, or nobody is there when the site asks for a person. Reliable agents fix those four things: they identify themselves with Web Bot Auth where sites support it, use a real browser the way a person does, keep logins in persistent profiles, and hand verification steps to a human. Kernel ships each of those as configuration on one runtime. This guide covers how to use them and how to measure whether they're working.

Why agents struggle on real websites

Websites have good reasons to be careful with automated traffic: fraud, fake accounts, credential stuffing, and scraping that ignores their terms. Most sites run a bot-management service that sorts sessions into people, known good bots, and everything else. Agents doing legitimate work for a user often land in the last group, not because of anything they did, but because nothing about the session tells the site what it is.

In practice, agent sessions stall for a handful of reasons:

• The site can't tell who the agent is. Without a verifiable identity, a site can only guess what a session is, and when it guesses wrong it asks for verification or turns the session away.
• The browser environment is unusual. A cloud server's IP, a viewport no laptop has, a timezone that doesn't match the IP, or software rendering instead of a GPU all make a session look unlike any real visitor.
• Every run starts cold. An agent that signs in from scratch each time looks like a new visitor every time, and repeated logins add friction for the site and for the account holder.
• Nobody is there for the verification step. Sites sometimes ask for a person: a CAPTCHA, a one-time code, a consent screen. An agent with no way to hand off stalls.
• The site changed. Login flows, page structure and access requirements change without notice, and a pipeline built on one fixed setup breaks when they do.

None of these are solved by hiding what the agent is. They're solved by identifying the agent where the site supports it, and by running a browser session that behaves the way the site was built to expect.

Identify your agent with Web Bot Auth

The most durable fix is identity. Web Bot Auth lets an automated client cryptographically sign its HTTP requests (Ed25519 keys and HTTP Message Signatures, per RFC 9421), so a site can verify who is behind the traffic instead of inferring it. The IETF has an active webbotauth working group, and its specifications are still Internet-Drafts, so expect details to move.

Identity changes the conversation between an agent and a website. A signed request tells the site which operator is behind the traffic and lets the site apply a policy to it: allow it, rate-limit it, or decline it. Sites that recognize and allow an identity can let that agent continue with fewer challenges and interruptions, and most major CDNs, WAFs and bot-management services verify Web Bot Auth signatures.

Kernel identifies its agents with Web Bot Auth and is listed in Cloudflare's bots and agents directory and Vercel's public directory. With Kernel's Web Bot Auth token, your browser agent signs its requests with Kernel's identity, so you don't have to register and maintain your own. The token is an opt-in feature for Start-Up and Enterprise customers, enabled on request: contact support with your use case and the sites your agent works with.

Two things to keep in mind. Identity is something the site evaluates, not a pass: each site still decides whether to allow a given agent, and Web Bot Auth doesn't replace logins, permissions or rate limits. And it pays off on the sites that verify it, so turn it on for the destinations your agent works with and confirm it's recognized there.

Use a real browser the way a person does

Identity tells a site who the agent is. The browser determines whether the site works for the agent the way it works for everyone else. The goal is fidelity: render pages, run scripts and send input the way a person's browser does, so the site behaves the way its developers intended.

Run a full browser with a real display

Headless browsers skip parts of the rendering pipeline. That's fine for simple, static pages, and it's cheaper, but sites built on canvas, WebGL, video or complex layouts can render differently or not at all. A headful browser renders to an actual display with GPU-backed compositing, so it runs the full web platform and draws the page exactly as a person's browser does, because it's the same browser doing the same work.

On Kernel, headful is the default, and it's also what powers live view and replays. Headless is a lighter option for simple sites, and headful with GPU acceleration is available on the Start-Up and Enterprise plans for WebGL, video, and visual agents that need real hardware rendering. Headful also matters for computer-use agents (Anthropic, OpenAI or Gemini models working from screenshots, mouse and keyboard), because the model needs a real rendered display to reason about.

Start from realistic defaults

Every Kernel browser launches with a consistent, realistic Chrome configuration, so the viewport, timezone, fonts and browser APIs line up the way they do on a real machine. Stealth mode adds two managed services on top: a default ISP proxy for a stable exit IP, and automatic CAPTCHA handling. It's a flag at session creation (stealth: true) on every plan, including the free one, and both services are opt-out. The browser build and runtime layers are open source, so you can see exactly how they're configured.

Send input the way a person does

How the agent drives the browser matters as much as the browser itself. Kernel's computer controls API drives the browser through OS-level mouse and keyboard input, like a person at a keyboard. Playwright execution runs your Playwright code inside the browser's own VM, next to the browser rather than across the network. Both work well for agent loops.

Pace multi-step flows the way a person would, too. An agent that fills a form faster than anyone could read it looks like a malfunction to the site, and it's the fastest way to run into rate limits.

Choose proxies for geography and stable IPs

Where traffic comes from is part of behaving like a normal visitor. Cloud-server IPs are a poor fit for sites built for consumers, and many workflows need a specific location or a stable IP. Pick the proxy type for the job:

• Datacenter: fast and inexpensive. Fine for public docs, your own sites, and pages that don't vary by visitor.
• ISP: static, residential-class IPs. The exit IP stays the same for the whole session, which is what allowlists, login flows and account security checks expect. Stealth mode uses an ISP proxy by default.
• Residential and mobile: consumer IPs for content that varies by location or network. Residential exits can change between connections, so they're a weaker fit for signed-in work.
• Custom: bring IPs you already own, or ones a site's operator has allowlisted for you.

Kernel supports all five, with health checks, geo-targeting and per-session selection; see the proxy docs. Configurable and bring-your-own proxies are on the Start-Up and Enterprise plans. If you have a relationship with a site's operator, the most reliable setup is often the simplest one: ask them to allowlist a static IP or your agent's signed identity.

Sign in once and stay signed in

Most useful agent work happens behind a login, and repeated logins are where agents lose the most time: one-time codes, email verification and security checks on every run. They're also friction for the site and for the account holder. Persist the session instead.

Kernel profiles save cookies and storage from a session, so the next browser that attaches the profile starts signed in. Attach a profile by name when you create a browser, and set save_changes: true on the run that signs in. For the login itself, Managed Auth runs the flow for you, through a hosted login page, an embeddable React component or a programmatic flow, and saves the result to a reusable profile, with health checks and automatic re-authentication when a session expires. If your agent needs to own the login, Fill from Vault fills stored credentials into the current session without putting them in your agent's prompt.

Keep one account per profile, and save changes only on the runs meant to update the session, so a failed run can't overwrite a good login. For the full pattern, including how to keep credentials away from the model, see Agent identity: how to authenticate an AI browser agent.

Bring in a person when a site asks for one

Some steps are meant for a person: a CAPTCHA, a one-time code, a consent screen, a payment confirmation. The reliable pattern is to hand those steps to a human instead of trying to automate around them.

Kernel's live view gives a person real-time access to the agent's browser so they can complete the step, and it embeds in your own app as an iframe, so your user can finish verification without leaving your product. The session keeps its state while it waits, and standby mode stops billing while the browser is idle, so holding a session open through a handoff costs little.

Stealth mode's automatic CAPTCHA handling covers common challenges in the session itself. For steps a site intends a person to complete, and for high-value flows where you want a person to confirm what the agent is about to do, use live view.

A minimal Kernel example

Here's a session set up the way this guide recommends: headful (the default), stealth mode with its default ISP proxy, and a saved profile so the agent starts signed in.

import Kernel from "@onkernel/sdk";
import { chromium } from "playwright";

const kernel = new Kernel(); // reads KERNEL_API_KEY from the environment

// Headful is the default. Stealth mode adds a static ISP proxy and automatic
// CAPTCHA handling. The profile carries cookies and storage from the run that
// signed in, so this session starts signed in too.
const session = await kernel.browsers.create({
  stealth: true,
  profile: { name: "acme-dashboard" },
  timeout_seconds: 3600, // inactivity timeout before Kernel deletes the session
});

// Share this with a person if the site asks for a verification step.
console.log("live view:", session.browser_live_view_url);

try {
  const browser = await chromium.connectOverCDP(session.cdp_ws_url);
  const page = browser.contexts()[0].pages()[0];

  await page.goto("https://dashboard.example.com/reports");
  await page.waitForSelector('[data-testid="report-ready"]', {
    timeout: 60_000,
  });

  console.log(await page.locator('[data-testid="metric"]').allInnerTexts());
} finally {
  // Without this the session lingers until the inactivity timeout fires.
  await kernel.browsers.deleteByID(session.session_id);
}

Create the profile with a sign-in run that sets save_changes: true, or let Managed Auth maintain it for you. On sites that verify Web Bot Auth, add it to the same session; the Web Bot Auth docs cover the setup.

Measure reliability per site

The teams whose agents run reliably measure the right things:

• Track success per site, not overall. An overall success rate averages away the sites that matter. Track task completion per site over time and alert when it drops, because sites change.
• Change one setting at a time. When a site starts failing, change one thing per run (proxy type, profile, display mode, identity), so you know which change fixed it. Kernel's Config Registry suggests browser and proxy settings that have worked for a given site, on the Hobbyist plan and up.
• Back off before retrying. A failed run retried immediately with the same setup usually fails the same way and adds load to the site. Wait, retry, and respect the site's rate limits.
• Record sessions and stream telemetry. Replays show exactly what the browser rendered when a run went wrong, and telemetry streams console, network, page and CAPTCHA events while the run is live.
• Watch for quiet failures. Not every failure is an error page. Check page content against a known-good baseline, so an empty result or an unexpected interstitial doesn't pass as success.

There's no honest universal success rate for agents on the web. Results depend on which sites you work with and how those sites change. The number that matters is task completion on your own sites, so run a proof of concept against them and hold any platform, Kernel included, to that.

Why Kernel

Kernel puts each of these pieces on one runtime, so you configure them per session instead of stitching vendors together:

• Web Bot Auth identity, listed in Cloudflare's and Vercel's directories, opt-in for Start-Up and Enterprise customers.
• Headful browsers by default, plus headless and GPU-accelerated modes.
• Realistic browser defaults on every browser, and stealth mode (a static ISP proxy and automatic CAPTCHA handling) on every plan, including free.
• Datacenter, ISP, residential, mobile and custom proxies, selectable per session.
• Profiles and Managed Auth for persistent logins, with health checks and automatic re-authentication.
• Live view for human handoff, embeddable in your own app.
• OS-level computer controls and in-VM Playwright execution.
• Replays and telemetry for debugging, and standby mode so idle browsers don't bill.
• Works with Playwright, CDP, WebDriver BiDi, Stagehand, Browser Use and computer-use agents.

FAQ

Why does my AI agent keep getting blocked?

Usually because the site can't tell what the session is. A cloud-server IP, an unusual browser environment, a cold login on every run, or an unanswered verification step all make an agent look unlike a normal visitor. Use a realistic, headful browser, keep logins in a profile, hand verification steps to a person, and identify your agent with Web Bot Auth on sites that support it.

What is Web Bot Auth?

A proposed IETF standard that lets automated clients sign their HTTP requests with a cryptographic key (HTTP Message Signatures, RFC 9421), so sites can verify who is behind the traffic and decide how to treat it. Kernel is listed in Cloudflare's and Vercel's bot directories, and Kernel's Web Bot Auth token is available on request for Start-Up and Enterprise customers.

Does Web Bot Auth guarantee my agent gets in?

No. It proves who your agent is, and each site still decides whether to allow it. It also doesn't replace logins, permissions or rate limits.

Do I need a headful browser?

For simple, static pages, headless is fine and cheaper. Use headful when a site depends on real rendering (canvas, WebGL, video, complex layouts), when you want live view or replays, or when a computer-use model needs to see the screen. On Kernel, headful is the default.

How should my agent handle CAPTCHAs and verification steps?

Stealth mode includes automatic CAPTCHA handling for common challenges. For one-time codes, consent screens and anything a site intends a person to complete, hand the step to a person through live view and let the agent continue once it's done.

Which proxy should I use?

Match it to the job. A static ISP IP suits signed-in work and allowlists, residential or mobile IPs suit location-specific content, and datacenter IPs suit public pages and your own sites. Stealth mode routes through a static ISP proxy by default, so many workflows need no proxy configuration at all.

How do I turn on stealth mode on Kernel?

Pass stealth: true when you create a browser. Every Kernel browser already ships with realistic defaults; stealth mode adds a default ISP proxy and automatic CAPTCHA handling, both opt-out. It's available on every plan, including free.

How do I debug a failed run on Kernel?

Stream telemetry while the run is live, open live view to watch the run and take over if needed, and record a replay so you can rewatch the moment it went wrong. Replays need a headful session.

Websites and browser platforms change often. Test against the sites your agent actually works with before you commit to a platform.