Building an AI Web Agent with Proxy Rotation

AI web agents hit detection faster than ordinary scrapers because they combine repeated page fetches, tool retries, browser automation, and multi-step flows like login, search, click, and extract. If every run shares one obvious route, or if one task changes IP halfway through a workflow, the agent starts to look synthetic.

The fix is not "rotate on every request." The fix is to bind one believable network identity to one agent task, keep it stable for the duration of that task, and rotate cleanly before the next task starts.

Why proxy choice matters for AI agents

AI agents usually fall into three buckets, and each one benefits from a different proxy policy.

• Simple fetch agents that call requests, httpx, or a LangChain tool can often use the default rotating gateway route.
• Browser agents using Playwright usually need sticky routing so login, cookies, and follow-up clicks all happen on one identity.
• Long-lived account agents may be better on assigned/static endpoints when consistency matters more than frequent rotation.

For the query "AI agent proxy", the common mistake is treating proxy rotation like a stateless scraper concern. Agents are stateful. Cookies, retries, memory, and follow-up actions all make route stability more important.

Step-by-step NinjaProxy setup

1. Open Portal → Rotating Gateway IPs and copy your rotating HTTP endpoint exactly as shown.
2. Open account settings and copy your portal username and per-user apiKey.
3. Keep the rotating endpoint fixed and add routing controls to the username only when you need sticky behavior.
4. For a browser-based agent task, use a structured username like:

<USERNAME>--session-agent-task-001--duration-600--provider-res--geo-country-us

That format matches the current public authentication docs. The session token pins the route for the workflow, duration bounds how long the gateway should try to keep it stable, provider-res is the safer default for stricter targets, and geo-country-us keeps the route aligned with the market you want to appear from.

If your agent runs from fixed egress IPs, IP whitelist mode can remove credentials from requests. For most cloud workers, local tools, and browser agents, username + API key is still the safer default.

Full working example: one proxy identity per agent task

The example below uses Playwright because browser agents are where rotation policy usually breaks down. It launches one browser per task, keeps one sticky residential route for the whole task, and closes the browser before the next identity starts.

import { chromium } from "playwright"

const ROTATING_HTTP_ENDPOINT = "<ROTATING_HTTP_ENDPOINT>"
const USERNAME = "<USERNAME>"
const API_KEY = "<API_KEY>"

function buildProxyUsername({
  sessionId,
  duration = 600,
  provider = "res",
  country = "us",
}) {
  return [
    USERNAME,
    `--session-${sessionId}`,
    `--duration-${duration}`,
    `--provider-${provider}`,
    `--geo-country-${country}`,
  ].join("")
}

async function browseTask({ sessionId, urls }) {
  const browser = await chromium.launch({
    headless: true,
    proxy: {
      server: `http://${ROTATING_HTTP_ENDPOINT}`,
      username: buildProxyUsername({ sessionId }),
      password: API_KEY,
    },
  })

  const context = await browser.newContext({
    locale: "en-US",
    timezoneId: "America/New_York",
    viewport: { width: 1440, height: 900 },
  })

  const page = await context.newPage()
  const results = []

  for (const url of urls) {
    await page.goto(url, {
      waitUntil: "networkidle",
      timeout: 30000,
    })

    results.push({
      url,
      title: await page.title(),
      text: (await page.textContent("body"))?.slice(0, 500) ?? "",
    })
  }

  await context.close()
  await browser.close()
  return results
}

async function main() {
  const researchTask = await browseTask({
    sessionId: "agent-research-001",
    urls: [
      "https://ip.ninjasproxy.com/",
      "https://example.com/",
    ],
  })

  const checkoutTask = await browseTask({
    sessionId: "agent-checkout-001",
    urls: [
      "https://example.com/login",
      "https://example.com/account",
    ],
  })

  console.log({ researchTask, checkoutTask })
}

main().catch((error) => {
  console.error(error)
  process.exit(1)
})

LangChain tool integration

If your agent framework uses tools, expose the proxy-aware browser function as one tool call per task. The critical design rule stays the same: create a new sessionId when the next job should look like a fresh identity, not halfway through the current job.

import { tool } from "@langchain/core/tools"
import { z } from "zod"

const browseWithProxy = tool(
  async ({ sessionId, urls }) => {
    return JSON.stringify(await browseTask({ sessionId, urls }))
  },
  {
    name: "browse_with_proxy",
    description: "Browse one task with a sticky NinjaProxy session",
    schema: z.object({
      sessionId: z.string(),
      urls: z.array(z.string().url()).min(1),
    }),
  }
)

const result = await browseWithProxy.invoke({
  sessionId: "langchain-research-001",
  urls: ["https://example.com/", "https://example.com/pricing"],
})

console.log(result)

Where MCP fits

MCP (Model Context Protocol) lets your IDE or AI assistant pull live docs and API references into the current task before it generates setup code.

If your editor or assistant supports MCP-style bootstrapping, point it at NinjaProxy's official resources before generating setup code or troubleshooting prompts.

• llms.txt
• MCP guide
• mcp.json bootstrap manifest

Those resources help the assistant stay grounded in the current auth format, endpoint model, and API references. Use the official docs first and avoid third-party MCP listings unless they are explicitly confirmed by NinjaProxy.

Common errors and fixes

• 407 Proxy Authentication Required usually means the username string or API key is wrong. Re-copy the base username and append routing controls to the username only.
• Agent gets challenged after login usually means the route changed mid-flow or the browser reused storage from a previous identity.
• No rotation between tasks usually means you kept reusing the same --session-... token.
• Country targeting looks wrong usually means the route geography and browser locale/timezone do not match.
• Requests fail only on cloud workers usually points to the wrong auth mode. Use username + API key unless that worker's egress IP is explicitly allowlisted.

Recommended implementation order

1. Verify one browser session against https://ip.ninjasproxy.com/.
2. Add a sticky session token and confirm the route stays stable through a multi-step flow.
3. Wrap that browser task in your agent framework or LangChain tool.
4. Only then raise concurrency and add more aggressive rotation.

Relevant docs

• Using NinjaProxy with AI Agents
• Authentication
• Rotating proxies
• Rate limits
• MCP guide
• API reference