An operator's take on the HN Swartz-vs-Meta scraping debate: why individual devs get banned, big AI labs get sued, and proxy vendors are now named as co-defendants.
A Hacker News thread hit 1,671 points this week over a single line: Aaron Swartz was prosecuted for scraping. Meta does it with impunity.
The thread ("Aaron Swartz was prosecuted for scraping, while Meta does it without consequence") is four days old and still climbing. Most of the discussion comes from developers who've scraped something, at some point, and want to know where the line is. Fair question. I don't have a law degree, but I run the infrastructure a meaningful chunk of that traffic — both the small-operator kind and the industrial kind — actually moves through. That's a specific, narrow vantage point, and it's a different one from "I read about this online." Here's what it actually looks like from underneath.
The best thread replies already flag this, and it's worth repeating because it changes the analysis: Aaron Swartz wasn't charged for scraping public data. He was charged under the Computer Fraud and Abuse Act for unauthorized access — connecting hardware to MIT's network in a closet, cycling through IP bans, pulling JSTOR content from a private, paywalled academic system he wasn't authorized to bulk-download from. Meta's large-scale web scraping, by contrast, has mostly targeted publicly accessible pages and (per ongoing litigation) datasets sourced elsewhere — not access-control circumvention on a private network.
That distinction — public-and-accessible vs. private-and-circumvented — is exactly the line that determines criminal exposure under the CFAA. It's also the line most HN commenters, and most developers emailing proxy vendors asking "is this legal," collapse into one bucket labeled "scraping." It isn't one bucket. And getting the two confused is the single most common way I see people talk themselves into more legal anxiety than their actual project warrants, or less than it deserves.
In my experience running this infrastructure, individual developers and small data teams don't generate criminal referrals. What shows up instead is the boring, unglamorous enforcement layer: rate limiting, CAPTCHA walls, IP bans, account terminations, and occasionally a cease-and-desist letter that goes to a lawyer, gets read once, and results in the target quietly stopping. That's the entire enforcement stack most solo devs and small teams will ever encounter — contractual and technical, not criminal.
The CFAA case that actually gets cited against individuals almost always still turns on the access-control question, not the scraping-public-data question: did you keep pulling data after a cease-and-desist explicitly revoked your authorization (the fact pattern in hiQ v. LinkedIn, later narrowed on remand), or did you bypass a login wall, an IP block, or a paywall to get in. Reading and parsing pages that are already open to the public — the thing most people mean when they say "scraping" — is a fundamentally lower-risk category than that, and always has been. The HN thread's anxiety is real, but it's mostly aimed at the wrong target.
In practice, "enforcement" for a solo dev or a two-person data team looks like this: your requests start returning CAPTCHAs, then 403s, then the IP is gone entirely and you're rotating to the next one. If you're persistent and visible enough about it — a recognizable user agent, a fixed IP range, an email address in your outreach — you might get a cease-and-desist. Almost nobody in that category gets a subpoena. The mechanism doing the actual work is rate limiting and IP reputation, not law enforcement, and it's worth designing around that reality instead of the criminal-prosecution scenario most people are actually worried about.
This is where it gets more interesting, and where the HN framing ("impunity") is only half right.
We've covered the broader legal landscape around scraping for AI training data before, and the short version holds: large-scale AI training scrapers aren't facing criminal exposure either — but they're not facing nothing. They're facing civil litigation, which moves slower, costs more, and settles or gets narrowed by motion practice instead of ending in an indictment. The clearest current example is Reddit, Inc. v. Perplexity AI, filed in the Southern District of New York in October 2025 and amended in February 2026. Reddit's theory isn't "you scraped us" in the abstract — it's a DMCA anti-circumvention claim: that the defendants pulled Reddit content out of Google Search results at industrial scale to route around Reddit's own licensing terms, a pattern Reddit's complaint calls "data laundering."
The detail that matters for anyone reading this from inside the proxy industry: Reddit didn't just name Perplexity. It named SerpApi, Oxylabs, and AWMProxy — the infrastructure vendors — as co-defendants alongside the AI company that bought the data. That's new. It's the first suit of this size to argue that the vendor supplying the scraping infrastructure carries exposure alongside the company using it, not just the end customer. And on July 31, 2026, that theory cleared its first hurdle: Judge Paul Engelmayer (SDNY) largely denied the motions to dismiss from Perplexity and SerpApi, holding that Reddit's DMCA anti-circumvention claims can proceed against both the AI company and its scraping vendor, which now sends the case into discovery. The court did trim the complaint — it dismissed a narrower DMCA "trafficking" theory against SerpApi and Reddit's state-law unfair-competition and unjust-enrichment claims — but the core circumvention claims survived against both defendants. That's the part worth tracking: a federal court has now let the vendor-liability theory — that the company providing the scraping tooling can be on the hook alongside the company using it — proceed past dismissal rather than throwing it out. The precedent is no longer hypothetical, even though the final outcome is still years away.
That's the actual asymmetry the HN thread is reaching for, just described imprecisely: it's not "individuals get prosecuted, companies get away with it." It's individuals get quietly rate-limited and banned; companies get sued, over years, by the parties whose data they used — and now the vendors those companies bought infrastructure from are getting pulled into the same suits.
There's a second data point worth adding, because it's moving in parallel and almost nobody outside compliance circles is watching it: the European Data Protection Board adopted Guidelines 03/2026 on web scraping in the context of generative AI on July 7, 2026, and opened it for public consultation through October 30, 2026.
The EDPB's framing is structurally different from the US criminal-law lens the HN thread is implicitly using. It doesn't ask "did you break into something." It asks "what's your GDPR legal basis for processing personal data you scraped, even from public pages." The draft guidance is fairly direct that consent isn't a workable basis for scraping at scale — nobody's opting in to being in a training set — which pushes most scrapers toward "legitimate interest," a basis that still requires a documented balancing test, a public privacy notice, and a pre-collection opt-out mechanism, even when individual notice to each data subject is impractical.
Put the two systems side by side and you get a real answer to "who actually faces consequences," just not the one the thread title implies: in the US, the exposure that materializes is civil litigation over IP/licensing (Reddit v. Perplexity) with vendor liability now spreading to infrastructure providers; in the EU, it's regulatory compliance exposure under GDPR that applies regardless of whether the target data was "public." Neither of those is a criminal prosecution. Both are more consequential, in practice, than most individual developers running a scraper will ever encounter — and less consequential, so far, than the HN thread's framing suggests for the AI labs themselves.
None of this is legal advice — talk to a lawyer for your specific situation, especially if you're operating at a scale or in a jurisdiction where any of the above applies to you directly. But from the infrastructure seat, three things are consistently true:
That last point is exactly why the "agent-friendly, documented, rate-limit-respecting" way of doing this is becoming the practical default rather than a nice-to-have — especially as more of this traffic starts coming from AI agents instead of hand-rolled scripts. If you're building agents that need to hit the web through a proxy layer, NinjaProxy's hosted MCP server gives Claude, Cursor, Codex, and other MCP-capable clients direct, documented access to your proxy account and endpoints — no reverse-engineered auth, no guessing at rate limits. See our AI agent use-case docs for the setup, and our published rate limits for what's actually enforced per plan, not just the marketing number.
The HN thread asked who gets prosecuted for scraping. From where I sit, the more useful question is who gets sued, who gets banned, and who's now being named as a co-defendant alongside their customers — because that's the version of this risk that's actually showing up in dockets and regulatory drafts right now, not just in comment sections.