B2B Ad Intelligence Only Comes From HTML: Writing a Parser That Survives Redesigns

You want to know which countries a B2B competitor is advertising in this quarter, for how long, roughly how much volume they're burning, and which audiences their targeting slices. That data is all sitting in their public ad library, which many platforms are required to publish for ad-transparency compliance. But you open DevTools, dig for a while, and find no clean JSON endpoint, just a full page of server-rendered HTML. You write a parser, it works, you get the data. Three weeks later the platform ships a redesign and the parser extracts not one field, without throwing an error. It quietly returns a pile of empty fields, and you nearly made a decision on empty data.

This piece is about keeping a parser like that alive after a redesign, and what a model can actually do in maintenance. First the data boundary: all data here is from each platform's public ad library and creative center, accessed by logging in normally with your own account, no signatures, nothing bypassed, no non-public endpoints. As you'll see, that's a line written explicitly into the parser's design.

Why B2B ad intelligence can only be parsed from HTML

Even among ad libraries, the way data is exposed splits three ways: some consumer libraries (Meta's ad library, say) offer structured search and JSON; another kind requires a session even to search; and most B2B platforms' ad libraries are pure server-rendered HTML with no JSON endpoint at all. The root of the difference is that these ad libraries are a compliance artifact, not a product API.

Their reason to exist is satisfying ad-transparency regulation, not being called by developers. No version number, no changelog, no backward-compatibility promise. The page is for humans, the server renders it and spits out HTML, and the only "API" you get is the web page itself.

So it's inherently fragile: you depend on the implementation details of someone else's UI, they change it whenever they like, and they have no obligation to tell you. A JSON API changing a field is at least a "change," but an HTML redesign is, in their eyes, just routine frontend iteration. There's no dodging it, and the only thing you can do is write the parser to break gracefully and be easy to fix after a redesign, rather than quietly return empty values.

Hand-writing a streaming parser saves not code but mental load

The first instinct is to grab lxml or BeautifulSoup, build the whole page's DOM tree, and .find() all the way down. It works, but for this kind of target it's wrong. The DOM tree is an intermediate product of the browser rendering the page (MDN defines the DOM as parsing the document into a tree of nodes that scripts access by structure), and all you want is to pull a few fields out. You don't need that tree, and you shouldn't bind yourself to its structure.

What I landed on is a subclass of the Python standard library's HTMLParser, a bit over eight hundred lines (881, to be exact), purely streaming: a few starttag / data / endtag callbacks drive a state machine, accumulating as it scans, and when it hits a card boundary it emits a record, clears, and continues, never building a full DOM tree.

What streaming saves is concrete. One is memory: a detail page's HTML is easily tens to hundreds of KB, a DOM tree builds the whole page structure in memory, streaming keeps only "where am I scanning, how far along is this card" as state. The other is mental load, which matters more. The moment you write .find('div').find('div')[2], you've bound the parse to the DOM's hierarchical position, and hierarchy is exactly what a redesign loves to move: wrap one more container, split off a wrapper, and every position shifts. A state machine forces you to ask only "is the thing I'm scanning, semantically, the start of a card, an impression number, a targeting tag." Position changes, semantics don't.

Three designs for surviving a redesign

It comes down to three, each taught by a redesign.

One: anchor on semantics, not position. The state machine advances on semantic signals: a field's label text (readable words like "total impressions," "run dates"), role-bearing markers, a block's semantic boundary, never "the third node from the top." The test is one line: if they move this element or wrap another layer around it, does my parse still hold? It holds, then it's a valid anchor. Position anchors shatter on the first redesign, semantic anchors survive most pure styling changes.

Two: degrade on a missing field, don't throw. Before accumulating each card, initialize from a template where every field is an empty default, empty string for text, None for numbers, empty array for lists. Fill what you can, leave empty what you can't. Any single field's extraction failing must not kill the whole card, let alone break the whole page. An ad missing its CTA copy, you still want its impressions and target countries. Letting one unimportant missing field destroy the intelligence the page could have given you is the worst design.

Three: tag results with completeness, don't let "empty" and "broken" look the same. This is the most easily missed and most costly one. "Parsed 0 ads" has two completely different meanings: genuinely no ads (this advertiser didn't run any this quarter), or the page structure changed and nothing anchored (the parser broke). Those two have to be distinguishable in the return value. The way is to attach corroborating evidence: the card count captured, the total the page itself declares, and the pagination state, returned together. So a combination like "card count 0, but the page metadata says there should be a batch and there's no next-page marker" can be judged a structure change rather than genuine emptiness, and throw a clear error rather than impassively return an empty list.

That data boundary from earlier lands through this layer too: the parser checks whether it got bounced to a login page, and the moment it finds the title is a login/signup page it errors out rather than parse on. It only handles public pages you can see normally with your own account, stops at a login wall, and never pushes through it.

The filter dimensions are where the intelligence value is

By here you might think the point is extracting every field of each ad cleanly. It isn't. A single ad's fields are dead, what's genuinely valuable is which dimensions you can slice these ads by. The ad library's search filters are themselves a ready-made list of intelligence dimensions, and organizing them into programmable query parameters gets you not "one ad" but "one competitor's launch slice":

• Country: which markets it advertises in and which it doesn't. A B2B company suddenly starting to advertise in some country often exposes an expansion move earlier than its own website.

• Run window (start and end dates): how long this creative ran. A long-running ad is the strongest signal, because nobody keeps paying for a creative that doesn't convert. The run length is itself an A/B test result validated with real money, run for you by the other side.

• Impression range (min/max): a coarse spend proxy. The absolute value isn't accurate, but it's enough to rank which ones are the priority buys.

• Targeting facets: which targeting is included or excluded. This is the most direct audience intelligence, who the other side thinks will buy its product.

Extracting fields is the means, these dimensions are the end. Writing the parser, think in reverse: to support querying and sorting on these dimensions, what's the minimum set of fields I need to extract reliably? The rest of the fancy fields can go unextracted without hurting the intelligence.

After a redesign, have the model diff old and new for repair suggestions

A parser like this will definitely break on a redesign, and how to fix it is where a model actually belongs, and note that it isn't in the parsing itself. Parsing is deterministic work, a hardcoded state machine, and shouldn't have model calls stuffed into it (don't hand deterministic work to a model, the same principle as on the reverse-engineering line). The model's job is maintenance.

The workflow: open the ad library on your own account, keep both a copy of the HTML saved before the redesign and the new HTML after, hand them to the model along with the field list the current parser extracts, and have it tell you, against the old-new diff, which fields' semantic anchors changed, where the new anchor should be, and which few lines are the minimal change. This is a textbook reasoning-tier scenario: it has to find whether the corresponding semantics still exist in the new structure and give a fixable plan, not restate "the structure was adjusted." These steps ask different things of a model, and one tier for the whole thing either costs money or costs precision:

The second tier is the core, and the only step where switching models visibly changes the result. Whether it's worth the reasoning tier, don't take my word, test it, the protocol is short:

1. Save a page's HTML before a redesign and one after (open the ad library on your own account and save the page).

2. Feed both HTMLs plus the current parser's field list to claude-opus-5 and gpt-5.6-sol.

3. Look at one thing: does its fix point to a specific semantic anchor change ("it used to anchor on the 'total impressions' label, the container role of that label changed in the new version, change it to anchor on X"), or a vague "the structure was adjusted, recommend re-adapting."

4. The former you can apply directly, the latter says nothing. That difference is your selection criterion, and it decides directly how many rounds of blind trial you do on redesign day.

One round shows you the difference, more directly than any benchmark.

The switching cost is the real obstacle

The four tiers come from three vendors, three SDKs, three auth schemes, three error formats. Wiring three clients for different steps, most people run the numbers, decide it isn't worth it for "the occasional parser fix plus bulk data cleaning," and end up on one model the whole way, using a tier that only says "recommend re-adapting" for the redesign fix, burning time without knowing why.

AIReiter flattens that layer: one key, one OpenAI-compatible interface, all four tiers behind it, switching by changing the model field in the request body.

# Redesign fix: the reasoning tier
curl https://aireiter.com/api/v1/chat/completions \
  -H "Authorization: Bearer $AIREITER_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "claude-opus-5",
    "messages": [{"role": "user", "content": "<old-new HTML diff + current field list, ask where the anchor drifted>"}]
  }'

# Bulk intelligence normalization: change the model field, leave the rest
#   "model": "claude-sonnet-5"
# Long context to read the whole page: "model": "kimi-k3"
# Difference attribution:               "model": "gpt-5.6-sol"

If you already use the OpenAI SDK, point base_url at https://aireiter.com/api/v1 and change nothing else. On the Anthropic SDK, hit POST /api/v1/messages with the same key.

On price, Claude models run at 30% off list, GPT models at half, and Kimi K3 is callable on the same key. For this flow the discount lands right on the main cost, and the main cost isn't the redesign fix (that's an occasional low-frequency high-value call), it's bulk intelligence normalization: you watch twenty competitor advertisers, dozens to hundreds of cards each, feeding them all to the model to read out promise / audience / run window, the most call-dense part, running on Claude Sonnet at 30% off. The long-context input for reading a whole page and aligning structure is the next densest. The two most expensive parts sit right on the discount.

• Get an API key

• Try it without signing up: hand one before-and-after HTML pair to both models by hand and compare who can actually point out where the anchor drifted, then decide whether to wire it in.

Once fixed, and the raw cards normalized in bulk, this intelligence's next step is feeding into the creative pipeline for decisions and asset generation. That's the job of the full keyword-to-finished-ad loop, and this piece is its earliest segment, reliably pulling public data in.

But whether the fields are right, the fixture has the final say

The fix the model gives is only a suggestion until you validate it. It says "the anchor should change to X," which sounds reasonable, but there's no guarantee X holds across all cards. B2B ads come as image-text, text-only, carousel, with and without a landing page, and the two samples the model looked at may not cover them all.

What stops this is the same thing that stops model hallucination in reverse engineering: fixed-vector comparison. Store a batch of known inputs (a few saved real page HTMLs) and their known-correct outputs (field results you hand-checked once) as a fixture committed to the repo, and every time you change the parser, whether you change it yourself or per the model's suggestion, re-run this fixture batch and compare field by field. It's the one thing that lets you quickly tell, after an upstream redesign, "did I apply the suggestion wrong, or did the page change again": all pass means the change is right, a few fail and those cases' fields tell you directly which layer the problem is at. The model generates repair suggestions, the fixture decides whether the suggestion is right, and the two must not be mixed. The full method of this differential comparison is laid out in the differential-testing piece, and the ad-library parser applies the same gate. Without this layer, you're taking the model's confidence for correctness, and it's easy to "apply the suggestion, see no error, ship, and find three days later that one country's data has been empty the whole time."

In closing

B2B ad intelligence can only be parsed from HTML, because the ad library is a compliance artifact rather than a product API, with no API contract and redesigns any time. Rot resistance rests on three designs: anchor on semantics not position, degrade on missing fields instead of throwing, tag results with completeness so "empty" and "broken" look different. What's genuinely valuable isn't a single ad's fields, it's the country / run window / impression range / targeting facet dimensions that slice a competitor's launch. The model's place is specific: not in parsing (a deterministic state machine), but in maintenance, where diffing old and new after a redesign for repair suggestions is where the reasoning tier helps, and turning cards into intelligence in bulk is the cheap high-concurrency tier's job. But whether the fields are right always goes to the fixture. Wire these tiers to one unified interface and the only friction left is changing a model field, solved by picking your model.