01Crawled
A bot fetches your URL. Not Googlebot: GPTBot, ClaudeBot, PerplexityBot, Google-Extended, each with its own rules.
Breaks whenrobots.txt blocks them. Most sites that block AI crawlers did it by accident, or inherited it from a plugin default, and never checked.
02Parsed
The HTML is turned into text and structure. Headings, lists and tables become an outline the model can navigate.
Breaks whenThe content only exists after JavaScript runs, so the parser sees an empty shell. Headings built from styled divs are invisible as headings.
03Chunked
The text is split into passages, usually a few hundred words, cut on the structure the parser found.
Breaks whenA wall of prose with no headings gets cut mid-argument, so no single chunk contains a whole answer to anything.
04Embedded
Each passage becomes a vector: a position in meaning space, near other passages that mean the same thing.
Breaks whenA passage that hedges, or covers three topics at once, lands near nothing in particular and is never the closest match to any question.
05Retrieved
Someone asks a question. The passages closest in meaning to that question get pulled, from anywhere on the web.
Breaks whenYours is close but a competitor's is closer, almost always because theirs answers in the first sentence and yours spends it warming up.
06Cited
The model writes an answer from what it retrieved, and names the sources it leaned on.
Breaks whenYou were retrieved and still not named, because nothing identified you as an entity worth naming. No Organization schema, no consistent brand name.
Move your cursor across to travel the pipeline, or use the arrow keys.