# try raw markdown before paying to scrape docs > most docs sites expose raw markdown. i built a 6-tier acquisition ladder that tries free methods first. here's how to extract docs without scraping. when you need a docs site in markdown, the obvious answer is "scrape the html." that's expensive, slow, and fragile. If you want to download a docs site for an AI coding agent, start by checking for a machine-readable source. A documentation crawler can use the pages it can access to build a Markdown bundle for browsing or download. here's the secret most people miss: **most modern documentation sites already expose raw markdown.** you just need to know where to look. i built a 6-tier acquisition ladder for agent cache. tier 1 is instant and free. tier 6 is paid extraction that i almost never need. here's the full ladder. ## why i leave html scraping until later scraping html costs compute. it costs bandwidth. it's slow. it breaks when sites redesign. more importantly, scraping puts you in an adversarial relationship with the docs site. you vs. cloudflare. you vs. rate limits. you vs. javascript rendering. html scraping is tier 5 of 6. i avoid it whenever possible. ## six places to look, cheapest first ### tier 1: llms.txt, instant, one request, free if a site has `/llms.txt` or `/llms-full.txt`, the entire docs structure is available in a single text file. no crawling needed. no parsing. no html. llms-full.txt (when available) contains the actual docs content, not just a table of contents. adoption: ~8% of sites. growing fast, especially on mintlify and fumadocs platforms. cost: zero. time: under 1 second. ### tier 2: github tree + raw cdn, 20-30 seconds, free if a docs site is open-source on github, i don't scrape it. i download it. 1. use the git tree api to list all files in the docs directory 2. filter for markdown files (.md, .mdx) 3. download each file via `raw.githubusercontent.com` github raw cdn serves files without rate limits for public repos. no auth needed. best result: 3,848 files from nango's docs in 87 seconds. posthog's docs: 1,915 files in 40 seconds. clerk's docs: 1,124 files in 28 seconds. adoption: ~30% of developer tools have docs in open-source repos. cost: zero. time: 20-60 seconds for typical sites. ### tier 3: direct .md endpoints, 1-2 seconds per page, free modern docs frameworks (mintlify, fumadocs, gitbook, read.me) expose raw markdown by appending `.md` to any docs url. - `hono.dev/docs/getting-started` → html version - `hono.dev/docs/getting-started.md` → raw markdown mintlify is the best at this. the markdown is clean, structured, and preserves the site hierarchy. fumadocs supports it too, though their url patterns vary. other platforms with .md support: gitbook (partial), read.me (good), nextra (good), mdbook (native). adoption: ~30% of sites i tested. cost: zero. time: 1 second per page, concurrent. ### tier 4: content negotiation, 1 request, free some sites serve markdown when you send the right http headers: ``` GET /docs/page Accept: text/markdown ``` if the server supports content negotiation, it returns raw markdown instead of html. no url modification needed. rare but worth probing. ~3% of sites support this. when they do, it's automatic and invisible to regular users. cost: zero. time: under 1 second per page. ### tier 5: html purification, 10-50x slower, compute cost when none of the above work, i fall back to html extraction. my stack: - jsdom for dom parsing - turndown for html-to-markdown conversion - custom cleaners per framework - regex-based noise removal this is the hard path. every framework needs its own extractor. mintlify puts nav in `