30 lessons learned from building a documentation extraction tool
i've extracted 100+ docs sites. hit cloudflare walls. debugged .md.md bugs. built the entire pipeline. here are 30 lessons.
lessons 1-10: about documentation
- most docs sites expose raw markdown. 61% of modern docs have
.mdendpoints,llms.txt, or github repos. scraping is the fallback. - docs sites redesign frequently. html selectors break. framework extractors need updates. extraction is maintenance, not a one-time thing.
- version selectors are the hardest ui element to handle. tabs, dropdowns, sidebar toggles. all invisible to simple extractors.
- code blocks are fragile. some frameworks split them across divs. some use custom syntax highlighting. some embed code in data attributes.
- navigation is the most useful structure. the page tree tells agents how docs relate. but it's often hidden in javascript or json apis.
- mobile and desktop versions differ. some sites serve different content. responsive design hides content with
display: none. extraction needs desktop user-agents. - search functionality is never extractable. search needs a backend. the frontend is just a query box. i ignore search.
- images and diagrams are usually outside the scope. i focus on text and code. visual content is hard to convert meaningfully.
- the "pretty" docs site is often a skin. the real content is markdown or json underneath. the rendered html is just one view.
- docs quality varies massively. some are clean, structured, well-maintained. others are outdated, broken-link-ridden messes.
lessons 11-20: about extraction
- framework detection is 80% of the work. once you know the framework, you know the selectors. everything else follows.
- html purification never produces perfect output. there's always some noise. the goal is "good enough for agents," not "pixel perfect."
- rate limiting is the biggest practical challenge. not parsing. not conversion. just getting permission to fetch the pages.
- headless browsers would solve most extraction problems. but they're 10x slower, 10x heavier, and unreliable at scale. i avoid them.
- github tree api is the fastest extraction method. when it works, nothing else comes close. always check for open-source docs repos.
- custom frameworks are the most time-consuming. every company that builds its own docs framework requires bespoke extraction logic.
- direct .md endpoints are the most reliable. mintlify, fumadocs, and others serve clean markdown. no parsing needed.
- content negotiation is underrated.
Accept: text/markdownworks on some sites. they just don't advertise it. - crawling ethics matter. respecting robots.txt and rate limits isn't just polite. it prevents your ip from being permanently blocked.
- the 80/20 rule applies. 80% of sites extract cleanly with simple rules. 20% need complex handling. but that 20% takes 80% of the time.
lessons 21-30: about building developer tools
- htmx + hono is a surprisingly capable stack. no react. no build step. fast, simple, reliable.
- zero-memory architecture works. stream everything to disk. replay events from a log. crash recovery is built in.
- dual-layer error messages are worth the effort. clear user messages + verbose debug logs. both audiences served.
- open source is the right business model. the extraction tool is free. the hosted service is convenient.
- vps beats serverless for crawling. cloudflare workers' 50 subrequest limit made serverless impractical. a $6 vps handles everything.
- turso (sqlite) handles metadata fine. no postgres needed for simple relational data. sqlite is enough.
- r2 saves money on downloads. zero egress fees vs. s3's 9 cents/gb. for zip-file downloads, that matters.
- user experience matters more than features. a clean form + status page + download button beats a dozen half-finished features.
- marketing is harder than engineering. the product works. distribution doesn't. seo content is my strategy.
- docs extraction should be free. charging for access to public documentation is wrong. open source extraction ensures it stays free.
raw markdown was already there on 61% of sites
the number of sites that already expose raw markdown. 61%. the frameworks are designed for it. developers built it in. i just need to know where to look.
finding the right extraction path took the most work
not extraction. not parsing. not html cleaning.
it's discovering the right extraction path. given a random docs url, which tier of the ladder works? that's 80% of the complexity.
what i'd do differently
- spend more time on framework detection. early versions had generic extraction. adding framework-specific extractors improved quality 10x.
- start with the cdn/github paths. i initially built html extraction first. reversing the order would have been faster.
- build meta.yaml early. metadata is what makes docs useful to agents. without it, it's just a folder of markdown files.
public docs still aren't always fetchable
building a docs extraction tool taught me more about how documentation works than i expected.
the diversity of docs frameworks. the creativity of web development. the gap between "publicly readable" and "programmatically accessible."
but also: most docs authors want their content to be accessible. they just need the right tools.
agent cache is one of those tools.
related: