llmstxt-gen
Live demo

See it work.

Replay a real crawl, read the two files it writes, then compare one page before and after extraction.

Output

Watch it run, then read the files.

Press run to replay a real crawl of the tool's test site: a small company site with docs, a blog, a robots.txt, a gzipped sitemap index, a noindex page, a duplicate and a 404.

Ready
Terminal
llms.txt, in progress

The finished files


    
Before and after

Only the content survives.

The same page as served and as extracted. Navigation, sidebar, cookie banner, footer, script and the heading permalink are removed. The code keeps its language, the table stays a table, and links become absolute.

HTML as served
Markdown extracted