Give AI a map of your site.
llmstxt-gen crawls your website and writes two files for language models: llms.txt, a tidy index, and llms-full.txt, every page as clean Markdown. Menus, banners and footers stay out.
python -m llmstxt_gen https://example.comClick the clutter to sweep it away. Move over the card to tilt it.
From a messy site to a clean index in four steps.
Each step does one job, and each can be tuned from the command line.
Discover
Finds every page worth reading without wandering off your site.
- robots.txt rules and delays
- sitemap indexes and .gz files
- link following, scoped by path
Fetch politely
Fast enough to finish, gentle enough not to bother your server.
- 8 parallel requests, 0.1 s apart
- retries with back-off
- 5 MB cap per page
Extract
Keeps the article, drops the chrome, and writes Markdown.
- main region detection
- nav, footer, banners removed
- duplicates and redirects merged
Compose
Writes an index a model can scan in seconds.
- clean titles and summaries
- sections in navigation order
- blog and legal under Optional
Pick up where you are.
Watch a crawl, then read the result
Replay a real crawl, open the two files it writes, and compare one page before and after extraction.
Open the demo Get startedBuild your command and run it
Choose a starting point, adjust a few options, and copy one line into your terminal.
Set it up SourceAll of it, readable in one sitting
The whole tool with its tests, file by file, ready to copy into a folder.
Read the code MeenoxNeed a tool built, or your marketing run?
llmstxt-gen is made by Meenox, a studio that builds AI-automated systems, websites and business software, and runs digital marketing.
Talk to Meenox