Get started

Up and running in a minute.

Copy the code, build your command, run it. You need Python 3.9 or newer and nothing else.

Three steps

From download to done.

1. Get the codeCopy the files from the Source page into a llmstxt-gen/ folder, using the same layout.
2. Run itpython -m llmstxt_gen https://your-site.com -o ./out
3. Check your copypython -m unittest discover -s tests runs 28 tests. None need a network.

Command

Set it up, copy one line.

Pick a starting point, adjust what you need, and paste the command into your terminal.

Start URL

Output folder

Maximum pages

Where to crawl

Skip these pathsComma-separated patterns

List as OptionalComma-separated patterns

Site name

One-line summary

Follow links only, ignore the sitemap Skip llms-full.txt Also save each page as a .md file Ignore robots.txt (only on a site you own)

Terminal

Run it from the folder that contains llmstxt_gen/. Results land in the output folder.

Good to know

Where it stops.

It reads HTML, not JavaScriptPages built only in the browser come out thin. They appear in the run summary as "too little content".

Tested on a local siteThe 28 tests crawl a built-in fixture site. It has not been run against live production sites yet, so check your first result.

Sections follow your URLsA flat site with no folders becomes one "Pages" section. Use the Optional and skip options to shape it.