Topic Hub
#robots-txt (5 Repositories)
Ranked open-source repositories tagged with #robots-txt, scored by pull request acceptance likelihood and maintainer engagement velocity.
Topic Avg Merge Rate
42.1%
Avg Review Latency
80.5h
Filter by language
5 repositories tagged #robots-txt
A TierJava 259
crawler-commons/crawler-commons
A set of reusable Java components that implement functionality common to any web crawler
95.2%
Merge Rate
2d
First Review
100%
1st-Timers
1
Maintainers
A TierTypeScript 912
thedaviddias/llms-txt-hub
🤖 The largest directory for AI-ready documentation and tools implementing the proposed llms.txt standard
65.1%
Merge Rate
7d
First Review
33%
1st-Timers
12
Maintainers
B TierJavaScript 167
samclarke/robots-parser
NodeJS robots.txt parser with support for wildcard (*) matching.
50.0%
Merge Rate
7d
First Review
0%
1st-Timers
0
Maintainers
D TierTypeScript 528
nuxt-modules/robots
Tame the robots crawling and indexing your Nuxt site.
0.0%
Merge Rate
<1h
First Review
0%
1st-Timers
0
Maintainers
D TierGo 791
PuerkitoBio/fetchbot
A simple and flexible web crawler that follows the robots.txt policies and crawl delays.
0.0%
Merge Rate
-
First Review
0%
1st-Timers
0
Maintainers