🚀 go-pugleaf

alright, the spider just crawled more than 10k different onion domains.
the jump from 5k to 10k happened real fast now, just after i deleted the ~4,8M links to cp sites from the database, and blocked the cluster of (huge) identical sites under the addresses of
http://invest******************.onion
guess the index is almost usable now, the only thing to add would be some kind of manual review option (the classification with the keywords works in most cases, but not always).

the speed of crawling is still surprisingly slow compared to my expectations. i assumed somehow that with the vast number of onion services estimated to operate already in 2021 would make this exercise faster (the number of onion domains was estimated by the tor team to be ~110k in 2021, using some kind of statistical analysis). but maybe a lot of them are not linked anywhere.

--
Posted on Rocksolid Light