-
@ Michael
2025-03-13 08:47:25
nostr:nprofile1qy2hwumn8ghj7un9d3shjtnddaehgu3wwp6kyqpqx5h5e6ma4jpuldzpxp2de6ewk7hgp6x2m208tup62ztt0fqx4mnqsl8qnw yeah, I find the same with 2 lot of missing replies.
I’m personally still not convinced it should adhere to robots.txt. At least not to the user-agent: * part, as it really doesn’t remotely behave or act like a bot. FediFetcher honouring this block is to me akin to Mastodon honouring it, which would be nonsense (and be an effective defederation from a large number of instances).
But the community has spoken…
I actually wanted to do an analysis to see how prevalent blanket disallows are, and if any instances with blanket disallows make exceptions for FediFetcher for quite some time. I’ll see if I get around to it eventually…