[FFmpeg-devel] trac spiders
Michael Niedermayer
michael at niedermayer.cc
Fri Nov 12 18:23:11 EET 2021
Hi all
i just made the following changes to robots.txt on trac
because trac seemed a bit unhappy and the logs showed some
spiders indexing strange things. I do not know if that was
the cause for occasional trac overload
@@ -15,9 +15,20 @@
Disallow: /timeline
Disallow: /vote
Disallow: /zip-attachment
+Disallow: /*?format
+Disallow: /*?version
+Disallow: /*?cversion
+Disallow: /*?action
+Disallow: /*?replyto
User-agent: Browsershots
Disallow:
User-agent: SemrushBot
Disallow: /
+
+User-agent: Sogou web spider
+Disallow: /
+
+User-agent: Sogou inst spider
+Disallow: /
--
Michael GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB
Those who would give up essential Liberty, to purchase a little
temporary Safety, deserve neither Liberty nor Safety -- Benjamin Franklin
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 195 bytes
Desc: not available
URL: <https://ffmpeg.org/pipermail/ffmpeg-devel/attachments/20211112/c9a7e273/attachment.sig>
More information about the ffmpeg-devel
mailing list