[FFmpeg-devel] [PATCH] ffmpeg-web/robots.txt: attempt to keep spiders out of dynamically generated git content

[Michael Niedermayer]

Signed-off-by: Michael Niedermayer <michael at niedermayer.cc>
---
 htdocs/robots.txt | 13 ++++++++++++-
 1 file changed, 12 insertions(+), 1 deletion(-)

diff --git a/htdocs/robots.txt b/htdocs/robots.txt
index eb05362..4bbc395 100644
--- a/htdocs/robots.txt
+++ b/htdocs/robots.txt
@@ -1,2 +1,13 @@
 User-agent: *
-Disallow:
+Crawl-delay: 10
+Disallow: /gitweb/
+Disallow: /*a=search*
+Disallow: /*/search/*
+Disallow: /*a=blobdiff*
+Disallow: /*/blobdiff/*
+Disallow: /*a=commitdiff*
+Disallow: /*/commitdiff/*
+Disallow: /*a=snapshot*
+Disallow: /*/snapshot/*
+Disallow: /*a=blame*
+Disallow: /*/blame/*
-- 
2.17.1