[FFmpeg-devel] [PATCH] ffmpeg-web/robots.txt: attempt to keep spiders out of dynamically generated git content

[Michael Niedermayer]

On Wed, Jul 14, 2021 at 04:00:53PM -0400, ffmpegandmahanstreamer at lolcow.email wrote:
> On 2021-07-14 14:51, Michael Niedermayer wrote:
> > Signed-off-by: Michael Niedermayer <michael at niedermayer.cc>
> > ---
> >  htdocs/robots.txt | 13 ++++++++++++-
> >  1 file changed, 12 insertions(+), 1 deletion(-)
> > 
> > diff --git a/htdocs/robots.txt b/htdocs/robots.txt
> > index eb05362..4bbc395 100644
> > --- a/htdocs/robots.txt
> > +++ b/htdocs/robots.txt
> > @@ -1,2 +1,13 @@
> >  User-agent: *
> > -Disallow:
> > +Crawl-delay: 10
> > +Disallow: /gitweb/
> > +Disallow: /*a=search*
> > +Disallow: /*/search/*
> > +Disallow: /*a=blobdiff*
> > +Disallow: /*/blobdiff/*
> > +Disallow: /*a=commitdiff*
> > +Disallow: /*/commitdiff/*
> > +Disallow: /*a=snapshot*
> > +Disallow: /*/snapshot/*
> > +Disallow: /*a=blame*
> > +Disallow: /*/blame/*
> LGTM based on my own personal experiences. But the robots.txt has to be

will apply


> applied for git.ffmpeg.org as well, and not just ffmpeg.org. Or else they
> will just do the same for git.ffmpeg since there are treated separately.

was expecting this a bit ...
i will look into that tomorrow or so unless someone else does before me

thx

[...]
-- 
Michael     GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB

Into a blind darkness they enter who follow after the Ignorance,
they as if into a greater darkness enter who devote themselves
to the Knowledge alone. -- Isha Upanishad
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 195 bytes
Desc: not available
URL: <https://ffmpeg.org/pipermail/ffmpeg-devel/attachments/20210714/6bc70ef7/attachment.sig>