[FFmpeg-devel] [PATCHv2] add signature filter for MPEG7 video signature

Gerion Entrup gerion.entrup.ffdev at flump.de
Wed Mar 30 22:57:47 CEST 2016


On Mittwoch, 30. März 2016 15:29:27 CEST Michael Niedermayer wrote:
> On Wed, Mar 30, 2016 at 01:57:24PM +0200, Gerion Entrup wrote:
> > Attached improved version of patch.
> > 
> > Differences to last time:
> > - reduce amount of errors in the signature (the last patch included some int
> > foo = a/b). This version replaces this with a rational.
> > - implement binary output.
> > - fixes in configure, some typos
> > 
> > I have found the conformance testfiles [1]. Both the binary and the xml output
> > passes the conformance test but are not bitexact. I wrote some python script
> > to prove this (see attachment). I don't see why this happens. If someone want
> > to help, the correspondent reference code is in the file
> > "ExtractionUtilities/VideoSignatureExtraction.cpp" beginning with line 1615,
> > that could be found here [2].
> > 
> > Then a few questions:
> > - The timebase of the testfiles is 90000. In the binary output unfortunately there
> > is only place for a 16 bit number, so this don't fit. Currently the code simply crop
> > remaining bits. Is there a better solution (devide with some number etc)?
> > 
> > - I try to use put_bits32 where it is possible, because I thought is is faster. Then
> > I saw it internally uses put_bits as well. Does it have a performance impact to
> > replace it with put_bits(..., 8, ...) (would simplify the code a lot)?
> > 
> > Gerion
> > 
> > [1] http://standards.iso.org/ittf/PubliclyAvailableStandards/c057047_ISO_IEC_15938-7_2003_Amd_6_2011_Conformance_Testing.zip
> > [2] http://standards.iso.org/ittf/PubliclyAvailableStandards/c056735_ISO_IEC_15938-6_2003_Amd_4_2011_Electronic_inserts.zip
> 
> >  Changelog                      |    1 
> >  configure                      |    1 
> >  doc/filters.texi               |   70 +++
> >  libavfilter/Makefile           |    1 
> >  libavfilter/allfilters.c       |    1 
> >  libavfilter/signature.h        |  574 ++++++++++++++++++++++++++++++
> >  libavfilter/signature_lookup.c |  527 +++++++++++++++++++++++++++
> >  libavfilter/version.h          |    4 
> >  libavfilter/vf_signature.c     |  774 +++++++++++++++++++++++++++++++++++++++++
> >  9 files changed, 1951 insertions(+), 2 deletions(-)
> > 18a73574782a4e5e576bed3857fd283a009ff532  0001-add-signature-filter-for-MPEG7-video-signature.patch
> > From c81db6a999694f01335ee0d88483f276f2d10d3f Mon Sep 17 00:00:00 2001
> > From: Gerion Entrup <gerion.entrup at flump.de>
> > Date: Sun, 20 Mar 2016 11:10:31 +0100
> > Subject: [PATCH] add signature filter for MPEG7 video signature
> > 
> > This filter does not implement all features of MPEG7. Missing features:
> > - compression of signature files
> > - work only on (cropped) parts of the video
> > ---
> >  Changelog                      |   1 +
> >  configure                      |   1 +
> >  doc/filters.texi               |  70 ++++
> >  libavfilter/Makefile           |   1 +
> >  libavfilter/allfilters.c       |   1 +
> >  libavfilter/signature.h        | 574 ++++++++++++++++++++++++++++++
> >  libavfilter/signature_lookup.c | 527 ++++++++++++++++++++++++++++
> >  libavfilter/version.h          |   4 +-
> >  libavfilter/vf_signature.c     | 774 +++++++++++++++++++++++++++++++++++++++++
> >  9 files changed, 1951 insertions(+), 2 deletions(-)
> >  create mode 100644 libavfilter/signature.h
> >  create mode 100644 libavfilter/signature_lookup.c
> >  create mode 100644 libavfilter/vf_signature.c
> > 
> > diff --git a/Changelog b/Changelog
> > index 1f57f5e..5b76607 100644
> > --- a/Changelog
> > +++ b/Changelog
> > @@ -12,6 +12,7 @@ version <next>:
> >  - ciescope filter
> >  - protocol blacklisting API
> >  - MediaCodec H264 decoding
> > +- MPEG-7 Video Signature filter
> >  
> >  
> >  version 3.0:
> [...]
> 
> > +typedef struct {
> > +    int x;
> > +    int y;
> > +} Point;
> > +
> > +typedef struct {
> > +    Point up;
> > +    Point to;
> > +} Block;
> 
> these are used for tables of small values, int which is 32bit
> would waste quite some space, can uint8_t be used too ?
Yes, all Points are < 32 .

> 
> 
> [...]
> > +/* bitcount[index] = amount of ones in (binary) index */
> > +static const int bitcount[256] =
> > +{
> > +  0, 1, 1, 2, 1, 2, 2, 3, 1, 2, 2, 3, 2, 3, 3, 4,
> > +  1, 2, 2, 3, 2, 3, 3, 4, 2, 3, 3, 4, 3, 4, 4, 5,
> > +  1, 2, 2, 3, 2, 3, 3, 4, 2, 3, 3, 4, 3, 4, 4, 5,
> > +  2, 3, 3, 4, 3, 4, 4, 5, 3, 4, 4, 5, 4, 5, 5, 6,
> > +  1, 2, 2, 3, 2, 3, 3, 4, 2, 3, 3, 4, 3, 4, 4, 5,
> > +  2, 3, 3, 4, 3, 4, 4, 5, 3, 4, 4, 5, 4, 5, 5, 6,
> > +  2, 3, 3, 4, 3, 4, 4, 5, 3, 4, 4, 5, 4, 5, 5, 6,
> > +  3, 4, 4, 5, 4, 5, 5, 6, 4, 5, 5, 6, 5, 6, 6, 7,
> > +  1, 2, 2, 3, 2, 3, 3, 4, 2, 3, 3, 4, 3, 4, 4, 5,
> > +  2, 3, 3, 4, 3, 4, 4, 5, 3, 4, 4, 5, 4, 5, 5, 6,
> > +  2, 3, 3, 4, 3, 4, 4, 5, 3, 4, 4, 5, 4, 5, 5, 6,
> > +  3, 4, 4, 5, 4, 5, 5, 6, 4, 5, 5, 6, 5, 6, 6, 7,
> > +  2, 3, 3, 4, 3, 4, 4, 5, 3, 4, 4, 5, 4, 5, 5, 6,
> > +  3, 4, 4, 5, 4, 5, 5, 6, 4, 5, 5, 6, 5, 6, 6, 7,
> > +  3, 4, 4, 5, 4, 5, 5, 6, 4, 5, 5, 6, 5, 6, 6, 7,
> > +  4, 5, 5, 6, 5, 6, 6, 7, 5, 6, 6, 7, 6, 7, 7, 8
> > +};
> 
> av_popcount()
> that also does 4 bytes at a time
>  
> 
> [...]
> > +static int get_l1dist(AVFilterContext *ctx, SignatureContext *sc, uint8_t *first, uint8_t *second)
> > +{
> > +    unsigned int i;
> > +    int dist = 0;
> > +    int f,s;
> > +
> > +    for(i=0; i < SIGELEM_SIZE/5; i++){
> > +        if(first[i] != second[i]){
> > +            f = first[i];
> > +            s = second[i];
> > +            do {
> > +                dist += FFABS((f % 3) - (s % 3));
> > +                f/=3;
> > +                s/=3;
> > +            } while(f > 0 || s > 0);
> 
> division and modulo are slow
> if this is speed relevant then please use a LUT
The already existing lut was meant for this. I improved the code and actually uses it.
BTW is division by 2 optimized out or it is better to use >> 1 ?

> 
> 
> [...]
> 
> > +static int filter_frame(AVFilterLink *inlink, AVFrame *picref)
> > +{
> > +    AVFilterContext *ctx = inlink->dst;
> > +    SignatureContext *sic = ctx->priv;
> > +    StreamContext *sc = &(sic->streamcontexts[FF_INLINK_IDX(inlink)]);
> > +    FineSignature* fs;
> > +
> > +
> > +
> > +    unsigned int pot3[5] = { 3*3*3*3, 3*3*3, 3*3, 3, 1 };
> > +    /* indexes of words : 210,217,219,274,334  44,175,233,270,273  57,70,103,237,269  100,285,295,337,354  101,102,111,275,296
> > +    s2usw = sorted to unsorted wordvec: 44 is at index 5, 57 at index 10...
> > +    */
> > +    unsigned int wordvec[25] = {44,57,70,100,101,102,103,111,175,210,217,219,233,237,269,270,273,274,275,285,295,296,334,337,354};
> 
> > +    unsigned int s2usw[25]   = { 5,10,11, 15, 20, 21, 12, 22,  6,  0,  1,  2,  7, 13, 14,  8,  9,  3, 23, 16, 17, 24,  4, 18, 19};
> 
> static const uint8_t
> 
> 
> > +
> > +    uint8_t wordt2b[5] = { 0, 0, 0, 0, 0 }; /* word ternary to binary */
> > +    uint64_t intpic[32][32];
> > +    uint64_t rowcount;
> > +    uint8_t *p = picref->data[0];
> > +    int inti, intj;
> > +    int *intjlut;
> > +
> > +    double conflist[DIFFELEM_SIZE];
> > +    int f = 0, g = 0, w = 0;
> > +    int dh1 = 1, dh2 = 1, dw1 = 1, dw2 = 1, denum, a, b;
> > +    int i,j,k,ternary;
> > +    uint64_t blocksum;
> > +    int blocksize;
> > +    double th; /* threshold */
> > +    double sum;
> > +
> > +    /* initialize fs */
> > +    if(sc->curfinesig){
> > +        fs = av_mallocz(sizeof(FineSignature));
> > +        sc->curfinesig->next = fs;
> > +        fs->prev = sc->curfinesig;
> > +        sc->curfinesig = fs;
> > +    }else{
> > +        fs = sc->curfinesig = sc->finesiglist;
> > +        sc->curcoursesig1->first = fs;
> > +    }
> > +
> > +    fs->pts = picref->pts;
> > +    fs->index = sc->lastindex++;
> > +
> 
> > +    for (i=0; i<32; i++){
> > +        for(j=0; j<32; j++){
> > +            intpic[i][j]=0;
> > +        }
> > +    }
> 
> memset
> 
> 
> > +    intjlut = av_malloc(inlink->w * sizeof(int));
> > +    for (i=0; i < inlink->w; i++){
> > +        intjlut[i] = (i<<5)/inlink->w;
> > +    }
> > +
> > +    for (i=0; i < inlink->h; i++){
> > +        inti = (i<<5)/inlink->h;
> > +        for (j=0; j< inlink->w; j++){
> > +            intj = intjlut[j];
> > +            intpic[inti][intj] += p[j];
> > +        }
> > +        p += picref->linesize[0];
> > +    }
> > +    av_free(intjlut);
> > +
> > +    /* The following calculate a summed area table (intpic) and brings the numbers
> > +     * in intpic to to the same denuminator.
> > +     * So you only have to handle the numinator in the following sections.
> > +     */
> > +    dh1 = inlink->h/32;
> > +    if (inlink->h%32)
> > +        dh2 = dh1 + 1;
> > +    dw1 = inlink->w/32;
> > +    if (inlink->w%32)
> > +        dw2 = dw1 + 1;
> > +    denum = dh1 * dh2 * dw1 * dw2;
> > +
> > +    for (i=0; i<32; i++){
> > +        rowcount = 0;
> > +        a = 1;
> > +        if (dh2 > 1) {
> > +            a = ((inlink->h*(i+1))%32 == 0) ? (inlink->h*(i+1))/32 - 1 : (inlink->h*(i+1))/32;
> > +            a -= ((inlink->h*i)%32 == 0) ? (inlink->h*i)/32 - 1 : (inlink->h*i)/32;
> > +            a = (a == dh1)? dh2 : dh1;
> > +        }
> > +        for (j=0; j<32; j++){
> > +            b = 1;
> > +            if (dw2 > 1) {
> > +                b = ((inlink->w*(j+1))%32 == 0) ? (inlink->w*(j+1))/32 - 1 : (inlink->w*(j+1))/32;
> > +                b -= ((inlink->w*j)%32 == 0) ? (inlink->w*j)/32 - 1 : (inlink->w*j)/32;
> > +                b = (b == dw1)? dw2 : dw1;
> > +            }
> > +            rowcount += intpic[i][j] *= a * b;
> > +            if(i>0){
> > +                intpic[i][j] = intpic[i-1][j] + rowcount;
> > +            } else {
> > +                intpic[i][j] = rowcount;
> > +            }
> > +        }
> > +    }
> > +
> 
> > +    for (i=0; i< ELEMENT_COUNT; i++){
> > +        const ElemCat* elemcat = elements[i];
> > +        double* elemsignature = av_malloc(sizeof(double) * elemcat->elem_count);
> > +        double* sortsignature = av_malloc(sizeof(double) * elemcat->elem_count);
> 
> missing alloc failure checks
How do I handle this in export() and lookup_signatures()? Both are called in uninit,
which returns void, so return AVERROR does not work.
> 
> 
> [...]
> > +static int request_frame(AVFilterLink *outlink)
> > +{
> > +    AVFilterContext *ctx = outlink->src;
> > +    SignatureContext *sc = ctx->priv;
> > +    int i, ret;
> > +
> 
> > +    for (i = 0; i < sc->nb_inputs; i++)
> > +        ret = ff_request_frame(ctx->inputs[i]);
> 
> ignoring the return code for all but the last call
hope I fixed it.
> 
> [...]
> 

Add improved patch.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0001-add-signature-filter-for-MPEG7-video-signature.patch
Type: text/x-patch
Size: 78449 bytes
Desc: not available
URL: <http://ffmpeg.org/pipermail/ffmpeg-devel/attachments/20160330/4c5c93dc/attachment.bin>


More information about the ffmpeg-devel mailing list