[FFmpeg-devel] [PATCH] libavfilter/f_select: switch to activate and properly handle EOF pts

[Li-Heng Chen]

On Wed, Sep 14, 2022 at 11:15 AM Nicolas George <george at nsup.org> wrote:
>
> Li-Heng Chen (12022-09-14):
> > Thanks for the comment, Nicolas! I've attached a new patch file which
> > should address the comments. I also want to mention that this patch is
> > similar to another one Paul had done for setpts filter:
> > http://git.videolan.org/?p=ffmpeg.git;a=commitdiff;h=2a546fb7d5722c306dd42c715137e5e493b0d5be
>
> It is similar in its form, but the logic is very different: Paul's
> commit has the function evaluated for the final timestamps.
>
> > The attached patch should have the commit message wrapped, and I
> > have also added a comment in the code to explain the logic:
> > /* Two conditions that current pts could be the EOF pts:
> > * - If the current frame N is not selected while the previous
> > * frame (N-1) is selected, frame N-1 could be the last frame,
> > * hence we update select->eof_pts.
> > * - Last input frame: we have EOF status, and the current (last)
> > * frame is selected */
>
> Thanks for explaining. Unfortunately, it confirms my analysis that your
> logic is slightly flawed.
>
> Consider a select filter that keeps only the frames with a PTS that is a
> prime number. The frame with PTS 13 used to extend in the timeline of
> the stream from 13 inclusive to 14 exclusive. Now it extends from 13
> inclusive to 17 exclusive; the timestamp 14 was completely skipped.
>

Please allow me to explain again with your example. That's say we have
an input with 20 frames (0-19), and we want to keep prime number frames
with select filter, -vf select='eq(n\,2)+eq(n\,3)+...+eq(n\,13)+eq(n\,17)'

In the activation function, select->eof_pts is updated to n at n=4, 6,
8, 12, 14, 18
However, select->eof_pts is passed into ff_outlink_set_status only at
n=20, since
ff_inlink_acknowledge_status returns 1.

> There is no reason that a EOF arriving a little after that would change
> that and make the filter re-consider 14: the frame 14 was skipped, its
> PTS should not be considered.
>

My point is to indicate EOF PTS to last-selected-frame (n=17) instead of
last-input frame (n=19). The reason why I added one to select->eof_pts is
because the existing EOF PTS is the pts value after the last frame (20).

> > This patch does not change the frame timestamps. Instead, we added
> > another variable select->eof_pts to track possible EOF pts. It sets the
> > EOF pts as the pts *after* the last selected frame (e.g. if the last frame
> > selected by the filter is frame 20, EOF pts is set as the pts of frame 21)
>
> Which means it sets the EOF PTS at the timestamp of a frame that was
> dropped. This is not logical.
>

In the existing behavior, the EOF PTS is always the pts after the last
input frame.
For example, if the input to ffmpeg has 20 frames, the EOF PTS passed to the
next filter is the pts of the "21th" frame, no matter the 20th frame
is dropped or not.

> > I have also tested the trim filter, which does not present this bug. However,
> > if you do -vf select='eq(n\,24)',fps=25/1 on the above example, this selected
> > frame will also be duplicated 21 times, which is also not an expected behavior
> > for the select filter.
>
> I consider it the correct behavior.
>

Regardless of my logic, this behavior looks weird to me. In my opinion,
-vf select='eq(n\,24),fps=25/1' should produce the same result as just having
-vf select='eq(n\,24)'. However, select='eq(n\,24),fps=25/1' generates
22 identical
frames while select='eq(n\,24)' only outputs one frame...

Thanks,
Li-Heng

> Regards,
>
> --
>   Nicolas George
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel at ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
>
> To unsubscribe, visit link above, or email
> ffmpeg-devel-request at ffmpeg.org with subject "unsubscribe".