[FFmpeg-devel] Allow interrupt callback for AVCodecContext

[Ronald S. Bultje]

Hi,

On Mon, Jan 6, 2014 at 2:48 PM, Don Moir <donmoir at comcast.net> wrote:

>
> ----- Original Message ----- From: "Ronald S. Bultje" <rsbultje at gmail.com>
> To: "FFmpeg development discussions and patches" <ffmpeg-devel at ffmpeg.org>
> Sent: Monday, January 06, 2014 9:52 AM
>
> Subject: Re: [FFmpeg-devel] Allow interrupt callback for AVCodecContext
>
>
>  Hi,
>>
>> On Mon, Jan 6, 2014 at 3:56 AM, Don Moir <donmoir at comcast.net> wrote:
>>
>>
>>> ----- Original Message ----- From: "Ronald S. Bultje" <
>>> rsbultje at gmail.com>
>>> To: "FFmpeg development discussions and patches" <
>>> ffmpeg-devel at ffmpeg.org>
>>> Sent: Monday, January 06, 2014 8:44 AM
>>>
>>> Subject: Re: [FFmpeg-devel] Allow interrupt callback for AVCodecContext
>>>
>>>
>>>  Hi,
>>>
>>>>
>>>> On Mon, Dec 16, 2013 at 1:21 AM, Don Moir <donmoir at comcast.net> wrote:
>>>>
>>>>  ----- Original Message ----- From: "Don Moir" <donmoir at comcast.net>
>>>>
>>>>>
>>>>>  To: "FFmpeg development discussions and patches" <
>>>>>> ffmpeg-devel at ffmpeg.org
>>>>>> >
>>>>>> Sent: Monday, December 16, 2013 2:03 AM
>>>>>> Subject: Re: [FFmpeg-devel] Allow interrupt callback for
>>>>>> AVCodecContext
>>>>>>
>>>>>>
>>>>>>
>>>>>>  ----- Original Message ----- From: "Ronald S. Bultje" <
>>>>>>
>>>>>>> rsbultje at gmail.com>
>>>>>>> To: "FFmpeg development discussions and patches" <
>>>>>>> ffmpeg-devel at ffmpeg.org>
>>>>>>> Sent: Wednesday, January 01, 2014 11:14 AM
>>>>>>> Subject: Re: [FFmpeg-devel] Allow interrupt callback for
>>>>>>> AVCodecContext
>>>>>>>
>>>>>>>
>>>>>>>  Hi,
>>>>>>>
>>>>>>>
>>>>>>>> On Mon, Dec 16, 2013 at 1:30 AM, Don Moir <donmoir at comcast.net>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>>  ----- Original Message ----- From: "Ronald S. Bultje" <
>>>>>>>>
>>>>>>>>> rsbultje at gmail.com>
>>>>>>>>> To: "FFmpeg development discussions and patches" <
>>>>>>>>> ffmpeg-devel at ffmpeg.org>
>>>>>>>>> Sent: Wednesday, January 01, 2014 10:52 AM
>>>>>>>>> Subject: Re: [FFmpeg-devel] Allow interrupt callback for
>>>>>>>>> AVCodecContext
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>  Hi,
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>  On Mon, Dec 16, 2013 at 2:07 AM, Don Moir <donmoir at comcast.net>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>  For now just seeing what you think about this. This is about
>>>>>>>>>> thread
>>>>>>>>>> based
>>>>>>>>>>
>>>>>>>>>>  apps where this makes sense.
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> When attempting to do a new seek or waiting to close a video, I
>>>>>>>>>>> find
>>>>>>>>>>> that
>>>>>>>>>>> I am waiting on avcodec_decode_video2 to return before I can
>>>>>>>>>>> continue.
>>>>>>>>>>> Depending on machine and video, this wait time can be up to about
>>>>>>>>>>> 50ms
>>>>>>>>>>> but
>>>>>>>>>>> normally wait time about 5 to 20 ms or so.
>>>>>>>>>>>
>>>>>>>>>>> You might say 'so what' and I would agree for a simple player app
>>>>>>>>>>> it
>>>>>>>>>>> does
>>>>>>>>>>> not make that much difference.
>>>>>>>>>>>
>>>>>>>>>>> If you are trying to stay on a timeline, or in case of scrubbing,
>>>>>>>>>>> or
>>>>>>>>>>> for
>>>>>>>>>>> editing apps, this wait time does make a difference. That is, you
>>>>>>>>>>> can not
>>>>>>>>>>> move on until avcodec_decode_video2 has returned.
>>>>>>>>>>>
>>>>>>>>>>> I can pretty much seek instantly to zero for any file except
>>>>>>>>>>> when I
>>>>>>>>>>> have
>>>>>>>>>>> to wait on avcodec_decode_video2 if that be the case.
>>>>>>>>>>>
>>>>>>>>>>> For me, it's normal for any intense process like decoding to be
>>>>>>>>>>> interruptible but this is not the case for AVCodecContext in
>>>>>>>>>>> ffmpeg.
>>>>>>>>>>> This
>>>>>>>>>>> is strange, don't you think?
>>>>>>>>>>>
>>>>>>>>>>> For AVFormatContext you have:
>>>>>>>>>>>
>>>>>>>>>>> typedef struct AVIOInterruptCB {
>>>>>>>>>>>     int (*callback)(void*);
>>>>>>>>>>>     void *opaque;
>>>>>>>>>>> } AVIOInterruptCB;
>>>>>>>>>>>
>>>>>>>>>>> I would use this model for AVCodecContext but change naming to:
>>>>>>>>>>>
>>>>>>>>>>> typedef struct AVInterruptCB {
>>>>>>>>>>>     int (*callback)(void*);
>>>>>>>>>>>     void *opaque;
>>>>>>>>>>> } AVInterruptCB;
>>>>>>>>>>>
>>>>>>>>>>> Then make name changes to whereever and add to AVCodecContext.
>>>>>>>>>>>
>>>>>>>>>>> This callback could be implemented piecemeal whereever needed
>>>>>>>>>>> over
>>>>>>>>>>> time,
>>>>>>>>>>> hitting the more intense processes first.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>  Just open a (potentially pre-cached) new AVCodecContext, it'll
>>>>>>>>>>> be
>>>>>>>>>>>
>>>>>>>>>>>  even
>>>>>>>>>> faster than your solution.
>>>>>>>>>>
>>>>>>>>>> Ronald
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>  hmmm. That's a thought for seeking I suppose but does not apply
>>>>>>>>>> to
>>>>>>>>>>
>>>>>>>>> waiting
>>>>>>>>> to close. Why do I care about close time ? Because another video
>>>>>>>>> has
>>>>>>>>> come
>>>>>>>>> in to replace it or variations of it. This can happen rapidly.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>> This is where more evolved languages have the concept of garbage
>>>>>>>> collection. For your purpose, you simply have a queue where you push
>>>>>>>> "AVCodecContexts I don't need anymore" into, and while the
>>>>>>>> application
>>>>>>>> is
>>>>>>>> in the idle loop, you pop it and destroy items left in it.
>>>>>>>>
>>>>>>>> Really, I understand your use case, but you don't want to add all
>>>>>>>> kind
>>>>>>>> of
>>>>>>>> clever hacks in AVCodecContext to get this kind of stuff done.
>>>>>>>> You're
>>>>>>>> not
>>>>>>>> using a shared I/O resource that may be protected by a cookie or
>>>>>>>> worse
>>>>>>>> for
>>>>>>>> pay-per-view video, and you're not in any sort of kernel wait, so
>>>>>>>> there's
>>>>>>>> no reason to add these kind of hacks. It's a logical thought, but
>>>>>>>> this
>>>>>>>> problem has been solved already and there's better, easier and
>>>>>>>> faster
>>>>>>>> solutions out there that do not involve adding hacks in every single
>>>>>>>> FFmpeg
>>>>>>>> decoder to actually support this.
>>>>>>>>
>>>>>>>> Ronald
>>>>>>>>
>>>>>>>>
>>>>>>>>  Yeah really did not like the notion of changing decoders and I
>>>>>>> don't
>>>>>>> like adding anything that might not be needed, but I will see what I
>>>>>>> can do
>>>>>>> with your suggestions. I never know what ffmpeg can tolerate. I had
>>>>>>> asked
>>>>>>> in libav-user but get the same old BS there when asking about things
>>>>>>> like
>>>>>>> this.
>>>>>>>
>>>>>>>
>>>>>>>  Ok tried some test code allocating new context and that worked
>>>>>> pretty
>>>>>> well.
>>>>>>
>>>>>> I had to do this to get consistent results:
>>>>>>
>>>>>> AVCodec *codec ... already have it
>>>>>>
>>>>>> AVCodecContext *new_context = avcodec_alloc_context3 (NULL);
>>>>>> avcodec_copy_context (new_context,old_context);
>>>>>> avcodec_open2 (new_context,codec,NULL);
>>>>>>
>>>>>> The following worked for at least one file but for failed for others
>>>>>> like
>>>>>> Theora etc.
>>>>>>
>>>>>> AVCodecContext *new_context = avcodec_alloc_context3 (codec);
>>>>>> avcodec_open2 (new_context,codec,NULL);
>>>>>>
>>>>>> For Theora it failed in avcodec_open2 saying 'missing side data' or
>>>>>> similiar.
>>>>>>
>>>>>> Using a cached context the wait time is zero. Executing the 3
>>>>>> statements
>>>>>> above on slower machine is about 1 to 4 ms. It's also not like it
>>>>>> always
>>>>>> stuck in avcodec_decode_video2 either. In this case I don't need a new
>>>>>> context and wait time is zero.
>>>>>>
>>>>>> Thanks Ronald.
>>>>>>
>>>>>>
>>>>>>  Ronald says:
>>>>>
>>>>>  Just open a (potentially pre-cached) new AVCodecContext, it'll be even
>>>>>
>>>>>  faster than your solution.
>>>>>>
>>>>>>>
>>>>>>>>>>
>>>>>>>>>>   and
>>>>>>>>>
>>>>>>>>
>>>>>
>>>>>  Really, I understand your use case, but you don't want to add all kind
>>>>> of
>>>>>
>>>>>  clever hacks in AVCodecContext to get this kind of stuff done. You're
>>>>>>
>>>>>>> not
>>>>>>>> using a shared I/O resource that may be protected by a cookie or
>>>>>>>> worse
>>>>>>>> for
>>>>>>>> pay-per-view video, and you're not in any sort of kernel wait, so
>>>>>>>> there's
>>>>>>>> no reason to add these kind of hacks. It's a logical thought, but
>>>>>>>> this
>>>>>>>> problem has been solved already and there's better, easier and
>>>>>>>> faster
>>>>>>>> solutions out there that do not involve adding hacks in every single
>>>>>>>> FFmpeg
>>>>>>>> decoder to actually support this.
>>>>>>>>
>>>>>>>>
>>>>>>>>   Using a cached and open AVCodecContext does work but it's like
>>>>>>> trying
>>>>>>>
>>>>>> to
>>>>> kill an ant with a sledgehammer. Using a cached and not opened context
>>>>> helps some but you still loose time when opening it. So best results
>>>>> are
>>>>> when using a pre-cached open context.
>>>>>
>>>>> This means you will have allocated resources and most likely opened
>>>>> threads in your cached context that are not doing anything.
>>>>>
>>>>> So you have to ask what the real hack is. Keeping an opened cached
>>>>> context
>>>>> or having an interrupt callback. An interrupt callback does not use any
>>>>> additional resources but then it has to be implemented for every
>>>>> decoder.
>>>>> An opened context works now, but uses addtional resources and mostly
>>>>> opened
>>>>> threads that are more or less dormant. Having an interruptible intense
>>>>> process is normal and not a hack and you should not have to beat it to
>>>>> death to get it to work.
>>>>>
>>>>> I restrict the context to have at most 2 threads. Yes I could limit it
>>>>> to
>>>>> no new threads, but I get better results with 2. So you have to be
>>>>> careful.
>>>>> If the thread_count is zero, which is the default, then it will choose
>>>>> the
>>>>> number of threads based on the cpu count. You will have this number of
>>>>> opened threads in a cached opened context. On one of my machines this
>>>>> would
>>>>> be 8 opened and unused threads for cached open context, but I set
>>>>> thread_count to 2, getting diminishing returns on greater number of
>>>>> threads.
>>>>>
>>>>>
>>>>
>>>> I'll bite. Please do define expensive, as Reimar also said. Do you mean
>>>> "cpu usage"? Or "memory usage"? Or something else?
>>>>
>>>>
>>> I was definning expensive as allocating threads and memory that do
>>> nothing
>>> for what should be a simple process.
>>>
>>
>>
>> But isn't that a completely academic concern?
>>
>
> Don't think so as something is not good somewhere. Did some more checking
> and tried to figure the cost of this more accurately. It varies on codec
> and some other things so trying to get a handle on it.
>
> With H264 file, 2 threads, 2 opened context, one in use and one cached.
>
> Initially all is good. Then when I seek, I swap the context. The 'in use'
> context which might still be in avcode_decode_video2 is flushed later but
> for test I waited on video and flushed both cached and original context.
>
> When the seek is complete, I see about double the amount of memory being
> used for the file. This a large video file and goes from about 30mb to 60mb
> after seek give or take.


Right, because you didn't flush the old context yet. You need to flush and
reset it for the reference frames to be released. Until then, it's as alive
as it could be, consuming any bit as much memory as the running one.

Ronald