[FFmpeg-devel] [PATCH] avformat/dv: fix timestamps of audio packets in case of dropped corrupt audio frames
Marton Balint
cus at passwd.hu
Sat Oct 31 23:15:48 EET 2020
On Sat, 31 Oct 2020, Dave Rice wrote:
>
>> On Oct 31, 2020, at 3:47 PM, Marton Balint <cus at passwd.hu> wrote:
>> On Sat, 31 Oct 2020, Dave Rice wrote:
>>
>>> Hi Marton,
>>>
>>>> On Oct 31, 2020, at 12:56 PM, Marton Balint <cus at passwd.hu> wrote:
>>>> Fixes out of sync timestamps in ticket #8762.
>>>
>>> Although Michael’s recent patch does address the issue documented in 8762, I haven’t found this patch to fix the issue. I tried with -c:a copy and with -c:a pcm_s16le with some sample files that exhibit this issue but each output was out of sync. I put an output at https://gist.github.com/dericed/659bd843bd38b6f24a60198b5e345795. That output notes that 3597 packages of video are read and 3586 packets of audio. In the resulting file, at the end of the timeline the audio is 9 frames out of sync and my output video stream is 00:02:00.020 and output audio stream is 00:01:59.653.
>>>
>>> Beyond copying or encoding the audio, are there other options I should use to test this?
>>
>> Well, it depends on what you want. After this patch you should get a file which has audio packets synced to video, but the audio stream is sparse, not every video packet has a corresponding audio packet. (It looks like our MOV muxer does not support muxing of sparse audio therefore does not produce proper timestamps. But MKV does, please try that.)
>>
>> You can also make ffmpeg generate the missing audio based on packet timestamps. Swresample has an async=1 option, so something like this should get you synced audio with continous audio packets:
>> ffmpeg -y -i 1670520000_12.dv -c:v copy \
>> -af aresample=async=1:min_hard_comp=0.01 -c:a pcm_s16le 1670520000_12.mov
>
> Thank you for this. With the patch and async, the result is synced and the resulting audio was the same as Michael’s patch.
>
> Could you explain why you used min_hard_comp here? IIUC min_hard_comp is
> a set a threshold between the strategies of trim/fill or stretch/squeeze
> to align the audio to time; however, the async documentation says
> "Setting this to 1 will enable filling and trimming, larger values
> represent the maximum amount in samples that the data may be stretched
> or squeezed” so I thought that async=1 would not permit stretch/squeeze
> anyway.
It is documented poorly, but if you check the source code you will see
that async=1 implicitly sets min_comp to 0.001 enabling trimming/dropping.
min_hard_comp decides the threshold when silence injection actually
happens, and the default for that is 0.1, which is more than a frame,
therefore not acceptable if we want to maintain <1 frame accuracy. Or at
least that is how I think it should work.
Regards,
Marton
>>>
>>>> Signed-off-by: Marton Balint <cus at passwd.hu>
>>>> ---
>>>> libavformat/dv.c | 16 ++--------------
>>>> tests/ref/seek/lavf-dv | 18 +++++++++---------
>>>> 2 files changed, 11 insertions(+), 23 deletions(-)
>>>> diff --git a/libavformat/dv.c b/libavformat/dv.c
>>>> index 3e0d12c0e3..26a78139f5 100644
>>>> --- a/libavformat/dv.c
>>>> +++ b/libavformat/dv.c
>>>> @@ -49,7 +49,6 @@ struct DVDemuxContext {
>>>> uint8_t audio_buf[4][8192];
>>>> int ach;
>>>> int frames;
>>>> - uint64_t abytes;
>>>> };
>>>> static inline uint16_t dv_audio_12to16(uint16_t sample)
>>>> @@ -258,7 +257,7 @@ static int dv_extract_audio_info(DVDemuxContext *c, const uint8_t *frame)
>>>> c->ast[i] = avformat_new_stream(c->fctx, NULL);
>>>> if (!c->ast[i])
>>>> break;
>>>> - avpriv_set_pts_info(c->ast[i], 64, 1, 30000);
>>>> + avpriv_set_pts_info(c->ast[i], 64, c->sys->time_base.num, c->sys->time_base.den);
>>>> c->ast[i]->codecpar->codec_type = AVMEDIA_TYPE_AUDIO;
>>>> c->ast[i]->codecpar->codec_id = AV_CODEC_ID_PCM_S16LE;
>>>> @@ -387,8 +386,7 @@ int avpriv_dv_produce_packet(DVDemuxContext *c, AVPacket *pkt,
>>>> for (i = 0; i < c->ach; i++) {
>>>> c->audio_pkt[i].pos = pos;
>>>> c->audio_pkt[i].size = size;
>>>> - c->audio_pkt[i].pts = c->abytes * 30000 * 8 /
>>>> - c->ast[i]->codecpar->bit_rate;
>>>> + c->audio_pkt[i].pts = (c->sys->height == 720) ? (c->frames & ~1) : c->frames;
>>>> ppcm[i] = c->audio_buf[i];
>>>> }
>>>> if (c->ach)
>>>> @@ -401,10 +399,7 @@ int avpriv_dv_produce_packet(DVDemuxContext *c, AVPacket *pkt,
>>>> c->audio_pkt[2].size = c->audio_pkt[3].size = 0;
>>>> } else {
>>>> c->audio_pkt[0].size = c->audio_pkt[1].size = 0;
>>>> - c->abytes += size;
>>>> }
>>>> - } else {
>>>> - c->abytes += size;
>>>> }
>>>>
>>>> /* Now it's time to return video packet */
>>>> @@ -444,13 +439,6 @@ static int64_t dv_frame_offset(AVFormatContext *s, DVDemuxContext *c,
>>>> void ff_dv_offset_reset(DVDemuxContext *c, int64_t frame_offset)
>>>> {
>>>> c->frames = frame_offset;
>>>> - if (c->ach) {
>>>> - if (c->sys) {
>>>> - c->abytes = av_rescale_q(c->frames, c->sys->time_base,
>>>> - (AVRational) { 8, c->ast[0]->codecpar->bit_rate });
>>>> - } else
>>>> - av_log(c->fctx, AV_LOG_ERROR, "cannot adjust audio bytes\n");
>>>> - }
>>>> c->audio_pkt[0].size = c->audio_pkt[1].size = 0;
>>>> c->audio_pkt[2].size = c->audio_pkt[3].size = 0;
>>>> }
>>>> diff --git a/tests/ref/seek/lavf-dv b/tests/ref/seek/lavf-dv
>>>> index 0000ff5abe..f63e4460be 100644
>>>> --- a/tests/ref/seek/lavf-dv
>>>> +++ b/tests/ref/seek/lavf-dv
>>>> @@ -7,9 +7,9 @@ ret: 0 st: 0 flags:0 ts: 0.800000
>>>> ret: 0 st: 0 flags:1 dts: 0.800000 pts: 0.800000 pos:2880000 size:144000
>>>> ret: 0 st: 0 flags:1 ts:-0.320000
>>>> ret: 0 st: 0 flags:1 dts: 0.000000 pts: 0.000000 pos: 0 size:144000
>>>> -ret: 0 st: 1 flags:0 ts: 2.576667
>>>> +ret: 0 st: 1 flags:0 ts: 2.560000
>>>> ret: 0 st: 0 flags:1 dts: 0.960000 pts: 0.960000 pos:3456000 size:144000
>>>> -ret: 0 st: 1 flags:1 ts: 1.470833
>>>> +ret: 0 st: 1 flags:1 ts: 1.480000
>>>> ret: 0 st: 0 flags:1 dts: 0.960000 pts: 0.960000 pos:3456000 size:144000
>>>> ret: 0 st:-1 flags:0 ts: 0.365002
>>>> ret: 0 st: 0 flags:1 dts: 0.360000 pts: 0.360000 pos:1296000 size:144000
>>>> @@ -19,9 +19,9 @@ ret: 0 st: 0 flags:0 ts: 2.160000
>>>> ret: 0 st: 0 flags:1 dts: 0.960000 pts: 0.960000 pos:3456000 size:144000
>>>> ret: 0 st: 0 flags:1 ts: 1.040000
>>>> ret: 0 st: 0 flags:1 dts: 0.960000 pts: 0.960000 pos:3456000 size:144000
>>>> -ret: 0 st: 1 flags:0 ts:-0.058333
>>>> +ret: 0 st: 1 flags:0 ts:-0.040000
>>>> ret: 0 st: 0 flags:1 dts: 0.000000 pts: 0.000000 pos: 0 size:144000
>>>> -ret: 0 st: 1 flags:1 ts: 2.835833
>>>> +ret: 0 st: 1 flags:1 ts: 2.840000
>>>> ret: 0 st: 0 flags:1 dts: 0.960000 pts: 0.960000 pos:3456000 size:144000
>>>> ret: 0 st:-1 flags:0 ts: 1.730004
>>>> ret: 0 st: 0 flags:1 dts: 0.960000 pts: 0.960000 pos:3456000 size:144000
>>>> @@ -31,10 +31,10 @@ ret: 0 st: 0 flags:0 ts:-0.480000
>>>> ret: 0 st: 0 flags:1 dts: 0.000000 pts: 0.000000 pos: 0 size:144000
>>>> ret: 0 st: 0 flags:1 ts: 2.400000
>>>> ret: 0 st: 0 flags:1 dts: 0.960000 pts: 0.960000 pos:3456000 size:144000
>>>> -ret: 0 st: 1 flags:0 ts: 1.306667
>>>> -ret: 0 st: 0 flags:1 dts: 0.960000 pts: 0.960000 pos:3456000 size:144000
>>>> -ret: 0 st: 1 flags:1 ts: 0.200833
>>>> +ret: 0 st: 1 flags:0 ts: 1.320000
>>>> ret: 0 st: 0 flags:1 dts: 0.960000 pts: 0.960000 pos:3456000 size:144000
>>>> +ret: 0 st: 1 flags:1 ts: 0.200000
>>>> +ret: 0 st: 0 flags:1 dts: 0.200000 pts: 0.200000 pos: 720000 size:144000
>>>> ret: 0 st:-1 flags:0 ts:-0.904994
>>>> ret: 0 st: 0 flags:1 dts: 0.000000 pts: 0.000000 pos: 0 size:144000
>>>> ret: 0 st:-1 flags:1 ts: 1.989173
>>>> @@ -43,9 +43,9 @@ ret: 0 st: 0 flags:0 ts: 0.880000
>>>> ret: 0 st: 0 flags:1 dts: 0.880000 pts: 0.880000 pos:3168000 size:144000
>>>> ret: 0 st: 0 flags:1 ts:-0.240000
>>>> ret: 0 st: 0 flags:1 dts: 0.000000 pts: 0.000000 pos: 0 size:144000
>>>> -ret: 0 st: 1 flags:0 ts: 2.671667
>>>> +ret: 0 st: 1 flags:0 ts: 2.680000
>>>> ret: 0 st: 0 flags:1 dts: 0.960000 pts: 0.960000 pos:3456000 size:144000
>>>> -ret: 0 st: 1 flags:1 ts: 1.565833
>>>> +ret: 0 st: 1 flags:1 ts: 1.560000
>>>> ret: 0 st: 0 flags:1 dts: 0.960000 pts: 0.960000 pos:3456000 size:144000
>>>> ret: 0 st:-1 flags:0 ts: 0.460008
>>>> ret: 0 st: 0 flags:1 dts: 0.480000 pts: 0.480000 pos:1728000 size:144000
>>>> — 2.26.2
>>>
>>> Best Regards,
>>> Dave Rice
>>>
>>> _______________________________________________
>>> ffmpeg-devel mailing list
>>> ffmpeg-devel at ffmpeg.org <mailto:ffmpeg-devel at ffmpeg.org>
>>> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel <https://ffmpeg.org/mailman/listinfo/ffmpeg-devel>
>>>
>>> To unsubscribe, visit link above, or email
>>> ffmpeg-devel-request at ffmpeg.org <mailto:ffmpeg-devel-request at ffmpeg.org> with subject "unsubscribe".
>> _______________________________________________
>> ffmpeg-devel mailing list
>> ffmpeg-devel at ffmpeg.org <mailto:ffmpeg-devel at ffmpeg.org>
>> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel <https://ffmpeg.org/mailman/listinfo/ffmpeg-devel>
>>
>> To unsubscribe, visit link above, or email
>> ffmpeg-devel-request at ffmpeg.org <mailto:ffmpeg-devel-request at ffmpeg.org> with subject "unsubscribe".
>
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel at ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
>
> To unsubscribe, visit link above, or email
> ffmpeg-devel-request at ffmpeg.org with subject "unsubscribe".
More information about the ffmpeg-devel
mailing list