[FFmpeg-devel] [PATCH] avcodec/videotoolboxenc: ignore encoded h264 SEI nalu by default
sharpbai at gmail.com
sharpbai at gmail.com
Fri Feb 19 00:09:28 EET 2021
To Andreas Rheinhardt,
You are right. The last byte 0x80 is rbsp_trailing_bits.
Before macOS 11 there is no problem of the generated SEI.
To Rick Kern,
After macOS 11, the encoded SEI nalu length changes from 20
to 25, the following code should help you to confirm it.
Place these code to copy_replace_length_codes function,
after CMBlockBufferCopyDataBytes calling copy nalu data.
```
if (nal_type == 6) {
char buf[1024] = {0};
int index = 0;
for(int i = 0; i < box_len; i++) {
index += sprintf(buf + index, "%.2x ", dst_box[i]);
}
av_log(avctx, AV_LOG_DEBUG, "Found sei nalu length: %zu value: %s\n", box_len, buf);
} else {
av_log(avctx, AV_LOG_DEBUG, "Found type %d nalu length: %zu\n", nal_type, box_len);
}
```
For example run this ffmpeg command
$ ffmpeg -f lavfi -i testsrc=d=5:s=1920x1080:r=24,format=yuv420p \
-ignore_internal_sei 0 -vcodec h264_videotoolbox -b:v 40m -y bar.h264
got a output like this
[h264_videotoolbox @ 0x7fa30d00ca00] Found sei nalu length: 25 value: 06 05 10 b9 ed b9 30 5d 21 4b 71 83 71 2c 10 a3 14 bb 29 80 00 00 03 00 01
[h264_videotoolbox @ 0x7fa30d00ca00] Found type 5 nalu length: 135751
This proves that the last 5 bytes are added by videotoolboxenc.
February 16, 2021 12:23 AM, "Rick Kern" <kernrj at gmail.com> 写到:
> On Mon, Feb 15, 2021 at 7:51 AM Andreas Rheinhardt <
> andreas.rheinhardt at gmail.com> wrote:
>
>> sharpbai:
>> Before macOS 11, the encoded h264 sample contains fixed SEI before
>> IDR frame, which the content is below
>>
>> 06 05 10 b9 ed b9 30 5d 21 4b 71 83 71 2c 10 a3
>> 14 bb 29 80
>>
>> The length of the encoded SEI nalu is 20 byte. According to
>> the ITU-T H.264 Recommendation section 7.3.2.3.1, the last byte
>> 0x80 means type. Then there is no length byte, which make the
>> SEI nonstandard.
>>
>> Wrong:
>> 06: forbidden_zero_bit + nal_ref_idc + nal_unit_type (6 for SEI)
>> 05: payload type 5 (user data unregistered) SEI message
>> 10: length of this SEI message is 16
>> b9-29: the payload of this SEI message (the first 12 byte are an uuid,
>> the last are user_data_payload_byte)
>> 80: rbsp_trailing_bits: the first bit is rbsp_stop_one_bit (equal to
>> one), the remaining bits are padding (with value zero)
>>
>> You can use trace_headers to inspect this.
>>
>> (Notice that one needs to know the length of the NAL unit in advance
>> before one can start parsing it; how one gets it depends upon the
>> framing. There are two common methods for this: Annex B and mp4. The
>> former works by using start codes 0x00 00 01 to mark the beginning of a
>> new NAL, the latter prefixes the length of the NAL units to the units.)
>>
>> On macOS 11, the encoded h264 sample still contains fixed SEI before
>> IDR frame, which length is 5 bytes longer than before.
>>
>> 06 05 10 b9 ed b9 30 5d 21 4b 71 83 71 2c 10 a3
>> 14 bb 29 80 00 00 03 00 01
>> ^^ ^^ ^^ ^^ ^^
>>
>> That looks as if there were a 00 00 00 01 startcode after the SEI that
>> some tool has tried to 0x03 escape. That is a bug. Can you share samples
>> containing these NAL units?
>
> This file is inserting the emulation prevention byte 0x03. I can take a
> look later today.
>
>> This time type the content of the SEI has type 0 payload but
>> invalid length payload data, which violates recommendation
>> appendix D.
>>
>> Whether it is a bug or not, these SEI nalus should be erased to keep
>> the output bytestream not violating the standard. But we may
>> not come across the situration we do need these encoded SEI
>> nalu, so we just reserve a switch and set not to output by default.
>>
>> Signed-off-by: sharpbai <sharpbai at gmail.com>
>> ---
>> libavcodec/videotoolboxenc.c | 51 +++++++++++++++++++++---------------
>> 1 file changed, 30 insertions(+), 21 deletions(-)
>>
>> diff --git a/libavcodec/videotoolboxenc.c b/libavcodec/videotoolboxenc.c
>> index cc08cf6a50..c240090f10 100644
>> --- a/libavcodec/videotoolboxenc.c
>> +++ b/libavcodec/videotoolboxenc.c
>> @@ -221,6 +221,7 @@ typedef struct VTEncContext {
>>
>> int64_t allow_sw;
>> int64_t require_sw;
>> + int64_t ignore_internal_sei;
>>
>> bool flushing;
>> bool has_b_frames;
>> @@ -1690,7 +1691,8 @@ static int copy_replace_length_codes(
>> CMSampleBufferRef sample_buffer,
>> ExtraSEI *sei,
>> uint8_t *dst_data,
>> - size_t dst_size)
>> + size_t dst_size,
>> + int ignore_internal_sei)
>> {
>> size_t src_size = CMSampleBufferGetTotalSampleSize(sample_buffer);
>> size_t remaining_src_size = src_size;
>> @@ -1707,8 +1709,8 @@ static int copy_replace_length_codes(
>> }
>>
>> while (remaining_src_size > 0) {
>> - size_t curr_src_len;
>> - size_t curr_dst_len;
>> + size_t curr_src_len = 0;
>> + size_t curr_dst_len = 0;
>> size_t box_len = 0;
>> size_t i;
>>
>> @@ -1740,7 +1742,8 @@ static int copy_replace_length_codes(
>> box_len |= size_buf[i];
>> }
>>
>> - if (sei && !wrote_sei && is_post_sei_nal_type(nal_type)) {
>> + if (sei && !wrote_sei &&
>> + (ignore_internal_sei ||
>> is_post_sei_nal_type(nal_type))) {
>> //No SEI NAL unit - insert.
>> int wrote_bytes;
>>
>> @@ -1775,27 +1778,30 @@ static int copy_replace_length_codes(
>> }
>>
>> curr_src_len = box_len + length_code_size;
>> - curr_dst_len = box_len + sizeof(start_code);
>>
>> - if (remaining_src_size < curr_src_len) {
>> - return AVERROR_BUFFER_TOO_SMALL;
>> - }
>> + if (!(ignore_internal_sei && nal_type == H264_NAL_SEI)) {
>> + curr_dst_len = box_len + sizeof(start_code);
>>
>> - if (remaining_dst_size < curr_dst_len) {
>> - return AVERROR_BUFFER_TOO_SMALL;
>> - }
>> + if (remaining_src_size < curr_src_len) {
>> + return AVERROR_BUFFER_TOO_SMALL;
>> + }
>> +
>> + if (remaining_dst_size < curr_dst_len) {
>> + return AVERROR_BUFFER_TOO_SMALL;
>> + }
>>
>> - dst_box = dst_data + sizeof(start_code);
>> + dst_box = dst_data + sizeof(start_code);
>>
>> - memcpy(dst_data, start_code, sizeof(start_code));
>> - status = CMBlockBufferCopyDataBytes(block,
>> - src_offset +
>> length_code_size,
>> - box_len,
>> - dst_box);
>> + memcpy(dst_data, start_code, sizeof(start_code));
>> + status = CMBlockBufferCopyDataBytes(block,
>> + src_offset +
>> length_code_size,
>> + box_len,
>> + dst_box);
>>
>> - if (status) {
>> - av_log(avctx, AV_LOG_ERROR, "Cannot copy data: %d\n",
>> status);
>> - return AVERROR_EXTERNAL;
>> + if (status) {
>> + av_log(avctx, AV_LOG_ERROR, "Cannot copy data: %d\n",
>> status);
>> + return AVERROR_EXTERNAL;
>> + }
>> }
>>
>> if (sei && !wrote_sei && nal_type == H264_NAL_SEI) {
>> @@ -1931,7 +1937,8 @@ static int vtenc_cm_to_avpacket(
>> sample_buffer,
>> sei,
>> pkt->data + header_size,
>> - pkt->size - header_size
>> + pkt->size - header_size,
>> + vtctx->ignore_internal_sei
>> );
>>
>> if (status) {
>> @@ -2543,6 +2550,8 @@ static const enum AVPixelFormat hevc_pix_fmts[] = {
>> { .i64 = 0 }, 0, 1, VE }, \
>> { "require_sw", "Require software encoding", OFFSET(require_sw),
>> AV_OPT_TYPE_BOOL, \
>> { .i64 = 0 }, 0, 1, VE }, \
>> + { "ignore_internal_sei", "Ignore SEI nalu generated by
>> videotoolbox", OFFSET(ignore_internal_sei), AV_OPT_TYPE_BOOL, \
>> + { .i64 = 1 }, 0, 1, VE }, \
>> { "realtime", "Hint that encoding should happen in real-time if not
>> faster (e.g. capturing from camera).", \
>> OFFSET(realtime), AV_OPT_TYPE_BOOL, { .i64 = 0 }, 0, 1, VE }, \
>> { "frames_before", "Other frames will come before the frames in
>> this session. This helps smooth concatenation issues.", \
>>
>> _______________________________________________
>> ffmpeg-devel mailing list
>> ffmpeg-devel at ffmpeg.org
>> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
>>
>> To unsubscribe, visit link above, or email
>> ffmpeg-devel-request at ffmpeg.org with subject "unsubscribe".
>
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel at ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
>
> To unsubscribe, visit link above, or email
> ffmpeg-devel-request at ffmpeg.org with subject "unsubscribe".
More information about the ffmpeg-devel
mailing list