[FFmpeg-devel] [PATCH 2/4] avcodec/liblc3: Add encoding/decoding support of LC3 audio codec
Stefano Sabatini
stefasab at gmail.com
Thu Mar 28 19:54:31 EET 2024
On date Wednesday 2024-03-27 22:21:43 +0000, ffmpeg-devel Mailing List wrote:
> The LC3 audio codec is the default codec of Bluetooth LE audio.
> This is a wrapper over the liblc3 library (https://github.com/google/liblc3).
>
> Signed-off-by: Antoine Soulier <asoulier at google.com>
> ---
> doc/encoders.texi | 57 ++++++++++++
> doc/general_contents.texi | 11 ++-
> libavcodec/Makefile | 2 +
> libavcodec/allcodecs.c | 2 +
> libavcodec/codec_desc.c | 7 ++
> libavcodec/codec_id.h | 1 +
> libavcodec/liblc3dec.c | 135 +++++++++++++++++++++++++++
> libavcodec/liblc3enc.c | 190 ++++++++++++++++++++++++++++++++++++++
> 8 files changed, 404 insertions(+), 1 deletion(-)
> create mode 100644 libavcodec/liblc3dec.c
> create mode 100644 libavcodec/liblc3enc.c
>
> diff --git a/doc/encoders.texi b/doc/encoders.texi
> index 7c223ed74c..66847191e1 100644
> --- a/doc/encoders.texi
> +++ b/doc/encoders.texi
> @@ -814,6 +814,63 @@ ffmpeg -i input.wav -c:a libfdk_aac -profile:a aac_he -b:a 64k output.m4a
> @end example
> @end itemize
>
> + at anchor{liblc3-enc}
> + at section liblc3
> +
> +liblc3 LC3 (Low Complexity Communication Codec) encoder wrapper.
> +
> +Requires the presence of the liblc3 headers and library during configuration.
> +You need to explicitly configure the build with @code{--enable-liblc3}.
> +
> +This encoder has support for the Bluetooth SIG LC3 codec for the LE Audio
> +protocol, and the following features of LC3plus:
> + at itemize
> + at item
> +Frame duration of 2.5 and 5ms.
> + at item
> +High-Resolution mode, 48 KHz, and 96 kHz sampling rates.
> + at end itemize
> +
> +For more information see the liblc3 project at
> + at url{https://github.com/google/liblc3}.
> +
> + at subsection Options
> +
> +The following options are mapped on the shared FFmpeg codec options.
> +
> + at table @option
> + at item b @var{bitrate}
> +Set the bit rate in bits/s. This will determine the fixed size of the encoded
> +frames, for a selected frame duration.
> +
> + at item ar @var{frequency}
> +Set the audio sampling rate (in Hz).
> +
> + at item channels
> +Set the number of audio channels.
> +
> + at item frame_duration
> +Set the audio frame duration in milliseconds. Default value is 10ms.
> +Allowed frame durations are 2.5ms, 5ms, 7.5ms and 10ms.
> +LC3 (Bluetooth LE Audio), allows 7.5ms and 10ms; and LC3plus 2.5ms, 5ms
> +and 10ms.
> +
> +The 10ms frame duration is available in LC3 and LC3 plus standard.
> +In this mode, the produced bitstream can be referenced either as LC3 or LC3plus.
> +
> + at item high_resolution @var{boolean}
> +Enable the high-resolution mode if set to 1. The high-resolution mode is
> +available with all LC3plus frame durations and for a sampling rate of 48 KHz,
> +and 96 KHz.
> +
> +The encoder automatically turns off this mode at lower sampling rates and
> +activates it at 96 KHz.
> +
> +This mode should be preferred at high bitrates. In this mode, the audio
> +bandwidth is always up to the Nyquist frequency, compared to LC3 at 48 KHz,
> +which limits the bandwidth to 20 KHz.
> + at end table
> +
> @anchor{libmp3lame}
> @section libmp3lame
>
> diff --git a/doc/general_contents.texi b/doc/general_contents.texi
> index f269cbd1a9..e7cf4f8239 100644
> --- a/doc/general_contents.texi
> +++ b/doc/general_contents.texi
> @@ -237,6 +237,14 @@ Go to @url{http://sourceforge.net/projects/opencore-amr/} and follow the
> instructions for installing the library.
> Then pass @code{--enable-libfdk-aac} to configure to enable it.
>
> + at subsection LC3 library
> +
> +FFmpeg can make use of the Google LC3 library for LC3 decoding & encoding.
> +
> +Go to @url{https://github.com/google/liblc3/} and follow the instructions for
> +installing the library.
> +Then pass @code{--enable-liblc3} to configure to enable it.
> +
> @section OpenH264
>
> FFmpeg can make use of the OpenH264 library for H.264 decoding and encoding.
> @@ -1300,7 +1308,8 @@ following image formats are supported:
> @tab encoding and decoding supported through external library libilbc
> @item IMC (Intel Music Coder) @tab @tab X
> @item Interplay ACM @tab @tab X
> - at item MACE (Macintosh Audio Compression/Expansion) 3:1 @tab @tab X
> + at item LC3 @tab E @tab E
> + @tab supported through external library liblc3
> @item MACE (Macintosh Audio Compression/Expansion) 6:1 @tab @tab X
> @item Marian's A-pac audio @tab @tab X
> @item MI-SC4 (Micronas SC-4 Audio) @tab @tab X
> diff --git a/libavcodec/Makefile b/libavcodec/Makefile
> index 9ce6d445c1..e70811dbd6 100644
> --- a/libavcodec/Makefile
> +++ b/libavcodec/Makefile
> @@ -1123,6 +1123,8 @@ OBJS-$(CONFIG_LIBILBC_ENCODER) += libilbc.o
> OBJS-$(CONFIG_LIBJXL_DECODER) += libjxldec.o libjxl.o
> OBJS-$(CONFIG_LIBJXL_ENCODER) += libjxlenc.o libjxl.o
> OBJS-$(CONFIG_LIBKVAZAAR_ENCODER) += libkvazaar.o
> +OBJS-$(CONFIG_LIBLC3_ENCODER) += liblc3enc.o
> +OBJS-$(CONFIG_LIBLC3_DECODER) += liblc3dec.o
> OBJS-$(CONFIG_LIBMP3LAME_ENCODER) += libmp3lame.o
> OBJS-$(CONFIG_LIBOPENCORE_AMRNB_DECODER) += libopencore-amr.o
> OBJS-$(CONFIG_LIBOPENCORE_AMRNB_ENCODER) += libopencore-amr.o
> diff --git a/libavcodec/allcodecs.c b/libavcodec/allcodecs.c
> index 2386b450a6..f4705651fb 100644
> --- a/libavcodec/allcodecs.c
> +++ b/libavcodec/allcodecs.c
> @@ -776,6 +776,8 @@ extern const FFCodec ff_libilbc_encoder;
> extern const FFCodec ff_libilbc_decoder;
> extern const FFCodec ff_libjxl_decoder;
> extern const FFCodec ff_libjxl_encoder;
> +extern const FFCodec ff_liblc3_encoder;
> +extern const FFCodec ff_liblc3_decoder;
> extern const FFCodec ff_libmp3lame_encoder;
> extern const FFCodec ff_libopencore_amrnb_encoder;
> extern const FFCodec ff_libopencore_amrnb_decoder;
> diff --git a/libavcodec/codec_desc.c b/libavcodec/codec_desc.c
> index 3bab86db62..7dba61dc8b 100644
> --- a/libavcodec/codec_desc.c
> +++ b/libavcodec/codec_desc.c
> @@ -3425,6 +3425,13 @@ static const AVCodecDescriptor codec_descriptors[] = {
> .long_name = NULL_IF_CONFIG_SMALL("QOA (Quite OK Audio)"),
> .props = AV_CODEC_PROP_INTRA_ONLY | AV_CODEC_PROP_LOSSY,
> },
> + {
> + .id = AV_CODEC_ID_LC3,
> + .type = AVMEDIA_TYPE_AUDIO,
> + .name = "lc3",
> + .long_name = NULL_IF_CONFIG_SMALL("LC3 (Low Complexity Communication Codec)"),
> + .props = AV_CODEC_PROP_INTRA_ONLY | AV_CODEC_PROP_LOSSY,
> + },
>
> /* subtitle codecs */
> {
> diff --git a/libavcodec/codec_id.h b/libavcodec/codec_id.h
> index c8dc21da74..0ab1e34a61 100644
> --- a/libavcodec/codec_id.h
> +++ b/libavcodec/codec_id.h
> @@ -543,6 +543,7 @@ enum AVCodecID {
> AV_CODEC_ID_AC4,
> AV_CODEC_ID_OSQ,
> AV_CODEC_ID_QOA,
> + AV_CODEC_ID_LC3,
>
> /* subtitle codecs */
> AV_CODEC_ID_FIRST_SUBTITLE = 0x17000, ///< A dummy ID pointing at the start of subtitle codecs.
> diff --git a/libavcodec/liblc3dec.c b/libavcodec/liblc3dec.c
> new file mode 100644
> index 0000000000..f9c047c03f
> --- /dev/null
> +++ b/libavcodec/liblc3dec.c
> @@ -0,0 +1,135 @@
> +/*
> + * LC3 decoder wrapper
> + * Copyright (C) 2024 Antoine Soulier <asoulier at google.com>
> + *
> + * This file is part of FFmpeg.
> + *
> + * Permission to use, copy, modify, and/or distribute this software for any
> + * purpose with or without fee is hereby granted, provided that the above
> + * copyright notice and this permission notice appear in all copies.
> + *
> + * THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
> + * WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
> + * MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
> + * ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
> + * WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
> + * ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
> + * OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
> + */
> +
> +#include <lc3.h>
> +
> +#include "libavutil/intreadwrite.h"
> +
> +#include "avcodec.h"
> +#include "codec.h"
> +#include "codec_internal.h"
> +#include "decode.h"
> +#include "internal.h"
> +
> +#define DECODER_MAX_CHANNELS 2
> +
> +typedef struct LibLC3DecContext {
> + int frame_us, srate_hz;
> + bool hr_mode;
> + void *decoder_mem;
> + lc3_decoder_t decoder[DECODER_MAX_CHANNELS];
> +} LibLC3DecContext;
> +
> +static av_cold int liblc3_decode_init(AVCodecContext *avctx)
> +{
> + LibLC3DecContext *liblc3 = avctx->priv_data;
> + int channels = avctx->ch_layout.nb_channels;
> + unsigned decoder_size;
> +
> + if (avctx->extradata_size < 2)
> + return AVERROR_INVALIDDATA;
> +
> + liblc3->frame_us = AV_RL16(avctx->extradata + 0);
> + liblc3->srate_hz = avctx->sample_rate;
> + liblc3->hr_mode = avctx->extradata_size >= 6 &&
> + AV_RL16(avctx->extradata + 4);
> +
> + av_log(avctx, AV_LOG_INFO,
> + "Decoding %.1f ms frames\n", liblc3->frame_us / 1000.f);
> + if (liblc3->hr_mode)
> + av_log(avctx, AV_LOG_INFO, "High-resolution mode enabled\n");
> +
> + decoder_size = lc3_hr_decoder_size(
> + liblc3->hr_mode, liblc3->frame_us, liblc3->srate_hz);
> + if (!decoder_size)
> + return AVERROR_INVALIDDATA;
> +
> + liblc3->decoder_mem = av_malloc_array(channels, decoder_size);
> + if (!liblc3->decoder_mem)
> + return AVERROR(ENOMEM);
> +
> + for (int ch = 0; ch < channels; ch++) {
> + liblc3->decoder[ch] = lc3_hr_setup_decoder(
> + liblc3->hr_mode, liblc3->frame_us, liblc3->srate_hz, 0,
> + (char *)liblc3->decoder_mem + ch * decoder_size);
> + }
> +
> + avctx->sample_fmt = AV_SAMPLE_FMT_FLTP;
> + avctx->delay = lc3_hr_delay_samples(
> + liblc3->hr_mode, liblc3->frame_us, liblc3->srate_hz);
> + avctx->internal->skip_samples = avctx->delay;
> +
> + return 0;
> +}
> +
> +static av_cold int liblc3_decode_close(AVCodecContext *avctx)
> +{
> + LibLC3DecContext *liblc3 = avctx->priv_data;
> +
> + av_freep(&liblc3->decoder_mem);
> +
> + return 0;
> +}
> +
> +static int liblc3_decode(AVCodecContext *avctx, AVFrame *frame,
> + int *got_frame_ptr, AVPacket *avpkt)
> +{
> + LibLC3DecContext *liblc3 = avctx->priv_data;
> + int channels = avctx->ch_layout.nb_channels;
> + uint8_t *in = avpkt->data;
> + int block_bytes, ret;
> +
> + frame->nb_samples = av_rescale(
> + liblc3->frame_us, liblc3->srate_hz, 1000*1000);
> + if ((ret = ff_get_buffer(avctx, frame, 0)) < 0)
> + return ret;
> +
> + block_bytes = avpkt->size;
> + for (int ch = 0; ch < channels; ch++) {
> + int frame_bytes = block_bytes / channels
> + + (ch < block_bytes % channels);
> +
> + ret = lc3_decode(liblc3->decoder[ch], in, frame_bytes,
> + LC3_PCM_FORMAT_FLOAT, frame->data[ch], 1);
nit++: weird indent (should be alignd to the first argument
> + if (ret < 0)
> + return AVERROR_INVALIDDATA;
> +
> + in += frame_bytes;
> + }
> +
> + frame->nb_samples = FFMIN(frame->nb_samples, avpkt->duration);
> +
> + *got_frame_ptr = 1;
> +
> + return avpkt->size;
> +}
> +
> +const FFCodec ff_liblc3_decoder = {
> + .p.name = "liblc3",
> + CODEC_LONG_NAME("LC3 (Low Complexity Communication Codec)"),
> + .p.type = AVMEDIA_TYPE_AUDIO,
> + .p.id = AV_CODEC_ID_LC3,
> + .p.capabilities = AV_CODEC_CAP_DR1,
> + .p.wrapper_name = "liblc3",
> + .priv_data_size = sizeof(LibLC3DecContext),
> + .caps_internal = FF_CODEC_CAP_INIT_CLEANUP,
> + .init = liblc3_decode_init,
> + .close = liblc3_decode_close,
> + FF_CODEC_DECODE_CB(liblc3_decode),
> +};
> diff --git a/libavcodec/liblc3enc.c b/libavcodec/liblc3enc.c
> new file mode 100644
> index 0000000000..5ac45b8cd2
> --- /dev/null
> +++ b/libavcodec/liblc3enc.c
> @@ -0,0 +1,190 @@
> +/*
> + * LC3 encoder wrapper
> + * Copyright (C) 2024 Antoine Soulier <asoulier at google.com>
> + *
> + * This file is part of FFmpeg.
> + *
> + * Permission to use, copy, modify, and/or distribute this software for any
> + * purpose with or without fee is hereby granted, provided that the above
> + * copyright notice and this permission notice appear in all copies.
> + *
> + * THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
> + * WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
> + * MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
> + * ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
> + * WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
> + * ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
> + * OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
> + */
> +
> +#include <lc3.h>
> +
> +#include "libavutil/intreadwrite.h"
> +#include "libavutil/opt.h"
> +
> +#include "avcodec.h"
> +#include "codec.h"
> +#include "codec_internal.h"
> +#include "encode.h"
> +
> +#define ENCODER_MAX_CHANNELS 2
> +
> +typedef struct LibLC3EncOpts {
> + float frame_duration;
> + int hr_mode;
> +} LibLC3EncOpts;
> +
> +typedef struct LibLC3EncContext {
> + const AVClass *av_class;
> + LibLC3EncOpts opts;
> + int block_bytes;
> + void *encoder_mem;
> + lc3_encoder_t encoder[ENCODER_MAX_CHANNELS];
> +} LibLC3EncContext;
> +
> +static av_cold int liblc3_encode_init(AVCodecContext *avctx)
> +{
> + LibLC3EncContext *liblc3 = avctx->priv_data;
> + bool hr_mode = liblc3->opts.hr_mode;
> + int frame_us = liblc3->opts.frame_duration * 1000;
> + int srate_hz = avctx->sample_rate;
> + int channels = avctx->ch_layout.nb_channels;
> + int effective_bit_rate;
> + unsigned encoder_size;
> +
> + if (frame_us != 2500 && frame_us != 5000 &&
> + frame_us != 7500 && frame_us != 10000 ) {
> + av_log(avctx, AV_LOG_ERROR,
> + "Unsupported frame duration %.1f ms\n", frame_us / 1000.f);
nit++: weird indent
> + return AVERROR(EINVAL);
> + }
> +
> + hr_mode |= srate_hz > 48000;
> + hr_mode &= srate_hz >= 48000;
> +
> + if (frame_us == 7500 && hr_mode) {
> + av_log(avctx, AV_LOG_ERROR,
> + "High-resolution mode not supported with 7.5 ms frames\n");
ditto
> + return AVERROR(EINVAL);
> + }
> +
> + av_log(avctx, AV_LOG_INFO, "Encoding %.1f ms frames\n", frame_us / 1000.f);
> + if (hr_mode)
> + av_log(avctx, AV_LOG_INFO, "High-resolution mode enabled\n");
> +
> + liblc3->block_bytes = lc3_hr_frame_block_bytes(
> + hr_mode, frame_us, srate_hz, channels, avctx->bit_rate);
> +
> + effective_bit_rate = lc3_hr_resolve_bitrate(
> + hr_mode, frame_us, srate_hz, liblc3->block_bytes);
> +
> + if (avctx->bit_rate != effective_bit_rate)
> + av_log(avctx, AV_LOG_WARNING,
> + "Bitrate changed to %d bps\n", effective_bit_rate);
> + avctx->bit_rate = effective_bit_rate;
> +
> + encoder_size = lc3_hr_encoder_size(hr_mode, frame_us, srate_hz);
> + if (!encoder_size)
> + return AVERROR(EINVAL);
> +
> + liblc3->encoder_mem = av_malloc_array(channels, encoder_size);
> + if (!liblc3->encoder_mem)
> + return AVERROR(ENOMEM);
> +
> + for (int ch = 0; ch < channels; ch++) {
> + liblc3->encoder[ch] =
> + lc3_hr_setup_encoder(hr_mode, frame_us, srate_hz, 0,
> + (char *)liblc3->encoder_mem + ch * encoder_size);
> + }
> +
> + avctx->extradata = av_mallocz(6 + AV_INPUT_BUFFER_PADDING_SIZE);
> + if (!avctx->extradata)
> + return AVERROR(ENOMEM);
> +
> + AV_WL16(avctx->extradata + 0, frame_us);
> + AV_WL16(avctx->extradata + 2, 0);
> + AV_WL16(avctx->extradata + 4, hr_mode);
> + avctx->extradata_size = 6;
> +
> + avctx->frame_size = av_rescale(frame_us, srate_hz, 1000*1000);
> +
> + return 0;
> +}
> +
> +static av_cold int liblc3_encode_close(AVCodecContext *avctx)
> +{
> + LibLC3EncContext *liblc3 = avctx->priv_data;
> +
> + av_freep(&liblc3->encoder_mem);
> +
> + return 0;
> +}
> +
> +static int liblc3_encode(AVCodecContext *avctx, AVPacket *avpkt,
> + const AVFrame *av_frame, int *got_packet_ptr)
> +{
> + LibLC3EncContext *liblc3 = avctx->priv_data;
> + int block_bytes = liblc3->block_bytes;
> + int channels = avctx->ch_layout.nb_channels;
> + uint8_t *data_ptr;
> + int ret;
> +
> + if ((ret = ff_get_encode_buffer(avctx, avpkt, block_bytes, 0)) < 0)
> + return ret;
> +
> + data_ptr = avpkt->data;
> + for (int ch = 0; ch < channels; ch++) {
> + int frame_bytes = block_bytes / channels
> + + (ch < block_bytes % channels);
> +
> + lc3_encode(liblc3->encoder[ch],
> + LC3_PCM_FORMAT_FLOAT, av_frame->data[ch], 1,
> + frame_bytes, data_ptr);
> +
> + data_ptr += frame_bytes;
> + }
> +
> + *got_packet_ptr = 1;
> +
> + return 0;
> +}
> +
> +#define OFFSET(x) offsetof(LibLC3EncContext, opts.x)
> +#define FLAGS AV_OPT_FLAG_AUDIO_PARAM | AV_OPT_FLAG_ENCODING_PARAM
> +static const AVOption options[] = {
> + { "frame_duration", "Duration of a frame in milliseconds",
> + OFFSET(frame_duration), AV_OPT_TYPE_FLOAT,
> + { .dbl = 10.0 }, 2.5, 10.0, FLAGS },
> + { "high_resolution", "Enable High-Resolution mode (48 KHz or 96 KHz)",
> + OFFSET(hr_mode), AV_OPT_TYPE_BOOL,
> + { .i64 = 0 }, 0, 1, FLAGS },
> + { NULL }
> +};
> +
> +static const AVClass class = {
> + .class_name = "liblc3 encoder",
> + .item_name = av_default_item_name,
> + .option = options,
> + .version = LIBAVUTIL_VERSION_INT,
> +};
> +
> +const FFCodec ff_liblc3_encoder = {
> + .p.name = "liblc3",
> + CODEC_LONG_NAME("LC3 (Low Complexity Communication Codec)"),
> + .p.type = AVMEDIA_TYPE_AUDIO,
> + .p.id = AV_CODEC_ID_LC3,
> + .p.capabilities = AV_CODEC_CAP_DR1,
> + .p.ch_layouts = (const AVChannelLayout[])
> + { { AV_CHANNEL_ORDER_UNSPEC, 1 },
> + { AV_CHANNEL_ORDER_UNSPEC, 2 }, { 0 } },
> + .p.supported_samplerates = (const int [])
> + { 96000, 48000, 32000, 24000, 16000, 8000, 0 },
> + .p.sample_fmts = (const enum AVSampleFormat[])
> + { AV_SAMPLE_FMT_FLTP, AV_SAMPLE_FMT_NONE },
> + .p.priv_class = &class,
> + .p.wrapper_name = "liblc3",
> + .priv_data_size = sizeof(LibLC3EncContext),
> + .init = liblc3_encode_init,
> + .close = liblc3_encode_close,
> + FF_CODEC_ENCODE_CB(liblc3_encode),
> +};
> --
> 2.44.0.478.gd926399ef9-goog
No more comments from me, it looks good to me apart for the nits.
More information about the ffmpeg-devel
mailing list