[FFmpeg-devel] [PATCH] libavfi/dnn: add LibTorch as one of DNN backend

[Guo, Yejun]

> -----Original Message-----
> From: ffmpeg-devel <ffmpeg-devel-bounces at ffmpeg.org> On Behalf Of
> wenbin.chen-at-intel.com at ffmpeg.org
> Sent: Monday, January 22, 2024 2:11 PM
> To: ffmpeg-devel at ffmpeg.org
> Subject: [FFmpeg-devel] [PATCH] libavfi/dnn: add LibTorch as one of DNN
> backend
> 
> From: Wenbin Chen <wenbin.chen at intel.com>
> 
> PyTorch is an open source machine learning framework that accelerates the
> path from research prototyping to production deployment. Official
> websit: https://pytorch.org/. We call the C++ library of PyTorch as LibTorch,
> the same below.
> 
> To build FFmpeg with LibTorch, please take following steps as reference:
> 1. download LibTorch C++ library in https://pytorch.org/get-started/locally/,
> please select C++/Java for language, and other options as your need.
> 2. unzip the file to your own dir, with command unzip libtorch-shared-with-
> deps-latest.zip -d your_dir 3. export libtorch_root/libtorch/include and
> libtorch_root/libtorch/include/torch/csrc/api/include to $PATH export
> libtorch_root/libtorch/lib/ to $LD_LIBRARY_PATH 4. config FFmpeg
> with ../configure --enable-libtorch --extra-cflag=-
> I/libtorch_root/libtorch/include --extra-cflag=-
> I/libtorch_root/libtorch/include/torch/csrc/api/include --extra-ldflags=-
> L/libtorch_root/libtorch/lib/
> 5. make
> 
> To run FFmpeg DNN inference with LibTorch backend:
> ./ffmpeg -i input.jpg -vf
> dnn_processing=dnn_backend=torch:model=LibTorch_model.pt -y output.jpg
> The LibTorch_model.pt can be generated by Python with torch.jit.script() api.
> Please note, torch.jit.trace() is not recommanded, since it does not support
> ambiguous input size.
> 
> Signed-off-by: Ting Fu <ting.fu at intel.com>
> Signed-off-by: Wenbin Chen <wenbin.chen at intel.com>
> ---
>  configure                             |   5 +-
>  libavfilter/dnn/Makefile              |   1 +
>  libavfilter/dnn/dnn_backend_torch.cpp | 585 ++++++++++++++++++++++++++
>  libavfilter/dnn/dnn_interface.c       |   5 +
>  libavfilter/dnn_filter_common.c       |  31 +-
>  libavfilter/dnn_interface.h           |   2 +-
>  libavfilter/vf_dnn_processing.c       |   3 +
>  7 files changed, 621 insertions(+), 11 deletions(-)  create mode 100644
> libavfilter/dnn/dnn_backend_torch.cpp
> 

I'm glad to see the libtorch as a new dnn backend personally, due to the fact that
more and more deep learning models are trained with PyTorch. PyTorch is a
necessary in the AI domain, including analysis/processing of image, video, audio
and subtitle (text) and even putting them together.