[FFmpeg-devel] [PATCH v1] libavfi/dnn: add Paddle Inference as one of DNN backend

Wed May 10 05:25:37 EEST 2023

Dear Madam or Sir,

Hope this email finds you well.

I am writing this email since i recently found FFmepg remove DNN native  backend, and i will be really grateful if you let me know if there is  any new plan on libavfilter/dnn.

I would like to explain to you again about the addition of dnn paddle backend.

At  present, ffmpeg only supports openvino and tensorflow backend. Among  the current deep learning frameworks, TensorFlow is the most active in  development. TensorFlow has 174k stars and pytorch has 66.5k. openvino  is 4.2k, and the models that openvino can implement are relatively few.  But in terms of attention on GitHub, there's no doubt that TensorFlow  and pytorch are more promising. Currently, the paddle framework has  reached 20.2k stars on github, which is much more widely used and active  than frameworks such as mxnet and caffe.

Tensoflow has a very  rich ecosystem. The TensorFlow models library updates very quickly and  has existing examples of deep learning applications for image  classification, object detection, image generation text, and generation  of adversus-network models. The dnn libavfilter module is undoubtedly very necessary for tensorflow  backend to support. But the complexity of the TensorFlow API and the  complexity of the training are almost prohibitive, making it a love-hate  framework.

PyTorch framework tends to be applied to academic  fast implementation, and its industrial application performance is not  good. For example, Pytorch framework makes a model to run on a server,  Android phone or embedded system, and its performance is poor compared  with other deep learning frameworks.

PaddlePadddle  is an open source framework of Baidu, which is also used by many people  in China. It is very consistent with the usage habits of developers,  but the practicability of the API still needs to be further  strengthened. However, Paddle is the only deep learning framework I  have ever used, which does not configure any third-party libraries and  can be used directly by cloning make. Besides, Paddle occupies a small  amount of memory and is fast. It also serves a considerable number of  projects inside Baidu, which is very strong in industrial application.  And PaddlePaddle supports multiple machine and multiple card training.

Users'  choice of different deep learning frameworks is a personal choice, and  the reason why most of us chose paddle is because of its better support  for embedded development and different hardware platforms and because  the community is very active and has proposed industrial improvements  and implementations for some advanced models. Especially for the GPU, it  supports cuda and opencl, which means we can optimize the model no  matter what kind of graphics card is used. In my opinion, more backend  support can better improve dnn libavfilter modules.

If there are any  new changes in dnn libavfilter module, I will be very willing to adjust our  implementation with the new planning and provide continuous maintenance.

Best Regards,
Wenzhe Wang

WenzheWang
wongwwz at foxmail.com

------------------ Original ------------------
From:                                                                                                                        "WenzheWang"                                                                                    <wongwwz at foxmail.com>;
Date: Tue, Apr 11, 2023 11:03 PM
To: "ffmpeg-devel"<ffmpeg-devel at ffmpeg.org>;

Subject: Re: [FFmpeg-devel] [PATCH v1] libavfi/dnn: add Paddle Inference as one of DNN backend

Could you please briefly introduce the reason why not adding any dnn backend? 

Do you have any plan for the maintenance and development of the dnn backend in the future? From my understanding, the current backend of dnn has tensoflow, openvino and native, but this cannot meet the needs of users.

Thus, I believe adding other dnn backends will be great for user experience, user growth, and industrial applications. In particular, various dnn backend can be adapted to different application environments, and there are some emerging inference engines that are faster and stronger, such as Pytorch and Paddle. In addition, from the practical point of view, it is not difficult for a deep learning practitioner to learn and use this framework, but how to choose a framework and apply it in practice, people pay more attention to the effect (recall and precision), and easy deployment, that is, high reasoning performance efficiency. The main reason why Paddle is relatively mainstream and why I want to add paddle backend is that it has a very high efficiency and performance. There are several projects maintained by Paddle, such as paddleDetection, paddleSeg, paddleGAN, paddleOCR and paddleCls have a lot of good pre-training models that migrate well to their own data and has excellent performance. Secondly, in terms of reasoning efficiency, Paddle supports many platforms and chips. Models trained using Paddle framework can be directly deployed, and custom device interfaces are open for independent development based on one's own hardware.

FFmpeg itself already has very extensive support for codec. If FFmpeg could support the deployment of more reasoning model backend, it would have a wider application.

In general, I hope that ffmpeg could support the backend of paddle or more. In any case that my code is not mature or proper, I would be grateful if professionals like you could offer me suggestions and comments. I will be absolutely honored if I could contribute to this project :)

Best,

Wenzhe Wang

WenzheWang
wongwwz at foxmail.com

------------------ Original ------------------
From:                                                                                                                        "FFmpeg development discussions and patches"                                                                                    <jb at videolan.org>;
Date: Sun, Apr 9, 2023 05:31 AM
To: "ffmpeg-devel"<ffmpeg-devel at ffmpeg.org>;

Subject: Re: [FFmpeg-devel] [PATCH v1] libavfi/dnn: add Paddle Inference as one of DNN backend

On Thu, 6 Apr 2023, at 12:36, wongwwz at foxmail.com wrote:
> PaddlePaddle (PArallel Distributed Deep LEarning) is a simple, 
> efficient and extensible deep learning framework that accelerates the 

Please don't add another DNN backend.

-- 
Jean-Baptiste Kempf -  President
+33 672 704 734
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel at ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request at ffmpeg.org with subject "unsubscribe".