[FFmpeg-devel] [PATCH v1] libavfi/dnn: add Paddle Inference as one of DNN backend
WenzheWang
wongwwz at foxmail.com
Wed May 10 05:25:37 EEST 2023
Dear Madam or Sir,
Hope this email finds you well.
I am writing this email since i recently found FFmepg remove DNN native backend, and i will be really grateful if you let me know if there is any new plan on libavfilter/dnn.
I would like to explain to you again about the addition of dnn paddle backend.
At present, ffmpeg only supports openvino and tensorflow backend. Among the current deep learning frameworks, TensorFlow is the most active in development. TensorFlow has 174k stars and pytorch has 66.5k. openvino is 4.2k, and the models that openvino can implement are relatively few. But in terms of attention on GitHub, there's no doubt that TensorFlow and pytorch are more promising. Currently, the paddle framework has reached 20.2k stars on github, which is much more widely used and active than frameworks such as mxnet and caffe.
Tensoflow has a very rich ecosystem. The TensorFlow models library updates very quickly and has existing examples of deep learning applications for image classification, object detection, image generation text, and generation of adversus-network models. The dnn libavfilter module is undoubtedly very necessary for tensorflow backend to support. But the complexity of the TensorFlow API and the complexity of the training are almost prohibitive, making it a love-hate framework.
PyTorch framework tends to be applied to academic fast implementation, and its industrial application performance is not good. For example, Pytorch framework makes a model to run on a server, Android phone or embedded system, and its performance is poor compared with other deep learning frameworks.
PaddlePadddle is an open source framework of Baidu, which is also used by many people in China. It is very consistent with the usage habits of developers, but the practicability of the API still needs to be further strengthened. However, Paddle is the only deep learning framework I have ever used, which does not configure any third-party libraries and can be used directly by cloning make. Besides, Paddle occupies a small amount of memory and is fast. It also serves a considerable number of projects inside Baidu, which is very strong in industrial application. And PaddlePaddle supports multiple machine and multiple card training.
Users' choice of different deep learning frameworks is a personal choice, and the reason why most of us chose paddle is because of its better support for embedded development and different hardware platforms and because the community is very active and has proposed industrial improvements and implementations for some advanced models. Especially for the GPU, it supports cuda and opencl, which means we can optimize the model no matter what kind of graphics card is used. In my opinion, more backend support can better improve dnn libavfilter modules.
If there are any new changes in dnn libavfilter module, I will be very willing to adjust our implementation with the new planning and provide continuous maintenance.
Best Regards,
Wenzhe Wang
WenzheWang
wongwwz at foxmail.com
------------------ Original ------------------
From: "WenzheWang" <wongwwz at foxmail.com>;
Date: Tue, Apr 11, 2023 11:03 PM
To: "ffmpeg-devel"<ffmpeg-devel at ffmpeg.org>;
Subject: Re: [FFmpeg-devel] [PATCH v1] libavfi/dnn: add Paddle Inference as one of DNN backend
Could you please briefly introduce the reason why not adding any dnn backend?
Do you have any plan for the maintenance and development of the dnn backend in the future? From my understanding, the current backend of dnn has tensoflow, openvino and native, but this cannot meet the needs of users.
Thus, I believe adding other dnn backends will be great for user experience, user growth, and industrial applications. In particular, various dnn backend can be adapted to different application environments, and there are some emerging inference engines that are faster and stronger, such as Pytorch and Paddle. In addition, from the practical point of view, it is not difficult for a deep learning practitioner to learn and use this framework, but how to choose a framework and apply it in practice, people pay more attention to the effect (recall and precision), and easy deployment, that is, high reasoning performance efficiency. The main reason why Paddle is relatively mainstream and why I want to add paddle backend is that it has a very high efficiency and performance. There are several projects maintained by Paddle, such as paddleDetection, paddleSeg, paddleGAN, paddleOCR and paddleCls have a lot of good pre-training models that migrate well to their own data and has excellent performance. Secondly, in terms of reasoning efficiency, Paddle supports many platforms and chips. Models trained using Paddle framework can be directly deployed, and custom device interfaces are open for independent development based on one's own hardware.
FFmpeg itself already has very extensive support for codec. If FFmpeg could support the deployment of more reasoning model backend, it would have a wider application.
In general, I hope that ffmpeg could support the backend of paddle or more. In any case that my code is not mature or proper, I would be grateful if professionals like you could offer me suggestions and comments. I will be absolutely honored if I could contribute to this project :)
Best,
Wenzhe Wang
WenzheWang
wongwwz at foxmail.com
------------------ Original ------------------
From: "FFmpeg development discussions and patches" <jb at videolan.org>;
Date: Sun, Apr 9, 2023 05:31 AM
To: "ffmpeg-devel"<ffmpeg-devel at ffmpeg.org>;
Subject: Re: [FFmpeg-devel] [PATCH v1] libavfi/dnn: add Paddle Inference as one of DNN backend
On Thu, 6 Apr 2023, at 12:36, wongwwz at foxmail.com wrote:
> PaddlePaddle (PArallel Distributed Deep LEarning) is a simple,
> efficient and extensible deep learning framework that accelerates the
Please don't add another DNN backend.
--
Jean-Baptiste Kempf - President
+33 672 704 734
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel at ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
To unsubscribe, visit link above, or email
ffmpeg-devel-request at ffmpeg.org with subject "unsubscribe".
More information about the ffmpeg-devel
mailing list