[FFmpeg-devel] [PATCH V2 1/2] libavfilter/dnn: add script to convert TensorFlow model (.pb) to native model (.model)

[Guo, Yejun]

> -----Original Message-----
> From: Guo, Yejun
> Sent: Tuesday, June 04, 2019 3:10 PM
> To: ffmpeg-devel at ffmpeg.org
> Cc: Guo, Yejun <yejun.guo at intel.com>
> Subject: [PATCH V2 1/2] libavfilter/dnn: add script to convert TensorFlow model
> (.pb) to native model (.model)
> 
> For example, given TensorFlow model file espcn.pb,
> to generate native model file espcn.model, just run:
> python convert.py espcn.pb
> 
> In current implementation, the native model file is generated for
> specific dnn network with hard-code python scripts maintained out of ffmpeg.
> For example, srcnn network used by vf_sr is generated with
> https://github.com/HighVoltageRocknRoll/sr/blob/master/generate_header_a
> nd_model.py#L85
> 
> In this patch, the script is designed as a general solution which
> converts general TensorFlow model .pb file into .model file. The script
> now has some tricky to be compatible with current implemention, will
> be refined step by step.
> 
> The script is also added into ffmpeg source tree. It is expected there
> will be many more patches and community needs the ownership of it.
> 
> Another technical direction is to do the conversion in c/c++ code within
> ffmpeg source tree. While .pb file is organized with protocol buffers,
> it is not easy to do such work with tiny c/c++ code, see more discussion
> at http://ffmpeg.org/pipermail/ffmpeg-devel/2019-May/244496.html. So,
> choose the python script.
> 
> Signed-off-by: Guo, Yejun <yejun.guo at intel.com>
> ---
>  .gitignore                                        |   1 +
>  libavfilter/dnn/python/convert.py                 |  52 ++++++
>  libavfilter/dnn/python/convert_from_tensorflow.py | 201
> ++++++++++++++++++++++
>  3 files changed, 254 insertions(+)
>  create mode 100644 libavfilter/dnn/python/convert.py
>  create mode 100644 libavfilter/dnn/python/convert_from_tensorflow.py

ping for review, thanks.

Here is my rough plan after this patch.
- move dnn relative .h/.c from libavfilter to libavfilter/dnn, it is expected there will be more files for dnn module (code for both model loading and execution).
- add a layer for padding (tf.pad) for native mode and its fate test.
- change the script to add tf.pad support, and so the native model and the tf model of vf_sr will be the same.
 in current implementation, the two models have a little difference, it makes the script not a general solution to convert tf model to native model.
- add layer maximum and fate test. This layer appears in tf model, but not in native model, of vf_sr.
- introduce operand concept in native mode (both execution and model), to support data split and merge/concat in the network, such split/concat is very common.
 it also makes possible to reuse memory for the intermediate data as the output of the hidden layers.
- tune conv2d layer performance (it is very slow now) or add more layers for native mode.