[FFmpeg-devel] native mode in FFmpeg DNN module

[Guo, Yejun]

> -----Original Message-----
> From: ffmpeg-devel [mailto:ffmpeg-devel-bounces at ffmpeg.org] On Behalf Of
> Steven Liu
> Sent: Friday, May 24, 2019 11:35 PM
> To: FFmpeg development discussions and patches <ffmpeg-devel at ffmpeg.org>
> Cc: Steven Liu <lq at chinaffmpeg.org>
> Subject: Re: [FFmpeg-devel] native mode in FFmpeg DNN module
> 
> 
> 
> > 在 2019年5月24日，20:34，Pedro Arthur <bygrandao at gmail.com> 写
> 道：
> >
> > Em qui, 23 de mai de 2019 às 00:06, Guo, Yejun <yejun.guo at intel.com>
> escreveu:
> >>
> >>
> >>
> >>>>>>>> Option 2)
> >>>>>>>> Write c code in FFmpeg to convert tensorflow file format (format 1)
> >>>> directly
> >>>>>>> into memory representation (format 3), and so we controls everything
> in
> >>>>>>> ffmpeg community. And the conversion can be extended to import
> more
> >>>> file
> >>>>>>> formats such as torch, darknet, etc. One example is that OpenCV uses
> >>>> this
> >>>>>>> method.
> >>>>>>>>
> >>>>>>>> The in memory representation (format 3) can still be current.
> >>>>>>>>
> >>>>>>>
> >>>>>>> Option 2 would be ideal, as it does not introduce any dependency for
> >>>>>>> using the native backend.
> >>>>>>> Yet I'm not sure  how complex implementing the tf model reader can
> >>> be,
> >>>>>>> If I remember correctly the student said it was not trivial at the
> >>>>>>> time.
> >>>>>>
> >>>>>> yes, it is not easy, but I think it is worthy to do it. Here is a reference
> >>>> example
> >>>>>> for the complexity, see
> >>>>>>
> >>>>
> >>>
> https://github.com/opencv/opencv/blob/master/modules/dnn/src/tensorflow/
> >>>>>> tf_importer.cpp.
> >>>>>>
> >>>>>>>
> >>>>>>> Is the tf model file stable? if not it will be a maintenance burden to
> >>>>>>> keep it working whenever tf releases a new version. This point makes
> >>>>>>> me think having control over our file format is good.
> >>>>>>
> >>>>>> imho, this issue is always there, no matter which method used, unless
> our
> >>>>>> format could be exported by tensorflow (it has little possibility).
> >>>>>>
> >>>>>> Whenever tf releases a new version with a new file format, we still
> have
> >>> to
> >>>>>> change the python script in phase 1 (convert tf file model to our format)
> >>>> which
> >>>>>> is even an external dependency at
> >>>>>> https://github.com/HighVoltageRocknRoll/sr,
> >>>>>>
> >>>>>> As from effort perspective, the current implementation is better since
> >>>> python
> >>>>>> script is simpler. But I think we are still worth implementing option 2 as
> >>> the
> >>>>>> ideal technical direction.
> >>>>>
> >>>>> I checked a bit more about https://github.com/HighVoltageRocknRoll/sr,
> it
> >>> is
> >>>> actually
> >>>>> not an converter (from tf model to native model), but hard code for given
> >>>> models.
> >>>>> And the native model is not exactly the same as tf model, it even
> changes
> >>> the
> >>>> behavior
> >>>>> of pad parameter of conv layer.
> >>>>>
> >>>>> If community is open to option 2, I'll try it.
> >>>>>
> >>>> Option 2 is fine for me.
> >>>
> >>> that's great, :)
> >>
> >> looks that option 2 is a bit complex, TF model file is in protocol buffers
> (protobuf) format and not easy to parse it with simple c code.
> >>
> >> Since there is no official c support for protobuf, let's first image how the
> work can be done via official c++ support.
> >>
> >> 1. get protobuf compiler protoc, .h header files and .so library files
> (download or build from
> https://github.com/protocolbuffers/protobuf/tree/master/src).
> >> 2. get tensorflow model's .proto files from
> https://github.com/tensorflow/tensorflow/tree/master/tensorflow/core/fram
> ework.
> >> 3. generate .cc/.h files from .proto files (see step 2) via protoc (see step 1).
> >> 4. let the generated .cc/.h files be part of ffmpeg source tree, and build with
> protobuf header/library files.
> >> 5. at run time, the protobuf libraries are invoked. It means that the system
> should have installed protobuf dev package.
> >>
> >> furthermore, there is a compatible problem between the protobuf compiler,
> header files and library files.
> >> So, as a practice to fix it, the method is to make the protobuf source code be
> part of ffmpeg source tree. (it is a common practice, so we can many other
> projects contain the protobuf source code).
> >>
> >> I guess the above method is not acceptable in ffmpeg. I would be glad to
> continue if the community embrace this change. :)
> > Indeed I think it is not acceptable.
> >
> >>
> >> While the current implementation has external dependency, my new
> suggestion is:
> >> -  add a python script under .../libavfilter/dnn/  (all other dnn source files
> will be also moved here later), so ffmpeg has the full control on it.
> > I'm not sure about the policy on putting secondary scripts with the
> > main code, but another option is to create a repo controlled by ffmpeg
> > maybe?
> > I think this option would also help GSoC students that work with dnn,
> > so they don't have to depend on previous students maintaining
> > independent repositories.
> 
> Yes, I agreed with you.
> I think this is a better way.
> maintaining the repositories at one repo controlled by ffmpeg.

ok, let me first finish the python script (general converter) and send out the patch, and the community
at that time would have more people involved to talk about it.

> >
> >> -  it is a script to convert tensorflow model file into native model file. (other
> formats such as caffe, torch can also be supported later if needed)
> >>
> >> thanks.
> >>
> >> _______________________________________________
> >> ffmpeg-devel mailing list
> >> ffmpeg-devel at ffmpeg.org
> >> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> >>
> >> To unsubscribe, visit link above, or email
> >> ffmpeg-devel-request at ffmpeg.org with subject "unsubscribe".
> > _______________________________________________
> > ffmpeg-devel mailing list
> > ffmpeg-devel at ffmpeg.org
> > https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> >
> > To unsubscribe, visit link above, or email
> > ffmpeg-devel-request at ffmpeg.org with subject "unsubscribe".
> 
> Thanks
> Steven
> 
> 
> 
> 
> 
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel at ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> 
> To unsubscribe, visit link above, or email
> ffmpeg-devel-request at ffmpeg.org with subject "unsubscribe".