[FFmpeg-devel] [PATCH] conversion of FFV1 specification from lyx to markdown

Mon Jul 20 20:18:18 CEST 2015

Hi Michael,

> On Jul 20, 2015, at 7:52 AM, Michael Niedermayer <michael at niedermayer.cc> wrote:
> 
> On Sat, Jul 18, 2015 at 11:23:16AM -0400, Dave Rice wrote:
>> 
>>> On Jul 10, 2015, at 4:55 PM, Michael Niedermayer <michael at niedermayer.cc> wrote:
>>> 
>>> On Fri, Jul 10, 2015 at 04:19:41PM -0400, Dave Rice wrote:
>>>> 
>>>>> On Jul 10, 2015, at 3:51 PM, Michael Niedermayer <michael at niedermayer.cc> wrote:
>>>>> 
>>>>> On Fri, Jul 10, 2015 at 03:47:45PM -0400, Dave Rice wrote:
>>>>>> 
>>>>>>> On Jul 10, 2015, at 2:06 PM, Michael Niedermayer <michael at niedermayer.cc> wrote:
>>>>>>> 
>>>>>>> On Fri, Jul 10, 2015 at 07:51:20PM +0200, Michael Niedermayer wrote:
>>>>>>>> On Fri, Jul 10, 2015 at 07:47:46PM +0200, Michael Niedermayer wrote:
>>>>>>>>> On Fri, Jul 10, 2015 at 11:52:24AM +0200, Michael Niedermayer wrote:
>>>>>>>>>> On Thu, Jul 09, 2015 at 02:14:36PM -0400, Dave Rice wrote:
>>>>>>>>>>> 
>>>>>>>>>>>> On Jul 9, 2015, at 1:53 PM, Michael Niedermayer <michaelni at gmx.at> wrote:
>>>>>>>>>>>> 
>>>>>>>>>>>> On Tue, Jul 07, 2015 at 11:03:39AM -0400, Dave Rice wrote:
>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> On Jul 7, 2015, at 10:35 AM, Michael Niedermayer <michaelni at gmx.at> wrote:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> On Thu, Jul 02, 2015 at 09:50:21AM -0400, Dave Rice wrote:
>>>>>>>>>>>>>>> Hi all,
>>>>>>>>>>>>>>> Last month I posted a draft of the FFV1 specification formatted in Markdown [1], whereas it currently lives in lyx. From that post there were responses in favor of the use of Markdown formatting and suggestions to not use mathml within a draft targeting an RFC specification and suggestions to update a few urls. I propose to proceed with a format conversion of the FFV1 specification from lyx to Markdown and then I'll file tickets or patches to explore simplifying the mathml expressions and updating the urls. Attached is a patch to convert the formatting (but not meaning) of the FFV1 specification from lyx to markdown.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>> Dave Rice
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> [1] http://comments.gmane.org/gmane.comp.video.ffmpeg.devel/193851
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> how can this be edited and converted ?
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Via pandoc you can get outputs that are close to the current ffv1.html and ffv1.pdf outputs of ffv1.lyx.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> pandoc --mathml -s --number-sections ../ffv1.md -c lyx.css -o ffv1.html
>>>>>>>>>>>>> pandoc -s --number-sections ../ffv1.md --latex-engine=xelatex -V geometry:margin=1in -o ffv1.pdf
>>>>>>>>>>>>> 
>>>>>>>>>>>>> I can move this info to a readme.
>>>>>>>>>>>> 
>>>>>>>>>>>> please do
>>>>>>>>>>> 
>>>>>>>>>>> Sure. Would you prefer a patch on ffmpeg-devel or a PR in github?
>>>>>>>>>> 
>>>>>>>>>> whatever you prefer
>>>>>>>>> 
>>>>>>>>> generated files like pdf should not be checked into git.
>>>>>>>>> instead add a Makefile to generate them, so a simple
>>>>>>>>> make
>>>>>>>>> or
>>>>>>>>> make ffv1.pdf
>>>>>>>>> generates the file
>>>>>>>> 
>>>>>>>> also the readme should document which version of markdown/pandoc
>>>>>>>> this needs
>>>>>>>> and something should check for that version ideally so the user
>>>>>>>> is nt left wondering why "make" produces broken tables
>>>>>>> 
>>>>>>> reviewing the pull req
>>>>>>> 
>>>>>>> The conventions section 3.
>>>>>>> lacks vertical alignment
>>>>>> 
>>>>>> Fixed in https://github.com/MediaArea/FFV1/commit/d25fb39f7514547f72cbc13ef3851f3c9d9c0c5f
>>>>>> 
>>>>>>> theres an empty line after
>>>>>>> for(i=0;i<e;i++) 4.6.1.2
>>>>>> 
>>>>>> Fixed in https://github.com/MediaArea/FFV1/commit/1291e6c3e90ea6816b543a9e12c08d1cbbc1f31c
>>>>>> 
>>>>> 
>>>>>>> 4.6.2.5:
>>>>>>> log2_run[41]={JPEGLS.
>>>>>>> 0,0,0,0,1,1,1,1,
>>>>>>> 
>>>>>>> that JPEGLS is supposed to be a link/reference/whatever but its a
>>>>>>> litteral word
>>>>>> 
>>>>>> I'm having trouble getting a footnote to work within a codeblock. It is feasible to use this footnote in a reference to JPEGLS in a sentence above the codeblock?
>>>>> 
>>>>> of course
>>>>> whatever works
>>>> 
>>>> I don't understand the relationship between the function and the reference to JPEG-LS. I could simply move the footnote reference to JPEGLS to outside of the codeblock, but it would be better to also explain the relationship at the same time. Could you provide some background as to why JPEGLS is referenced here.
>>> 
>>> This table is also used by jpegls, see ff_log2_run in libavcodec
>>> 
>>> 
>>> some other things
>>> nested TOC and numbering in TOC is lost
>> 
>> Switching here to use pandoc's TOC features instead of maintaining them manually. https://github.com/MediaArea/FFV1/commit/64f643534c10ac6d680b63c75fa1b7cb670bd9c4
>> 
>>> in "4.6.1.4 State transition table" there should be a newline
>>> before one_statei = ...
>> 
>> Added here: https://github.com/MediaArea/FFV1/commit/701edaf67ad1f0933cac1fc74f4921a209461164
>> 
>>> 4.6.2 Huffman coding mode
>>> the content and headings seem to mismatch
>>> the prefix table is above the prefix heading
>>> the examples empty
>> 
>> Can you review again in git master. I've already made changes in this section according to Timothy Gu's comments.
>> 
> 
>>> the stuff inside
>>> ConfigurationRecord()
>>> lost 1 level indention
>> 
>> I haven't found a way to preserve white space within a table. As a workaround I moved it to a syntax-highlighted code block, which may be better anyhow. https://github.com/MediaArea/FFV1/commit/38c50dda4ed892fcd1a8701598058266fbcadcbb
> 
> this contains backslashes in the pdf output
> at least with pandoc 1.12.4.2

My mistake, escaping with backslashes is not necessary within a code block. Fixed here: https://github.com/MediaArea/FFV1/commit/1272d2094142ac5753d03eaba5d0078145e28ab0

>>> in
>>> QuantizationTable( i ) {
>>> 
>>> scale = 1 is indented diferently than the next line
>> 
>> I took the same above as ConfigurationRecord here: https://github.com/MediaArea/FFV1/commit/772919919703189e6743461e427ff190fa7555bd <https://github.com/MediaArea/FFV1/commit/772919919703189e6743461e427ff190fa7555bd>. Let me know if this approach works and I could change the other code-tables to code blocks.
>> 
>>> above review based on "pandoc 1.12.4.2" output, you might want to
>>> update the needed version numbers
>>> 
>>> "Propering PDF and HTML rendering has been tested with requires pandoc version 1.13.2.1 and higher."
>>> IMHO something like:
>>> PDF and HTML rendering has been tested with pandoc version ..., some older versions are known to produce very poor output, please ensure your pandoc is recent enough
>> 
>> Updated here: https://github.com/MediaArea/FFV1/commit/f85ce05a816d98bea0e15f1dd96afe23d78b1ef8 <https://github.com/MediaArea/FFV1/commit/f85ce05a816d98bea0e15f1dd96afe23d78b1ef8>.
>> 
>> [...]
>> 
>> At this point, I think I've respond or patched according to all notes from you and Timothy. Please review.
> 
> with old pandoc it fails totally before displaying the version message:
>    pandoc --toc --mathml -s --number-sections  -c "http://elyxer.nongnu.org/lyx.css" -o "ffv1.html" "ffv1.md"
>    pandoc --toc -s --number-sections --latex-engine=xelatex -V geometry:margin=1in -o "ffv1.pdf" "ffv1.md"
>    pandoc: Error producing PDF from TeX source.
>    ! Missing delimiter (. inserted).
>    <to be read again>
>                    \let
>    l.432 $context=Q_{0}[l-tl]+\left|
> 
>    make: *** [ffv1.pdf] Error 43
> 
> it should probably display something about the version so the user
> knows where to look for fixing the issue

Adjusted here: https://github.com/MediaArea/FFV1/commit/0036f2f04ce461cdc3dd5e309230808a44582641 <https://github.com/MediaArea/FFV1/commit/0036f2f04ce461cdc3dd5e309230808a44582641>. It will now print the warning before running pandoc.

> "Order of operation precedence" should mention that this is the
> "standard C" order so people who know that dont need to read the
> actual list and memorize it

Added a comment in. Preferably I'd like to cite to the correct specification that outlines standard C order of operations, but it seems that they're behind an ISO paywall. Do you know which spec it is?
Updated here https://github.com/MediaArea/FFV1/commit/a289aa6dfa3770f5e5374370b78c2941863d6b38

> The table in "Border 4.1" is missing a seperator betwen the top
> slice and the current slice (it does have seperators for the left
> and right ones

Added: https://github.com/MediaArea/FFV1/commit/07bf8ed09cfd2790573e06c47a549bb791151639

> The huffman coding mode section is just wrong
> the examples section is empty and the rest is shuffled around

Within the examples section of huffman coding mode (section 4.6.2.3) I see a table. Actually the whole Huffman Coding Mode section seems to have identical content to what is here http://www.ffmpeg.org/~michael/ffv1.html#toc-Subsubsection-3.6.2 <http://www.ffmpeg.org/~michael/ffv1.html#toc-Subsubsection-3.6.2>. I also checked http://www.ffmpeg.org/~michael/ffv1.html#toc-Subsubsection-3.6.2 <http://www.ffmpeg.org/~michael/ffv1.html#toc-Subsubsection-3.6.2> and things appear in the right order although the tables are not properly rendered.

> a seemingly random subset of entries in log2_run[] are cyan/green
> colored

Timothy Gu suggested using syntax highlighting in the codeblocks. It's easier turned off by changing ```c to ``` at the start of the code block. Remove it?

> The indention depth for ConfigurationRecord( ) is
> inconsistent and different indention levels are not a multiple of
> spaces of some constant

Fixed in https://github.com/MediaArea/FFV1/commit/0339ffaca67f46c4782f4b080c5035e67acf0961 <https://github.com/MediaArea/FFV1/commit/0339ffaca67f46c4782f4b080c5035e67acf0961>. Using 4 spaces as indentation.

> In 5.3 Slice
> the "if( i || version > 2 )" has the || interpreted somehow and
> missing in the pdf output

Good catch. Fixed in https://github.com/MediaArea/FFV1/commit/85b5197515e30429e4d311396f8dac886d8d3cc8 <https://github.com/MediaArea/FFV1/commit/85b5197515e30429e4d311396f8dac886d8d3cc8>.

> the colorspace_type table is split accross 2 pages in the pdf
> output, considering its just 4 rows, it would be nice to avoid
> that split if its possible easily

I've been looking into this. I may need to make a custom pandoc template to customize the rules more granularity. I'll do some investigation. Except for this all comments above are responded to and there is an open question about syntax highlighting in code blocks.

[...]

Dave Rice