[Ffmpeg-devel] [PATCH] flacenc - md5

[Justin Ruggles]

Michael Niedermayer wrote:
> Hi
> 
> On Sun, Jul 02, 2006 at 12:56:24PM -0400, Justin Ruggles wrote:
> 
>>Hi,
>>
>>This patch enables writing the MD5 checksum in the FLAC header.  It also
>>writes the total number of samples.  The solution is based on the way
>>that the WAV muxer seeks back & writes over portions of the header after
>>encoding is finished.
>>
>>I am not 100% happy with how I had to do the md5 calculation, but I
>>can't think of any other way...  The problem I ran into is that there is
>>no way for the encoder to know when it is encoding the last frame.
> 
> 
> set CODEC_CAP_DELAY and you should get a last zero size frame to indicate the
> end

oh yeah...forgot about that capability. that's one problem solved. :)

> 
> 
>>Looking for a smaller frame would do it most of the time, but not if the
>>sample count is an even multiple of the block size.  So the solution I
>>came up with was to write the MD5 value to the extradata when encoding
>>every frame, as if every frame was the last one.  To do this, I had to
>>copy the working MD5 struct to a temporary struct before running
>>av_md5_final().  Since 'struct AVMD5' is an incomplete type, the
>>solution looks a bit ugly, but it works.
> 
> 
> hmm, seems flac is almost beating ogg in the most idiotic design category
> * putting a checksum of the whole stream at the begin makes flac unuseable for
>   streaming, and anything realtime
> * doing the checksum over the unencoded data instead of the encoded makes
>   it impossible to check without decoding, impossible to add the checksum
>   at the muxer level where it belongs too and its slower too as the
>   unencoded data is larger then the encoded not to mention that without
>   specifiying the byteorder or channel interleaving of the input the
>   checksum is of course not well defined

There are also several other annoying design flaws in the FLAC specs.  I
think Josh Coalson is aware of most or all of them, but I have the
feeling compatibility is taking precedence over making improvements.  As
far as streaming...the format is designed to work even without the
global header.  The header just gives extra useful information.  Whether
decoders support this, I do not know, but each frame is indeed its own
independently decodable entity as long as the file conforms to the
Subset format (which reminds me, the encoder doesn't always do this
right now...I'll need to put in a warning. Subset compliance will be a
separate patch).

> my first idea for a solution is to simply set the md5 to 0, what will break?
> players cannot check the md5 so they cant notice it, its only tools
> which transcode the whole file which might complain about a md5 missmatch

This is what the encoder is currently doing.  It's actually within spec
to set the md5 and total sample count to 0 if they are unknown.
Decoders don't complain, they just can't verify the integrity of the
file or know the playing time.

> * set extradata at the end and have a special case in the raw muxer, what
>   about ogg, avi, nut, mov and all other general propose containers?

This isn't that much different from my other solution.  Once I take out
the constant updating of the extradata, it will only be set at the start
and end of encoding.  As far as other containers, there is a standard
mapping of flac in ogg which puts all the flac headers in ogg comment
packets, but even the reference encoder does not write the md5 checksum
when writing ogg-flac files.  Concerning other formats, I don't think
there are standard ways of encapsulting flac in anything except ogg or
matroska, and FFmpeg does not have a matroska encoder.  I did try adding
ogg-flac support to the ogg muxer, but I couldn't get it to work.  The
ogg muxer is very vorbis-specific right now, so it would require some
significant changes.

Actually though, the flac format is specifically designed to be
streamable. Like mentioned above, you can leave off the global header
and theoretically still have a 100% decodable stream.  I think this
would make it quite easy to put it in any container.  Most of what I
have seen in various sources uses fourcc="fLaC" or wFormatTag=0xF1AC.

On to the patch.  I decided to still update the md5 in the extradata.  I
think it makes sense.  extradata is still used only for the global
header, it's just set once at the start of encoding and again at the
end.  If the output is streamed, the header cannot be rewritten, but it
is still valid.

I also added writing of more complete header info with a vendor string
and padding.  The padding is there so that other stuff like tags and
cuesheets can be added by the user later without rewriting the entire
file.  Here is an example of the metaflac output.

METADATA block #0
  type: 0 (STREAMINFO)
  is last: false
  length: 34
  minumum blocksize: 4608 samples
  maximum blocksize: 4608 samples
  minimum framesize: 0 bytes
  maximum framesize: 19022 bytes
  sample_rate: 44100 Hz
  channels: 2
  bits-per-sample: 16
  total samples: 1323001
  MD5 signature: 12ad168f9e68257e84f5ab59f586e3d1
METADATA block #1
  type: 4 (VORBIS_COMMENT)
  is last: false
  length: 18
  vendor string: libavcodec
  comments: 0
METADATA block #2
  type: 1 (PADDING)
  is last: true
  length: 4096

I could make it so the user can set the padding size, but I really don't
think that's necessary.  An extra 4k of file size is not too bad, and if
someone needs to add more than 4k of metadata, well, it will just be slower.

I first thought of putting the lavc version in the vendor string like in
the vorbis encoder, but then realized this would break regression tests
at every version increment, so I just used "libavcodec" instead.  I've
also been debating about whether to set the vorbiscomment and padding in
the encoder or in the raw muxer.  Right now it is all in the encoder,
but it works either way.  Personally, I like it better in the muxer, but
I'm not sure if it belongs there from a design standpoint.  I have tried
it both ways, so I have a patch standing-by if it should be done in the
raw muxer instead.

all suggestions are welcome. :)

-Justin
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: flac-md5.diff
URL: <http://lists.mplayerhq.hu/pipermail/ffmpeg-devel/attachments/20060703/bba2315f/attachment.txt>