[FFmpeg-devel] [RFC] fate rsync switch to git

[Vittorio Giovara]

On Wed, Feb 21, 2024 at 3:50 PM <epirat07 at gmail.com> wrote:

>
>
> On 21 Feb 2024, at 15:38, Niklas Haas wrote:
>
> > On Tue, 20 Feb 2024 20:43:30 +0200 Jan Ekström <jeebjp at gmail.com> wrote:
> >> Do note that the idea was that this would only be for management of
> >> the main archive, so it would not affect clients/runners rsync'ing
> >> from the main archive.
> >>
> >> Of course clients which want to sync directly from git could do that,
> >> but the idea would be to keep the sync requirements same for FATE
> >> clients/runners: if you are only running tests, rsync is enough.
> >>
> >> As after all, the primary reasons for having the samples in git would
> >> be versioning, more concrete known states in a public archive (I would
> >> probably not call this a "backup", but it would mean we would have the
> >> history in multiple places at least), as well as - if we utilize
> >> something like git{lab,hub} - easier workflow to adding new samples by
> >> means of f.ex. merge/pull requests.
> >>
> >> This idea originated from looking at how the dav1d project handled
> >> their reference sample suite, which seems to have served them well
> >> enough: https://code.videolan.org/videolan/dav1d-test-data
> >>
> >> Regards,
> >> Jan
> >
> > Is there any reason (besides efficiency hit) not to make the FATE repo
> > a `git submodule` of the FFmpeg git repo? That way, commits which depend
> > on certain additions to fate-samples can explicitly depend on the
> > commits adding those files, developers can more easily see (e.g. via
> > `git status`) if the fate samples are out-of-date (or use `git pull
> > --recurse-submodules` to automate the process).
>
> I am all for having it in git but do not like the idea of a git submodule
> at all as they are a nightmare to work with, sometimes create absolutely
> unworkable conflicts when rebasing and other oddities…
>
> (We use submodules for the Icecast project, it was my idea back then and
> I regret it…)
>
> >
> > It will also make the samples repo historically consistent, e.g. if
> > somebody changes a detail about a file in a later commit, older commits
> > referencing the unmodified version will continue passing FATE tests. I'm
> > not sure if this has ever been a concern in the past, but it may well
> > be one in the future.
> >
> > Worrying about the performance impact of rsync vs git-lfs (or equivalent
> > solutions) seems like premature optimization to me; and the ease of
> > maintenance, historical consistency, transparency in process, and
> > end-user convenience of a git repository seems to far outweigh the
> > drawbacks.


I don't mind submodules but I understand the complications. Maybe we could
design a system in which samples have attached a hash/branch/ref so that
when checking out older/outdated samples the correct sample itself can
still be used.

But overall I agree it would be much preferable to have the management
system in git over rsync.
-- 
Vittorio