[FFmpeg-devel] CI

[Timo Rothenpieler]

On 22/08/2025 07:00, Kieran Kunhya via ffmpeg-devel wrote:
> On Thu, 21 Aug 2025, 11:33 Michael Niedermayer via ffmpeg-devel, <
> ffmpeg-devel at ffmpeg.org> wrote:
> 
>> Hi
>>
>> On Thu, Aug 21, 2025 at 12:31:46AM +0200, Timo Rothenpieler via
>> ffmpeg-devel wrote:
>>> On 8/20/2025 9:25 PM, Michael Niedermayer via ffmpeg-devel wrote:
>>>> Hi
>>>>
>>>> On Wed, Aug 20, 2025 at 05:56:27PM +0200, Timo Rothenpieler via
>> ffmpeg-devel wrote:
>>>>> On 8/20/2025 1:26 AM, Michael Niedermayer via ffmpeg-devel wrote:
>>>>>> Hi
>>>>>>
>>>>>> It seems the forgejo CI takes about
>>>>>> 13min to do fate on aarch64 and x86-64 and build on win64
>>>>>>
>>>>>> Locally i run
>>>>>>        fate + install on x86-64
>>>>>>        build on x86-32, mingw64, arm32, mips, ppc, x86-64 + shared
>> libs
>>>>>>        testprogs alltools examples build on x86-64, x86-32 and arm32
>>>>>>        in 2min 44sec
>>>>>>
>>>>>> can we improve the speed vs amount of tests ratio ?
>>>>>> (its not a problem ATM, i did in fact not even notice as i never
>> waited on it)
>>>>>>
>>>>>> Iam just seeing the difference in time and i think there is
>> potential for
>>>>>> optimization here
>>>>>>
>>>>>> I dont think my box here is really special, just a
>>>>>> AMD Ryzen 9 3950X 16-Core + Samsung SSD 970 PRO
>>>>>
>>>>> Well, the test runners are 4 cores and 8GB of RAM. So that'll be the
>> primary
>>>>> difference in speed.
>>>>> I think they're performing pretty good for being just that.
>>>>>
>>>>> We could of course throw money at the problem and turn them into 16
>> core
>>>>> machines. That would up the hosting cost of the runners from
>> currently
>>>>> 3*7.5€ a month to 3*30€ a month. Just for the runners.
>>>>>
>>>>> imo the current CI turnaround times are fine. 15-20 minutes per job
>> is fine,
>>>>> as long as they can all run in parallel.
>>>>
>>>> Option 1: 15-20 min CI turnaround,  270 € per year
>>>> Option 2:  4-5? min CI turnaround, 1080 € per year
>>>>
>>>> we have over 150k $ it seems
>>>>
>>>> Good use of capital can also lead to more donations
>>>>
>>>> I think the main question is, "would we benefit from the faster
>> trunaround"?
>>>> or not ?
>>>
>>> You have to keep in mind, 4 Core 8GB is also the swarm of runners we get
>> for
>>> free from Microsoft via GitHub.
>>>
>>> So the choice is actually "Be able to process 20+ jobs in parallel that
>> take
>>> 15-20 minutes each" vs. "Be able to process 3 or so at a time (roughly
>> one
>>> PR/push) in 5 minutes".
>>> So realistically, unless we also pay for an actual swarm of runners
>>> ourselves(which would cost 10k or more a year while being idle 95% of the
>>> time) the total turnaround time including wait for a free runner is
>> probably
>>> still better with more of the smaller runners than less of the big ones.
>>>
>>> It'd also make it a lot more pressing to think about every single CI job
>> we
>>> add, vs. having a bit of leeway due to the over-abundance of runners.
>>
>> for 1-2k$ you can buy a box that runs fate once and build on 6 times in
>> under 3minutes.
>>
>> if one is not enough buy 3, use the extra capcity for fuzzing or rent out
>> to other projects
>>
>> I must be stupid, because to me this looks cheaper, its also one time
>> expense
>> these boxes can be used for 10 years
>>
>> also no need to be reliable expensive servers, if you have 3.
>>
>> thx
>>
> 
> In the end you're paying for hosting, 24/7 electricity and not having to
> worry about it.

Yeah, in the end someone has to put these 3 servers somewhere, monitor 
them for hardware failures, and pay for their electricity.
Which isn't exactly cheap for 3 servers either.

> I agree with both viewpoints. For "mission critical" stuff like CI we
> should host at a proper hosting company.

There's also the option of renting a dedicated server from Hetzner or 
some other company, and put a bunch of runners on there.
Hetzner rents you an 8 core Zen4 Ryzen with 64GB of RAM for ~55€ a month.

But compared to the current virtual servers we rent from Hetzner, that's 
a pretty poor "performance per money" ratio, given we get one server 
with 4 cores and 8GB of RAM, which is plenty for our needs, for 7.5€ a 
month.
So we can rent up to 7 of those, while still paying less than for one 
dedicated server.
And dedicated servers are again more effort, because you do need to 
manually monitor them for any hardware faults, and then coordinate the 
replacement with Hetzner, and they do eventually die of old age.

There's also the option of writing some tool that automatically scales 
runners up/down at some cloud provider that bills minutely/secondly 
(Hetzner bills Servers per Hour, making this a lot less interesting).
That would require some serious cost calculation. But I feel like we 
have plenty of CI downtime to make it worthwhile.

But then again, GitHub/MS gives us 20 parallel runners for free, and we 
can freely pick if they're running on x86_64 or aarch64, Linux, Windows 
or even OSX.
As long as they do that, we only need to host a baseline of runners 
ourselves, and can scale out into that whenever there's a peak in usage.