[FFmpeg-user] "Leftover" ffmpeg instances eating RAM. Solutions?

[Steven Kan]

Hi all,

I have 3 LaunchDaemons running on macOS nearly 24/7, each of which calls a script to handle instances of ffmpeg (binaries from https://evermeet.cx/ffmpeg) to pull 1 or 2 RSTP streams from IP cameras, optionally hstack them, and then push them to 3 different YouTube channels:
https://www.youtube.com/channel/UCE0jx2Z6qbc5Co8x8Kyisag/live
https://www.youtube.com/channel/UCIVY11504PcY2sy2qpRhiMg/live
https://www.youtube.com/channel/UCcIZVSZfzrxS6ynL8DQeC-Q/live
Each script runs ffmpeg in a loop and then repeats, due to the fact that some cameras have no audio, but YouTube streaming requires it, so I have a playlist of mp3 files that runs for 1:47:02. Scripts are of the general form StreamToYouTube.sh:
#!/bin/bash
cd /usr/local/bin/
while true
do
./ffmpeg -thread_queue_size 2048 -i "rtsp://anonymous:password@192.168.1.11:554" -f concat -safe 0 -i playlist.txt -vcodec copy -acodec copy -t 01:47:02 -f flv "rtmp://a.rtmp.youtube.com/live2/<my-youtube-streaming-key>"
echo Stream1Complete
sleep 5
done
The LaunchDaemon restarts a script if it dies for some reason, which it sometimes does for reasons I’m still searching for. 
I also pause the scripts for a few minutes at 07:00 and 19:00 so that YouTube can create an archive of the footage and start a new video instance. 
If I don’t do that, and a video gets longer than 12 hours, it will often get stuck in YouTube’s “processing purgatory” and never become viewable.
Finally, I have a watchdog timer that periodically checks if each stream is Live, and if it doesn’t it runs through a procedure to reset the YouTube stream and start the ffmpeg script again.
Most of this is working reliably, now, but I have the following problem: when I check the machine running all of this, I sometimes see 6 or more instances of ffmpeg (and their associated VTDecoderXPC processes) instead of the 3 that I should see.
Only the 3 desired instances are using any appreciable CPU time, but they all consume significant RAM.
As a result, sometimes this machine locks up or things start shutting down due to RAM starvation
One possible solution I thought of is to copy the ffmpeg executable to ffmpeg1, ffmpeg2, ffmpeg3, etc., and using a different copy for each of my streaming services/scripts/daemons
This would allow to me to “killall ffmpegX” during my daily pause/cleanup routines
I don’t want to just “killall ffmpeg” because sometimes I want to reset only 1 of the 3 channels that are running
Would a copy of ffmpeg as ffmpeg1 or ffmpeg2 work properly? Or do I have to build it somehow?
Or is there another way to identify which instance(s) of ffmpeg were created by each service, so that I can kill them by PID?
Or am I doing something wrong with my services that’s causing them to leave “ghost” ffmpeg instances behind?
Services are initiated via “sudo launchctl load StreamToYouTubeX.plist”
Services are terminated via “sudo launchctl unload StreamToYouTubeX.plist”
The plists contain just a simple call to, for example:
    <key>ProgramArguments</key>
    <array>             
        <string>/usr/local/bin/StreamToYouTube2.sh</string>
    </array>

Thanks!

Console dump follows:

ffmpeg version N-109676-g40512dbd96-tessus  https://evermeet.cx/ffmpeg/  Copyright (c) 2000-2023 the FFmpeg developers
  built with Apple clang version 11.0.0 (clang-1100.0.33.17)
  configuration: --cc=/usr/bin/clang --prefix=/opt/ffmpeg --extra-version=tessus --enable-avisynth --enable-fontconfig --enable-gpl --enable-libaom --enable-libass --enable-libbluray --enable-libdav1d --enable-libfreetype --enable-libgsm --enable-libmodplug --enable-libmp3lame --enable-libmysofa --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenh264 --enable-libopenjpeg --enable-libopus --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvmaf --enable-libvo-amrwbenc --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxavs --enable-libxvid --enable-libzimg --enable-libzmq --enable-libzvbi --enable-version3 --pkg-config-flags=--static --disable-ffplay
  libavutil      57. 44.100 / 57. 44.100
  libavcodec     59. 57.100 / 59. 57.100
  libavformat    59. 36.100 / 59. 36.100
  libavdevice    59.  8.101 / 59.  8.101
  libavfilter     8. 54.100 /  8. 54.100
  libswscale      6.  8.112 /  6.  8.112
  libswresample   4.  9.100 /  4.  9.100
  libpostproc    56.  7.100 / 56.  7.100
Input #0, rtsp, from 'rtsp://anonymous:password@192.168.1.11:554':
  Metadata:
    title           : Session streamed by "preview"
    comment         : 
  Duration: N/A, start: 0.000000, bitrate: N/A
  Stream #0:0: Video: h264 (High), yuv420p(progressive), 2304x1296, 120 tbr, 90k tbn
  Stream #0:1: Audio: aac (LC), 16000 Hz, mono, fltp
[mp3 @ 0x7fea1f007140] Estimating duration from bitrate, this may be inaccurate
Input #1, concat, from 'playlist.txt':
  Duration: N/A, start: 0.000000, bitrate: 320 kb/s
  Stream #1:0: Audio: mp3, 44100 Hz, stereo, fltp, 320 kb/s
Output #0, flv, to 'rtmp://a.rtmp.youtube.com/live2/<my-youtube-streaming-key>':
  Metadata:
    title           : Session streamed by "preview"
    comment         : 
    encoder         : Lavf59.36.100
  Stream #0:0: Video: h264 (High) ([7][0][0][0] / 0x0007), yuv420p(progressive), 2304x1296, q=2-31, 120 tbr, 1k tbn
  Stream #0:1: Audio: mp3 ([2][0][0][0] / 0x0002), 44100 Hz, stereo, fltp, 320 kb/s
Stream mapping:
  Stream #0:0 -> #0:0 (copy)
  Stream #1:0 -> #0:1 (copy)
Press [q] to stop, [?] for help
frame=    0 fps=0.0 q=-1.0 size=       0kB time=-577014:32:22.77 bitrate=  -0.0kbits/s speed=N/A    
[flv @ 0x7fe9ed809e40] Timestamps are unset in a packet for stream 0. This is deprecated and will stop working in the future. Fix your code to set the timestamps properly
frame=   53 fps=0.0 q=-1.0 size=    1175kB time=00:00:01.77 bitrate=5419.1kbits/s speed=3.21x    
frame=   71 fps= 64 q=-1.0 size=    1677kB time=00:00:02.37 bitrate=5778.6kbits/s speed=2.15x    
<etc, where speed eventually settles at ~1.00 after a few dozen frames>