[FFmpeg-devel] Number of files issue

[Wolfgang Meyerle]

Hi,

I'm writing this because I'm desperate. I'm trying to use ffmpeg in a 
completely different environment that it's probably intended or 
developed for.

I use it to compress slightly changing time series (arrays) in python.

The problem that I have at the moment with it is that opening a single 
h264 encoded mkv file seems to open dozens of file handles and the 
operating system can't handle this anymore.

I'd like to understand what's happening under the hood and how to avoid 
this.

I experimented already with ulimit -n in the linux environment but with 
more than 80k files open at the same time the operating system is at its 
boundaries.

Basically I want to open 132 files at once with 132 ffmpeg threads

For this I'm using the ffmpeg-python library but as I found out it's 
basically just forwarding command line parameters and calling ffmpeg via 
a command line.

Is there any parameter in ffmpeg available to reduce the number of open 
file handles it's opening for a single mkv file?

Is there any workaround available?


The code snippet I'm testing atm is this (which leads to 3751 files open 
at the same time after and 444 before I run the program in python:

import ffmpeg
import os
import time

suffix =".mkv" loglevel ="error" # folder = 
r"D:\testData\159540a71a46b29461d6484e58d7d0f7d97c9595bcb84ad16288986191b259a9" 
folder =r"/mnt/d/testData/159540a71a46b29461d6484e58d7d0f7d97c9595bcb84ad16288986191b259a9" 
name ="entity_encoder_feats" file = os.path.join(folder, '') + name + suffix

# Prepare FFMPEG Pipes ''' process = ( 
ffmpeg.input(filename=os.path.join(folder, '') + name + suffix, 
**{'threads:0': 1, 'loglevel': loglevel}) .output('pipe:', 
format='rawvideo', pix_fmt='rgb24') .run_async(pipe_stdout=True) # 
capture_stdout=True) ) ''' f =open(file, "rb")
content = f.read()

process = (
     ffmpeg.input('pipe:', **{'threads:0':1, 'loglevel': loglevel})
         .output('pipe:', format='rawvideo', pix_fmt='rgb24')
         .run_async(pipe_stdin=True, pipe_stdout=True)# capture_stdout=True) )

process.communicate(input=content)

while True:

     time.sleep(10)




Regards,


Wolfgang