[FFmpeg-user] Compiling FFmpeg for nvidia hardware encoding on Debian 12

[Lowell Friesen]

Hi there,

I’m trying to compile FFmpeg on Debian 12 to enable hardware encoding 
with an Nvidia graphics card, but am having some issues. I have 
installed the nvidia driver and cuda toolkit. This is the output of 
|nvidia-smi|:

|+-----------------------------------------------------------------------------+ 
| NVIDIA-SMI 525.125.06 Driver Version: 525.125.06 CUDA Version: 12.0 | 
|-------------------------------+----------------------+----------------------+ 
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan 
Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | | | | MIG 
M. | 
|===============================+======================+======================| 
| 0 NVIDIA GeForce ... On | 00000000:01:00.0 On | N/A | | 0% 45C P2 46W 
/ 163W | 1052MiB / 4096MiB | 2% Default | | | | N/A | 
+-------------------------------+----------------------+----------------------+ 
+-----------------------------------------------------------------------------+ 
| Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID 
Usage | 
|=============================================================================| 
| 0 N/A N/A 1697 G /usr/lib/xorg/Xorg 275MiB | | 0 N/A N/A 1862 G 
/usr/bin/gnome-shell 55MiB | | 0 N/A N/A 2086 G 
...ud-3.10.1-x86_64.AppImage 2MiB | | 0 N/A N/A 3096 G 
/usr/lib/firefox/firefox 448MiB | | 0 N/A N/A 3905 G 
...b/thunderbird/thunderbird 183MiB | | 0 N/A N/A 5558 G 
/usr/bin/nautilus 41MiB | | 0 N/A N/A 6235 G 
...936111296816443610,262144 12MiB | | 0 N/A N/A 117598 G 
...RendererForSitePerProcess 15MiB | 
+-----------------------------------------------------------------------------+ 
|

Consulting the FFmpeg’s compilation guide 
<https://trac.ffmpeg.org/wiki/CompilationGuide/Ubuntu> I am given a list 
of dependencies to install. Two of them (|libsdl2-dev| and 
|libvdpau-dev|) give me the following error when I attempt to install them:

|The following packages have unmet dependencies: libx11-dev : Depends: 
libx11-6 (= 2:1.8.4-2+deb12u1) but 2:1.8.7-1 is to be installed |

I would install the older version of |libx11-dev| except that |sudo apt 
install libx11-6=2:1.8.4-2+deb12u1| gives me such a long list of 
packages that will be removed that I have the strong suspicion I would 
be left with an unusable system if I went ahead with it.

|Reading package lists... Done Building dependency tree... Done Reading 
state information... Done The following packages were automatically 
installed and are no longer required: apg avahi-utils bogofilter 
bogofilter-bdb bogofilter-common brasero-common ca-certificates-java 
cdrdao cheese-common chromium-sandbox coinor-libcbc3 coinor-libcgl1 
coinor-libclp1 coinor-libcoinmp1v5 coinor-libcoinutils3v5 
coinor-libosi1v5 default-jre-headless docbook-xml espeak-ng-data 
evolution-common evolution-data-server-common firebird3.0-common 
firebird3.0-common-doc firebird3.0-server-core firebird3.0-utils 
folks-common fonts-dejavu fonts-dejavu-extra fonts-font-awesome 
gimp-data gir1.2-accountsservice-1.0 gir1.2-atspi-2.0 gir1.2-evince-3.0 
gir1.2-gck-1 gir1.2-gcr-3 gir1.2-gdm-1.0 gir1.2-geocodeglib-2.0 
gir1.2-gmenu-3.0 gir1.2-gnomebluetooth-3.0 gir1.2-gnomedesktop-3.0 
gir1.2-grilo-0.3 gir1.2-gstreamer-1.0 gir1.2-gtksource-4 gir1.2-ibus-1.0 
gir1.2-javascriptcoregtk-4.0 gir1.2-javascriptcoregtk-4.1 
gir1.2-json-1.0 gir1.2-malcontent-0 gir1.2-mediaart-2.0 gir1.2-nm-1.0 
gir1.2-nma-1.0 gir1.2-polkit-1.0 gir1.2-rest-1.0 gir1.2-rsvg-2.0 
gir1.2-shumate-1.0 gir1.2-soup-2.4 gir1.2-soup-3.0 gir1.2-totem-1.0 
gir1.2-totemplparser-1.0 gir1.2-tracker-3.0 gir1.2-upowerglib-1.0 
gnome-control-center-data gnome-session-common 
gnome-settings-daemon-common gnome-shell-common graphviz 
grilo-plugins-0.3 gsfonts gstreamer1.0-x i965-va-driver ibus ibus-data 
ibus-gtk ibus-gtk3 ibus-gtk4 ibverbs-providers im-config 
imagemagick-6-common intel-media-va-driver java-common libaa1 
libabw-0.1-1 libamd2 libann0 libasyncns0 libavc1394-0 libbabl-0.1-0 
libblas3 libbotan-2-19 libbox2d2 libbrasero-media3-1 libbrlapi0.8 
libbs2b0 libburn4 libcaca0 libcamd2 libcamel-1.2-64 libccolamd2 
libcdr-0.1-1 libcdt5 libcgraph6 libcholmod3 libcjson1 
libclutter-1.0-common libcmark0.30.2 libcodec2-1.0 libcogl-common 
libcolamd2 libcolord-gtk4-1 libdbus-glib-1-2 libdc1394-25 libdca0 
libdecor-0-0 libdecor-0-plugin-1-cairo libdee-1.0-4 libdmapsharing-3.0-2 
libdotconf0 libdouble-conversion3 libdrm-amdgpu1 libdrm-intel1 
libdrm-nouveau2 libdrm-radeon1 libdv4 libdvdnav4 libe-book-0.1-1 
libebackend-1.2-11 libebook-1.2-21 libebook-contacts-1.2-4 libecal-2.0-2 
libedata-book-1.2-27 libedata-cal-2.0-2 libedataserver-1.2-27 
libel-api-java libenca0 libepubgen-0.1-1 libetonyek-0.1-1 libevdev2 
libfaad2 libfbclient2 libfftw3-double3 libflatpak0 libflite1 libfolks26 
libfreeaptx0 libfreehand-0.1-1 libgail-3-0 libgail-common libgail18 
libgbm1 libgdm1 libgegl-common libgfortran5 libgl1-mesa-dri 
libglapi-mesa libgles-nvidia1 libgles-nvidia2 libgles1 libgles2 
libglu1-mesa libglvnd0 libgme0 libgnome-autoar-gtk-0-0 
libgnome-bluetooth-ui-3.0-13 libgnome-menu-3-0 libgnome-rr-4-2 
libgom-1.0-0 libgpod-common libgpod4 libgrilo-0.3-0 libgsl27 
libgslcblas0 libgsm1 libgssdp-1.6-0 libgtk2.0-0 libgtk2.0-bin 
libgtk2.0-common libgtksourceview-4-0 libgtksourceview-4-common 
libgts-0.7-5 libgts-bin libgupnp-1.6-0 libgupnp-av-1.0-3 
libgupnp-dlna-2.0-4 libgupnp-igd-1.0-4 libgvc6 libgvpr2 
libhsqldb1.8.0-java libhwy1 libib-util libibumad3 libibus-1.0-5 
libibverbs1 libiec61883-0 libigdgmm12 libimath-3-1-29 libinput-bin 
libinput10 libinstpatch-1.0-2 libisofs6 libjack-jackd2-0 
libjavascriptcoregtk-4.1-0 libjavascriptcoregtk-6.0-1 libjsp-api-java 
libjte2 libjxl0.7 libjxr-tools libjxr0 libkate1 liblab-gamut1 liblapack3 
liblc3-0 libldacbt-abr2 libldacbt-enc2 liblibreoffice-java liblilv-0-0 
liblirc-client0 libllvm15 liblqr-1-0 liblrdf0 libltc11 libmad0 
libmagickcore-6.q16-6 libmagickcore-6.q16-6-extra libmagickwand-6.q16-6 
libmalcontent-ui-1-1 libmanette-0.2-0 libmbedcrypto7 libmd4c0 
libmediaart-2.0-0 libmetis5 libmfx1 libminizip1 libmjpegutils-2.1-0 
libmng1 libmodplug1 libmpcdec6 libmpeg2encpp-2.1-0 libmpg123-0 
libmplex2-2.1-0 libmspub-0.1-1 libmtdev1 libmwaw-0.3-3 libmypaint-1.5-1 
libmypaint-common libmysofa1 libneon27 libnice10 libnma-gtk4-0 libnorm1 
libnss-myhostname libnvidia-allocator1 libnvidia-egl-gbm1 
libnvidia-egl-wayland1 libnvidia-eglcore libnvidia-encode1 
libnvidia-glcore libnvidia-glvkspirv libnvidia-rtcore liboauth0 
libodfgen-0.1-1 libopenal-data libopenal1 libopenexr-3-1-30 libopengl0 
libopenh264-7 libopenmpt0 libopenni2-0 libostree-1-1 libotr5 
libpagemaker-0.0-0 libpathplan4 libperl4-corelibs-perl libpfm4 
libpgm-5.3-0 libphonenumber8 libplacebo208 libportal-gtk3-1 
libprotobuf32 libpst4 libqt5concurrent5 libqt5core5a libqt5dbus5 
libqt5network5 libqt5qml5 libqt5qmlmodels5 libqt5xml5 libqxp-0.0-0 
librabbitmq4 libraw1394-11 libraw20 librdmacm1 libreoffice-base-core 
libreoffice-base-drivers libreoffice-java-common 
libreoffice-sdbc-firebird librest-1.0-0 librhythmbox-core10 librist4 
librnp0 librubberband2 librygel-core-2.8-0 librygel-db-2.8-0 
librygel-renderer-2.8-0 librygel-renderer-gst-2.8-0 
librygel-server-2.8-0 libsbc1 libserd-0-0 libservlet-api-java 
libservlet3.1-java libset-scalar-perl libsgutils2-1.46-2 libshine3 
libshout3 libshumate-1.0-1 libshumate-common libsnapd-glib-2-1 
libsnappy1v5 libsndfile1 libsndio7.0 libsodium23 libsonic0 libsord-0-0 
libsoundtouch1 libsoup-gnome2.4-1 libsoxr0 libspa-0.2-bluetooth 
libspandsp2 libspeechd2 libsratom-0-0 libsrt1.5-gnutls libsrtp2-1 
libssh-4 libssh-gcrypt-4 libstaroffice-0.0-0 libsuitesparseconfig5 
libtag1v5 libtag1v5-vanilla libtommath1 libtotem0 libtspi1 
libtss2-tctildr0 libtwolame0 libu2f-udev libucx0 libumfpack5 
libunity-protocol-private0 libunity-scopes-json-def-desktop libunity9 
libunoloader-java libv4l-0 libv4lconvert0 libva-drm2 libva-wayland2 
libva2 libvdpau1 libvidstab1.1 libvisio-0.1-1 libvo-aacenc0 
libvo-amrwbenc0 libvorbisidec1 libvpx8 libvulkan1 libwacom-common 
libwacom9 libwavpack1 libwayland-bin libwayland-dev libwayland-server0 
libwebsocket-api-java libwildmidi2 libwinpr2-2 libwireplumber-0.4-0 
libwmf-0.2-7 libwmflite-0.2-7 libwnck-3-common libwpd-0.10-10 
libwpe-1.0-1 libwpebackend-fdo-1.0-1 libwpg-0.3-3 libwps-0.4-4 
libxatracker2 libxcb-damage0 libxcb-dri2-0 libxcb-dri3-0 libxcb-glx0 
libxcb-icccm4 libxcb-image0 libxcb-keysyms1 libxcb-present0 
libxcb-randr0 libxcb-render-util0 libxcb-res0 libxcb-sync1 libxcb-util1 
libxcb-xinerama0 libxcb-xinput0 libxcb-xkb1 libxcb-xv0 libxcvt0 
libxfont2 libxkbcommon-x11-0 libxres1 libxshmfence1 libxss1 libxv1 
libxvidcore4 libxvmc1 libxxf86dga1 libytnef0 libz3-4 libzbar0 libzimg2 
libzmf-0.0-0 libzmq5 libzvbi-common libzvbi0 libzxcvbn0 libzxing2 
lp-solve malcontent malcontent-gui media-player-info mutter-common 
nsight-compute-target nsight-systems-target nvidia-driver-bin 
nvidia-egl-common nvidia-settings nvidia-vdpau-driver 
nvidia-vulkan-common openjdk-17-jre-headless openjdk-8-jre-headless 
pocketsphinx-en-us power-profiles-daemon python3-brlapi python3-gi-cairo 
python3-ibus-1.0 python3-louis python3-mako python3-markupsafe 
python3-pyatspi python3-speechd python3-xdg qttranslations5-l10n realmd 
rhythmbox-data rtkit sgml-data shotwell-common sound-icons 
switcheroo-control system-config-printer timgm6mb-soundfont totem-common 
ure-java x11-session-utils xbitmaps xbrlapi xcvt xdg-dbus-proxy 
xfonts-100dpi xfonts-75dpi xfonts-base xfonts-scalable xinit xkbset 
xserver-common xserver-xorg-legacy yelp-xsl zenity-common Use 'sudo apt 
autoremove' to remove them. The following additional packages will be 
installed: libreoffice-core-nogui openjdk-8-jre-headless 
policykit-1-gnome Suggested packages: fonts-nanum fonts-ipafont-gothic 
fonts-ipafont-mincho fonts-wqy-microhei fonts-wqy-zenhei fonts-indic The 
following packages will be REMOVED: cheese chrome-gnome-shell chromium 
chromium-common default-jre evolution evolution-data-server 
evolution-plugin-bogofilter evolution-plugin-pstimport evolution-plugins 
firefox gdm3 gimp gir1.2-clutter-1.0 gir1.2-cogl-1.0 
gir1.2-coglpango-1.0 gir1.2-gst-plugins-bad-1.0 
gir1.2-gst-plugins-base-1.0 gir1.2-gtkclutter-1.0 gir1.2-mutter-11 
gir1.2-rb-3.0 gir1.2-webkit2-4.0 gir1.2-webkit2-4.1 gir1.2-wnck-3.0 
gnome gnome-2048 gnome-browser-connector gnome-calendar gnome-contacts 
gnome-control-center gnome-core gnome-games gnome-initial-setup 
gnome-maps gnome-music gnome-nibbles gnome-online-accounts 
gnome-remote-desktop gnome-session gnome-session-bin 
gnome-settings-daemon gnome-shell gnome-shell-extension-prefs 
gnome-shell-extensions gnome-sound-recorder gnome-sushi gnome-tweaks 
gnome-user-docs gnome-video-effects gstreamer1.0-clutter-3.0 
gstreamer1.0-gl gstreamer1.0-gtk3 gstreamer1.0-libav 
gstreamer1.0-pipewire gstreamer1.0-plugins-bad gstreamer1.0-plugins-good 
keepassx keepassxc libatk-wrapper-java libatk-wrapper-java-jni 
libavcodec59 libavfilter8 libavformat59 libavutil57 libcanberra-pulse 
libcheese-gtk25 libcheese8 libchromaprint1 libclutter-1.0-0 
libclutter-gst-3.0-0 libclutter-gtk-1.0-0 libcogl-pango20 libcogl-path20 
libcogl20 libdirectfb-1.7-7 libedataserverui-1.2-4 
libedataserverui4-1.0-0 libegl-mesa0 libegl-nvidia0 libegl1 
libespeak-ng1 libevolution libfluidsynth3 libfolks-eds26 
libfreerdp-server2-2 libfreerdp2-2 libgegl-0.4-0 libges-1.0-0 libgimp2.0 
libgl1 libgl1-nvidia-glvnd-glx libglx-mesa0 libglx-nvidia0 libglx0 
libgoa-backend-1.0-1 libgstreamer-gl1.0-0 libgstreamer-plugins-bad1.0-0 
libmutter-11-0 libpcaudio0 libpipewire-0.3-modules libpocketsphinx3 
libpostproc56 libpulse-mainloop-glib0 libpulse0 libqt5gui5 libqt5quick5 
libqt5svg5 libqt5waylandclient5 libqt5waylandcompositor5 libqt5widgets5 
libqt5x11extras5 libreoffice-base libreoffice-calc libreoffice-core 
libreoffice-draw libreoffice-gnome libreoffice-gtk3 libreoffice-impress 
libreoffice-math libreoffice-sdbc-hsqldb libreoffice-writer 
libsdl1.2debian libsdl2-2.0-0 libsphinxbase3 libstartup-notification0 
libswresample4 libswscale6 libva-dev libva-glx2 libva-x11-2 
libvdpau-va-gl1 libwebkit2gtk-4.0-37 libwebkit2gtk-4.1-0 
libwebkitgtk-6.0-4 libwnck-3-0 libx11-xcb1 libyelp0 mesa-va-drivers 
mesa-vdpau-drivers mesa-vulkan-drivers mplayer nsight-compute 
nsight-systems nvidia-driver nvidia-driver-libs nvidia-egl-icd 
nvidia-openjdk-8-jre nvidia-visual-profiler nvidia-vulkan-icd 
openjdk-17-jre orca pipewire pipewire-alsa pipewire-audio pipewire-bin 
pipewire-pulse qt5-gtk-platformtheme qtwayland5 quadrapassel rhythmbox 
rhythmbox-plugin-cdrecorder rhythmbox-plugins rygel rygel-playbin 
rygel-tracker shotwell smplayer smplayer-l10n smplayer-themes 
speech-dispatcher speech-dispatcher-audio-plugins 
speech-dispatcher-espeak-ng swell-foop task-desktop task-gnome-desktop 
thunderbird totem totem-plugins va-driver-all vdpau-driver-all 
wireplumber x11-apps x11-utils xorg xserver-xephyr xserver-xorg 
xserver-xorg-core xserver-xorg-input-all xserver-xorg-input-libinput 
xserver-xorg-input-wacom xserver-xorg-video-all 
xserver-xorg-video-amdgpu xserver-xorg-video-ati 
xserver-xorg-video-fbdev xserver-xorg-video-intel 
xserver-xorg-video-nouveau xserver-xorg-video-nvidia 
xserver-xorg-video-qxl xserver-xorg-video-radeon xserver-xorg-video-vesa 
xserver-xorg-video-vmware xwayland yelp zenity The following NEW 
packages will be installed: libreoffice-core-nogui 
openjdk-8-jre-headless policykit-1-gnome The following packages will be 
DOWNGRADED: libx11-6 0 upgraded, 3 newly installed, 1 downgraded, 204 to 
remove and 0 not upgraded. Need to get 57.9 MB of archives. After this 
operation, 2,512 MB disk space will be freed. Do you want to continue? 
[Y/n] |

What should I do?

Thanks for helping me with this.

​