What Should You Check When H.264 Hardware Transcodes but AV1 Does Not?
H.264 success proves the GPU path works for H.264, but it does not prove the same hardware supports AV1 decode, encode, profile, or bit depth.
Hardware transcoding is a chain of codec-specific operations: decode the source, apply scaling or tone mapping, and encode the output format selected for the client. Older and mid-generation GPUs commonly accelerate H.264 while supporting only AV1 software decode, AV1 hardware decode without AV1 encode, or neither. Start by identifying which AV1 stage fails before changing global acceleration settings.
Determine Whether AV1 Decode or AV1 Encode Is Failing
Inspect the active FFmpeg command and transcode reason. Record the input codec, hardware decoder, filter path, output codec, hardware encoder, and the first error line.
A server may decode AV1 and encode H.264, or decode H.264 and attempt to encode AV1. Those are different capabilities. A Jellyfin hardware discussion includes a system where AV1 QSV and VA-API paths were unsupported on the installed GPU while other hardware paths remained usable.
Run one AV1-input-to-H.264-output test and one H.264-input-to-AV1-output test only when the media server exposes both choices. If the first succeeds and the second fails, AV1 decode exists but AV1 encode does not.
Check Hardware Capability by Codec Direction and Profile
Look up the actual GPU generation and device ID, then compare decode and encode support separately. Include AV1 Main profile, 8-bit versus 10-bit, chroma format, resolution, and maximum level.
Codec support can be partial even on newer devices. An Intel media-driver report shows AV1 hardware decoding failing on Tiger Lake while software decoding worked, demonstrating that a present AV1 code path may still fail for a particular kernel and driver combination.
Do not infer AV1 capability from H.264 utilization, GPU model family, or an application checkbox. Confirm the exact source profile and requested output entry point against what the installed stack reports.
Read the Hostโs Exposed Decode and Encode Profiles
Run the platformโs capability tool against the intended render or CUDA device. For VA-API, capture profile and entry-point output; for NVIDIA, capture the installed driver and FFmpegโs available decoders and encoders.
AV1 may appear for decode but not encode, or a low-power encode mode may require firmware that H.264 does not. Intelโs media-driver repository notes that AVC, HEVC, VP9, and AV1 low-power bitrate control can depend on HuC firmware availability.
If the host does not expose the required AV1 entry point, stop at the host layer. Changing container permissions or media-server settings cannot create a codec block that the kernel and userspace driver do not provide.
Compare Kernel, Firmware, and Userspace Driver Versions
Record the kernel, firmware package, GPU userspace driver, libva or CUDA stack, container runtime, and FFmpeg build. H.264 can remain stable across an update while a newer AV1 path regresses.
Early Intel Arc deployments reported VA-API decode and encode failures affecting AV1 along with other codecs until the surrounding kernel and media-driver stack matured. The case illustrates why driver-stack compatibility matters more for a newer codec path.
Compare the current versions with the last known-good configuration and the distributionโs supported package combination. Avoid mixing a new userspace media driver with an incompatible old kernel or replacing packaged FFmpeg components individually.
Test the Same Device and FFmpeg Build Inside the Container
Enter the media-server container and verify the render node or NVIDIA device, numeric groups, driver libraries, and bundled FFmpeg codec list. Host capability does not prove the container uses the same stack.
Run a short known-good AV1 sample through hardware decode and a simple H.264 output without HDR, subtitles, or scaling. Then repeat with the applicationโs FFmpeg binary and its actual device selection.
The ZimaSpace guide to checking hardware transcoding provides the adjacent test for proving that the container uses the expected device rather than silently falling back to software.
Separate AV1 Video Processing From Delivery-Container Problems
An AV1 source may decode correctly but fail when the client requires audio conversion, a different HLS segment type, HDR processing, or a delivery container that does not carry the chosen combination.
A Jellyfin Web issue found AV1 playback with audio transcoding failed under one HLS container choice but worked when fMP4-HLS was enabled, showing that the delivery container can be the failing layer.
Retest with a simple SDR AV1 file, compatible audio, no subtitles, and H.264 output. Add audio conversion, HDR tone mapping, subtitles, and the normal client profile one variable at a time.
Use Software Fallback or a Compatible Output Only After Classification
If the GPU supports AV1 decode but not encode, keep hardware decode and encode the client output as H.264 or HEVC when supported. If AV1 decode is absent, software decode may work for lower resolutions but can be too slow for high-bitrate 4K.
Client support also affects whether AV1 output is useful. Jellyfin Web has tracked browser profiles that continue selecting H.264 because AV1 client support remains profile dependent.
The repair is complete when the tested AV1 file uses the intended hardware decoder or a measured software fallback, the output codec matches the client, transcode speed remains above real time, and H.264 sessions continue working after the AV1-specific change.
Recommended products
Support & Tips
More to Read
Why Does a Docker Volume Restore Recreate File Contents but Drop Extended Attributes?
A volume-restore diagnosis covering xattr inventory, tar and Rsync options, namespaces, destination support, privileges, labels, app metadata, and tests.
Why Does a Running Container Keep Its Old Memory Limit After the Compose File Changes?
A memory-limit diagnosis covering live cgroups, restart versus recreation, Compose fields, hard and soft limits, parent scopes, swap, and runtime heaps.
Why Does Restarting a Reverse Proxy Invalidate Every Session for One Self-Hosted App?
A session-loss diagnosis covering restart scope, cookie ownership, secret rotation, cache-backed sessions, sticky routing, auth gateways, and recovery.
