This was predictably nightmarish, given how ridiculous mpeg4 is. I had to stare at the cuvid parser output for a long time to work out what each field was supposed to be, and even then, I still don't fully understand some of them. Particularly:
vop_coded: If I'm reading the decoder correctly, this flag will always be 1 as the decoder will not pass the hwaccel any frame where it is not 1. divx_flags: There's obviously no documentation on what the possible flags are. I simply observed that this is '0' for a normal bitstream and '5' for packed b-frames. gmc_enabled: I had a number of guesses as to what this mapped to. I picked the condition I did based on when the cuvid parser was setting flag.
Also note that as with the vdpau hwaccel, the decoder needs to consume the entire frame and not the slice.
6b77a10e43 avcodec: Implement mpeg4 nvdec hwaccel
Changelog | 2 +-
configure | 2 +
libavcodec/Makefile | 1 +
libavcodec/allcodecs.c | 1 +
libavcodec/h263dec.c | 3 ++
libavcodec/nvdec.c | 1 +
libavcodec/nvdec_mpeg4.c | 121 +++++++++++++++++++++++++++++++++++++++++++++++
libavcodec/version.h | 2 +-
8 files changed, 131 insertions(+), 2 deletions(-)