Scaling bugs when using HW encoder

[MikeIndiaAlpha]

As described here: livepeer/studio#1239 there are several problems with HW encoder scaling video prior to encode. Two we know for sure:

• In 854x480 profile (I had to add one to transcoding, like that: ERROR = VideoProfile{Name: "ERROR", Bitrate: "1000k", Framerate: 30, AspectRatio: "16:9", Resolution: "854x480"}): green bar appears on the bottom of the image when encoded using Nvidia HW. No problem when encoded using SW encoder
• When doing transcoding cat.mp4 P360p30fps16x9 mode output resolution depends on the mode. "sw" generates 202x360 stream while "nv 0" generates 360x640 one.
• There is also suggestion of problems with QSV hardware encoder, such as green bar. Not sure if I can test it, would rather focus on the problems I can reproduce

I strongly suspect that the cause of second problem is different scaling logic used for Nvidia encoders (I vaguely remember code in ffmpeg.go inserting some magic options that handle scaling on Nvidia HW). In any case, green bar shouldn't arise so my guess is we also have Nvidia scaler bug, and/or are using said component improperly.

Will now read scaling code carefully to learn how scaling is handled in different cases.

[MikeIndiaAlpha]

I did some testing and logging. Here is how the Profile changes during the configuration phrase when Nvidia encoder is active (no such problems on SW encoder)

0 :  {ERROR 1000k 30 0 854x480 16:9 0 3 0s 0 0 0}
0 :  {ERROR 1000k 30 0 480x853 16:9 0 3 0s 0 0 0}

Please note that not only width and height got swapped, also 854 changed to 853. I need to check in standard but AFAIR h.264 doesn't support odd cropping rectangles and so scaling/encoding may be going south on this one (showing green rectangle)

Notes: it looks like ensureEncoderLimits function may be reason for all this. Because:

1. In Clamp method resolution is swapped if it is portrait, and...
2. ...there are also floating point operations which may contribute to the dimensions being decreased by one due to rounding errors

[cyberj0g]

Great find @MikeIndiaAlpha. Tagging @AlexKordic as he is the one who last worked on landscape-portrait and resolution clamping.

[MikeIndiaAlpha]

I spent some more time working on this and come up with the solution that I hope is not too disruptive: #354

[Quintendos]

Diederick working on this - dec' 22