You will also need to make sure that the VENC configuration matches the expected range for the HDMI core.
This clamping that you are seeing looks like the VENC is range expanding, possibly with the color space converter.
How are you generating your 'source'? Are you placing known values in the frame buffer or are you using the complete capture-display path?
To analyze exactly what is going on I suggest placing known values directly in the frame buffer initially as a test.
Next, can you create a mapping table where you cycle through input values from 0 to 255 and list what output value you get? This will help me understand if this is a scaling issue or a clipping issue or both.
I am not on the software team so I don't know how software configures the path, but there is a lot of flexibility in each stage and each stage needs to be aligned with the others.
Someone from the software team may be able to directly answer. Hopefully they will chip in :)
BR,
Steve