You can check this post http://e2e.ti.com/support/dsp/davinci_digital_media_processors/f/717/p/307578/1071547.aspx#1071547
You can also limit qpMax value to small number like 10 and check.
Print the static and dynamic params and attach the encoded stream at 10 mbps to confirm you are not making any mistake in encoder configuration.