I found the answer. When changing to encodingPreset = XDM_USER_DEFINED, one needs to also set some other parameters or it will complain about bad parameters.
tStaticParam.videoStaticParams.h264EncStaticParams.numTemporalLayer = IH264_TEMPORAL_LAYERS_1; /* for base profile */ tStaticParam.videoStaticParams.h264EncStaticParams.transformBlockSize = IH264_TRANSFORM_4x4; tStaticParam.videoStaticParams.h264EncStaticParams.entropyCodingMode = IH264_ENTROPYCODING_CAVLC; /* for base profile end */