If scaler performance is causing bottleneck it will only result in reduced fps. It will not cause segmentation fault to A8 app thread.
Check the segfault core dump to determine cause of segfault.
Once s/w issue is eliminated couple of other issues could be DDR instability on your board or NAND bitflip in filesystem causing libc corruption.
For DDR issue follow the s/w leveling procedure detailed in http://processors.wiki.ti.com/index.php/DM816x_C6A816x_AM389x_DDR3_Init is followed.
For nand bitflip issue there are some Linux psp nand driver fixes that are part of DVRRDK 4.1. Pls check if you see the issue with DVRRDK 4.1 Linux kernel.
