Some more bits of information
I am using Slot 0 in both TDM and SPDIF modes. So, checking for XDATA clearing should be fine according to Section 25.2.4.1.2 of OMAP-L138 TRM, I am guessing.
I could reproduce the hang even the first time if I do not enable the Write FIFO.
I am using only one serializer in the tx direction. When things work the first time (i.e. when write FIFO is enabled), the WNUMEVT and WNUMDMA are both set to 1 in WFIFOCTL register.
Why is the behavior different when write fifo is disabled?