Hi Steve,
Good catch. I believe you are correct - the top row of Table 1-11 in SPRUGH5A is invalid because PLLOUT must be greater than 60MHz to maintain clean jitter and duty cycle. Running PLLOUT less than 60MHz goes against the datasheet - you would have configure PLLOUT > 60MHz then use the output divider.
Test your PLL values in the PLL calculator: http://processors.wiki.ti.com/images/f/f0/C5505_PLL_Calculator_060210.zip
You might try using a 12.288 MHz clock source into CLKIN (CLK_SEL = 1), then bypass the PLL all together.
I'll file a bug on this document.
Hope this helps,
Mark