Yes its a recursive function and has what looks like an unrolled loop with around 18 iterations in it. It copys "Bits " into "VLC " blocks in memory (var length code MDEC blocks?) but almost all of the MDEC emulation functions have a debug string in them and they don't get called when processing a cam file.
I Have a text file with what I think is almost or is all of the hard coded array that it applies the phase 1 processing to I can upload it if you like?
Just to make it clear its this data
1. Game decodes a small hard coded table into a bigger one
2. This bigger table is the one I ripped from the games memory and format a little to make it look nicer
3. Each 16x240 pixel segment of cam file is then processed using this big table
4. Finally the big recursing unrolled loop function of doom does a lot of bit manipulation on it to get R G B values and throws em in the emulated psx vram.
Even though I don't think this might be MDEC it could be because of the following reasons:
1. They did write an MDEC emulator for the game to play videos maybe? Not sure what it is actually used for.
2. They also wrote masher as a separate static lib, now this one might not have had debug strings in it and might be ANOTHER copy of similar code in the binary since the old school linkers where too stupid to know it was the same thing and remove references to it (although if the debug print isn't there then I guess it actually is not the same code!)
I do think the function was checking for the files of FF FE that you mention in your FG1 docs though! Also this is not the same as the AO ones because the AO ones are not using MDEC or whatever compression. If it is MDEC then the uncompressed table of lookup data must be in other psx emulators or something?
Edit: Attached the array