.lvl files

[TheAdmiester]

I haven't had time to have a proper look, but viewing CAM files is working perfect so far.
Also, is it possible to Import/Export them to or from BMP in the GUI version yet?

[Paul]

I haven't had time to have a proper look, but viewing CAM files is working perfect so far.
Also, is it possible to Import/Export them to or from BMP in the GUI version yet?

Not yet I'm going to add export cam image / import image in the next version

[mlg man]

WOW, good work on the editor! its so noob friendly! if someone complains about not knowing how to convert cams now, there the dumbest thing on the planet.

[ThibG]

Hi,
I've been (and I'll be) busy, but I've managed to understand most of FG1 format (and it's on the hg repo).
By the way, Paul, can you tell me everything you discovered about the Bits (if you've find something more than me), Path (same), Anim, and Font formats?

[Paul]

I did have Font figured out but lost the docs.. Bits in AO is very simple in AE in has a small 1kb or so hard coded table that is then decompressed or something into a much bigger 8kb table. This table is then used to decompress each segment of the cam file. After this something else is done to the segment in some recursive complex function that extracts the RGB values

Anim isn't too hard but i lost the docs on that too, I'll get it back eventually so don't worry about it

Path has a hard coded offset for every path id to where the "real" data starts. It appears that each chunk of data is an offset into an array of function pointers (possibly a class VTable) and also some arguments to go to that given function. I managed to replace a slig with a mud from changing the values at run time with the debugger.

Also I can't see your FG1 docs yet Hopefully they are the same as the AO versions.

[ThibG]

This sounds bad... So many hard-coded values...
The "something else" that is done to each Bits segment, are you sure it involves a recursive function? Considering that the game was originally on playstation and that the segments are 16px-wide, isn't the "something else" done by the Motion Decoder?
http://jpsxdec.googlecode.com/files/...format0-56.txt (2.3 MDEC emulation)

Ok, I'll wait for Font and Anim docs, then

Again, values hardcoded in the exe... Damn. Function pointers... The hard part begins, it seems...

Sorry for the FG1 docs, I forgot to push my changes to the server. That's done, now!

[Paul]

Yes its a recursive function and has what looks like an unrolled loop with around 18 iterations in it. It copys "Bits " into "VLC " blocks in memory (var length code MDEC blocks?) but almost all of the MDEC emulation functions have a debug string in them and they don't get called when processing a cam file.
I Have a text file with what I think is almost or is all of the hard coded array that it applies the phase 1 processing to I can upload it if you like?

Just to make it clear its this data
1. Game decodes a small hard coded table into a bigger one
2. This bigger table is the one I ripped from the games memory and format a little to make it look nicer
3. Each 16x240 pixel segment of cam file is then processed using this big table
4. Finally the big recursing unrolled loop function of doom does a lot of bit manipulation on it to get R G B values and throws em in the emulated psx vram.

Even though I don't think this might be MDEC it could be because of the following reasons:

1. They did write an MDEC emulator for the game to play videos maybe? Not sure what it is actually used for.
2. They also wrote masher as a separate static lib, now this one might not have had debug strings in it and might be ANOTHER copy of similar code in the binary since the old school linkers where too stupid to know it was the same thing and remove references to it (although if the debug print isn't there then I guess it actually is not the same code!)

I do think the function was checking for the files of FF FE that you mention in your FG1 docs though! Also this is not the same as the AO ones because the AO ones are not using MDEC or whatever compression. If it is MDEC then the uncompressed table of lookup data must be in other psx emulators or something?

Edit: Attached the array

[ThibG]

Huffman tables are not in the MDEC part, but in the common movie file format (STR).
So, the table haven't anything to do with the emulator. It's in the software, not the hardware, and althought there is a common table, each game may implement its own format.
But if it's processed by a recursive function, I don't really think it's MDEC...
However, 16*240 is 15 blocks of 16*16 pixels, and that makes sense for MDEC, which works by 16*16px macroblocks.

I didn't look at your table yet, but I'll. By the way, have you a convinient way to extract Path offsets, strings, and things like that from the exe?

I don't think the Bits decoding part use FE FF, since in FG1, its meaning is "the specified macroblock is completly opaque" and there is no way a 16*16px macroblock would contain pixels of the same exact color.
I'm almost sure it's using a VLC, since messing with a byte will corrupt everything from this byte. So, if your table is correct, it should be useful (but a way to extract this table would be far better!).

Edit: Is this table from the PS1 version or the PC version? Can you post the corresponding compressed table? For now, I haven't been capable of anything with it

[Paul]

The only way to extract the table is from reversing the code since that table is generated by a function in the game. The function that does that pulls it from some other smaller table so I guess you could pull the data from that offset, copy what the function is doing to make the bigger table (which looks far from simple) and then you have your data.

FYI the loop to decode the last stage of the pixels is doing something like this:

if ( g_left7 <= 0 ) // First if batch
{
right25Copy2 = g_right25;
vlc_buf_ptr = *g_pointer_to_vlc_buffer;
vlc_buf_ptr1 = g_pointer_to_vlc_buffer + 1;
++g_pointer_to_vlc_buffer;
left7M1 = (unsigned int)vlc_buf_ptr >> 7;
right25_new = (signed __int16)vlc_buf_ptr << 25 >> 25;
g_left7 = left7M1;
g_right25 = right25_new;
}
else
{
right25_new = g_right25;
vlc_buf_ptr1 = g_pointer_to_vlc_buffer;
left7M1 = g_left7 - 1;
right25Copy2 = 0;
--g_left7;
}

There is one case in the function where it will call itself 3 times!

The function of intrest is the one that uses the array i posted which looks something like this:

signed int __cdecl decode_vlc_segment(int camSegPtr, int vlcBufferPtr)
{
signed int result; // eax@1
int index; // ebx@1
int segDWord; // esi@1
__int16 *retPtr; // edi@2
int retDWord; // ecx@2
signed int forCounter; // ebp@4
int v8; // [sp+14h] [bp+4h]@1

index = vlcBufferPtr;
segDWord = *(_WORD *)(camSegPtr + 2) | (*(_WORD *)camSegPtr << 16);
v8 = camSegPtr + 4;
result = 0;
loop_start:
while ( 2 )
{
retDWord = (unsigned __int16)g_cam_look_up_table_q[4 * ((unsigned int)segDWord >> 21)];
retPtr = &g_cam_look_up_table_q[4 * ((unsigned int)segDWord >> 21)];
result += retDWord;
segDWord <<= retDWord;
if ( result > 15 )
{
result &= 0xFu;
segDWord |= *(_WORD *)v8 << result;
v8 += 2;
}
for ( forCounter = 0; ; ++forCounter )
{
++retPtr;
if ( forCounter >= 3 || !*retPtr )
goto loop_start;
if ( (unsigned __int16)*retPtr == 0xFFFF )
break;
*(_WORD *)index = *retPtr;
index += 2;
}
*(_WORD *)index = (unsigned int)segDWord >> 19;
if ( (unsigned int)segDWord >> 19 != 1 )
{
result += 13;
index += 2;
segDWord <<= 13;
if ( result > 15 )
{
result &= 0xFu;
segDWord |= *(_WORD *)v8 << result;
v8 += 2;
}
continue;
}
return result;
}
}

Where g_cam_look_up_table_q is the array. Looks like I was wrong and its actually checking for 0xFFF.

Also I found an old function I wrote to decode part of a specific path file which looks like this:

void ParseFile( std::vector& aBuf )
{
int numBytes = aBuf.size();
int numDwords = aBuf.size() / 4;

DWORD* ptr = (DWORD*)&aBuf[0];

// Read the last dword
int pos = 1;
DWORD end = ptr[numDwords-pos];
do
{
// Process the dword
if ( end != -1 )
{
struct rec
{
WORD u1;
WORD u2;
DWORD funcPtr;
DWORD u3; // Used in asm up to here
};


rec* ptr = (rec*)&aBuf[end];

cout << ptr->u1 << " ";
cout << ptr->u2 << " ";
cout << hex << ptr->funcPtr << " ";
cout << ptr->u3 << " ";
cout << "\n";
}

// Get next last dword
pos++;
end = ptr[numDwords-pos];
}
while ( end != 1 );

}

I've attached the file thats in "aBuf" thats passed to this function.

Edit: Also attached an image of when I replaced a slig with a mud from changing the op codes at run time.

[Varrok]

I haven't had time to have a proper look, but viewing CAM files is working perfect so far.

On my Steam AO works good too Will it be difficult to add an option (like check/uncheck one) to make images 2x higher in viewer?

[ThibG]

@Paul, nice! Thanks for the RE-ed code! One little question, though... Is this code the PS1 code or the PC code? I mean, the system on which this code is ran is in Little Endian or Big Endian?
Edit: and what's the type/size of g_cam_look_up_table_q? byte? word? double word? I guess word...?

[Paul]

@Paul, nice! Thanks for the RE-ed code! One little question, though... Is this code the PS1 code or the PC code? I mean, the system on which this code is ran is in Little Endian or Big Endian?
Edit: and what's the type/size of g_cam_look_up_table_q? byte? word? double word? I guess word...?

It appears to be a word from what I've seen. Its from the PC, so whatever byte-sex an x86 cpu is

@Varrok: Nope that wouldn't be too hard, I'll likely add it in the next version if I can (there is a ton of stuff that needs fixing or sorting out before that though)

[ThibG]

@Paul, ok, thanks.
I'm trying to get it work, but for now, I haven't managed to do anything. So, some questions:
Are you 100% sure that the table is correct? Do you think it could change with the locale of the game? camSegPtr is a pointer to the cam segment (just after the size information), isn't it?

[Paul]

I think its correct and I don't expect it would change the only problem is that it might not be all of it, e.g there could be X amount more bytes needed to complete it.

As for camSegPtr

segDWord = *(_WORD *)(camSegPtr + 2) | (*(_WORD *)camSegPtr << 16);
v8 = camSegPtr + 4;

Yeah that looks like what it is, its switching the two words of the cam segment dword around and then advancing to the next cam segment dword (so that suggests the cam data is an array of either words or dwords).

[ThibG]

I think its correct and I don't expect it would change the only problem is that it might not be all of it, e.g there could be X amount more bytes needed to complete it.

As for camSegPtr

segDWord = *(_WORD *)(camSegPtr + 2) | (*(_WORD *)camSegPtr << 16);
v8 = camSegPtr + 4;

Yeah that looks like what it is, its switching the two words of the cam segment dword around and then advancing to the next cam segment dword (so that suggests the cam data is an array of either words or dwords).

Yeah, that's why I asked if it's Little Endian. It would make sense if it's Little Endian, to read a double word instead of a word (already in LE).

Anyway, I've tried this function, and it doesn't work it seems (either a segfault, or an infinite loop, depending on the endianness I use).

[Paul]

Yeah, that's why I asked if it's Little Endian. It would make sense if it's Little Endian, to read a double word instead of a word (already in LE).

Anyway, I've tried this function, and it doesn't work it seems (either a segfault, or an infinite loop, depending on the endianness I use).

I think there is a lot more work to be done before it "works" as is. Prob needs a few iterations of reversing to make it work and be readable.

[ThibG]

@Paul, the table may indeed be incomplete. Can you post a bigger dump?
If my code is right, when decoding the first cam segment of MIP01C04.CAM, it tries to access the 2037th chunk of the lookup table, but the one you posted only contains 1986 chunks.

If the code is right, it should just "work", I mean, the output may not be directly usable, but at least, it'll read all the input without segfault or infinite loop.

Edit: the maximum number of chunks is 2048, even is my code is wrong. So, please provide a table of 2048 chunks (2048 * 4 words, that is, 16kB).

[Paul]

I'll try to re-rip it later if I get time, I tested it with STP01C25.CAM which is the very first cam file with the black and white text that the game displays and decoding the first 16x240 segment worked with it (Although i lost my converted version of the RE'ed code)
So its likely worth trying that one to see if your code works, if not then you know you have an issue. It will be strange if some is missing since I ripped the array from the first and last element written to the heap.

Edit: Found an old text file that might be useful. It shows decoded seg1 of the above cam file with some of my silly ramblings in it. Don't ask me what half of it means cause its from over a year ago

[mlg man]

The only way to extract the table is from reversing the code since that table is generated by a function in the game. The function that does that pulls it from some other smaller table so I guess you could pull the data from that offset, copy what the function is doing to make the bigger table (which looks far from simple) and then you have your data.

FYI the loop to decode the last stage of the pixels is doing something like this:



There is one case in the function where it will call itself 3 times!

The function of intrest is the one that uses the array i posted which looks something like this:



Where g_cam_look_up_table_q is the array. Looks like I was wrong and its actually checking for 0xFFF.

Also I found an old function I wrote to decode part of a specific path file which looks like this:



I've attached the file thats in "aBuf" thats passed to this function.

Edit: Also attached an image of when I replaced a slig with a mud from changing the op codes at run time.

How do you change the slig into mud?! Whats an op code?

[Paul]

How do you change the slig into mud?! Whats an op code?

Reverse the function in the game that decodes path files, or just look at path files and figure out how they work.

[mlg man]

Paul, take a look at this. http://www.oddworldforums.net/showthread.php?t=18954

Why don't we use saves to make levels, they contain where muds are, and items. But we can only change position of things. Its a start, right?

[Nate]

This thread confuses me. For all that I know, you guys could be Al Qaeda operatives planning your next horrific attack by using an innocent web forum and a set of codewords.

[Paul]

Paul, take a look at this. http://www.oddworldforums.net/showthread.php?t=18954

Why don't we use saves to make levels, they contain where muds are, and items. But we can only change position of things. Its a start, right?

Because the save file is just restoring the state of some variables, unless part of the path data is saved in there then you can't change it. You need to look at XXPath.BND since these are the files that define the layout and whats in any given "path".

[mlg man]

Ok, Heres what i'm gonna do. Using ddcheat in abes oddysee i will see where the floor are placed. x and y value, look for values in it, and hopefully change something. Just a question, is dos more easy to reverse engineer, if so, the abe's oddysee pc comes with the dos files in a folder called dos. why dont u decompile that.

@Nate

[Paul]

Ok, Heres what i'm gonna do. Using ddcheat in abes oddysee i will see where the floor are placed. x and y value, look for values in it, and hopefully change something. Just a question, is dos more easy to reverse engineer, if so, the abe's oddysee pc comes with the dos files in a folder called dos. why dont u decompile that.

@Nate

The DOS version is evil and I've already done tons of work on the AO win32 version. Found something interesting Part of the path data IS saved into the game save. I'm not sure why though it must keep track of what muds on a given level are dead? Seems rather strange that it would save that data...

So in AO when you save a game and you kill a slig in the 2nd screen for example.. what happens when you load the game save? Is that slig still dead until abe dies and restarts that part of the level?

Edit: To make hexing it more clear 256bytes of the game save are path data.

[ThibG]

@mlg man, interesting stuff, I'll have a look
I've already started that, though. You can have a look at doc/formats/nxtp.h to have an overview of what I have figured out.

@Paul, ok, gonna try STP01C25.CAM then.
Edit: I can't decode this, even with the code you've posted...

[Paul]

@mlg man, interesting stuff, I'll have a look
I've already started that, though. You can have a look at doc/formats/nxtp.h to have an overview of what I have figured out.

@Paul, ok, gonna try STP01C25.CAM then.
Edit: I can't decode this, even with the code you've posted...

It does work for at least the first segment of it so likely however you've converted the code is slightly off. That function is executed per segment, not for the whole file.

[mlg man]

@ThibG But i have no idea where the values are. it doesn't say where they are and stuff. Its hard to hex edit without Addresses. @paul But the dos version has what the names of values are called. And i think a decompresser called. dcc or something can give you source code.

[Paul]

@ThibG But i have no idea where the values are. it doesn't say where they are and stuff. Its hard to hex edit without Addresses. @paul But the dos version has what the names of values are called. And i think a decompresser called. dcc or something can give you source code.

Its impossible to get the original source back, the win32 version is easier to reverse because it actually runs on my computer and I can debug it. Also since the win32 version was compiled with the MSVC 6.0 compiler its easier to identify vtables and construct/destructors (and hence the inheritance tree).

Edit: By the way for offsets..

struct PlainSavFile { // 8096 bytes
char unknown1[32]; // Unknown, zeros
char unknown2[32]; // Unknown, non-zero
char unknown3[448]; // Unknown, zeros
char unknown4[4]; // Unknown
uint32_t gnframes; // Global frame counter
char unknown5[2];
uint16_t level_part; // first integer in sublevel name %sP%02dC%02d
uint16_t level_component; // second integer in sublevel name
char unknown6[2];
uint16_t hero_x;
uint16_t hero_y; // In fact, hero_y - 1
char unknown7[2];
int16_t score_saved; // Number of escaped mudokons
int16_t score_killed; // Number of killed mudokons
// 0x021d
char unknown8[937];
// 0x05c6
uint16_t have_ring_1;
uint8_t number_of_projectiles;
uint8_t unknown9; // WTF?!
// 0x05ca
char unknown10[82];
// 0x061c
uint16_t have_soulstorm_fart;
// And many other unknown things...
};

You calculate it from the size of each element so:

uint32_t gnframes; // Global frame counter

is at offset

char unknown1[32]; // Unknown, zeros
char unknown2[32]; // Unknown, non-zero
char unknown3[448]; // Unknown, zeros
char unknown4[4]; // Unknown
uint32_t gnframes; // Global frame counter

32+32+448+4 = 516, or in hex 0x204

[ThibG]

@Paul, I know that, I'm trying to decode the first segment.
@mlg man, try and search for character identifiers. Abe is 0x45 0x00. Now that you've found that, it corresponds to uint16_t id