Map TGAs

[Just Crazy Jim]

I've been looking at the TGA portion in GMP files and it appears there is very little uniformity when it comes to how water areas are treated on the texture.

Example 1:

Coast-to-Coast Ireland

Example 2:

Japan Trembles

Example 3:

Coast-to-Coast Pacific NW

Example 4:

TexMex

Look closely at the "ocean" portions of the map for the most striking differences.

What I am getting at is this: In most instances, it appears that PopTop injected the final TGA after creating the maps. Most probably, a TGA made by a 3rd party artist.

[Gumboots]

That's interesting. The treatment of water is one of the known bugs with the RT3 editor. It tends to accumulate crud that borks the border between water and land, which reduces map quality after a lot of testing and saving (or even after not that much). So injecting an unborked TGA into the GMP after testing is finished would make a lot of sense for production.

[RulerofRails]

I agree about painting being injected, that's what the bmp2gmp utility does for us. You can get those same shots as bmps with the related gmp2bmp utility (unless that's what you did?). That utility results in map texture being referred to as the map's "bmp", even though in fact the format may be TGA (the file generated when loading a map seems to indicate this). Although why would Pjay mis-name his utilities?

If you convert the ocean to land (I had the "change color" option un-checked, didn't test if that's important) in the map editor you will see the underlying texture. Here's a before and after:

Even with average graphics, one should see different shades of blue on each of your example maps corresponding to the texture color beneath the ocean.

[Just Crazy Jim]

I copied out the hexes after the last (non-random) instance of FF D9 from the GMPs and saved them to a new hex file. I saved the resulting files with the file extension BMP and TGA. Adobe Photoshop could not open the file with the BMP file extension, but opened the TGA version without any problem. Further testing with a couple of other image editors and Cerious Software's Thumbsplus confirm that the map texture is indeed a TGA missing the optional Truevision-Xfile footer and other optional extension data. There may be a thumbnail buried in the GMP (also in TGA format) as well.

It was in those first 18 bytes after the FF D9 that started me thinking. I've noted this 18-byte cluster in a number of other games I've modded. Specifically, the 3rd byte always being 02 that suggested to me that the format was TGA. In a TGA header, the 3rd byte indicates compression type. 02 in that position indicates an uncompressed true-colour image. Once my suspicion was set off, I checked to see if the rest of the header file info conformed to TGA header specifications. It does.

Every GMP file I have scoped has this series of values after the final (non-random) FF D9:

Code: Select all

00 00 02 00   00 00 00 00   00 00 00 00   00 04 00 04
20 xx 

02 at position 0002 (3) indicates uncompressed true-colour image
00 04 at positions 000c-000d (13-14) indicates 1024 pixels width
00 04 at positions 000e-000f (15-16) indicates 1024 pixels height
20 at position 0010 (17) indicates 32-bit colour
xx at position 0011 (18) indicates alpha channel depth and direction and this varies in the files. In the files I have scoped, the value at position 18 is usually 08 (8-bit alpha channel), but in some it is 20 (32-bit alpha channel).

Ergo, the GMP map texture is TGA.

The same pattern can be found in saved games (GMS files), uncompressed 32-bit colour image with whichever type of alpha channel the original GMP specified. However, the degradation is visible, the EXE is applying some manner of compression before injecting the then decompressed data into the GMS

If there is a thumbnail in the GMP file, I haven't found it yet. My suspicion is that the EXE generates the preview thumbnail on demand from the embedded TGA. And I suspect that it is this function of the EXE that in some manner relates to map texture degradation. It's possible that the EXE reads the uncompressed TGA, resizes it for the thumbnail preview, then resizes it a 2nd time (e.g. back to 1024x1024) to paint to the terrain, rather than doing a second read of the GMP's embedded TGA.

[Gumboots]

It'd be daft if it was done that way. It'd make far more sense to generate the thumbnail once, and not bother compressing and re-expanding the map image.

[Just Crazy Jim]

It'd be daft if it was done that way. It'd make far more sense to generate the thumbnail once, and not bother compressing and re-expanding the map image.

True enough, daft. But I've seen the similar done in other games: Slop shop coding for some fundamental aspect that leads to progressive data corruption. I'll known more when I've disassembled every PopTop GMP

[RulerofRails]

Interesting stuff. If it helps your investigation, when you go to start low_grade's Chile map it doesn't have a map preview (at least it appears to be black/blank). Not sure if that is what you mean by "thumbnail"?

Makes me wonder if somebody tried hacking in a TGA larger than 1024x1024? . . . probably instant crash. . . .

[Just Crazy Jim]

Interesting stuff. If it helps your investigation, when you go to start low_grade's Chile map it doesn't have a map preview (at least it appears to be black/blank). Not sure if that is what you mean by "thumbnail"?

Makes me wonder if somebody tried hacking in a TGA larger than 1024x1024? . . . probably instant crash. . . .

I just scoped Lowgrade's Chile map - it had a 512x512 texture file embedded. I resized the TGA to 1024x1024 and replaced it, then did a save in the editor (making no other changes), then copied over the TGA in the re-saved GMP again with the 1024x1024 version. Presto, chango, problem solved.

And this gave confirmation that there is a preview (thumbnail) embedded in the GMP. The map texture TGA doesn't have tracks on it, but the preview (thumbnail) does. This clearly indicates that there is a separate preview texture based on a resized version of the 1024x1024 map texture in the GMP. Somewhere. Now I just have to locate that silly thing in a bajillion bytes of nonsense.

fixed version of Lowgrade's Chile map attached.

[RulerofRails]

Congrats, on fixing the Chile map preview.

In the meantime, I remembered/checked that saved games on the old version of Chile are displaying the preview correctly.
Is that significant?

[Just Crazy Jim]

TBH, sir, I don't know. I've tried two resource finder programs trying to locate the map preview you see in the map selection tool during start-up. So far, the GMP defies the best hacking tools I know of. In terms of image data, I've seen patterns of bytes that suggest they are meaningful, but it's not TGA RGB data and it's not DXT compress data. And I can't find a proper BMP in there either, but there may be an RLE-compressed BMP in there... maybe...

I've even tried a variety of game archive extractor applications at the GMPs, hoping it was a known format using an unknown file extension. Nothing came of that effort. So far, my best guess - and it is a guess - is it's a spreadsheet or database with embedded images in a proprietary format. Sort of, kind of, like Microsoft Works files... meaning overly complicated and bloated nightmares, with no compression and no encryption, but lots and lots of extraneous bytes as empty fields.

[Gumboots]

What about jpg? Any chance they used that for the preview? It's ideally suited to small images with a broad colour palette.

[Just Crazy Jim]

What about jpg? Any chance they used that for the preview? It's ideally suited to small images with a broad colour palette.

I would think not.

In 2003, the GIF and JPEG/JFIF formats were still zealously guarded tech patents with steep pricing for for-profit licensure, as in a fee plus additional costs on a "per sale unit" scale (i.e., a cut of the profits). IIRC, even AOL got into that game with its own format (.ART) that no one bothered to support or license. PNG was around, but wasn't an ISO standard format until 2004 and was (and still is) pretty unpopular and in 2003 most image editors didn't ship with support for PNG to get around the delicate matter of the then raging war of open-source vs for-profit. The early 2000s were a tech lawyer's paradise.

So it comes down to the money. Using JPEG/JFIF would have cost PopTop cash upfront and a slice of the profits, whereas TGA, BMP, and DXT may not have cost them anything but a straight fee they may have already paid.

[Gumboots]

But the .exe contains a fair number of references to jpg, so they were obviously supporting it for something.

[Just Crazy Jim]

Well, paint me green and call me a shrubbery. I did a hex search for FF D8 in a random GMP and found the thumbnail. The thumbnail starts at the FF D8 immediately before the TGA. And the ironic part is the FF D9 I have been using as a marker to locate the start of the TGA is the termination of the JPG thumbnail... *head desk*

It's a 256x256 JFIF (JPG) sure enough. 16,940 bytes (hexes) in the first one I found. 7,070 bytes (hexes) in the second. So there's no standard length to that portion of the GMP.

The string to search for is:

Code: Select all

FF D8 FF E0 00 10 4A 46 49 46

That's the header of the JFIF, then all the data until terminating with:

Code: Select all

FF D9

It makes no sense that they'd use JPG on account of the licensure costs, but there's probably more than one reason PopTop ain't with us no more.

Thank you, sir, for jabbing at it until I crawled back into the code.

[Gumboots]

Well, paint me green and call me a shrubbery.

You're a shrubbery.

I did a hex search for FF D8 in a random GMP and found the thumbnail. The thumbnail starts at the FF D8 immediately before the TGA. And the ironic part is the FF D9 I have been using as a marker to locate the start of the TGA is the termination of the JPG thumbnail... *head desk*

[Gumboots]

Anyway, I think this probably rules out game map degradation being caused by taking a 1024x1024 tga, shrinking it down into a 256x256 thumbnail and converting to jpg, then re-enlarging it back to 1024x1024 and re-converting it back to tga. I know coders do some daft things at times, but that combination is just too daft to be plausible, IMO.

It's more likely, AFAICT, that the exe just does some unwanted compression on the tga and stores the result. Is it possible that what they are doing is using a tga base image, but running an on the fly conversion to dds for use in-game? And then stashing that data from the (obviously compressed) dds back into tga format when the game is saved?

Which would be bonkers anyway, but maybe not quite as bonkers, and might explain the observed results.

[Just Crazy Jim]

You're probably right about the DDS. Although if you resize the JPG from 256x256 to 1024x1025 using nearest neighbour, you'd get the same effect. 6 one way, half-dozen the other. Either way, it's daft.

[Just Crazy Jim]

Did a series of experiments using RLE compression on map TGAs and can safely say that RLE is not the bugaboo mauling saved maps. Firstly, the RGB values are almost identical to those of an uncompressed TGA. Secondly, the pixelation pattern is not the same.

Did a series of similar tests converting the map TGAs to DXT3 and DXT5 then back to TGA. There's a bit drift in those, but the loss and pixelation pattern is just different enough to make it unlikely. Certainly, the loss isn't nearly as high as what we see in the game.

Then, I did a series of tests resizing the TGA to 256x256 and 512x512, saved as a 60% JPG, opened the JPGs and resized them to 1024 x 1024. That was more like what you see in terms of degradation of map TGAs, although not perfectly, exactly the same. There is probably some value other than 60% JPG compression being used by the EXE, but short of trying every possible value (which is just too tedious for me), I can't say what value is being used. Still, from what I saw, I'm pretty sure that a JPG conversion is what is the problem with map lossy save.

Even if it's not JPG, the conversion to 256x256 and being resized back to 1024x1024 is all that comes close to the sort of loss we see in the game.

Edit: As an afterthought, It is possible that the EXE is generating a DXT compressed file with a mipmap, then pulling the 256x256 and resizing it to 1024x1024. I am not sure why it would use mipmapping in that instance when the devs eschewed mipmaps in every other instance of DDS formating.

[Gumboots]

I am not sure why it would use mipmapping in that instance when the devs eschewed mipmaps in every other instance of DDS formating.

Because they're evil?

[Just Crazy Jim]

I am not sure why it would use mipmapping in that instance when the devs eschewed mipmaps in every other instance of DDS formating.

Because they're evil?

There may be some truth in that, but I think Hanlon's razor may be closer to the truth: "Never attribute to malice that which is adequately explained by stupidity."

I think in some mad effort to economize on the usage of system memory, the PopTop devs overshot the mark.