Heightmap precision in MicroDEM

[RedMaple986]

I've been getting a feel for using heightmaps as the basis for custom maps. The PopTop-supplied MapBuilder program is pretty easy to use and produces color TGAs with fine precision elevation-wise, but they suffer from distortion (appearing "squashed") that can only be approximately compensated for by resizing the map in an image editor. So with most folks here recommending the use of MicroDEM to create heightmaps, I figured I'd give it a try, using the old tutorial by Wolverine@MSU as a guide.

I downloaded and installed the latest full-install package, swapped in the latest 32-bit executable (since the 64-bit version is the "main" version these days), and copied in the special color table for RT3 (ELEV_COLORS_0-9999.dbf), and downloaded the GTOPO30 tiles I wanted from the USGS (registering as required). Of course there was the obligatory fumbling around to find some things that had moved since the old version used in the tutorial, but I figured it out well enough. There's just one problem with the output though:



Yep, it's garbage. Even a greyscale map has better precision than that: IrfanView reports only 50 unique colors in the image, and the low
precision is evident when importing the map into RT3. Compare to this MapBuilder heightmap of a roughly similar area, which has 1,700 colors:



I figured this wasn't right, but if there is an option buried somewhere in the program to enable a fuller color depth when using a color table, it eluded me. Anyone know what I missed?

So I decided to try an older version of MicroDEM. The Internet Archive Wayback Machine unfortunately doesn't have any archives of the microdem_setup.exe download dating after October 31, 2001 and before January 9, 2009, so I downloaded the copy from January 2009, uninstalled the latest MicroDEM and installed this one, which turned out to be version 10 build 2008.12.30.1. This version seemed less buggy, but still gave the same low quality output using the special color table. This version expanded the dialog when I selected the table to show a preview of the table, and also gave a "Table > 256 colors" checkbox, but checking this box didn't improve the rendered map.

Then I tried swapping in the old "special" executable (version 8.0 build 15.4.12.2) from the ZIP package provided by Wolverine@MSU. It so happens that this version starts up fine with the support files in the version 10 installation, and produced this output:



Much better, right off the bat. It's still visibly inferior to the MapBuilder output though, having only 190 unique colors and looking comparable to a greyscale map of the area. I found that's because the special color table only gives you 1/10 precision, specifying one color for every 10 elevation units. I'm thinking of trying to make higher-precision versions of the color table. Should be easy enough using the series/auto-fill features in Microsoft Excel.

[Gumboots]

I wouldn't worry about it. The thing is that you can't use all the precision available from the DEM's anyway, because to get a usable heightmap from the editor you will have to use the default smoothing factor, and that is a rather blunt tool.

Grayscale maps may nominally have a similar number of colours, but in practice they produce utter crap if you try to make RT3 maps from them. They're probably ok for inland areas, but they are useless for coastal areas because the editor won't recognise the correct values for ocean and coast. The result is that you get radical cliffs where there should be beaches, as well as the terrain in general being off the charts.

OTOH, any heightmap exported from MicroDEM will produce a good result, in practice, in the actual RT3 editor.

[RedMaple986]

Yeah, the precision of the MicroDEM 8.0 output with the existing table isn't too bad, and you make a good point about the effect of the RT3 smoothing function. Though the tutorial does advise that a smoothing factor of 0 would work fine on a map without the sharp elevation changes you get in mountainous areas. Of course, I can imagine how well that would actually work out in gameplay would depend on the bumpiness of the ground.

I'd still argue the output from the latest MicroDEM, and the MicroDEM 10 build I tried, is pretty bad. The elevation changes in 40-unit steps in that map, which is noticeable inside RT3. So the question still remains about what needs to be configured to get it to spit out heightmaps that are at least as good as 8.0's. Till that's resolved though, at least one can still get the 8.0 executable working using a 10.0 installation.

Thing is, the whole point of going to the trouble to use MicroDEM is for better accuracy. Otherwise, one could just take a MapBuilder map and resize it using the miles-per-pixel or suggested resize figures given by the program to determine the proper new size, then crop to an RT3-compatible size. Granted, that mountain peak over there might not be in exactly the same place relative to other stuff as in a MicroDEM map, but the map probably won't be very noticeably off, and that would be an overall easier way of making heightmaps.

[Gumboots]

TBH I just used the latest 64 bit MicroDEM for the Latvia map I'm working on (includes bit of other countries too). It has stuff all colours in it, but seems to produce a map that matches the real terrain of that area remarkably well. Mind you, that area is not exactly mountainous.

If you can manage to get the latest MicroDEM spitting out better heightmaps for RT3 then sure, that would be great. The reason I'm using the latest is because the earlier versions were playing up on my box, and producing weird results inside RT3 (radically stupid terrain heights at the coasts, etc) while the latest version seems to behave nicely. This is on Windows 7 64 bit with an nVidia card, if that makes any difference (dunno if it does).

Without any smoothing factor at all I've found maps are just to jagged to use, but again if you can get around this by finangling a better output then I certainly wouldn't grumble about it.

The thing I like about MicroDEM is that you can crop for latitude and longitude easily and accurately. This helps when it comes to laying out cities (using lat/long from GeoHack) and terrain features like rivers (sat images and/or maps overlaid onto the .gmp temporarily). Things are where they are supposed to be, and personally I like that. It also means getting the most accurate east/west/north/south proportions for a given latitude is simple. Just take the latitude halfway up and run it through something like this.

[Gumboots]

I've just been experimenting a little bit, by using the RT3 Map Builder instead of MicroDEM. I have to say that at the moment it seems to do just as good a job on the whole. It even does better in some locations, due to the usual SRTM dataset lacking information for some regions (for example, the eastern coast of Romania, on the Black Sea, is missing).

I just grabbed a selection in the Map Builder, from 0E to 30E and from 52N to 40N, and saved it as 3720x1560 TGA, with no snap to 64+1. Obviously this is way off for horizontal and vertical scale (0.4 miles/pixel E-W and 1.6 miles/pixel N-S) so just for a quick test I resized it to 1025x1025 in Photoshop and saved that. This gives a horizontal scale of 1.41 miles/pixel (taken at 46N latitude, which is halfway up the map) and a vertical scale of 0.81 miles/pixel. This is still way off for proportions, of course, but I just wanted to see the effect of a massive resize on the results the game engine would generate.

I used the default value of 1 for all map generation modifiers (overall height, mountain tops, smoothness) and it turns out pretty good. Amazingly good in places. Switzerland is unbuildable, but the Dardanelles are close to perfect. The Carpathian Basin is pretty good, and even has the correct pass from Brasov over to Romania. The Iron Gate is a bit of a mess, but gives the general idea.

Overall it's very encouraging, so I think I'll try some more in-depth tests. Given that the Map Builder will make TGA's in any size you like, and that it always crops to the nearest degree, it should be fairly simple to get any region cropped to a custom size that is accurate for vertical and horizontal scale. If I happen to want 1.4E to 16.6E (French coast to Vienna, with a bit of margin) that would simply be (1.4/30)(3720) = 173.6 and (16.6/30)(3720) = 2058.4, so I'd just have to crop between 174 and 2058 pixels before resizing to the appropriate width, then sort out the vertical dimension to match that scale.

Edit: Got some interesting results using 0.7 for overall height, 0.3 for mountaintops, and 0 for smoothing. Using 0 for smoothing has some potential, even in mountainous areas, as long as you did some manual smoothing wherever you intended rail to be built. For things like river canyons it does a better job than a 1.0 smoothing value, and if the mountaintop modifier is knocked back some the overall terrain isn't too bad. Some work with the smoothing tool would be required, but given the amount of work rivers and lakes can take (ha!) a bit of route smoothing for rail would be luxury by comparison.

[RulerofRails]

Generally I would agree that smoothing is often overdone. While reading I had an idea that you could also use in your experiment. In the tools for raise/lower there is the OTHER category. In that you can raise/lower the land with the distinction of uniformly, mountains most, or low areas most. My thought was that if you were using a lower modifier for mountains in lieu of so much smoothing, you might achieve a similar effect on the actual mountains if using a "raise mountains most" option later?

[Gumboots]

Yeah sure, you could do that. I have done something similar before. On the Latvia map I was using the "lower mountains most" at one stage, to knock some of the high points back to more like Latvia (kinda flat, y'know) without losing (much) detail on the lower sections. All of the tools are worth playing with.

What I've found so far is that if you want sharp peaks, it's hard to get them if you use any smoothing. Smoothing is a bit like taking a rasp to the end of a pencil. Makes it really blunt, really fast. It's a real pity that the map editor only accepts integers for the smoothing factor. You can use any decimal value for the other two factors, right down to 0.01 if you want to, but with smoothing there is no fine control.

*adds to ever-lengthening 1.07 wishlist*

At the moment I'm thinking the best solution will be to use no smoothing when initially generating the map. You can smooth large areas manually pretty quickly if you use a large brush, so it shouldn't really be a drama to sort the most likely routes out.

[Gumboots]

I just figured out something by checking this map against Google Earth: the crop function in the Map Builder app is not accurate.

As far as I can tell, it appears to be accurate for the left edge (at 0E, which obviously is also 0W) and the top edge (52N) but it's out of whack for the right edge and the bottom edge. The right edge seems to be at 31E (should be 30, according to the crop function) and the bottom edge at 39N (crop function said it was at 40). It could even be slightly out from those estimates, but not by very much (<0.1 degrees).

I vaguely remember something, from the last time I used it ages ago, about the crops being off, but I haven't done any quantitative comparisons before now. I don't know at this stage if the 1 degree error, on two sides only, will be consistent over all crops but it's certainly enough to throw things out of whack if you didn't know how to compensate for it.

Edit: Ran a few tests, checking against DEM's, and the Map Builder error seems to be consistent. This makes things pretty easy. If you are using the RT3 Map Builder, and you want 40W to 30W and 50N to 40N, then you should select 40W to 31W and 50N to 41N. The Map Builder will then add its standard 1 degree error to the right and bottom edges and make you a nice heightmap that covers 40W to 30W and 50N to 40N. Easy.

I'll whip up a similar map from MicroDEM data and see how it compares. I know the cropping will be more accurate, but I'll be interested to see how the terrain compares when using the same modifier values.

---

Edit: Here's one for the lulz and rofl's department. I was just grabbing some of the latest SRTM's from CGIAR-CSI. They have a new set out now, with better detail in some regions. I thought I'd whip up a spreadsheet to list which numbers (39_04, etc) correspond to which latitude and longitude, so I can just grab the right ones whenever I want to make a map.

The fun part is that their download page doesn't list latitude and longitude correctly. Say you select two SRTM's in the middle of Africa, one each side of the equator (40_12 and 40_13). These run from 5N => Equator => 5S and from 15E to 20E. However, when you get to the download page you find this:

Description
Product: SRTM 90m DEM Version 4
Data File Name: srtm_40_12.zip
Mask File Name: srtm_mk_40_12.zip
Latitude Min: 15 N Max: 20 N
Longitude Min: 0 E Max: 5 E
Center Point Lat: 17.5 N Long: 2.5 E

Description
Product: SRTM 90m DEM Version 4
Data File Name: srtm_40_13.zip
Mask File Name: srtm_mk_40_13.zip
Latitude Min: 15 N Max: 20 N
Longitude Min: 5 W Max: 0 E
Center Point Lat: 17.5 N Long: 2.5 W

What they are doing, and I find this extremely helpful, is mixing up the longitude and latitude in the description.
When the description says "Latitude - North" it actually means "Longitude - East".
"Latitude - South" means "Longitude - West".
"Longitude - East" means "Latitude - North".
"Longitude - West" means "Latitude - South".

I think I might send them an email. If enough people do it, they might debug the page.

---

Edit again: Got the spreadsheet sorted. It lists every v4 DEM available from CGIAR-CSI (there are 874 of them) and which longitude and latitude they cover. The real longitude and latitude, not the mixup on the CGIAR-CSI download page.

Also has rough indication of a few major countries and regions, just to give a general idea of which DEM's you might want for a particular map. This is only a rough indication, because obviously DEM's aren't made to match country borders, but it should be a useful reminder anyway. And don't grumble if your country got left out, because I 'm too lazy to try and cover them all.

Spreadsheet (.ods format) is in the attached zip.

[Gumboots]

I downloaded and installed the latest full-install package, swapped in the latest 32-bit executable (since the 64-bit version is the "main" version these days), and copied in the special color table for RT3 (ELEV_COLORS_0-9999.dbf), and downloaded the GTOPO30 tiles I wanted from the USGS (registering as required). Of course there was the obligatory fumbling around to find some things that had moved since the old version used in the tutorial, but I figured it out well enough. There's just one problem with the output though:



Yep, it's garbage. Even a greyscale map has better precision than that: IrfanView reports only 50 unique colors in the image, and the low
precision is evident when importing the map into RT3. Compare to this MapBuilder heightmap of a roughly similar area, which has 1,700 colors:

Been thinking about this, after playing with test maps. I've just made a rough crop of Europe and exported that from MicroDEM. Resized it to a 1025x1025 TGA for a test map. Imported into the RT3 editor as a game map, using the same modifier values as I used for my previous test map (same size, same area, but from RT3 Map Builder). As far as I can tell, they generate near enough to the same terrain, and I did check pretty carefully.

Much better, right off the bat. It's still visibly inferior to the MapBuilder output though, having only 190 unique colors and looking comparable to a greyscale map of the area. I found that's because the special color table only gives you 1/10 precision, specifying one color for every 10 elevation units. I'm thinking of trying to make higher-precision versions of the color table. Should be easy enough using the series/auto-fill features in Microsoft Excel.

The existing table gives heights every ten metres, up to a maximum of ten thousand metres (about the height of Everest). DEM's do not have 1 metre resolution anyway. They're simply not that accurate. An accuracy of ten metres is pretty good in practice, especially by the time it's scaled down to a 1025x1025 (at most) image that the RT3 editor can use.

190 colours indicates an elevation range from 0 up to 1900 metres. If the area in question doesn't have any terrain higher than 1900 metres (which is a decent height) then AFAICT the output is correct. My second test map includes the Alps, which go as high as 4,800 metres, and a quick check on the image shows it has at least 300 colours (go to save as PNG-8, shows maxed out at 256, so obviously a lot more than 256 colours).