Creating a Map from Digital Terrain Data and Google Maps

[Gumboots]

I was thinking, if Bing uses the same projection as Google Maps there's always the option of using Google to get some grid references, then throwing Bing images on top if you like them better. As long as they have the same zoom settings, it should work.

Google does do the deserts well. However, the thing is that it's not very realistic to have a sprawling railroad in the middle of the desert. Realistically, how many maps are we going to make in the Sahara, Gobi desert, etc.?

https://en.wikipedia.org/wiki/Indian_Pacific
https://en.wikipedia.org/wiki/The_Ghan
https://en.wikipedia.org/wiki/Broken_Hill

For all intents and purposes the places where people live tend to have seasonal changes in vegetation. Whether this is natural as in a summer/winter or a wet/dry, or man-made as in crops planted/harvest etc. an average in my opinion just doesn't look right.

Sure, but we're stuck with an average for RT3 maps. It can't do seasons or harvests.

Incidentally, I've been idly playing with routes through the Alps on that map. The Swiss railways are quite amazing bits of engineering, even if they are nuts and like cuckoo clocks. Anyway, the Simplon tunnel route is done suprisingly well on that map at that scale. Has the right ravines in the right places. Bit of a mongrel trying to get a smoothish route that still has a fairly steep grade (the track leading to the Simplon tunnel has shorter spiral tunnels and all sorts of stuff). I think it'll need careful selection of vertical scale and lots of temporary lake tool hits.

Edit: Meh. On second thought it needs a D9 Cat, dynamite, and a large hammer. RT3 terraforming tools are utter rubbish.

[Gumboots]

Found a small trap with the Google projection. It's easy enough to tile up a big screenshot without having to do all the tedious matching of corners that Google Earth captures require. If you shot the sat view and the map view in pairs, you can use the cities, etc to line up each layer group to the exact pixel. This makes it fast and easy to tile up big images accurately.

The original image is 10,000 x 6,000 pixels, with the idea being that if I make a few of these monsters I'll have a handy stash for whenever I want to make maps. All I'll have to do is crop out the chunk I want and scale it to 1024x1024. Easy.

The catch is the latitude lines. They are at the same spacing when viewed in your monitor, but they are not the same spacing in actual latitude. This is where the Web Mercator (as they call it) deviates from old Mercator (which has constant spacing for latitude lines). What is happening is that the spacing between each latitude line, when viewed at the "20 km" scale, is decreasing by about 1.1% for every line north. The result is that latitude lines near the bottom of the image are spaced about 58 minutes of latitude apart, while at the top of the image they are only spaced about 41 minutes apart.

This is quite a difference, and could cause problems with maps that cover large areas. Fortunately there is no problem in the other direction. Longitude lines are equally spaced at any latitude, so no worries on that score. The different spacing for latitude lines could be ignored for small areas, and can be corrected for large areas by slicing the image horizontally and vertically scaling each slice to suit, before merging the slices into one humungous wallpaper thing.

If anyone was wondering why I was worried about Bing not providing grid references, this is a good example of why. If there were no grid references available I wouldn't have spotted this problem, and wouldn't have known how to fix it. Now I'm aware of it, I can easily compensate when necessary.

(Presumably it does the same thing going south of the equator, with latitude line spacing decreasing towards the Antarctic, but I haven't checked that yet, and presumably there is some sane reference spacing at the equator, but I haven't checked that yet either.)

[Gumboots]

Couple more tests. These are basically just terrain checks.

The idea kicking around in my head is to do an Orient Express map that covers the entire route from the French coast to Istanbul. That's too big a distance to do it all on one map with reasonable detailing, so it makes sense to split it into two maps and two scenarios, and it makes sense to cover the route of the Simplon Orient Express* as well as the original (both were running simultaneously in the 1920's and 1930's). This should give more scope for scripting and more replay value.

So, have made a few more test maps. These are both to the same scale (1.18 pixels per real life mile) and are sized at 832x512 for the western Stage 1 and 832x640 for the eastern Stage 2 .I threw a few quick bucket-fill-above's at them to make assessing the terrain easier. Both of these imports are done with overall height modifier set at the default value of 1, ditto for the mountaintop modifier, but 0 for smoothing.

This works brilliantly in some areas. The Danube drainage basin from Belgrade down to the Black Sea is just about perfect as is, important mountain passes are where they should be, and most of the terrain is workable already. These are things that get lost if you import with smoothing. Smoothing is fine in open spaces, but tends to wreck things like narrow gorges, which then makes life harder when you want to cut them in again and run things through them.

It doesn't work quite so well in the Alps. It still works for keeping gorges where they should be, but the peaks are just over the top ridiculous. I figure this is easier to fix than having to fix borked gorges. It should be just a matter of grabbing the smoothing tool with whatever size brush, and running it over the worst offenders while leaving the important stuff alone. The lower areas are still fine though. The Rhine valley looks right, as does most of the map.

I'll play around a bit more with height modifiers and see if I can come up with anything better.

*The Simplon Orient Express swung south from Paris to Lausanne, through the Alps and the Simplon Tunnel to Milan, then across to Zagreb and Belgrade, then on to Sofia and Istanbul. This is the train that Murder on the Orient Express is set on.

Belgrade-Sofia-Istanbul was the route taken by the original as well, except of course it ran north of Switzerland and came through Vienna first. The extra track that ran into Romania never had a rail connection to Istanbul (but did have a ferry connection from Constanta).

[Gumboots]

Note: After doing some more research, and figuring out some more details about how RT3 works, I no longer recommend this procedure.
The short version is that heightmaps should not be resized after being exported from MicroDEM, unless the "resizing" is limited to cropping off a few pixels to get it to 64+1. Stretching or squashing the heightmap, even slightly, will create artifacts in the game map.

Code: Select all

I've done a few more experiments, and have figured out something useful. !*th_up*! 

My usual procedure is to export the DEM as .BMP, then resize it in Photoshop to get the proportions matching for vertical and horizontal scale (DEM's are always a bit out for scales on those two axes). Doing this gets the final .TGA in scale overall, so the map is a free from stretching or squashing as it is possible to get it. I know a lot of people don't bother with this resizing, and just use maps however they get spat out, but I like doing it.

Anyway, this causes problems around coastlines. Resizing the image results in blurred pixels where the pure red of the ocean joins the blue/green of the elevation pixels, giving a fringe of pixels that don't have valid values for the height scale. Getting around that is easy enough. I used to frig around with more complex ways of doing it, but the simplest way is to use the Nearest Neighbour algorithm.

The Nearest Neighbour algorithm always gives pixels that are either pure red or completely transparent, meaning the new layer will make a clean ocean regardless of scaling. This part is fine, but the catch is that [i]Nearest Neighbour is total crap for resizing the blue/green elevation pixels[/i]. This is where I've had to get cunning.

To get the best results, [i]duplicate the .BMP before resizing it[/i]. This way you can use two different resize algorithms on the two separate images: Nearest Neighbour on the image you'll take the red ocean from, and the default Bicubic on the image you'll take the blue/green elevation pixels from.

Why? Well, if you use Nearest Neighbour on the whole thing in one go it will import just fine, but if you are making a map with fairly gentle slopes the result without smoothing will be a series of terraces, rather like shallow mesas stacked on top of one another. It took me a while to realise the cause, and I'm pretty sure it's down to error margins on the DEM's themselves.

The table of colours we use with MicroDEM is theoretically capable of a 1 metre elevation resolution, but DEM's are only accurate to 10 or 15 metres elevation. This means if all the raw data gets shoehorned into blocks of 10 or 15 metres, then that shoehorning into those blocks would create the stacked terraces when imported into RT3. It has to, if the raw data is rationalised so that the finished DEM has all its pixels within the error bars. This seems to be what is happening. I found this out when I figured out Nearest Neightbour was the solution to ocean edges, and tried it on the whole thing. Terraces everywhere, if you don't use the default smoothing of 1 and you set it to 0.

Ok, so how was I getting good results on the trial Orient Express maps without smoothing? That was just before I figured out the Nearest Neighbour trick, so all the elevation pixels on those maps were resized with Bicubic. Having thought about that too, it became clear that this would also blur the elevation pixels at boundaries between two colours but, and this is the tricky bit, [i]in the case of elevations it actually appears to be an advantage.[/i]

What it must be doing is effectively injecting intermediate values of the "bluescale" in between the values that would ordinarily result in the terraced effect. In other words, it's automatically applying a low level smoothing by interpolating extra elevation values, but without going the full melted blob that the RT3 default smoothing gives. It is basically a way of getting a fractional smoothing value that RT3 ordinarily doesn't allow on import, and the amount of it will be proportional to the amount of blurring caused by the resizing the .BMP.

There's another minor catch here, of course, in that sometimes the intermediate values it injects are not quite valid for that pixel, meaning it produces noticeable artifacts when imported. These usually appear as very small and very distinct needle-like peaks on an otherwise smooth surface. These are not very common, and are easy to spot and easy to knock back when spotted. Even allowing for such artifacts, I still think the overall result is better for maps where you want accuracy on complex terrain.

The upshot of all this is that I've nailed down my procedure like this:

1/ Select the cropping limits for the map, based on whatever latitudes and longitudes will give an equal vertical and horizontal scale once the image is forced to the desired multiples of 64. This requires a bit of calculation, but nothing particularly difficult.I use a calculator and this tool to figure it out: http://www.csgnetwork.com/degreelenllavcalc.html

2/ Before importing the DEM into MicroDEM set the default map size (Options > Maps) to whatever you want for the larger dimension of finished TGA heightmap. The reason for doing this is it means MicroDEM will automatically export the temporary .BMP with it's larger dimension correct for the finished TGA. Note that this will [i]only sort out the larger dimension[/i]. You cannot use MicroDEM's default map size to force a different aspect ratio. That still has to be done by resizing in Photoshop.

3/ Now import the single DEM or multiple DEM's (which will automatically be merged). Then right click on the image > Modify map area > Keyboard corners > type in whatever values give you your correct cropping limits (one 5 degree DEM equals 6000 units in the "Keyboard corners" inputs). Hit enter. Hey presto, map cropped to the right limits.

4/ Make sure you are using the correct colour scale (right click on the image > Elevation colours > Colours from table > ELEV_COLORS_0-9999.dbf).

5/ File > Save image. You now have a BMP with one dimension correct. (0!!0) 

6/ Open Photoshop or whatever beastie you use. Open the BMP. Duplicate it now.

8/ Resize the incorrect dimension to make it fit the 64+1 thang. Resize both images, but use the two different resize algorithms: Nearest Neighbour on one, and Bicubic on the other.

9/ For added insurance, on the image that was resized Bicubic you can ditch the red channel (Image > Adjustments > Channel mixer > Red to -200). This will make sure that any traces of red that may have bled past the ocean will be removed from any visible pixels, leaving just the desired blue and green channels.

10/ Select the red from the Nearest Neighbour image (tolerance set to 0) and copy paste that selection onto the other (Bicubic-resized) image.

11/ Do a final check around coasts for tiny islands and indentations that are too small to be useful. Either paint them out with pure red on the red layer (if trying to get rid of a small island) or crop them out of the red layer (if wanting to get rid of a tiny patch of useless ocean inside the shoreline).

12/ Now you can save the thing as 24 bit TGA and import it into RT3. Instant game map. Mess around with height modifers until you get something you like.

This is all a lot easier to do than it is to read about, once you've done it once or twice. It's actually a pretty simple procedure and goes quickly. !*th_up*!

[Gumboots]

Right, more experiments.

Just for fun I whipped up a small map of the Parenzana route, just to see if it had potential for a workable scenario. After a bit of thinking about the results that RT3 generated, combined with some PM chatting with RoR, I figured I should run the ultimate test for accuracy.

The natural resolution of standard DEM's is 1200 pixels per degree of latitude or longitude. So, to see what RT3 is capable of making from a DEM under optimum conditions, the way to go is a simply crop out a 1025x1025 chunk of DEM and save it as a heightmap. This equates to a real life area that is 0.854 degrees in latitude and longitude ranges.

DEM's are tiled at even 5 degree increments, so for this test I grabbed 39_03 (the DEM for the general Parenzana area) and cropped out my 1025x1025 chunk, from 45N to 45.854N and from 13.4E to 14.254E. This was then saved as BMP, and MicroDEM automatically saved it as 693x1025. I assume this is because MicroDEM is coded to use some reference geoid when saving cropped DEM's, and thinks this is the correct aspect ratio for that general latitude. According to the lat/long calculator I use the aspect ratio should be a bit different, and for RT3 it needs to be based on 64+1 anyway, so I scaled it out to 705 wide using the procedure detailed in the previous post.

Ran a few test imports to RT3 editor, and for now have settled on an overall height modifier of 2.785, with a mountaintop modifier of 1, and smoothing of 0. The overall height modifier was chosen to give 2 RT3 height units per metre of elevation, at the known elevations of Baldasi and Grosnjan. This means the rest of the map can be quickly checked for accuracy of heights, as long as you have something like Google Earth to give you elevations and you know how to multiply by 2 (which is something I happen to know). The results are educational.

1/ In large areas of shallow slopes, the DEM data will produce a series of flat terraces. These are set approximately 77 height units apart. In other words, a standard DEM will, under optimum conditions, only give the heights of shallow slopes in about 40 metre steps. For anyone who still thinks in feet and inches, that's about 130 feet. This is as accurate as the standard 3 arc second DEM series gets. Anything that looks more accurate than that is probably an illusion.

Edit: The above turns out to not be correct. It was the result of the faulty elevation table that was supplied for MicroDEM. With a correctly functioning elevation table it is possible to get a resolution of 1 metre (although that small an interval is not actually useful).

2/ How can we tell it's probably an illusion? Well, on a shallow slope you will have large areas of the same colour pixels on your DEM. This is why it generates terraces when imported into RT3. The terraces are the areas in between the contour lines defined by the pixel colours. In real life those areas are sloped, but when limited by a pixel's colour they all turn out the same until you get to the next step.

3/ This is on a maximum sized test map of a comparatively small area. If making a map of a larger real life area, or making a smaller map of it, the resolution will be reduced. This means that, in general, on most maps, the heights will not even be accurate to 40 metres.

4/ Obviously the ocean is always accurate, since it's height is always 0. Not even DEM's and RT3 can screw that up. However, low elevations that are not 0 are out of whack by quite a bit. For example, on this test map I know that Livade has an elevation of approximately 10 metres. That would be 20 RT3 height units at this scale. However, the map generates with the valley floor at a height of 77 units. In other words the DEM is telling RT3 that the valley floor is roughly 30 metres higher than it really is. This comes back to the 40 metre resolution on shallow slopes. It knows it aint 0, but it can't tell it's only 10 metres, so it rounds up to 40. Genius stuff.

5/ So where are we, then? The standard DEM series we all know and love is built of out 130 foot tall Lego blocks. We may have a lovely .dbf file that theoretically allows 1 metre height resolution, but there's no way you will get that from the standard DEM series. The standard DEM series will only use 1/4 of the .dbf file's capabilities, even under optimum conditions, and far less than that under less than optimum conditions.

6/ This is without using smoothing on import. Using smoothing on import will exponentially borkificate the 130 foot Lego blocks, and your resulting heights will be anyone's guess.

7/ Ok, is it possible to get more accurate results? Yes, in theory, if you're lucky. There are all sorts of data floating around the world of science and engineering. Surveys of all sorts of things are routinely done to an accuracy way beyond anything we'll ever need for RT3. The hard part is getting the right data, in a usable form, and without forking out hundred of thousands of bucks for it.

8/ Or if you're a hard core lunatic who wants a map to be as good as it can get in a few critical places, it would be possible to start with the DEM's output, import that into RT3, and then use the RT3 terraforming tools in combination with Google Earth to wallop a few chunks of the map into shape. A somewhat daunting prospect, but frankly no worse than doing RT3 rivers anyway.

9/ Having said all of that, I will say that the results at this large a scale are still quite usable, and the main features are clearly recognisable. The 1.06 test map is attached. This has a few basics thrown in, but it only currently suitable for sandboxing.

Edit: Zip removed. The following posts will explain why.

I'll get onto testing the old RT3 Map builder as well, and see how that compares for accuracy.

[Gumboots]

Ok, ran two more tests.

1/ Tried the old RT3 Map builder. It's rubbish for this, since the biggest image it will create for this area is 102x120. RT3 will import that as 129x129, and obviously at that scale the available detail is completely hopeless. It also makes a map which is too small to be useful, and it's stretched out of shape. Blah blah etc etc. Lesson #1: this is why people went to using DEM's.

2/ Went looking for better DEM's. Found the ALOS AW3D30 set. These have much better resolution than the standard CGIAR-CSI set. Their DEM's have 3600 pixels per degree instead of 1200, and their heights are good to less than 5 metres.

Sounds promising yes? Much better resolution should equal much better generated maps, right? You can see where this is heading...

No chance. Forget it. Despite the ALOS set's better much better height resolution, when you import it into RT3 with the same height modifier it generates the same 77 unit terraces. IOW, this appears to be a limitation of RT3 itself, regardless of how good your DEM's are. Even if you had DEM's accurate to three fifth's of knee high to a gnat, after it had been run over by a steamroller, RT3 would still give you 130 foot tall Lego Blocks.

Yay for RT3. We luvs it, we does.

Now it is possible, I suppose, that this is a fault with the height table we usually use, so I could try some other height tables and see what explodes. Famous last words.

---

Ok, tests #3 and #4 others: Tried other height tables. RT3 does not like them. Makes stupendously awesome cliffs though.

Test #5: Tried saving as 32 bit TGA instead of 24 bit. No difference (shouldn't be anyway).

Test #6 and #7 and whatever: Tried saving the same crop from MicroDEM with other colour tables, and doing a comparison.

Here's the real problem!

Saved with a couple of MicroDEM's default colour tables I get an image with around 100 colours. Highest point on the map with the 2 units/metre height modifier is just over 1,000 units, which equates to 500 metres or so. 100 colours into that would give a 5 metre height resolution, which is what it is supposed to have (using the ALOS data here).

The same crop, from the same ALOS DEM, but saved as BMP with the famous made-for-RT3-it-does-a-thousand-metres-with-one-metre-resolution ELEV_COLORS_0-9999.dbf gives...

wait for it...

16 colours.

Which, when you do the maths on it with a 500 metre high point, gets you back to the 130 foot tall Lego blocks. This is even before anything gets imported into RT3. It's straight out of MicroDEM and opened in Photoshop.

Conclusion: that custom table, which we have to told to rely on, aint working. It may theoretically allow a thousand graduations, but when it's run though MicroDEM it will dump 85% of the heightmap information and leave you stuck with those 130 foot tall Lego blocks.

It's not the fault of MicroDEM.
It's not the fault of RT3.
It's the fault of ELEV_COLORS_0-9999.dbf.
That is where the information loss is happening.

Question: did anyone ever test that thing to see if it actually did do an accurate job of heightmaps? Because it sure doesn't seem like it was ever tested.

Next question: does anyone have a clue how to fix it so it actually does what it says on the tin? Because if it can be fixed the results should be pretty impressive.

Edit: Found something which may or may not be relevant. The MicroDEM help pages (which probably haven't been updated since the Palaeolithic) have this tasty bit of info: https://www.usna.edu/Users/oceano/pguth ... _table.htm

Only one problem. That option no longer exists in the current 64 bit build, or at least not in that location. So, I went hunting through the entire GUI. Still couldn't find it. Looked all through the Help section. Nope, no luck there either.

The reason this may be relevant is that if colour tables will only handle 256 colours, and if the upper limit of elevation is around 10,000 metres, then 10,000/256 is getting pretty close to 40 metres, which just happens to be the height of our rotten Lego blocks. It's looking very much like MicroDEM is compressing things back to 256 colours for Mount Everest, and anything shorter than Everest only gets a handful of colours. And, as far as I can tell at the moment, there doesn't appear to be any way around it.

[Gumboots]

Ok, tried another test. Might as well while I'm up for this stuff. Grayscale.

Made a grayscale heightmap of the same area, using ALOS data again. Imports just fine. Generates reasonable terrain, but with caveats.

First point is that is does very well indeed with the higher areas. Checking the DEM itself told me the highest elevation was around 1,300 metres, and sure enough when I checked that point against some of the other mid-altitude points it was scaling to the right height. It was doing better in this respect than the custom 9999.dbf, which thought the high point was only around 500 metres. The grayscale heightmap also imported with smoother terrain and fewer artifacts.

Lower areas are still too high, by a factor of about 4. IOW, by about the same amount as the custom 9999.dbf generates. The other drawback with using grayscale is that it won't do oceans automatically. You have to put them in yourself, but putting in oceans isn't hard (one of the least tedious parts of mapmaking) and overall, as things stand at the moment, grayscale does a better job of the terrain.

[Gumboots]

Welldang and Hornswoggle. Darntootin and Ornery.

I'm in one of my determined moods, so decided to check things myself. Set up a basic PSD with rectangular chunks on it, with red borders between them to separate them with ocean. It looks like a block of chocolate when imported into RT3, and the clear gaps keep things easy on my brain.

So, start off with RGB 0 0 255, step down to RGB 0 0 239, then keep going down in steps of 16 until I get to RGB 0 0 15. That's the first two rows. Next two rows start at RGB 0 1 255 and go down to RGB 0 1 15. The two after that are RGB 0 2 255 to RGB 0 2 15. The final two rows are obviously RGB 0 3 255 to RGB 0 3 15, again in steps of 16 Blue.

So, results.

With a bit of rounding error, each series of two rows steps up by 11 or 12 in RT3 height units for every additional 15 units of Blue.

However, the next series doesn't start at RGB 0 X 0, so the next test to do is RBG 0 0 0 to 0 0 15, then 0 1 0 to 0 1 15, etc and see what that generates.

And, when you start adding green in, it acts as a straight multiplier on the highest value in each series. So RGB 0 0 255 is 178 RT3 height units. RGB 0 1 255 is 357 RT3 height units, which allowing for rounding error is double the previous value. Then it goes to 536 RT3 height units for RGB 0 2 255 and 716 RT3 height units for RGB 0 3 255. This is clearly a straight progression: 178 to 178x2 to 178x3 to 178x4.

Now for the fun part. RGB 0 3 255 is exactly 4 times the height of RGB 0 0 255 when imported as part of a heightmap. However, that custom table for MicroDEM doesn't even have either of those RGB values in it. The closest values it has are 0 0 250 and 0 3 252.

It assigns 0 0 250 to the 240-250 metre height range, and it assigns 0 3 252 to the 1010-1020 metre height range. This is actually pretty close to the 4:1 ratio that 0 0 255 and 0 3 255 give for actual RT3 heights, but it's not quite there and it's rather odd that the table omits values that RT3 will use. You'd think that if it was counting up in chunks of 10 metres, and if RT3 is counting up in chunks of 1 RT3 unit, then all values should be used. There definitely appears to be some degree of error. More tedious checking to follow.
---------------------------------
Edit: Have found a pattern. Will have to confirm it by checking all relevant hex codes but it's pretty clear already.
Obviously it starts at 0 height for RGB 0 0 0, which is why black works for oceans if you are using a greyscale heightmap. It then goes like this:

Code: Select all

RT3 Height  Red Green Blue        RT3 Height  Red Green Blue        RT3 Height  Red Green Blue      RT3 Height  Red Green Blue
      0      0    0     0             179      0    1     0             358      0    2     0           537      0    3     0
      0      0    0     1             179      0    1     1             359      0    2     1           538      0    3     1
      1      0    0     2             180      0    1     2             359      0    2     2           539      0    3     2
      2      0    0     3             181      0    1     3             360      0    2     3           539      0    3     3
      2      0    0     4             182      0    1     4             361      0    2     4           540      0    3     4
      3      0    0     5             182      0    1     5             361      0    2     5           541      0    3     5
      4      0    0     6             183      0    1     6             362      0    2     6           541      0    3     6
      4      0    0     7             184      0    1     7             363      0    2     7           542      0    3     7
      5      0    0     8             184      0    1     8             364      0    2     8           543      0    3     8
      6      0    0     9             185      0    1     9             364      0    2     9           543      0    3     9
      7      0    0    10             186      0    1    10             365      0    2    10           544      0    3    10
      7      0    0    11             186      0    1    11             366      0    2    11           545      0    3    11
      8      0    0    12             187      0    1    12             366      0    2    12           546      0    3    12
      9      0    0    13             188      0    1    13             367      0    2    13           546      0    3    13
      9      0    0    14             189      0    1    14             368      0    2    14           547      0    3    14
     10      0    0    15             189      0    1    15             368      0    2    15           548      0    3    15

So basically it's going in groups of three, with two RGB values out of every three giving the same in-game height in RT3 units. That means going up to 0 5 255 will be more than enough to fill up the 1,024 available values in the table. That would nominally allow 1,536 values, but then you have to ditch 1/3 of them because they duplicate for height anyway, so you end up with 1,024. And then you have to take another one off anyway because you have to leave a slot for pure red.

Something like this should do the trick:

Code: Select all

RT3 Height  Red Green Blue        RT3 Height  Red Green Blue        RT3 Height  Red Green Blue      RT3 Height  Red Green Blue
      0      0    0     0             179      0    1     0             358      0    2     0           537      0    3     0
      1      0    0     2             180      0    1     2             359      0    2     1           538      0    3     1
      2      0    0     3             181      0    1     3             360      0    2     3           539      0    3     3
      3      0    0     5             182      0    1     4             361      0    2     4           540      0    3     4
      4      0    0     7             183      0    1     6             362      0    2     6           541      0    3     6
      5      0    0     8             184      0    1     7             363      0    2     7           542      0    3     7
      6      0    0     9             185      0    1     9             364      0    2     9           543      0    3     9
      7      0    0    11             186      0    1    10             365      0    2    10           544      0    3    10
      8      0    0    12             187      0    1    12             366      0    2    12           545      0    3    11
      9      0    0    13             188      0    1    13             367      0    2    13           546      0    3    13
     10      0    0    15             189      0    1    14             368      0    2    14           547      0    3    14

[Wolverine@MSU]

Saved with a couple of MicroDEM's default colour tables I get an image with around 100 colours. Highest point on the map with the 2 units/metre height modifier is just over 1,000 units, which equates to 500 metres or so. 100 colours into that would give a 5 metre height resolution, which is what it is supposed to have (using the ALOS data here).

The same crop, from the same ALOS DEM, but saved as BMP with the famous made-for-RT3-it-does-a-thosuand-metres-with-one-metre-resolution ELEV_COLORS_0-9999.dbf gives...

wait for it...

16 colours.

Which, when you do the maths on it with a 500 metre high point, gets you back to the 130 foot tall Lego blocks. This is even before anything gets imported into RT3. It's straight out of MicroDEM and opened in Photoshop.

Conclusion: that custom table, which we have to told to rely on, aint working. It may theoretically allow a thousand graduations, but when it's run though MicroDEM it will dump 85% of the heightmap information and leave you stuck with those 130 foot tall Lego blocks.

It's not the fault of MicroDEM.
It's not the fault of RT3.
It's the fault of ELEV_COLORS_0-9999.dbf.
That is where the information loss is happening.

Question: did anyone ever test that thing to see if it actually did do an accurate job of heightmaps? Because it sure doesn't seem like it was ever tested.

Next question: does anyone have a clue how to fix it so it actually does what it says on the tin? Because if it can be fixed the results should be pretty impressive.

Edit: Found something which may or may not be relevant. The MicroDEM help pages (which probably haven't been updated since the Palaeolithic) have this tasty bit of info: https://www.usna.edu/Users/oceano/pguth ... _table.htm

Only one problem. That option no longer exists in the current 64 bit build, or at least not in that location. So, I went hunting through the entire GUI. Still couldn't find it. Looked all through the Help section. Nope, no luck there either.

The reason this may be relevant is that if colour tables will only handle 256 colours, and if the upper limit of elevation is around 10,000 metres, then 10,000/256 is getting pretty close to 40 metres, which just happens to be the height of our rotten Lego blocks. It's looking very much like MicroDEM is compressing things back to 256 colours for Mount Everest, and anything shorter than Everest only gets a handful of colours. And, as far as I can tell at the moment, there doesn't appear to be any way around it.

I think you're right Gumboots. I never tested the coloring scheme for vertical resolution, but I made a DEM of the coast of the Netherlands and it looks like colors only change every 40 M (see attachment). Even checking the box in an earlier 32-bit version (V12) doesn't work. There is a workaround that will be limited only by the range of elevations on a particular DEM (or subset), and that is to make an Elevation Table with 256 colors covering only the range of elevations on a map. I made an Excel worksheet to do just that when I was exploring mapping of the sea floor for import into RT3.

[Gumboots]

Oh bother. I was hoping I was wrong.

Greyscale works just fine for height res, in that at least it always gives you the 256 heights that it should, assuming a full-size test that hasn't lost information by resizing the DEM before exporting, and assuming the relevant area has sufficient elevation range to make use of 256 shades once the DEM 's accuracy is taken into account. The only real problem with greyscale is that it doesn't handle oceans and other low areas well, so it'd be cool if the "bluegreenscale with added red oceans" could be made to work.

I like the idea of having a table that deals with the actual height range if we are going to be permanently limited to 256. I'd already thought of something similar, since it's obviously what greyscale does. My initial thought was that even if we can get 1,024 values working somehow there could still be value in having at least two tables: one for everything-including-the-Himalayas (0-10,000 metres) and one for just-about-everything-else (0-5,000 metres). Seems easy enough in principle. You could just use the same RGB values and double the height range columns. And, having looked into this now, in general terms I can see how you could adjust things to incorporate bathymetry. The same principle has obvious applications if anyone wants to include something like the Dead Sea.

I've even thought that there could be a great way of doing rivers. Laying rivers through gnarly terrain is one of the all-time worst RT3 map making jobs. However, if we have all the relevant RGB values and the known heights they generate then in principle you could do some quite useful trickery in an image editor. For example, say your resizing down to a usable game map has lost sufficient information to lump up the course of a river. If you know the correct heights for it in a few places, then you could do something like lay down a wiggly line in Photoshop and apply a simple colour gradient to it before saving the TGA. Once imported into RT3, this should cut the river through at the correct height. After doing that !!!&@%# Latvia map, I'm all for anything that makes rivers easier.

---

Just thought of something else. There's no need for us to be restricted to 256 height values, even if MicroDEM is. The solution is simple.

All that's required is to whip up a table that covers from 0 to 9,000 metres (Everest is 8,850) in 5 metre increments (accuracy of ALOS .tiff's). That table can then be divided into 8 tables, each of which can easily cover a 1,200 metre range while still leaving space for a few odds and ends. Each table then gets a simple bottom end value for heights below Z=whatever, and a top end value for heights above Z=whatever's-brother. These can be colour coded to be blindingly obvious in MicroDEM (pure green would be a good choice).

So you'd open your DEM, crop it where you wanted it cropped, and take a look with the first table hooked up. Export the thing as a .BMP. If you can see any lime green pixels, switch to the next table in the series before exporting another .BMP. Repeat as necessary.

Open all your .BMP's in your image editor. Stack them as layers in the right order. Use the magic wand on 0 tolerance and no anti-alias to knock out the unwanted base colour on each layer. Flatten image. Save as .TGA.

Import that into RT3 and you should end up with a model of Everest, from sea level to the peak, in 5 metre increments. Easy.

---

Also, I thought I'd see if it was possible to get negative height values from an imported heightmap, since we know RT3 will make negative heights with its terraforming tools. The obvious ploy was to try adding some red channel into the blue-green height scale. Turns out this has no effect whatsoever.

0 0 255 gives height of 178 units, which we already knew, but 255 0 255 also gives height of 178 units. Tested 0 22 0, and that gives height of 3,942 units. Tested 222 22 0, and that also gives a height of 3,942 units. Looks like the red channel is basically a nothing channel except when it is pure red, in which case it automatically makes oceans. This unfortunately means we cannot get negative heights by import, but on the other hand it's a useful bit of idiot-proofing if you happen to blur the edges of your red before importing. Those pixels will just be assigned whatever height value matches their blue and green channels, and the editor won't get indigestion.

Went looking for the upper limit too. That turns out to be 10,000 height units, which equates to an RGB value of 0 55 206. Anything further up the scale than that, like 0 55 232 for example, will just be capped at 10,000 height units anyway. Which is kinda interesting because it loosely implies they were thinking in SI units in relation to heightmaps, and that one RT3 height unit is meant to represent 1 metre elevation in real life. This would make sense, since on a very large map a height modifier of 2 or 3 gives realistic-looking terrain if you assume 2 or 3 RT3 height units per metre, so on a smallish map (like most of the defaults) a height modifier of 1 should give decent results at 1 unit = 1 metre.

By the way, this is a completely different unit to the RT3 modelling unit for assets, which equates to 10". They apparently did their locomotives in feet and inches (makes sense in terms of US plans) but their heightmaps in SI units (which makes sense in terms of DEM's).

[Gumboots]

Woohoo!

Nailed it. Took a bit of head scratching, because it's not the way I usually think of doing numbers, but it turns out that the formula behind the blue-green height scale is really simple. Stunningly doofus simple. Your cat could do it.

All it does is start at 0, of course, with an RGB value of 0 0 0.
It then proceeds one step at a time to 0 0 1, then 0 0 2, etc, all the way up to 0 0 255.
Then it starts again, with 0 1 0 taking the place of what would have been 0 0 256 (if the hex scale went that far).
It then goes 0 1 1, 0 1 2, all the way up to 0 1 255, at which point it starts again, at 0 2 0.

So, if you take the position on the scale, which for 0 1 0 is 256, then all you need to do to get the RT3 height value is multiply that by 0.7 and round it down to the nearest integer.

The upper limit is 10,000. That has a hex code of 0 55 206. So its position on the scale is 55(256)+206 = 14,286.
Multiply that by 0.7 and you get 10,000.2 and if you round that down as you normally would anyway you get 10,000.

Taking another example, there's hex code 0 3 127. Its position on the scale is 3(256)+127 = 895.
Multiply that by 0.7 and you get 626.5. I would normally round that up, but RT3 rounds it down, giving a height of 626 units.

This works perfectly for any value between 0 0 0 and 0 55 206. Which is a vast relief, because now I won't have to waste any more time screwing around making patches of colour in Photoshop and loading them into RT3. I can now do the entire humungous beastie via basic arithmetic, which is a lot faster and easier.

Edit: Lolz. Maybe the cat is smarter. I just twigged to how it could be written succinctly.
If you take the green channel value as G, and the blue channel value as B, the formula for finding height value for any value of G or B is:

ROUNDDOWN(0.7*((256*G)+B);0)

in standard spreadsheet syntax.

And another one. Got the formula for greyscale too. That one is even simpler.

Height = ROUNDDOWN(0.7*32*V;0)

where V equals the RGB value (which for greyscale is the same number for all three channels).

Note that this one tops out at 5712 height units, instead of the 10,000 units allowed by the Green-Blue scale.
Also note that all game maps imported from a greyscale heightmap will have a highest point of 5,712 units.

That's just the way greyscale works, and it means importing from a greyscale heightmap will require massively different height modifiers compared to importing from a green-blue scale heightmap. Also note that if you accidentally mix a red ocean with a greyscale heightmap it will multiply all heights by a factor of 8. Which tends to be rather surprising if you're not expecting it.

---

Just found out something. The first series of the blue scale, from 0 0 1 to 0 0 255, work perfectly as long as there is at least one pixel of pure red somewhere on the heightmap. They need that as a reference. If you make a patch of any of those 255 colours, without at least one square of red for a reference of 0, it will result in a height of 0. Not only that, but in the case of pure blue (0 0 255) it will automatically be made into an ocean! Just like as if it was pure red. Go figure.

All the other green-blue colours, from 0 1 0 up to 0 55 206, work perfectly with or without a pixel of red for reference. As soon as there is at least one unit of green, the red is no longer necessary. The obvious catch is that without that reference pixel the heights from 0 to 178 won't be available. This wouldn't matter for a map of an inland area that had a minimum altitude up over 178 anyway, but could be a consideration for low-lying but dry areas. If necessary it would be easy to incorporate just one pixel of red somewhere inconspicuous, and just fill in the divot later (lake tool reset would pull it into place easily).

Anyway, I made a proof of concept tile to see what this stuff is capable of. The TGA was 321x193 and looks like the attached .jpg. The result it generates in RT3 is shown in the screenshot. It's a perfect ramp, on an even gradient of about 1 in 85. No terracing at all. Even slope from top to bottom.

321_ramp_test.jpg (3.37 KiB) Viewed 21555 times

[Gumboots]

Ok, tried a test on an actual map.I started by making a custom elevation table with 5 metre resolution. Cropped a DEM, and exported it at its default resolution (ie: no resizing losses). Turned it into a suitable TGA (513x833) and imported into RT3 editor with an overall height modifier of 4.5. Hey presto, one map. This is an area .231 degrees north to south and 0.207 degree east to west, with a scale of about 52 pixels per mile. IOW, each pixel is about 30 metres (100 feet) square.

This appears to be meshed at around 10 metre height resolution. This is a lot better than the 40 metre resolution we had before, but still not as good as it should be. Since I did the ramp test I know RT3 is capable of utilising all 14,286 possible height values, with height resolution down to 0.7 metres, so I checked the number of colours in the TGA. There are 32 colours.

The maximum elevation in the cropped area is around 280 metres, so there theoretically should be around 55 colours (give or take a few). IOW, the image as exported from MicroDEM only has about half the colour range that it should have. Not sure why at the moment, but will keep investigating. It's obviously some sort of problem with the custom elevation table, and hopefully will not be too hard to track down.

I did check exporting the image from MicroDEM with a few of its standard elevation colour options. These come out with 48 colours, which is about where they should be if resolution is 5 metres.

The game map is attached if anyone wants to check it out.

[Gumboots]

Yay! Got it working.

Turns out that when the MicroDEM help pages helpfully say that elevation .dbf files can have 1,024 values, what that really means in English (or any other language, come to think of it) is that when viewed in OpenOffice or a similar app, the .dbf can have a total of 1,024 cells used for data, and that includes the headings of each of the four columns.

Edit: This is not quite right, because the elevation tables have 5 columns instead of 4 columns. I blame lack of sleep at the time.
However, the key point is the next one, about the limit of 255 heights. Once the header row is taken into account, 255 heights require a total of 256 rows. This is the actual limitation, and equates to 1,280 cells.

IOW, you can have up to 255 heights. If you attempt to enter more heights, what MicroDEM does is drop every second one and do a recount. This is why the previous test turned out to have 10 metre res: that one had 256 heights, so every second value was being dropped. For this test I knocked the upper limit back to 1,250 metres, which reduced the number of heights to 252 (250 for 0 to 1250, plus one for below 0 and one for above 1250) and suddenly they all work just like they should.

This also explains why the old 0-9999.dbf only achieved 40 metre height resolution: it had 1,003 heights plus the one extra line for headers, so 5,020 cells in total. MicroDEM ditched half of the height cells and found there were still 2,013 cells, so ditched half of them again and then found there were only 1,255 cells left. It was happy with 1,255 cells, since that was less than 1,280, but of course by that time it only had 1/4 the resolution it was meant to have. Result: 130 foot tall Lego blocks.

The good news is that it is now clear that my cunning plan of splitting things into a range of sheets, to handle every elevation from 0 up to the peak of Everest in 5 metre slices, will work. And, if you import a heightmap to RT3 and want to tweak some part of it slightly, you will be able to do that with an image editor and a colour picker if the game's editor is making things difficult for you.

Screenshot of the new map:

The exported image has 56 colours. One of those is red for the ocean, leaving 55 for the heights. 55 heights times 5 metres = 275 metres maximum elevation, plus up to another 4.99 before it would bring in another colour, so highest point on the map is somewhere between 275 and 279.99 metres, which is spot on where it should be.

That was made with the same cropped area as the previous test and at the same native DEM resolution, but with an extra 10% on the RT3 overall height modifier. It's still as smooth as silk, even with 0 smoothing applied. Zip is attached, for comparison with the previous one. Doubling the height resolution makes a big difference. I'll get onto making the rest of the .dbf tables so we can cover the rest of the world.

[RulerofRails]

Good work. That one looks very nice. The foot of the hills especially look far better to me than on any typical map.

[Gumboots]

Yup, I think we're on a roll now.

I gave it some more thought before that last test. The RGB scale is based on multiples of 8 anyway (256 possibly values for each channel) so it all works really well if I assign the 5 metre elevation contours in steps of 8 RGB height units. This allows using 8 elevation tables to cover everything from sea level to just past the peak of Mount Everest, with a little bit left over before hitting the game's hard-coded height limit. If some nutter wants to map Everest and then build a hotel on top, this scale would just allow them to get away with it.

It's easy on the brain too. I've attached a spreadsheet to show how it works.

BTW, I'm kicking myself that I didn't figure this stuff out before doing that Latvia map. I could easily have had the heightmap done to 2 metre resolution instead of 40. That would have made those !%@$@ rivers and lakes a lot easier.

[Gumboots]

I made the rest of the tables, and gave them a full testing. They work.

Obviously, the way to test them is with Mount Everest. Aint nothing taller than that. So, I grabbed the DEM for that area and exported 8 BMP images, using one elevation table after another. Stacked the BMP's in Photoshop, knocked out the green pixels, and saved as a TGA heightmap.

Imported that into RT3, and it looks like this...

I've attached the TGA in a zip, so anyone can play with it themselves.

Ok, so how did I make this beastie? I used the new elevation tables which are in the other zip.
They come with a readme file, and you are advised to read it, but I'll describe the procedure in the next post.

Edit: Just for fun, I threw a satellite shot on it.

[Gumboots]

The procedure goes like this:

Due to MicroDEM being limited to 255 heights in every elevation table, each table covers a range of 1,250 metres.
They are all clearly named, so even I can tell which one I want when my brain isn't working.

All of the elevations are done with the usual RT3 colour scale, which looks like blue and black but is actually blue and green.
Red is for oceans, as is also usual for RT3. To get your oceans red:
- right click on the image,
- select Display parameter > Reflectance > Water > Set to red > Ok.

Now switch back to elevation view by:
- right click on the image,
- select Display parameter > Elevation > Color from table > Pick your table >
- then, in the same pop-up > Enable Sea Check > Leave Lake Check disabled > Ok.

The colours to watch out for are yellow, lime green and pure white. These are warnings that you need to do something.
These colours should not be visible in the finished heightmap, or RT3 will do weird things when you import it.
If you want to see RT3 do weird things, go ahead and import it with some of those colours (those pixels will be 10,000 high).

Taking the warning colours in order:

Yellow means the DEM thinks that area is below sea level. It might be, or it might just be an artifact in the DEM. Satellite data is not perfect, and the sea goes up and down, so sometimes a DEM will have pixels registering as below sea level when the area in question is at or above sea level. If you see any yellow pixels when using the 0-1250 table, you should check the area out on a map or a satellite shot and decide what you want to do with it. If you want it as ocean, fill it with red. If you want it as land at sea level, fill it with black.

White means those pixels are at an elevation that is higher than the current elevation table will cover. If you see white pixels, you'll need to also save another BMP with the next table up. You don't have to remove white pixels, because the next layer will cover them.

Green means areas that are below the range covered by the current table. If you see green pixels, you'll also need the next table down (and another BMP).

Once you have all the BMP's you need, stack them up as layers in your image editor and knock out the green pixels with the Magic Wand tool. Make sure that anti-aliasing is not enabled, because if it is the cuts will be furry. No anti-aliasing means you get clean cuts exactly around each pixel. That's what you need for this job.

Once all layers are free of green pixels, you can save the whole thing as your TGA heightmap.

This is the first layer for the Everest heightmap:

Lotsa white there. This means Everest is taller than this (in case anyone didn't know).
So, next layer up...

Lotsa green there, so knock it out with the magic wand and the two layers together will look like this:

Pretty straightforward. Only takes few seconds. Keep going like that until there's no white left. Easy.

[Wolverine@MSU]

Greaat work Gumboots . I especially like the way you've color-coded places that are below the range of the elevation table (green) and those above (white). Makes touching up the stacked BMPs really easy as long as the layers are in the correct order.

[Gumboots]

Glad you like it. I wanted it easy. I don't like playing guessing games.

White means up to most people, since terrain maps usually have white on top of high mountains (for obvious reasons). Green means land, so down. Both are nice and clear against the rest of the heightmap, without burning your eyes out. Yellow stands out well against red, so is good for sea level artifacts. And, most importantly, none of these colours will be used anywhere else in the heightmap, so you can't accidentally knock out pixels you want to keep by selecting any of these colours.

I've run a few other tests with this scale and I think it's a good general purpose one, but I can see specific cases where someone might still want a different scale. Now that we know how to do them, that should still be easy. One detail I did notice is that apparently the elevation .dbf's won't accept fractions of a metre. I did experiment with setting a range for 0-0.1 metres and then 0.1-5 metres (in the hope it would be handy for small artifacts around sea level) but after saving the file and running MicroDEM those decimal values were reverted to 0. So it looks like the smallest range possible is one metre, which is still ludicrously precise for RT3 maps given that DEM data is often out by more than that.

[Gumboots]

I came up with an idea last night. It's one of those ideas that's so obvious in retrospect that I'm wondering why it only just occurred to me.

MciroDEM doesn't just do TGA heightmaps. It's capable of a lot more than that. One of the easy things it will do is make reflectance maps, which are the basis of the terrain shading that is commonly used on terrain maps. These turn out to be very useful, because since they are generated directly from the DEM they match it exactly. This means that if you export one with it zoomed in, so that the exported size is greater than 1024x1024, you can then use it as a guide for positioning and scaling a large composite satellite shot. The advantages here are:

a/ You don't have to guess how it will fit your game map. You can see it right in front of you. This is a very welcome change.

b/ There's no need to worry about exact coordinates to get your satellite shot overlay fitting your game map. As long as the image(s) you start with cover more area than you need to cover, that's good enough. Cut the edges anywhere that's easy.

The current test case is a map of the Darjeeling Himalayan Railway. This was chosen because it's a possible good scenario at some stage, the area has highly complex terrain that provides a good test case, and I already had the DEM for that region anyway. The game map is 448x576 pixels and the area it covers is from 26.60042 North to 27.20042 North, and from 88.09958 East to 88.62042 East.

Why the funny decimal values? Because that's the closest that MicroDEM would get to simple values while giving me the finished map size I wanted. MicroDEM does the cropping by using values of pixels on the DEM, and you just put these at the most convenient values. A basic reflectance map of the area, with standard IHS colours enabled, looks like this:

That's a handy starting point, but it can be tweaked to make it more useful. Desaturating it and then colorizing gives this:

When overlaid on top of the previous image, with layer blending set to Color Burn, you get this:

Which looks a bit lurid, but turns out to be useful. Use the Magic Wand, on a suitable tolerance level, to select the river valleys (blue areas) and ridges (reddish areas). Then "Copy merged" and paste those selections as a new layer. This makes a very good guide for resizing your satellite image. You can just stick the satellite shot underneath the guide layer and then manually resize it, without worrying about any coordinates or other numbers, until it fits the way you want it to fit.

Once you have that sorted, resize the thing to 1024x1024 and save as a BMP. Apply to the game map with bmp2gmp, and you get this:

Which is pretty good for a first stab at it, given the complexity of the area. It's not pixel perfect everywhere. You can find small anomalies if you look for them, but it's close enough to be a very good starting point for further tweaking, and it's a process that is a lot easier on the brain than either guesstimating or trying to calculate it all accurately and hoping it works.

Now for the next step, which I haven't looked into yet but has great potential. MicroDEM will also import stuff other than DEM's, like shapefiles. These can be used for things like this: Polylines of 687 rivers, associated to the basins from the river mouths.

The idea here is that if you can directly overlay accurate river courses onto your DEM, you can obviously export that as an image to your image editor. This will then allow you to colour the rivers with whatever colour fits the elevation you want for the river bed, using the values from the RGB elevations spreadsheet back here.

This method should ensure that you can cut rivers through any terrain at all, at the correct heights, regardless of the game map resolution. It should all be done automatically when you import the heightmap into the RT3 editor.