Creating a Map from Digital Terrain Data and Google Maps

[Gumboots]

Ok, off to a good start with lakes and rivers too. I tried doing various things with MicroDEM but quite a few functions seem to be buggy. It's still good for heightmaps and relief maps, at least for the moment, but not so good for some other things.

So, I went and grabbed QGIS. Spent a bit of time swearing at the interface, but fairly quickly got a useful result. The same SRTM data that is the basis for the usual DEM's also comes in other datasets, like this - https://dds.cr.usgs.gov/srtm/version2_1/SWBD/ - which is a vector map of just about every lake on the planet, and this - https://hydrosheds.cr.usgs.gov/datadown ... ata=15rivs - which does the same for rivers.

So if you download a bunch of files from those pages and import the shapefiles into QGIS, you can make a nifty overlay for your game map that will contain more information that you'll ever need and will match the DEM projection perfectly. Which would have been handy to know some time back, but nobody ever mentioned it.

Basic example using the heightmap for my Latvia scenario, since it was handy anyway and that area is chockers with lakes and rivers.

[Gumboots]

Ok, here's some more RT3 weirdness.

When I did that ramp test back here, to check out the lowest smooth gradient the game was capable of, I noticed that even though it had a gradient of approximately 1 in 85 the track was shown as being at 0%. So, I decided to run a few tests to see what the game thought various gradients were, compared to what they actually are in terms of in-game mesh. The results are amusing.

Using the same test ramp heightmap, and just playing with the overall height modifier on import, I generated a range of in-game ramps with modifiers varying from 2.6 up to 48.0. Up until 2.6, the game will still show the track as being at 0%. In other words, this supposed 0% track gradient seems to apply to ramps as steep as 1 in 32, in terms of the actual game mesh geometry. That's an actual 3% ramp, still displaying to the player as 0% track.

Track starts getting displayed as a solid 1% at around a height modifier of 8, with a solid 2% kicking in around 16, etc. It looks like the internal calculations are done on the basis that 8 internal height units (which remember are 0.7 of the height units the game displays to you) over the length of one map pixel (the smallest squares visible when laying track) gives you a "track gradient" of "1%" shown in the interface, with the actual gradient of said track being around 8%.

Note that this is still an assumption, because changing ramp height via the overall height modifier on import is not as accurate as changing it directly via the heightmap RGB values. Large modifier values screw things up. Since the original ramp was 1 height unit per map pixel, and was dead smooth, obviously a modifier of 48 should make a smooth ramp of 48 height units per map pixel. It doesn't, though. It gets the top plateau height correct, but the ramp between that and the ocean is badly striated. It's mostly a grade of 6, but grades vary from 1 to 10 in places. At some point I'll probably make a few more RGB ramps to test some of this stuff without having to use height modifiers (IOW, ramps will still be dead smooth) but given the observed results I'm pretty sure the scale suggested is the correct one.

So, in real terms, "0% track" is anything under an actual 1 in 32. "1% track" is about an actual 1 in 12, although with a bit of a range either side of that. "2% track" is about an actual 1 in 6, etc. By the time you're up to "12% track" the actual gradient is near to 45 degrees, which if you check things out in the game is about right.

---

That's today's first lot of weirdness. The second lot of weirdness is the editor's smoothing function. The ramp testing was all done with 0 smoothing, but just to see what would happen I also did a couple of import tests with smoothing values of 1.0 and 2.0. It definitely doesn't do what I would expect it to do. I would expect it to round off the edges of the top plateau, and round off the internal edges between the ramp and the steep sides up to the plateau.

Instead, what it does is only round off one edge of the plateau, and only round off the opposite internal edge between ramp and sides. The other two relevant edges, one edge of the plateau and the ramp-to-side internal edge opposite it, are not smoothed at all and remain angular.

Given this behaviour it is no wonder I've found that the map "smoothing" modifier tends to make a mess of things. I wouldn't trust is as far as I could throw it. It's clearly going to be better to import terrain with a fine resolution (good elevation tables) and a smoothing value of 0, then do any minor route smoothing manually.

[Gumboots]

Been doing a bit more testing. I've found the 5 metre scale is very good for some maps. It works well for small areas done at large scale. For some maps even a 4 metre scale could be handy (ie: setting 8 of RT3's internal height units to be 4 real life metres). For example, a 4 metre scale would have been perfect for the Latvia map I made last year, because it's large and flat and you want some extra detail at low elevations. But minor tweaks to the overall height modifier when importing the heightmap would do just as well in that case (which is what I did when I made it) so a 5 metre scale would have been fine.

OTOH the 5 metre scale is not so good for larger areas that have wide ranges in elevation. The first Mount Everest test model was really too exaggerated at that scale, and looked more realistic with a height modifier of around 0.4 or 0.5. The Orient Express test maps I made the other day were done with the old 0-9999.dbf, which even though it didn't work properly (40 metre Lego blocks) was about right for overall height range on a map of that real life area and in-game pixel size, if imported with the height modifiers left on the default value of 1.

The other thing is areas below sea level. The Dead Sea is the most obvious example, but there are plenty of others: Sea of Galilee at -210 m, Afar Triangle at -150 m, parts of the Netherlands at whatever, central valley of California, etc. Sooner or later somebody is going to want maps of those places too, and they can't use tables that start at sea level. Obvious thing to do: make a few sets of tables in handy ranges, so you can pick whichever ones work best. So I did.

The old monster Lego block table turns out to have been done on an 8 metre scale (ie: 8 of RT3's internal height units = 8 real life metres) or would have been if it had been working properly. That's basically what the RGB values were set for. So I whipped up a general sea level to 10,000 metres table on an 8 metre scale, and another -400 to 9,600 metres table on the same scale for handling the Dead Sea. Ran a test on that one to make sure it works, and it does.

That was done so that the Dead Sea itself was at 0 height for heightmap purposes, so was "ocean" in RT3 terms, with the Mediterranean being a manually-done lake at a nominal 400 metre altitude (the Dead Sea was around -400 metres before 1980, but has dropped a bit since). The same principle would work for any below sea level starting point. You could even just use this set of tables if you weren't worried about having an automatic "ocean", and your central Cali map (for example) could have the Salton Sea added in as a lake (obviously the Pacific would have to be a lake too). I like automatic oceans when I can get them, so personally I'd only use the -400 to 9,600 tables if I really needed to start below sea level.

Anyway I've zipped up both new sets: 0 to 10,000 m, and -400 to 9,600 m, both in 8 metre increments, for anyone who wants to use them. This is not as fine a resolution as the 5 metre tables, but is still 5 times better resolution than the old 0-9999.dbf.

Also included is the TGA heightmap for the Dead Sea test map, if anyone wants to try it out. I found that looked best if imported with an overall height modifier of about 2, since the real life area is quite small and the test map size is quite large. It would obviously be possible to make a set of tables with the same starting point and a 4 metre resolution, but it would mean a/ more tables to mess with and b/ you'd run out of RGB values at +6,743 metres, so no Himalayas.

[FlorianF]

I made the rest of the tables, and gave them a full testing. They work.

Obviously, the way to test them is with Mount Everest. Aint nothing taller than that. So, I grabbed the DEM for that area and exported 8 BMP images, using one elevation table after another. Stacked the BMP's in Photoshop, knocked out the green pixels, and saved as a TGA heightmap.

Imported that into RT3, and it looks like this...


Everest_import_testing.jpg


I've attached the TGA in a zip, so anyone can play with it themselves.

Ok, so how did I make this beastie? I used the new elevation tables which are in the other zip.
They come with a readme file, and you are advised to read it, but I'll describe the procedure in the next post.

Edit: Just for fun, I threw a satellite shot on it.


Everest_satshot.jpg

Wow! Did you actually made this into a map to play? :O

[Gumboots]

No. It was just an obvious test case for proof of concept. I figured if it will handle Everest, it will handle anything.

Edit: By the way, if you want to turn it into a scenario or sandbox I'm happy to give you a copy of the version with the satellite shot.

[Gumboots]

I've been giving this stuff more thought, now that I'm at the stage where finishing eye candy for the NSW map is the next step.

At the intended scale for default RT3 heightmaps, 8 internal units equates to 8 metres (ie: 1 metre per unit) so each click on the up or down tool will change height by 8 metres. Obviously this will change if you use a different height scale. If you use a .dbf set to 8 units = 4 metres (useful for large maps of small areas) then each click will raise or lower that vertex by 4 metres in real life height.

This scale thing could also be useful for checking altitudes of any critical points, perhaps if they have been messed up by a smoothing algorithm. If you know what they are supposed to be (which you can get from Google Earth easily just by hovering over them) then you can use a simple formula to convert that to RT3 game map height (altitude in metres x map scale x .7) and then the relevant vertex or area/lake can be set exactly on the .gmp.

---

The other thing is getting satellite shot overlays to match around nasty geographical details like coasts and rivers. I had an idea here too, which should take all the guesswork out of it. The actual river course on the .gmp will not exactly match real life due to the game's limits, and ditto for coastlines. However, you can get their course on the .gmp quite easily.

You know how many pixels there are on the .gmp, and each section of coast or river exactly follows .gmp pixels, as do oceans and lakes, so these can be transferred to a .psd in Photoshop or .xcf in GIMP. This .psd/.xcf can then be used as a guide for tweaking sat shot overlays so they match the tricky bits. There's no need to load things into the game by trial and error. The whole lot can be done in an image editor, and it can be made to match the .gmp exactly.

This is worth doing, because the resulting .bmp that will be applied to the .gmp is going to give more accurate terrain painting on most maps than trying to paint in the editor. This is because the .bmp will always be 1024x1024, while the terrain painting in the editor follows .gmp pixels. If you have a 512x512 map, the .bmp for the satellite shot will have double the resolution of the .gmp, and will therefore give much better detailing.

So, the way to do it would be to make a reference .psd/.xcf that matches the proportions of the .gmp but is at a larger scale. For a 512x512 .gmp the logical choice would be 1024x1024, which the .bmp can be built straight on top of. All pixel dimensions for coasts and rivers can simply be doubled.

For other sizes of .gmp's it would be easiest to use integers to scale them (because that way pixels still work for placement of features) so for the NSW map it would make sense to scale its 512x448 to 2048x1792. Apply rivers courses and coasts from the .gmp with a simple x4 multiplier. Sort out sat shot to match that. Scale resulting image down to 1024x1024. Apply to .gmp. Sorted.

PS: Obviously this same trick would also work for any other overlays: reflectance maps, terrain contour maps, etc.

[Gumboots]

Hey I just tried something out of curiosity. Greyscale heightmaps get imported with no ocean created, and you can't combine a greyscale heightmap with a red ocean. Well you can, but all heights get multiplied by 8, so it ends up being ridiculous.

Anyway, I remembered how blue behaved on a blue-green heightmap (the standard RT3 scale) if there were no red pixels present (it makes an ocean at 0 altitude) and that got me idly wondering what would happen if you put a blue ocean on a greyscale heightmap.

Turns out that it works perfectly. It automatically creates a zero altitude ocean for you, and it does this without borking the heights of the land vertices. The land gets created at the normal heights for greyscale: if imported at the default 1.0 height modifier the highest point on the map will always be 5,712 internal height units, which will display in the game interface as 3,998.

IOW, whatever height modifiers someone is used to using for any size greyscale map will still work normally, but you get the bonus of automatic oceans. It's easy to do this in MicroDEM by setting the reflectance colour for water to be blue instead of red. The standard ocean check will then add the blue where it's supposed to be, and the rest of the image stays as greyscale, and you just import that to the RT3 editor.

Slight drawback is that it's not as accurate at very low elevations as the blue-green scale, so your coastline will lose some details, but it's still pretty good and would be fine for areas that don't have a complex coastline anyway.

Personally I'll stick with blue-green, but I thought this trick was worth mentioning for anyone who prefers to work with greyscale.

[Gumboots]

Found a way to get sat shots matching coasts and rivers without much effort.

It's still a bit fiddly, and requires some trial and error adjustment to get it all lined up, but results are very good. Eye candy is pretty much sorted. It will still need a little bit of touching up here and there, but it's close straight out of the image editor.

First point: don't use Google Earth for the sat shots. Use this page: https://earthexplorer.usgs.gov/

This has a cylindrical projection which will match your heightmap. Take a succession of screenshots of the area you want, with two shots of each section: one with grid on and one with grid off. These all get grouped in pairs in Photoshop (or GIMP or whatever) and you use the grid to line up all the groups. This is quick and easy in practice. Once they're lined up, merge them into two layers (one with grid, one without). These composite shots are then used to make your final .bmp.

The short version is I got a copy of the heightmap and resized that to 1024x1024. This gets used as a guide for the sat shot. I made several guide layers for different purposes: one had the ocean cut out with the magic wand, then 0% fill applied to that layer and a translucent 3px red outside stroke. What this does is give a clear but see-through guide for aligning the sat shot coast. You can get it spot on everywhere just by eye, with no measuring required.

It's necessary to tile the sat shot in several horizontal strips stacked north to south, just because the projection varies slightly over the latitude range, so trying to do it all with one big sat shot doesn't work as well. I used four strips, each 256 px high on the finished image. The joins between turned out pixel perfect.

To get things lined up east to west, I used copied/pasted some magic wand selections from the heightmap to extra layer. These were obvious things like river bed contours, which can be used to align the sat shot horizontally. It's basically just a matter of stretching and squashing the four strips until they fit everywhere, then saving the whole kaboodle as a 1024x1024 .bmp.

Only catch is that although everything fits, the ocean from the sat shot was too dark around the beaches, etc. To fix that I made three more layers on top of the base layer. Two had the ocean cut off with the magic wand. One of these stayed as the top layer, and the next one underneath was given a 3px Gaussian blur. The third layer under that was given a sandy-coloured drop shadow (angled to match the general run of the coastline). This combo sorts it out so there's a pretty natural gradient from land colour to ocean colour around the edges. After that it's just a matter of sweeping the beaches with a sandy brush in the editor (or a rocky one if you want rocks) to finalise the last bits of odd pixels.

Screenshot shows the result, which has had a few minutes of fiddling with brushes in the editor but is otherwise straight out of Photoshop.

[Gumboots]

Found a fix for the seams that RT3 generates in the bitmap for the terrain.

These appear at multiples of 128 on the map X axis, which is obviously some bug related to processing powers of 2. I'd seen them before (the Latvia map springs to mind) but had never been able to fix them properly. Brushing over the seam doesn't work. No idea why not, but it doesn't.

What does work is selecting a colour from a pixel on one side of the seam, then spot clicking that colour onto the corresponding pixel on the other side of the seam. To drop the colour into place you have to click around the middle of where the map pixel would be, not right against the seam. If you click right against the seam nothing happens. If you move away from the seam slightly, the colour will drop into place with a perfect match across the seam.

It's a bit tedious, but it only has to be done in the worst and most obvious places.

Edit: Thought of something which might be worth testing. I might try applying the same bitmap to a 576x512 version of the same heightmap. I'm curious as to whether the seams will only appear if the X or Y axis is a multiple of 128 (IOW, total axis length = 128, 256, 384, 512, 640, 768, 896, 1024).

The reason I'm wondering about this is the Latvia map was 768x512 and that had seams (will check if they are on both axes, but they were definitely on the X axis 256 multiples) and the NSW map is 512x448, and only has seams on the X axis positions. If the bug is caused by choosing map dimensions that are multiples of 128 that would be good to know.

I've already tested applying the same bitmap directly to the .gmp via hex editing, instead of via Pjay's bmp2gmp tool. That's very easy to do (no harder than using bmp2gmp) but unfortunately it makes no difference to the 256 seams problem. The bug is obviously within RT3 itself, rather than being an artifact of bmp2gmp.

[Gumboots]

Just thought of something, which should have been obvious years ago.

We can't be the first mugs who have wanted satellite shots to match the projection used for DEM's. When you think about it, given that there are tools to put river shapes and all sorts of things on top of DEM's it makes sense that someone would have figured out how to geo-reference satellite shots. So I ran a quick search for "geo-referenced satellite shots" and a lot of useful stuff turns up. There are even tutorials for referencing old maps to whatever projection you want to use.

Georeferencing Satellite Images
Fundamentals of georeferencing a raster dataset
Georeference imagery

This one looks particularly useful:
Download Georeferenced Satellite Images

SAS.Planet is a free application used to view and download satellite images submitted by such services as Google Maps, Bing Maps, Yandex.maps, Yahoo! Maps and many more.

The satellite images can be downloaded and then loaded into OCAD as georeferenced background maps.

As soon as I get time I'll have a play with this stuff. Doing the rivers, etc as shape overlays on DEM's and BMP's is a great labour saver. If we can get a fairly simple process for referencing sat shots to DEM projection as well, that would make map texturing much easier.

[Gumboots]

Got sidetracked a bit. The question that sidetracked me is an oldie but a goodie.

Once you finish that project, perhaps you could turn your enthusiasm toward identifying where heightmap info is stored in the .gmp files. I've always thought it would be useful to have utilities like PJay's BMP2GMP / GMP2BMP duet to inject/extract heightmap info from a map.

I found it. Even better, it's not hard to find once you know what you're looking for. Milo's notes are a bit confusing (may have been WIP) but I've nailed down the format in terms average mugs can understand.

I made a test heightmap: 65 x 65 in size (smallest the editor will accept) and with a constant height rim 4 pixels wide, surrounding a central ocean. This meant there were only going to be 2 known values in the heightmap data, so easy(ish) on the brain. The raised rim was coloured as RGB(0,0,248), which on the usual "1 internal height unit = 1 metre" scale means an altitude of 248 metres above sea level.

But, it turns out that RT3 does that multiplication by 0.7 before storing the height values in the .gmp, so the initial 248 ends up being stashed as 173.6. It gets stashed as a four byte float, so the hex is actually 99 99 2d 43. So that's a point to note: if you happen to be looking for a non-zero height value in the heightmap data, its value will be recorded as 0.7 of whatever your .dbf gives as the nominal height for that particular colour on your DEM/TGA.

Anyway, by referring to Milo's notes, scratching my head, looking around, and ignoring Milo's notes where they didn't make sense, while taking notice of them where they did make sense, I got a result. The heightmap data starts just after the end of the locomotives list.

The locos are 65 bytes each. Next shot shows the last loco's bytes with extra crimson highlighting.

Straight after the last locomotive, there are 16 bytes (yellow) with weird stuff in 'em (no idea what it is at the moment). These are followed by 8 bytes, which record the width and height of the map, plus 1. In other words, they are recording the number of vertices coded by the heightmap instead of the number of pixels on the map (verts = pixels + 1 for any row). These width and height values are integers (not floats) in RT3's standard little-Endian hex. They are followed by another 8 yellow bytes of more weird stuff. After that, it's straight into the heightmap (highlighted in red).

The good news here is that the weird stuff in yellow appears to be a standard chunk of heading code. It's exactly the same in my 64x64 test pool and in my 512x448 New South Wales map, so should always be easy to find via search. The bytes in both cases are (dd 32 00 00) (e2 2e 00 00) (f9 2a 00 00) (ba 0b 00 00) in front of the height declaration, and (fa 2a 00 00) (fc 2a 00 00) between the width declaration and the start of the heightmap data.

The heightmap data is coded to have a four byte padding, all zeroes, at the end of each row of verts. The verts are counted left to right, starting at the bottom left corner of the map. After every 65 verts on this small test map, we get this (dark greenish - four zeroes row ending):

After the last row of verts and its finishing four zero padding, there's a tricky bit thrown in. It's RT3. There have to be tricky bits. There are another 65 instances of four bytes of zero tacked on the end, after the padding at the end of the last row of heightmap verts. So in the screenshot above, the actual heightmap data ends with 99 99 2d 43 (vertex height) then 00 00 00 00 (standard row ending), then the last lot of zeroes is added on.

Edit: Correction. I made a mistake here. Got it figured out now. Ignore the part in small italics.
Note that while the number of bytes in each row of verts will obviously depend on the width of the map, this end chunk of zeroes is independent of map size. It's always 65 chunks, of four bytes each, all zeroes. After that you'll see ED 2C 00 00, which marks the start of what Milo calls the "ground map".

The correct version goes like this: The number of zero bytes between the last heightmap line ending, and the ED 2C 00 00 hex at the start of the "groundmap", is always 4 + 4*((width of map in pixels) + 1).

Which is another way of saying there are four bytes of zero for each vert in a row (ie: 00 00 00 00) and then there is another of those (00 00 00 00) as a standard line ending.

So if your map is 640 px wide, there will be 2,568 bytes of zeroes between the end of the last row of height verts and the start of the groundmap, because 4(640 + 1) = 2564, and then you add another 4 to that for the standard line ending, and you get 2568.

For a map 512 wide (like my NSW map) there are 2056 bytes of zeroes: 4(512 + 1) = 2052. Then add another 4, and you get 2056.

I've cross-checked all of this by doing the same searching and bookmarking in the hex of my New South Wales map (edit: went and checked a few more maps, ergo the correction) and it all works (really does now). Even ran a few checks on which vertex is the last land one going east, and which is the first ocean, and those values match in the TGA heightmap and in the hex. So it all seems solid.

---

Edit: Just thought of something else. The number of bytes in the actual heightmap block (before the extra zeroes are tacked on the end) is always equal to 4*[((Map width in pixels) + 2) * ((Map height in pixels) + 1)]

Why does it equal that? Because the number of vertices is always pixels + 1, then you have the four byte 00 00 00 00 line ending for each row, so total number of bytes per horizontal row = 4(pixels + 2). Vertically, number of rows = pixels + 1, so multiply that figure by the previous figure. Then multiply the lot by 4, because there are four bytes for each heightmap value or line ending.

This can be used to check that you have found the exact end of the heightmap verts. If your map has some ocean in the top right corner, the last heightmap values will all be zeroes. That makes it hard to tell where the heightmap stops and where the extra zeroes are tacked on. Your hex editor should give a readout of the number of bytes in any selected block, so that can tell you exactly where the last vert is.

[Gumboots]

Ok, finding the heightmap data in the hex is all very well, but is it actually useful? I went and found it just because I was messing with .gmp hex anyway and I got curious, but since then I've been wondering if this knowledge is any use. It might be.

Blender will create a 3D model from a heightmap. The way it works is that you can use the value of any RGB channel as a displacement modifier (see Displace Modifier - Blender Manual) which means you could easily use the blue channel to make a 3d model of your heightmap, or you could use greyscale if that was your preference.

The catch with RT3 is that its coloured heightmaps also use the green channel as a multiplier on heights (basically, 1 green = 256 blue) and Blender won't let you use a combination of two channels to modify height on one axis. So, it would have to be done in two steps:

1/ Import blue channel values only, and apply the Displace modifier based on those.
2/ Import green channel values, and apply the modifier again, based on those.

This should be accurate, and would give you a true model of your .gmp in Blender. So what? Well, one thing it might be good for is rivers. They're a real nuisance to cut inside the game editor, but Blender will make a plane anywhere you want, at any dimensions and any gradient. With rivers you'll usually be aiming for an smooth gradient from source to mouth (barring discontinuities like rapids or waterfalls) and a sloped plane is an ideal guide for that, so you could use it to shift river valley vertices to where you want them.

Alternatively, if you import the course of the river as a shape overlay on your heightmap (not hard to do) then it could be coloured to automatically set vertices to the correct height when you import the heightmap into Blender and apply the Displace modifier. You'd simply mark out that layer as a series of straight or angled chunks of pixels in your image editor, then apply a gradient overlay based on the height at the source and the height at the mouth. The result would be an even gradient between the two once the 3d model was created.

But then, you can do this just in the image editor and then import the result straight into RT3, so what's the point of an intermediate step in Blender? It could be handy for tweaking things. For example, the default RT3 smoothing algorithm is a bit of a blunt instrument, notorious for making a total mess of river courses. Blender will also smooth any series of vertices, and give you much better control over the process. It will also scale the model vertically, to any increment you want, if you want to increase or decrease any heights. You can easily revert any changes too.

Another thing is applying satellite shots. These are notoriously tricky in places, but Blender includes UV mapping. That means you can shift your model around on a larger satellite shot to get the best match, then crop and/or scale the shot to those boundaries before applying it to to your .gmp. It also means you can use the new possibility of higher res map textures if you think your map needs it. Blender will let you see exactly how your texture will look on your map at any zoom level, so if it's getting too blocky you might think a 2048 is in order instead of a 1024.

It could also be handy for placing assets like buildings and ports. Once a suitable scaling was worked out (should be easy enough) you'd be able to import any assets from a stash, and plonk them wherever you liked to see how they fit.

So it does seem like being able to import heightmap data directly into the .gmp might be useful, if you were going to use it in conjunction with another modelling app.