I figured out why I was getting weird results with TGA heightmaps exported from MicroDEM.
If I use the old PopTop MapBuilder to generate a TGA heightmap it will import into the RT3 editor without any problems. The heights and terrain will all look normal even if all modifiers are left on the default value of 1. It may not be the best solution, but it will be close enough that a bit of tweaking with modifiers will get it really good. You won't need to use crazy values.
The catch with the MapBuilder is that it's not as accurate as MicroDEM. The latitude/longitude scale is off by a degree or so, so you don't get the area it's telling you you'll get. The data is also coarser than the more modern DEM's, so you get blockier coastlines, etc.
If I (previously) exported a TGA heightmap from MicroDEM (using the blue/black 0-9999 table) it didn't behave the same way when imported into the RT3 editor. It was dramatically different. If all modifiers were left on 1, the terrain generated by the editor would have
massive cliffs around all the coastlines, and inland areas would have mountains that make the Himalayas look like molehills.
To get a playable map from a MicroDEM heightmap used to require using an overall height modifier around 0.005 or so (yes, 5/1000) and then playing around with the other modifiers to tweak coastal heights and mountaintop heights. And naturally it ended up losing detail due to the extremely small value of the initial height modifier.
So this is fixed now.
All I did was uninstall the old version of MicroDEM that I had, including deleting the remaining old C:\microdem folder and its contents, then do a complete clean install of
the latest 64 bit version.
That obviously needed the 0-9999 DBF (see Wolverine's post above) dropped into the C:\microdem root folder to give the correct heightmap scale for RT3.
Then all it needed was adjustment of the settings inside MicroDEM. From the top menu:
Modify > Elevation > Check "Colors from table" > Select "Elevation table" > Select ELEV_COLORS_0-9999.dbf
> Select Missing (upper right of the pop-up) and set the color to pure red
> Click OK.
You will now have your DEM in the correct colour format.
Crop out the section you want, and save that as a .bmp.
Open in Photoshop or GIMP or whatever.
Resize to a suitable RT3-compatible 64+1 sizing, including scaling laterally to correct for latitude.
You can use an online latitude/longitude calculator to get the correct scaling.
This one is a good one.
Use the standard bicubic algorithm for resizing. Don't mess around with sharpening or smoothing.
Once you have the proportions sorted, simply save/export as a 24 bit TGA. This will then import into RT3 without problems.