GPSFileDepot Forums

We have pretty extensive coverage in Southern New Jersey with USGS NED 1/9 arc second LIDAR DEM, so I'm trying to find a way to use this in a map. It's especially nice around here where everything is so flat because it reveals a lot of surface detail in the forest.

It would be very easy to make a garmin "custom map" (.kmz raster image) from this data, but it's pretty limiting both in size and in the number of devices that can use it. So my thought was to vectorize it using Globalmapper's function to create polygons based on RGB values.

I have all the data, so for starters I cropped it to the outline of the map I want to make, then exported it as gridded 8 bit grayscale TIFF's that are ~500x500 pixels. Globalmapper converts these into anywhere between 50,000 to 60,000 polygons when using a match distance of 16. Smaller values for match distance create way too many polygons.

Here's a comparison of the original TIFF to the vector image. Globalmapper really does a great job - it's hard to tell them apart until you zoom way in.

(http://stephencreek.com/gpsfd/lidar/raster.png)


(http://stephencreek.com/gpsfd/lidar/vector.png)


Next I export a shapefile and use Filemaker Pro to do some processing of the .dbf, parsing the RGB values and mapping them to MP_TYPES in a custom .typ file. I decided to go this route instead of using a program like MOAGU or Mapwel because it gives me full control over everything.

I then import the modified shapefile back into Globalmapper, combine with other vector features like roads, then export an .mp file to compile with cgpsmapper. My current prototype breaks a USGS 24k quad into 8x8=64 tiles, each of which is ~1MB. So far so good. Here's what it looks like in Globalmapper

(http://stephencreek.com/gpsfd/lidar/gm01.png)


And the compiled map in Mapsource looks surprisingly good - bet ya didn't think Mapsource could do this!  ;)

(http://stephencreek.com/gpsfd/lidar/ms01.png)


But then the fun begins... there are two problems on the GPS. First, it practically brings the unit to its knees due to all the little polygons (I tried on a Nuvi 205 and Montana 600). But worse, the GPS doesn't seem to render the image with as much precision as Mapsource, even with detail set to max. Basically, it's a mess. Kind of an interesting abstract art image, but nothing like the original.

(http://stephencreek.com/gpsfd/lidar/mt01.png)


So I'm going to need to re-think this and do more experimentation.  :)

Anyway, it's an interesting although time consuming and sometimes frustrating project that I wanted to share. I have also recently refined my technique for processing the Landsat NLCD 2006 landcover data in a similar fashion and that is working out really well - but that source imagery is about 10 meters pixel. I'll post some details about this in another thread soon.

Hope everyone is enjoying the holidays - here's wishing a happy New Year to all! :)

All I can say is wow.  I bet Garmin never thought they'd need more processing power in the GPS units.

Thanks! I'm very impressed that Mapsource can cope with this so well. Once I got the tile size right, the map works very well in Mapsource and scrolls/zooms quickly. I guess this is because it uses the graphics card to help. When I open the same map in nRoute it works, but it's slow motion as each row of pixels is drawn. At the resolution used in the examples above, one USGS 24k quad consists of over 3 million polygons!

Am doing some more testing now with lower resolution images and will post the results later today.

I'll watch this with interest, but it's a pity nobody knows how to turn this into DEM data.  The sexy thing about LIDAR is the ability of GM to view it in 3D.  Does your data use a 1 foot grid or something smaller?  Have you tried using it to generate contours?

What are you trying to accomplish with that map?  Why are you doing it that way instead of creating topo lines?  It might be neat to create a transprent overlay with a 2 foot contour interval for flat areas. 

Yeah, I really wish we knew how to put DEM into Garmin's format. But I wonder is this high res stuff would just choke the GPS anyway? But my technique promises to be the next best thing IMO.

I am mapping each elevation range into one of 72 custom types I've created. As you know, when you "mouse over" a map object (in either mapsource or the GPS), it will display any custom text you include in the type definition. So, if I put the elevation into the custom text for each type definition, it will display elevations. Of course, it can't associate that with track profiles, etc., but still kind of cool. Also, I can create difference custom .typ files with different color schemes that have effects similar to the different shaders available in globalmapper. That makes it easy to change the map appearance without recompiling anything - just change the .typ file.

I'm not sure what you mean by "1 foot grid". I really haven't worked a lot with DEM in the past. I downloaded the LIDAR from USGS and in the examples above I used it in the raw format. The source resolution is ~3 meters/pixel (ie: 10 ft per pixel).

Yes, I have used this same data to generate contours for my most recent map - http://www.gpsfiledepot.com/maps/view/294/ . I experimented with different intervals and posted a bunch of examples here, seeking input from my "audience" :) http://forums.njpinebarrens.com/threads/boyds-map-of-the-pines-beta-available.5103/page-7

I haven't yet looked at any LIDAR from mountainous areas. I'm thinking it might be a bit of a mess for generating contours however.... the "too much information" syndrome perhaps? LIDAR is a really different beast from 1/3 arc second data. But it's great around here where jmost of the elevations fall within a range around10-70 meters.

My techniques are tuned for this "bumpy" kind of data. When I use it with the traditional 1/3 arc second DEM, it looks pretty bad. Smooth gradients aren't so easily vectorized, and you hit the limits of the allowable number of custom types available to represent grayscale in Garmin's format.

Ah jbensman, you just feel that transparent maps are the answer to everything. ;) Don't get me wrong - your work is terrific and a great contribution. Just not my own style, and variety is the spice of life.

Actually I spent quite awhile playing with different ways to generate contours, but they really don't capture the subtlety of the terrain as well. See all the screenshots in the thread I linked to in the previous post at njpinebarrens.com. All those contour lines get very busy.

(http://stephencreek.com/njpb/con13.PNG)

For me, "a picture is worth a thousand words". I can just cop out and make a "custom map" (.kmz), but then people with older units can't use it. And if I use the source resolution LIDAR at 3m/pixel, Garmin's format maxes out with a map about 6 miles x 6 miles. The Montana can use bigger .kmz files, but not many people have one of those.

You should certainly try experiments of your own with contouring LIDAR. With your expertise, I don't doubt that you could come up with some nice overlays that could enhance City Navigator or OSM maps.

I live about a mile from the Mississippi River.  What I think would be nice is LDAR 2 ft contour for the floodplain.   The reason why I think transparent is the only way to reasonably do this is right next to the floodplain there are bluffs and lots of steep topoography.  If there was a 2 ft interval there, the map would be a solid contour line.  That way when you were out on the flodplain you could turn on the detailed 2 ft overlay and turn it off when you leave the floodplain.  The last time I checked they did not have any ldar for my area but I know they are working on getting it.  I've been wanting to try it.

Another issue ldar might help with is lots of the other dem data messes up bluffs.  Instead of showing a bluff, the contour lines will show a slope.  I've spent many hours trying to fix this on lots of my maps. 

My understanding is the ldar gets two data sets, the first reflection is the height of the vegetation and the second is the ground.  It might be interesting to make a map of vegetation height. 

Quote from: Boyd on December 28, 2011, 09:25:24 AM
I'm not sure what you mean by "1 foot grid". I really haven't worked a lot with DEM in the past. I downloaded the LIDAR from USGS and in the examples above I used it in the raw format. The source resolution is ~3 meters/pixel (ie: 10 ft per pixel).

Grid actually may not mean much for LIDAR.  I don't know how a "point cloud" is defined.
I seem to remember generating some contours for an airport using LIDAR, and thought I remembered it had a grid, but maybe I was confusing that with an imagery resolution.

Quote from: Boyd on December 28, 2011, 09:25:24 AM
I haven't yet looked at any LIDAR from mountainous areas. I'm thinking it might be a bit of a mess for generating contours however.... the "too much information" syndrome perhaps? LIDAR is a really different beast from 1/3 arc second data. But it's great around here where jmost of the elevations fall within a range around10-70 meters.

A few years ago I downloaded some LIDAR of Mt. St. Helens.  Viewed in 3D in GM you can make out trees at the bottom of the mountain.

If you can make out trees, my understanding of LDAR (which could be wrong) is that would be the wrong data.  My understanding is with LDAR it first gets vegetation height and then it gets ground elevation.  So if its showing trees, would it be vegetation height layer, not the ground elevation layer?

No doubt the wrong data (or meaningless data anyway).  I was looking at the whole point cloud in a 3D rendering without any way of distinguishing between vegetation and topography.  When I turned off the veg layers the trees went away.  The data has an incredible number of layers available though.

Just did a whole bunch of comparisons using different settings so that I could mull them over. The "match" settings come from the dialog when you right-click the raster image and choose "create area features from equal values" in the globalmapper layer dialog.

I also tried resampling the original 3m/pixel raster image to both 6m/pixel and 10m/pixel. The number of polygons refers to the whole image which is one tile from a USGS 24k quad exported in an 8x8 grid. These tiles are about 500x500 pixels. If I try this with a 1000x1000 image, globalmapper eats up all the resources on my computer - task mgr showed 96% RAM use and 90% CPU.... I had to reboot! My computer has a 2.8 ghz core 2 duo CPU with 6GB RAM.

In this first series of tests, I used the original 3m resolution data and tried different match settings

(http://stephencreek.com/gpsfd/lidar/match16-03m.png)

(http://stephencreek.com/gpsfd/lidar/match32-03m.png)

(http://stephencreek.com/gpsfd/lidar/match48-03m.png)


Next I resampled the raster image at 6m/pixel and tried different match settings

(http://stephencreek.com/gpsfd/lidar/match16-06m.png)

(http://stephencreek.com/gpsfd/lidar/match32-06m.PNG)


Same thing, but with the raster image resampled at 10m/pixel

(http://stephencreek.com/gpsfd/lidar/match16-10m.png)

(http://stephencreek.com/gpsfd/lidar/match32-10m.png)


But maybe I'm expecting too much? All those examples show the image at 1:6000 scale (in Globalmapper, View > Zoom to Scale) which is probably equivalent to about the 300 zoom on the GPS. Here's a composite comparing a view of the full tile at 1:24,000 scale. Pretty big difference in the number of polygons, but not as big of a visual difference at this scale

(http://stephencreek.com/gpsfd/lidar/compare01.png)


So this gives me some things to think about and try. Somewhere in here there's a compromise that should result in reasonable performance on the GPS unit. My original prototype mapset (shown in the first post of this thread) used match=16 at 3m/pixel and had 56,000 polygons. The example with match=32 at 3m/pixel reduces the polygon count to 15,700 while still retaining the full resolution of the source image. Maybe that's enough?

OTOH, it may be that the GPS itself doesn't calculate with enough precision to do justice to 3m/pixel imagery. As discussed earlier, the .img format (as supported by cgpsmapper) supposedly maxes out at 5m/pixel....

Quote from: Boyd on December 28, 2011, 11:36:07 AM
OTOH, it may be that the GPS itself doesn't calculate with enough precision to do justice to 3m/pixel imagery. As discussed earlier, the .img format (as supported by cgpsmapper) supposedly maxes out at 5m/pixel....

Get a chance to show my ignorance here.  If I understand correctly at Level 0 the circumference of the Earth's ellipsoid (24902 miles) is divided into 2²⁴ (16,777,216)segments.  24902 miles/16777216 = 0.015+/- miles = 7.8370 feet = 2.4 meters.  And cgpsmapper complains strenuously if I try to compile a map with routing nodes closer than 2.4 meters.

The Public Land Survey mapset is transparent and the polygons are invisible.  When you view it on the GPS, you do not see the polygons (but you see the polylines at the boundaries).  When you click on the map, the GPS will identify the label of the invisible polygon.

How about you create a transparent map with invisible polygons.  Since you have the polygons labeled with the elevation, if you then click on the map it should give you the LDAR elevation for that spot.  Since the polygons are invisible, maybe it would not grind the GPS to a halt trying to draw the polygons.  While it would not do other things DEM does, it would be a neat map to have.

Seldom: yeah, that math always confuses me but what you say makes sense. I thought the error was +/- 2.5 meters, which means any two points could be 5 meters apart. Remember this thread? it is pretty graphic evidence that some rounding or truncation is taking place: http://forums.gpsfiledepot.com/index.php/topic,1335.msg8387.html#msg8387

jbensman: that's a good idea, but I need to follow through on what I started first. Anyway, what fun is it unless you can push something to its breaking point.  ;) But I'm not sure... do you think an invisible polygon is easier to draw than a visible one? Seems like the "heavy lifting" is the math to calculate its proper location, although maybe there's some overhead related to actually drawing the pixels.

Quote from: Boyd on December 28, 2011, 01:00:23 PM
Seldom: yeah, that math always confuses me but what you say makes sense. I thought the error was +/- 2.5 meters, which means any two points could be 5 meters apart. Remember this thread? it is pretty graphic evidence that some rounding or truncation is taking place: http://forums.gpsfiledepot.com/index.php/topic,1335.msg8387.html#msg8387

Getting off topic again, but what would be really instructive would be to see a graphic scale on your pink and cyan illustration.

No problem, just quickly threw this together from the old files. But I tried a new test and didn't see this happen, so maybe something was wrong with my earlier method? Weird. IIRC, I also needed to check the option for saving data in double precision when writing the .gmp file, so maybe it's related to that.

Then I realized I was getting sucked back into this old project, so that will have to wait.  ;)

Try for yourself.... draw something curvy at close zoom, export as .mp and reimport. Here's a screenshot with a scale using the same files from the original thread.

(http://stephencreek.com/gpsfd/lidar/error.png)

Thanks, I'll start a new thread when I have something to show.

Boyd - very interesting.

FYI - my West Virginia 10' Contour Overlay mapset, uploaded in Aug, 2009, used a 10' contour interval created from USGS 3 meter NED data derived from mass points/breaklines.
My North Carolina 10/5 ft Contour Overlay mapset, uploaded in Sept, 2009, used a 10' ci in western' NC and 5' in the central areas and was created from USGS 3 meter NED data derived from LIDAR.

Well this is certainly a learning experience. The main lesson so far is: Mapsource is a lot more powerful than I ever expected. I re-did my prototype map using the match distance of 32 and full 3m resolution. This reduced the polygon count to about 1/4 of the previous test and compiled .img tile are only around 500KB.

In Mapsource, this works really well. Response is snappy zooming and scrolling and it looks great.

(http://stephencreek.com/gpsfd/lidar/ms02.png)

(http://stephencreek.com/gpsfd/lidar/ms03.png)


Much to my surprise, Basecamp looks really bad, even at the max detail setting. To be fair, I am breaking all the rules for Garmin vector maps, so I can't really blame Garmin. But it's strange to see such a big difference from Mapsource. I guess Basecamp has been tweaked for other things, like Birdseye imagery?

(http://stephencreek.com/gpsfd/lidar/bc03.png)


Now, on my Montana this map is more responsive than the previous version although still sluggish to zoom and pan. But the same problem remains - the rendering quality is much worse. It almost looks like it's smoothing the rectangular pixels. The two screenshots below show the exact same area as the two Mapsource screenshots above. If you compare, you'll see they are completely different. This map just isn't usable on the GPS.

(http://stephencreek.com/gpsfd/lidar/mt02.png)


(http://stephencreek.com/gpsfd/lidar/mt03.png)


Oh well, time to try some more things and see what happens. I am just blown away by how good this works in Mapsource though. Maybe it's useful for that by itself. I tried it with nRoute, and this new map works ok there now as long as you don't zoom out too far. So at least that provides one way to navigate using the map.

Have tried a lot of different things, and the best result so far involved using the Photoshop pixellate filter to process the TIFF before vectorizing in Globalmapper. Still needs more work, but it has improved. Clearly, it will never approach the quality that I can get in Mapsource however. I think the next step will be to come up with a color table more similar to the Atlas Shader in Globalmapper. At least that would give more visual cues to the elevations than greyscale.

(http://stephencreek.com/gpsfd/lidar/mt04.png)

(http://stephencreek.com/gpsfd/lidar/ms04.png)


(http://stephencreek.com/gpsfd/lidar/mt05.png)

(http://stephencreek.com/gpsfd/lidar/ms05.png)

I am not sure about everything, but to me it is looking more and more like you did not do what you think you did.

When GM displays a DEM, it displays a combination of the elevation data and slope shading it creates from the adjacent elevation values.  Turn off hill shading in configure/vector options to see just the elevation data.  For me the Daylight shaded gives various shades of cyan; the slope shaded gives shades of grey.

When you exported the file in 8 bit grey scale, you were exporting the slope shading not the elevation values.  For elevation you would need to use the elevation - 16 bit or elevation -32 bit format. 

When I open a file saved as 8-bit greyscale, the RGB values are representing the shading, lightest for flat to deepest for steepest.  The three values I see are the same for the various shades of grey, or 000,xxx,xxx for the various shades of cyan.

You are making 72 ranges of slopes, not elevations.  Elevations would be in bands of similiar shades on opposite sides of streams/valleys.

I do not understand why MapSource, BaseCamp and the GPSr display the data so differently.

The number of significant digits exported in the .mp format is often less than the what was in the source data.  The 4-way intersection being converted to 2 3-ways is weird; however, I have seen a lot of things GM kinda does right, but not exactly correct.

No, I fully understood what I was doing. I converted to a TIFF image and discarded the elevation data because I wanted the map to look like the results of the shader on the original DEM data. I also tried a number of different ways of making polygons from the actual elevation data. When you vectorize that, you get just one value, ELEVATION. When you vectorize a TIFF you RGB(XX XX XX) in the file.

I may in fact make a different kind of map using DEM shading too, but that looks very different, which much smoother gradients that don't reveal the terrain detail as well. Believe me, I have tried lots of different things, and will try many more before I'm through no doubt!  ;)

It's just frustrating that I can make exactly the map I want in Mapsource, but the GPS renders it so differently. Probably has something to do with compromises they made because of the small CPU and limited RAM.

But now I've got to put this to rest for a few days while I clean up my disaster area of a house so I can party with my kids on New Years Eve. Happy New Years everyone!

Wow. Not all GPS'es are created equal. I almost didn't bother, but figured "what the heck"... Here's the same map on my Nuvi 3790. Rendering is virtually identical to Mapsource. Totally blown away by this... why can't my expensive Montana do the same thing?  ;D

(http://stephencreek.com/gpsfd/lidar/3790-04.png)


(http://stephencreek.com/gpsfd/lidar/3790-05.png)


And this effect is being enhanced by the North America DEM map on the device (gmapdem.img). But the fun really begins when you switch to 3d mode. Now I feel just a little better about this. :)


(http://stephencreek.com/gpsfd/lidar/3790-06.png)

I've been lurking on this thread... and I have to ask now, are you going to send this information to Garmin and ask why the Montana performs so poorly?

I know I want to know :)

Quote from: Boyd on December 29, 2011, 11:29:18 PM
Wow. Not all GPS'es are created equal. I almost didn't bother, but figured "what the heck"... Here's the same map on my Nuvi 3790. Rendering is virtually identical to Mapsource. Totally blown away by this... why can't my expensive Montana do the same thing?  ;D

Does your Montana do 3D buildings?

Well clearly there were different design goals for these devices. My tongue was implanted in my cheek when I made the Montana comment.  ;)  All along I've been impressed by the 3d capabilities of the 3790 (the 3490 is the new version, but I haven't seen one myself). The Montana doesn't do 3d buildings and doesn't do "real" 3d like the 3790. However it does a nice job with *real* raster imagery, and I am "faking" it here with this project. I am breaking all of Garmin's rules here and pushing their format to the breaking point.

Also remember that the 3790 screen is 800x480 pixels - almost 3x the pixel count of the Montana, but not transreflective. Each unit has its own strengths, and the Montana is still my choice "real" navigation.

Pondering this, I assume that Garmin made some compromises when rendering vector maps on screens with low pixel counts. I'm sure they never thought people would use maps with so many tiny polygons.

FWIW, my first attempt at this resulted in ~3 million polygons in one 24k quad. This most recent map cuts that down to around 60,000.

I did not realize you statement about the 72 elevation ranges was refering to a diferent method than all the 3D slope type illustrations you were describing and providing.

Hope Garmin will provide some info on why the same data displays so differently.  I have noticed some differences with Garmin's default types between my 76 and OR300. 

I used the daylight shader in GlobalMapper to create the source TIFF. The slope direction shader also works very well with this sort of flat terrain. Remember, it is very flat around here. In my earlier examples, the full range of elevations is only about 20 meters.

To make shaded terrain in a mountainous area, I'm sure I would approach it differently. But the original goal was to make a map that shows the subtle variations in a flat coastal area. Using an 8 bit grayscale TIFF, there are 256 steps that should correspond to the full range of elevations. I map these to 72 steps because that was the most custom types I could re-purpose without getting into marine colors, etc. It's an approximation, but these steps each represent a range of elevations. That should be relatively accurate in a flat area, but not so much if you are mapping the Rockies.  ;)

Clearly this doesn't solve the problem of making *real* DEM for Garmin. But we just don't know how to do that....

Wow, the nuvi does look really good.

I laugh at the Rockies comment; the grand canyon is a nightmare of topography.

It would be nice to make our own dem for the units but the good news is many of the newer ones include it so our maps can just lay on top.

Quote from: -Oz- on December 30, 2011, 11:15:52 AM
Wow, the nuvi does look really good.

I laugh at the Rockies comment; the grand canyon is a nightmare of topography.

It would be nice to make our own dem for the units but the good news is many of the newer ones include it so our maps can just lay on top.

Alas for those of us too cheap to spend the extra $100 for the DEMs.  AFAIC the Grand Canyon is more of a dream than a nightmare.

Third party support of the .jnx format would also do.

Soon I will make a test using my technique for an area with real mountains. There's about a 3GB file pre-installed on the 3790 called gmapdem.img that is just the DEM data. Pretty cool, if you connect the unit and look at that map in Basecamp, it looks like the lunar surface. Too bad they don't sell this as a separate product. My *guess* is that it's the same data as topo 100k, but also contains the rest of North America. Here's your Grand Canyon as seen with City Navigator :) Maybe I will use that as a target for my next test. The way the 3790 maps images on terrain is very impressive.

(http://stephencreek.com/gpsreview/3790/3d01.jpg)

Quote from: babj615 on December 30, 2011, 12:47:35 AM
I have to ask now, are you going to send this information to Garmin and ask why the Montana performs so poorly?

If I did, it would probably go something like this:

Boyd:
When I make a map containing millions of pixel-sized polygons using unapproved software, why doesn't the Montana make it look pretty?

Garmin:
Are you serious?....

;D

Quote from: maps4gps on December 30, 2011, 11:42:05 AM
Third party support of the .jnx format would also do.

Mike's todo list on GlobalMapperForum.com (Preferred GM site now) has JNX export on it. 

I recall someone mentioned that last Spring.
If it can be added to GM, I was hoping someone would write a utility program to do it.

Interesting that Globalmapper would do this, seeing as there is no approved way to use such a file on a Garmin device. AFAIK, the firmware only supports Garmin's own Birdseye .jnx files.

Quote from: Boyd on January 01, 2012, 10:18:39 AM
Interesting that Globalmapper would do this, seeing as there is no approved way to use such a file on a Garmin device. AFAIK, the firmware only supports Garmin's own Birdseye .jnx files.

Could you elaborate?  I don't use Birdseye, so I don't know.  IIRC using JNX files involved a firmware hack. Is your point that GM's JNX files would be useless without a firmware hack?

Yes, exactly. It's also my understanding that, for this reason, the topic is not appropriate for discussion here.

After a lot of trial and error, I now have

* Color terrain based on the slope direction shader
* Very similar rendering between Mapsource and a variety of devices
* Map tiles based on USGS 24k quads with ~110,000 polygons

These are "blockier" than I would like, but they seem to need to be like this in order to render correctly on everything but the Nuvi 3790. IT works on the 60csx, but has pretty questionable value... it really needs a high resolution screen.

I processed the full resolution TIFF file in Photoshop where I used the Pixellate > Mosaic filter with a setting of 8, then applied a custom 32 color palette. This was imported back into Globalmapper and vectorized with a match setting of 16. Using the 32 color palette seemed to be the trick that finally made this work, by creating larger polygons due to a more limited number of colors to match.

This probably sounds harder than it really is. Now that I have a methodology, I can put together the whole map relatively quickly - it will only contain 12 tiles. Not quite what I had in mind when I started out, but not sure how much better I can get - although I'm going to continue trying.  :)

(http://stephencreek.com/gpsfd/lidar/slope01.png)

Looks good to me. Congratulations.

I have been spending WAY too much time on this, but I've learned quite a bit along the way....

* Mapsource does a really nice job of rendering the map, both in terms of color and detail, not to mention speed.

* While some of the earlier screenshotslook nice, it just doesn't look very good on the Montana's screen. I guess it's a trade-off when making a good transreflective screen - colors are very de-saturated and lack contrast. My Oregon 400t was much worse. 60csx is not too different either, but the nuvi's with their bright backlights look more like the computer screen.

* Small polygons just don't render correctly on everything but the Nuvi 3790 - they turn into blobs. Some rounding of coordinates must be happening.

Now I have managed to get a good workflow at making this kind of map - about 20 minutes to make one finished .img map tile. So I made a test map in the previous style with 6 USGS quads, and it looked great in Mapsource but terrible on my Montana. So back to the drawing boards once again...

I decided that the map needed to be more colorful to look good on the GPS and also convey information about elevation, so I turned to the Globalmapper Atlas shader. It was a long process sorting out the best way to do this, and I will skip the details and only say, "it's all about the palette". Here I was struggling to map 256 colors into 72 custom types before, but that was exactly the wrong approach.

Instead, the best results are produced with very small palettes. I settled on only 16 colors after a lot of experimentation. And the whole key to making this work is to use a setting of match distance=0 when vectorizing in Globalmapper. With a small palette, this produces more pixels that match and therefore larger polygons. The tiles I'm currently using have ~50,000 polygons each and result in .img files ~2.5MB each. They are pretty responsive on my Montana, although it's probably Garmin's fastest unit.

I am using a little trick to help keep the polygon count down. I chose a color that seems to appear a lot in these maps - and often as very small polygons. When processing in Filemaker, I flag these polygons, then delete them after re-importing the shapefile. I have my map background defined as this color in the custom .typ file, so you see through the missing polygons to the background, giving the same effect as if they were still there. This results in ~15% few polygons in my tests so far. Further study might improve this.

I also have another method, but not really using it yet. I can use Globalmapper to generate an area and perimeter attribute before exporting the shapefile. I could then use this data in filemaker to discard small polygons within a given range of colors (close to the background color). I suspect there's a lot of image processing like this that's possible, but not sure if it's needed since I've already greatly reduced polygon count.

Still tweaking this, but here are some examples comparing Mapsource to the Montana. No matter what I do, the Montana just refuses to render as much detail as Mapsource. However, in the sequence below you will notice that as you zoom in on the Montana, the detail seems to actually be there. You just need to zoom in much farther to get it to show.

(http://stephencreek.com/gpsfd/lidar/atlas01.png)



(http://stephencreek.com/gpsfd/lidar/atlas02.png)



(http://stephencreek.com/gpsfd/lidar/atlas03.png)

Quote from: maps4gps on December 29, 2011, 07:43:09 PMWhen GM displays a DEM, it displays a combination of the elevation data and slope shading it creates from the adjacent elevation values.

Just wanted to add that you are quite correct here. Nevertheless, I like the effect of hillshading.  :) The following graphic shows the range of elevations (in feet) in the images above. This scale is valid over the full range of the map I'm making - which is basically the whole southern part of New Jersey. As I said, it's really flat around here!

(http://stephencreek.com/gpsfd/lidar/elevationlegend.png)

This sure is an interesting thread and I am enjoying reading the posts. Maybe I will even learn something somewhere along the way.

Not sure I believe the Montana processor to be faster than a Nuvi 3790, however (but that is another thread)  ;-)

Thank you for the enlightenment!

Thanks. For some reason, the Nuvi 3790 balks when using a highly detailed map as soon as you zoom out past .2 miles. Have seen this with several maps I've made. Seeing the same thing here too.

"Still tweaking this, but here are some examples comparing Mapsource to the Montana. No matter what I do, the Montana just refuses to render as much detail as Mapsource. However, in the sequence below you will notice that as you zoom in on the Montana, the detail seems to actually be there. You just need to zoom in much farther to get it to show."

How many levels and what setings?  If you zoom in and the detail is there, it sounds like a level issue. 

You may be right - thanks, will have a look. The whole concept of zoom levels is something I've just never grasped very well.  :-\

But wouldn't that also affect the way the objects look in Mapsource?

I've never really understood them either.  I just experminted a lot and found the settings that worked the best.

Maybe you do like I do, set my gps to normal detail and MapSource to most.  If MapSource is set to med detail, the different layers kick in at the same zoom as the GPS.  But when mapsource is set to highest, it shows the greatest detail at amuch higher levels.

I think there are some limits if you set it to 24 bits on how far you can get it to display.  For My Trails, I set zoom 0 to 23 bits as I had too many problms with zoom if I set it to 24 bits.  So I would suggest trying zoom 0 23 bits. 

Quote from: jbensman on January 05, 2012, 05:25:16 PM
I've never really understood them either.  I just experminted a lot and found the settings that worked the best.

Haha, here I assumed you were the master of zoom levels, based on all the things you've posted in the past! You certainly seem to have a better understanding than I do.

Nevertheless, after lots of study I'm not sure that's the issue. Seems more like different rendering engines on different platforms. But I've made some major progress today, at the expense of considerable time. It really is all about the palette. Between this and simple use of the gaussian blur filter in Photoshop, I'm getting results I like a lot more. It looks more like a map.  :) I apply the blur filter in RGB colorspace, then convert to my 16 color indexed palette. The result is banding, which creates nice smooth, curvy shapes.

Polygon count is greatly reduced... but the polygons have gotten bigger and more complex, so it's a bit of a trade off. But these look more like the kind of shapes that a GPS is intended to render. Mapsource still shows a bit more detail, but I'm starting to get convergence between PC and GPS.

(http://stephencreek.com/gpsfd/lidar/atlas04.png)


And it looks much more attractive when you zoom in.

(http://stephencreek.com/gpsfd/lidar/atlas05.png)

Wow, Boyd, the MapSource vs Montana images are almost indiscernible!

I can't even remember how many times I've thrown everything away and started over from scratch again on this project! But it's finally paying off and I'm getting results that work well on the Montana as well as Mapsource. I have sure learned a lot along the way, all of which makes me more interested than before in converting raster imagery to Garmin's vector format.

Will want to work on this for a little while longer, but will upload the finished map here before too long. BTW, I just started using TypWiz 2 Beta yesterday with my custom types file. It has a few little quirks, but overall a terrific program - get it here: http://pinns.co.uk/osm/ostyp.html

This new version has its own .typ compiler. This addresses my issues with the old version that used cgpsmapper to compile the .typ files. Cgpsmapper does not support all of the advanced styles, such as text sizes and colors so that was an issue with my existing files. But the new version seems to handle this very nicely, although I had to manually make some changes that didn't seem to import correctly from the file I created with the online editor.

(http://stephencreek.com/gpsfd/lidar/mt06.jpg)


(http://stephencreek.com/gpsfd/lidar/mt07.jpg)


(http://stephencreek.com/gpsfd/lidar/mt08.jpg)


(http://stephencreek.com/gpsfd/lidar/mt09.jpg)

I have been gradually expanding my color palette in order to capture more detail in the lowlands (which were mostly just flat green looking before) and have worked back up from to 32 colors). Seemed to be going well until I hit a quad that was especially "bumpy" and I ended up with over 73,000 polygons.

Cgpsmapper seemed to just roll over and go to sleep over this - hadn't completed after an overnight compile. Kind of surprised me since everything else I've done recently has compiled within about 5 minutes, albeit with less polygons. So I've spent today implementing some basic image compression algorithms in Filemaker. After importing the shapefile I can control parameters that discard polygons which meet certain criteria.

(http://stephencreek.com/gpsfd/lidar/filemaker2.png)

This really pays off - just reduced my test 24k quad from 73,000 polygons to 19,000 with almost no loss in quality - see below. Could probably improve further on this, but I think I'm just gonna make this map now. :)

(http://stephencreek.com/gpsfd/lidar/compress5.png)

(http://stephencreek.com/gpsfd/lidar/compress6.png)

(http://stephencreek.com/gpsfd/lidar/compress1.png)

(http://stephencreek.com/gpsfd/lidar/compress2.png)

(http://stephencreek.com/gpsfd/lidar/compress3.png)

(http://stephencreek.com/gpsfd/lidar/compress4.png)

Still searching for the magic formula :)

(http://stephencreek.com/gpsfd/lidar/compress7.png)

(http://stephencreek.com/gpsfd/lidar/compress8.png)

(http://stephencreek.com/gpsfd/lidar/compress9.png)

Well I have come a long way in the past couple days, but it hasn't been easy. Using every trick I could come up with, I've gotten a lot more detail into the map. I seem to have hit the limits of cgpsmapper and was having trouble getting these to compile at all. After messing around most of the day troubleshooting, I concluded that the polygons are just too complex. I finally solved that by gridding the terrain into tiny pieces, then reassembling it back into a 24k quad. This slices everything into small polygons and cgpsmapper will compile a full quad in about 3 minutes.

At this point, I just need to sit back and be sure that this is the correct level of detail. The map tiles are pretty big - 3 to 4 mb for a 24k quad. But I like the way they look. :)

(http://stephencreek.com/gpsfd/lidar/gm+ms.jpg)

Have you tried using mkgmap to compile these maps? As long as you set the memory usable by Java to a high-enough level (use the Java control panel), it seems to do better on files with a large number of polygons than cgpsmapper, and it's faster as well.

Thanks, will have to look at that. I tried Mapwel and it was pretty fast but the .img file didn't have all the polygons in it. I may just stick with my current cgpsmapper kludge until I finish this project though, since the compiles are now very fast and I've never used mkgmap.

I tried slicing each of my quad-sized tiles into 4 small ones and that gets the individual .img files down to 1MB or less. But it didn't seem to perform any faster on the GPS. So I'm going to resample the images with a bit less resolution, and maybe filter out more of the really little objects. My general sense is that a 24k quad-sized .img tile shouldn't be more than ~2.0 to 2.5MB for acceptable performance on the GPS. Too bad, because Mapsource can handle 5 or 6MB .img tiles with no problem.

Quote from: leszekp on January 12, 2012, 09:55:31 AM
Have you tried using mkgmap to compile these maps? As long as you set the memory usable by Java to a high-enough level (use the Java control panel), it seems to do better on files with a large number of polygons than cgpsmapper, and it's faster as well.

What version of mkgmap are you using, and are you using it on MP files or OSM?  I follow the mkgmap mailing list, and it still sounds like a work in progress.  Are you generating routable maps, and if so, is address search working reliably for you?

The last time I tried it on MP files I got some funky edges between adjacent tiles.  Since then, I just use it to compile OSM data that I convert to MP and compile w/ cgpsmapper.

I'm not doing routable maps, just maps with custom TYP files (including subtypes) and lots of polygons. I've been using release 1995, though I may try the most recent release series (21xx) in the near future.

My kludge is working well now, so I'm just sticking with it. I uset the export > grid option to create a lot of little tiles, then re-import and combine into a single layer. These compile really fast in cgpsmapper, typically less than a minute. The longest was about 7 minutes.

I have 24 out of 55 quads finished and the .img files are mostly 1mb - 2mb, with the largest being 2.4mb. These seem to perform well on my Montana, even in the car but will do some more serious testing soon.

(http://stephencreek.com/gpsfd/lidar/big2.jpg)

Did a brief road test today, and it works very well, even on the lowly Nuvi 205. Also works on the 60csx, but screen updates are rather slow.

(http://stephencreek.com/gpsfd/lidar/roadtest1.jpg)


3d mode on the 3790 looks pretty cool. Would love to see how it looks on the new 3550 which has a 5" glass multi-touch screen. :) https://buy.garmin.com/shop/shop.do?cID=401&pID=74086

(http://stephencreek.com/gpsfd/lidar/bearswamp.png)

Well if you read back to the beginning of this thread you'll see that my original goal was to produce a map covering 12 USGS 24k quads. I got more ambitious and upped that to 48 quads, then 55 quads... and the final product will now contain 76 quads which is about half of New Jersey. :)

(http://stephencreek.com/gpsfd/lidar/coverage.png)

Each quad that I finished made me wonder what the next one would look like, and so forth. I'm pretty much at my limit here due to the fact that my elevation color scale only accommodates terrain up to 260 feet, so this map needs to stay in the lowlands. But I wanted to do a big enough map to be a "proof of concept" that a large area of raster imagery could be converted to garmin's vector format and still yield decent performance.

Took a longer test drive yesterday and it worked really well on my Montana, even with City Navigator also active to provide routing (my map has a draw priority of 31 and hides City Navigator).

(http://stephencreek.com/gpsfd/lidar/mt10.jpg)

It works on the 3790, but in 3d mode with City Navigator also enabled it really taxes the poor nuvi to its limit and the device is very slow to respond to screen taps. Changing to 2d mode and/or turning off City Navigator provides better performance.

So I have a few more quads to crank out and this project will then be ready for upload. But I'm already eyeing some future raster conversion projects. NJ has some really nice black and white orthoimagery from 1930 available with a source resolution of about 2m/pixel. Just did a quick test and Iit should be very similar to this LIDAR project in terms of tile size.

From what I've learned so far, you should convert imagery to ~40 ft/pixel in order to keep the polygon count within reason. This makes a USGS 24k quad about 1100x1100 pixels. These look good on the GPS at zoom settings up to about .2 miles but start to become abstract art as you zoom in further. The real trick is HOW to process the raster image before converting to polygons. :)

Is that really 2m orthoimagery from 1930 - 80 years ago?

Yes, quite a remarkable resource. You can download by setting up a wms entry in globalmapper for http://njwebmap.state.nj.us/njimagery

I suppose you could debate what the true spatial resolution is (maybe 3m?), but that is the resolution it's being served at (according to Globalmapper). It is certainly not as good as the 2007 1 ft/pixel imagery, but considering the age... it ain't bad. :)

There's also a lot of cool stuff at historic aerials, if you've never looked there. http://www.historicaerials.com/aerials.php?op=home

I purchased a bit of their imagery from the 1950's that covers an area near me with some historic ruins. This was very high quality stuff, probably 1 meter/pixel. Fun website to explore using the free viewer.

Quote from: Boyd on January 21, 2012, 12:30:05 PM
There's also a lot of cool stuff at historic aerials, if you've never looked there. http://www.historicaerials.com/aerials.php?op=home

Wow, thanks for that link!

I see many hours of my life slipping away..... quickly.... :)

Quote from: maps4gps on December 29, 2011, 07:43:09 PMWhen GM displays a DEM, it displays a combination of the elevation data and slope shading it creates from the adjacent elevation values.

I have finished all my map tiles - it keeps getting bigger, and I have to stop somewhere  ;) . So today I revisited the idea of using the text displayed by custom types as elevation labels. Turns out that you *can* do this; the shaded areas appear to only have their brightness altered and not their hue, so they can still be matched. But again, "it's all about the palette" and mine was chosen for visual effect, not elevation accuracy. I generated this with GlobalMapper and Filemaker.

(http://stephencreek.com/gpsfd/lidar/elev01.jpg)


Just as I was starting to refine this, it hit me that my approach was completely wrong. GlobalMapper can calculate all kinds of elevation and slope statistics for any polygon if you also load DEM on another layer. So that's what I did, then exported as a shapefile where Filemaker massaged the data.

(http://stephencreek.com/gpsfd/lidar/elev02.jpg)


This gets imported back into GlobalMapper and the elevation becomes the name for the polygons. Note that, if desired, I could also have included other statistics on the map such as the range of elevations in each polygon, its average slope, etc. What is actually displayed is the result of how I format the data for export in Filemaker. In the examples below I have just rounded off the values GlobalMapper calculated for average elevation.


(http://stephencreek.com/gpsfd/lidar/elev03.jpg)


Notice how this is actually very high resolution - since it was generated from the 1/9 arc second LIDAR DEM

(http://stephencreek.com/gpsfd/lidar/elev04.jpg)


The "jaggy" polygons above are the result of me simplifying them with a 4 meter threshold. No elevation data is available in the "Swiss cheese" holes. These are part of my data compression scheme, where polygons have been removed to allow the background to show through. With a lower resolution map (one with less polygons), this wouldn't have been necessary.

Now on the GPS, I don't think I'd display the elevations on the map because it will be very cluttered. But if I use invisible text in the custom type file, I think the elevation will be displayed when clicking or mousing over the map.

But I am going to save this technique for my *next* map, since I don't want to recompile the 80 some quads in the current project right now.  :) I should be on track to put this map online sometime this week.

Have been testing how well elevation can be estimated by palette colors: I analyzed over 125,000 polygons (15 quads) with filemaker to compare my estimated elevation range for each color with actual DEM data.

For example, the table below shows 2509 polygons are filled with a dark blue/green color coded as custom type 0x20. I have chosen a range of 20-35 feet, which is correct 87% of the time.

The true range of elevations represented by that color is actually 2-42 feet, but I'm settling for  being right 87% of the time, in order to narrow down the range of elevations. Kind of an interesting study in statistics... look at 0x32 for example; the actual range of elevations in the 2502 polygons analyzed is 8-38 feet, but 89% (2227) fall within the 20-25 ft range. As you can see, some colors are better at predicting elevation than others.

Might tweak this just a bit more, but it's basically what I'll use, so I can upload version 1 this weekend. The next version will have much more accurate elevation data. :)

(http://stephencreek.com/gpsfd/lidar/analyze_elev.png)

A universal version of this map is now available for download here: http://www.gpsfiledepot.com/maps/view/276

:)

Now that the first version is finished, I'm wondering whether SRTM would have been a better choice of imagery for the areas without LIDAR coverage: http://eros.usgs.gov/#/Find_Data/Products_and_Data_Available/SRTM

Resolution is only 1 arc second as compared to NED 1/3 arc second data, but it has a "look" that's more similar to LIDAR. In the examples below, the images on the top show a full USGS quad and the images below show a full size (pixel for pixel) view of both kinds of data after pre-processing with my techniques. Maybe blending both together might produce something that looks more like LIDAR.

(http://stephencreek.com/gpsfd/lidar/SRTMvsNED.png)

BTW, getting back to some discussion way back at the beginning of this thread, have you seen the USGS Center for LIDAR Information Coordination and Knowledge website (CLICK)? Some cool stuff.  8) http://lidar.cr.usgs.gov/

Once again, nice work Boyd. 
Your are approaching what cartographers call a shaded hypsometric map.  Shading to give the impression of relief and hypsometry by using colors to denote elevation ranges. 



Some observations on your recent posts.

Mp_type is showning na accuracy of 284%.  Whatever is causing this, may also be effecting the accuracy values for other my_type.

QuoteResolution is only 1 arc second as compared to NED 1/3 arc second data, but it has a "look" that's more similar to LIDAR.

1. You started this thread using 1/9 arc second LIDAR data; did I miss where you changed to 1/3 arc second? 
2. If have seen the professionals state the 95% limits of vertical accuracy of SRTM data is 13.6 meters.  Which is more important, the 'look' or the accuracy of the data? 

How do all these colors affect the land-use displays you were doing?  Or are they not intended to be used together?

Thanks!  :) That's a big word I hadn't heard before (hypsometric). Yes, I noticed the impossible 284% number but I believe it has to do with bogus data from polygons with no elevation values. I think the others are correct, but should check the function I wrote to analyze this because that shouldn't be happening. There's clearly a flaw in the logic somewhere.

But that was just sort of a statistical exercise I got interested in, and a way to get some handle on how accurately the color represented elevation. As I mentioned above, my next update will have very accurate elevation data averaged for every polygon on the map from the source DEM. The current version only has 37 palette colors, so it can only represent 37 different ranges of elevation - not very accurate by any measure.

Yeah, the SRTM data is not very high resolution, but didn't realize the vertical res was so poor. Have just been playing with a test quad however and I used Globalmapper's function to combine terrain layers to average the SRTM and NED 1/3. Visually, it looks a lot more like LIDAR, but of course it really isn't. Among other things, SRTM is what they call "first return" data that bounces off whatever the highest object is (as I understand it) whereas LIDAR shows "bare earth".

LIDAR data is 1/9 arc second, but I don't have it for the whole map so I used NED 1/3 arc second data in those areas. The SRTM data would only be used to augment the 1/3 arc second DEM.

For this particular project, I am definitely emphasizing "look" over accuracy. After a career as an artist/designer, that's just the way I'm wired up. :) Really, this project is largely about ways to convert raster imagery (any kind) to Garmin's vector format. The whole map covers almost 5,000 square miles and I'm not sure whether anyone has created a Garmin vector map from raster imagery on a comparable scale before.

I did some early experiments with combining landcover data - like forest shading - with the shaded terrain. It seemed kind of confusing from a visual standpoint, so I decided to just let this map concentrate on the terrain.

I am going to use my averaged SRTM and NED 1/3 data for future projects where NED 1/9 isn't available. There could be issues with accuracy, but the SRTM just "feels" better to me. This comparison shows Mays Landing, near where I live. The ground is definitely "bumpy" here and not as smooth as NED 1/3 would imply.

(http://stephencreek.com/gpsfd/lidar/srtm01.jpg)

Have gone back to the beginning of this thread and started playing with grayscale imagery again, to see if I can create more detailed maps. It's a mixed bag... using a small number of shades of gray produces fewer polygons however they get huge and complex. This is a bigger problem than producing lots of little polygons because it can't be compiled by either cgpsmapper or mkgmap.

To make a long story short, I can get a little better detail with grayscale and may be able to improve slightly more, but I'm really up against the limits of how complex a Garmin vector map can be while still performing acceptably on a GPS.

In the example below, I have only used 4 shades of gray. I'm pretty surprised (and happy) that GlobalMapper and Mapsource render these exactly the same.

But there's another completely unrelated thing that I find interesting here. Even though map was made with Lat/Lon WGS 84 and Mapsource is also set that way, I have to set GlobalMapper to State Plane NAD 27 to get the two programs to match on the computer screen.

(http://stephencreek.com/gpsfd/lidar/gray4.jpg)

Was the source NJ LIDAR data in State Plane 27?
Are you using a template .mp file and telling GM to output the data in SP27?

I think everything is fine with my map. Mapsource just doesn't display it in the correct proportions like Globalmapper. But it's all coming back to me now.... this was discussed at length in Garmin's forums as well as other place. Go to the Garmin forum and search for "projection" in the Mapsource forum.

This gives a clue as to what I'm seeing.... from one of the developers. Garmin made some kind of change to this in an earlier version, causing circles to become ellipses on the screen and this upset a number of people.  ;)

https://forums.garmin.com/showthread.php?t=7049

Quote from: FALAGARA few remarks about the projection issue: our ultimate goal is to fix the projection for all regions (just like 6.13.7). We are not quite there yet, so we use band-aids trying to fix the projection for the majority of our users.

For small map products we now (as of 6.16.0.2) use the center of the map as a reference point. So there will be no distortion at the center. The further North or South from the center you go the worse the distortion will get. This works fairly well for most of our products.

For larger products this is not ideal. For Canada especially this did not work so we put in a special fix that should make Canada-only maps less distorted where it matters (in the South).

We still allow users to change the projection angle in the registry. We decided not to openly provide a user interface for this, but made it a bit easier to change the angle without having to meddle directly in the registry. You now can hit Ctrl-Alt-A in MapSource to set the projection angle for the current map product. You will need to restart MapSource for this to have any effect.

Changes to the projection angle that you made in 6.16.0.2 will not be used (the registry location for the angle has changed). You will have to set the angle again. This was done so that the projection angle adjustment can be shared amongst Garmin applications (BaseCamp 3.0.x & MapSource 6.16.x will now both use the angle you set).

So, Mapsource is using the center of the map for projection and, if I'm not mistaken, that would look pretty much the same as State Plane.

Quote from: Boyd on February 06, 2012, 03:37:27 PM
Have gone back to the beginning of this thread and started playing with grayscale imagery again, to see if I can create more detailed maps.

Believe it or not, ten years have passed, I'm still here and still doing "Experiments with LIDAR"!  ;D Still working with grayscale imagery too, but with 1-meter resolution DEM in 3d with my own web app. Garmin devices and software just can't support the maps I want to make anymore.

https://boydsmaps.com/#13.00/40.357081/-77.046754/mbx3dmidatl/69.60/70.20

(https://stephencreek.com/gpsfd/lidar/desktop.png)

My web app is free, there's no advertising, no registration and it works on any device.

(https://stephencreek.com/gpsfd/lidar/mobile.png)

This has been my most ambitious map-making project yet, ten million map tiles (over 260gb), covering a portion of the US Mid-Atlantic region. Am planning to expand the map to a much larger area later this summer. This is the current coverage

(https://stephencreek.com/gpsfd/lidar/midatlantic.png)

There's also a 2d version of the map which loads faster and uses less data, since there's no 3d elevation data.

https://boydsmaps.com/#13.00/40.991210/-75.132460/midatlidar/0.00/0.00

Still using Globalmapper for almost everything, finally upgraded to version 23 this year. My app uses the open source Maptalks API however the 3d maps use Mapbox GL/JS inside a Maptalks container. Has been quite an education, needed to figure most things out for myself as there is not much documentation on building 3d maps from your own data and hosting them yourself.

I put all the DEM together in 32-bit floating point TIFF files which are imported into the Mapbox rasterio command-line program, then exported in Mapbox RGB elevation format. I wrote my own code to read elevation from these files and show it in a realtime "heads up" display on the map.

As a companion project, I used NASA ASTER 1 arc-second DEM to make a lower-resolution 3d map covering all of North America. Used USGS National Landcover Dataset and ESA Worldcover data (outside the US) with my own pseudo natural color pallete to produce a more "artistic" 3d view, along with atmospheric effects.

https://boydsmaps.com/#12.80/44.099767/-71.572680/mbx3dnalc/48.60/76.80

This was more of an "art project" for fun, but the elevation data is used in the same way as the other map, providing a realtime elevation display anywhere in North America as you move the map around. Still a big project, but "only" about 4 million tiles / 100gb.

(https://stephencreek.com/gpsfd/lidar/aster.png)

There are over 200 other maps available in my app, from servers in many different US states, Canada and even Switzerland. Also OpenTopo, USGS National Map, live weather forecasts and radar plus a number of maps from HERE (the company that provides data to Garmin for City Navigator). I support waypoint import/export in Garmin-compatible GPX files, plus my own format that can be customized and imported/exported in Excel. Not supporting tracks yet, but will get around to it eventually.

Anyway... that's what I've been up to. How did you spend the last ten years?  ;D  Is anybody from this thread still here? Quite a nostalgia trip revisiting it now, I do miss the old days.

Quote from: Boyd on June 07, 2022, 11:45:24 AM
Am planning to expand the map to a much larger area later this summer.

The new, expanded map is now available! This is the new coverage area - over twice the size of the original map, more than 600 gb with almost 22 million map tiles!  8)

(https://stephencreek.com/gpsfd/lidar/coverage-v2.png)


Here's Shenandoah, Virginia in 3d

https://boydsmaps.com/#11.71/38.454567/-78.602919/mbx3dmidatl/40.20/73.20 (https://boydsmaps.com/#11.71/38.454567/-78.602919/mbx3dmidatl/40.20/73.20)

(https://stephencreek.com/gpsfd/lidar/shenandoah.png)

3d view of the Catskills

https://boydsmaps.com/#14.00/41.998428/-74.386694/mbx3dmidatl/-174.00/72.00 (https://boydsmaps.com/#14.00/41.998428/-74.386694/mbx3dmidatl/-174.00/72.00)

This is how it looks on a phone.

(https://stephencreek.com/gpsfd/lidar/poconos.png)

You can view the 3d map in 2d North-up mode if you prefer - just click the compass to toggle between 2d/3d. But there's a separate 2d version of the same map without the elevation data if you aren't interested in 3d views, it loads much faster (note that 3d maps appear in blue text in the map menu)

Here's the 2d map of Martha's Vineyard, for example.

https://boydsmaps.com/#11.71/41.398941/-70.647387/midatlidar/0.00/0.00 (https://boydsmaps.com/#11.71/41.398941/-70.647387/midatlidar/0.00/0.00)

(https://stephencreek.com/gpsfd/lidar/martha.png)

I would like to continue expanding this map to cover an even larger area in the future. I still have over 200gb free disk space on the server but need to find the best use for it. Ultimately, I'll need to lease additional disk space to support future expansion.

It's not practical to create a map this large in one piece, so I divide it into smaller blocks of 100gb to 150gb of uncompressed 32-bit floating point DEM data. After processing this, I end up with blocks of a few hundred thousand .png map tiles which are ~1/4 the size of the source files. Then there is a companion set of .png tiles that contain the elevation data in Mapbox RGB format. The elevation tiles only use about half the space of the map tiles.

Before creating the RGB tiles, the geoTIFF DEM source file must be processed with Mapbox rasterio, which is a rather complex command-line program

https://blog.mapbox.com/rasterio-gdal-especially-for-python-programmers-553dabf7dd71 (https://blog.mapbox.com/rasterio-gdal-especially-for-python-programmers-553dabf7dd71)

This encodes the DEM data as a .tif raster image which can be exported as .png map tiles. It took awhile to figure out exactly how to do all of this, Mapbox does not really support creating your own RGB DEM, they expect you to use their own pre-packaged DEM. But their DEM is about 5-meter resolution and I wanted 1-meter DEM. Using their own content can also result in usage charges but it's free to use their API with your own data. But you're pretty much on your own, I could find very little information about how to implement all of this locally but eventually figured it out through trial and error.

I create all the the tiles with Globalmapper using the option to only export fully-covered tiles, this prevents gaps between different blocks of data when I merge them with the full map. Finally, I zip the tiles, upload them to the server and unzip.

There are so many steps to the process and so many different files, I had to create a spreadsheet to track my progress!

Quote from: Boyd on June 07, 2022, 11:45:24 AMBelieve it or not, ten years have passed, I'm still here and still doing "Experiments with LIDAR"!

Another year has passed... still doing "Experiments with LIDAR"  :)  You can also do some "experiments", just click the button with the blue cube on my site (https://boydsmaps.com/#14.00/37.823792/-79.914047/legacy24k/0.00/0.00) to open the Terrain Viewer. You can also link directly to the viewer, for example:

https://boydsmaps.com/terra/#37.823766/-79.914004/-877/1766/1578/0/292/0/2000/775/-2000/56.99999999999999/2/3d/shader40/0/2/z15 (https://boydsmaps.com/terra/#37.823766/-79.914004/-877/1766/1578/0/292/0/2000/775/-2000/56.99999999999999/2/3d/shader40/0/2/z15)

(https://stephencreek.com/gpsfd/lidar/terra1.png)

The Terrain Viewer takes a "slice" of the map and exports it as a 3d model where you have many options for controlling the appearance.

There are sliders for the XYZ coordinates and intensity of the "sun" (a directional light source). Observe the effect as you move the sliders, the changing lighting will reveal different details in the terrain. Another slider changes the vertical scale of the terrain (vertical exaggeration) in realtime, which can make a big difference in appearance.

You can choose 6 different map types: aerial imagery from NAIP (the National Agriculture Imagery Program)

https://boydsmaps.com/terra/#40.615801/-74.822234/-1273/1059/-2/0/46/0/1000/437/1014/75/2/3d/ortho/0/0/z15 (https://boydsmaps.com/terra/#40.615801/-74.822234/-1273/1059/-2/0/46/0/1000/437/1014/75/2/3d/ortho/0/0/z15)

(https://stephencreek.com/gpsfd/lidar/terra3.png)


The grayscale option is just a white surface and the shadows create the shading.

https://boydsmaps.com/terra/#41.460554/-71.944232/-676/454/891/0/250/0/0/465/-1577/52/2/3d/grayscale/0/0/z17 (https://boydsmaps.com/terra/#41.460554/-71.944232/-676/454/891/0/250/0/0/465/-1577/52/2/3d/grayscale/0/0/z17)

(https://stephencreek.com/gpsfd/lidar/terra4.png)

The shaders (5, 10, 20 or 40 colors) use my own code to set the color of each pixel based on its position between the minimum and maximum elevation of the map.

Each map consists of 4.2 million elevation samples in a 2048 x 2048 grid. The spacing between samples determines the map resolution and size, the highest resolution has a sample spacing of 4 feet (1.2 meters) and coverage area of about 1.5 x 1.5 miles (about 2.5 km).

https://boydsmaps.com/terra/#39.251821/-76.735124/0/67689/0/0/314/0/-986/600/-986/45/3/2d/shader20/0/0/z17 (https://boydsmaps.com/terra/#39.251821/-76.735124/0/67689/0/0/314/0/-986/600/-986/45/3/2d/shader20/0/0/z17)

(https://stephencreek.com/gpsfd/lidar/terra2.png)

The viewer uses webGL with the three.js API (https://threejs.org/manual/#en/fundamentals) and the same RGB-encoded elevation dataset as my main web app. I use the MapTalks API (https://maptalks.org/) to cover an HTML canvas with RGB tiles, then populate an array using elevation values decoded from each pixel's RGB values. That array is then used to create a 3d mesh. For the color shaders, I iterate through the array, creating another HTML canvas where each pixel is colored according to its elevation. That canvas is then mapped onto the 3d mesh.

The detail available in the terrain viewer revealed some major flaws in my original Mid-Atlantic elevation dataset that were not so obvious before. Took quite a bit of troubleshooting to understand the problem, but eventually I found it. The issue was caused by re-sampling artifacts when I exported the RGB elevation tiles with Globalmapper. Using "nearest neighbor" sampling would have fixed the problem, but that is not an option in Globalmapper. Finally, I discovered that exporting my DEM files at the *precise* resolution for each zoom level would prevent Globalmapper from re-sampling.

I ended up re-processing my entire elevation dataset from almost 2 terabytes of uncompressed TIFF source files, producing 9 million 256x256 map tiles.Took around a week of 24/7 processing but made a huge difference in the quality of the 3d views in both the terrain viewer and the main web app.

My next project is expanding the 3d terrain coverage area to include all of New York, Massachussets, New Hampshire, Vermont and Maine, which should be available later in 2023. I've expanded the server to 2 terabytes of disk, so we'll see how much space is left when this is completed!