Experiments with LIDAR

[Boyd]

Well if you read back to the beginning of this thread you'll see that my original goal was to produce a map covering 12 USGS 24k quads. I got more ambitious and upped that to 48 quads, then 55 quads... and the final product will now contain 76 quads which is about half of New Jersey.



Each quad that I finished made me wonder what the next one would look like, and so forth. I'm pretty much at my limit here due to the fact that my elevation color scale only accommodates terrain up to 260 feet, so this map needs to stay in the lowlands. But I wanted to do a big enough map to be a "proof of concept" that a large area of raster imagery could be converted to garmin's vector format and still yield decent performance.

Took a longer test drive yesterday and it worked really well on my Montana, even with City Navigator also active to provide routing (my map has a draw priority of 31 and hides City Navigator).



It works on the 3790, but in 3d mode with City Navigator also enabled it really taxes the poor nuvi to its limit and the device is very slow to respond to screen taps. Changing to 2d mode and/or turning off City Navigator provides better performance.

So I have a few more quads to crank out and this project will then be ready for upload. But I'm already eyeing some future raster conversion projects. NJ has some really nice black and white orthoimagery from 1930 available with a source resolution of about 2m/pixel. Just did a quick test and Iit should be very similar to this LIDAR project in terms of tile size.

From what I've learned so far, you should convert imagery to ~40 ft/pixel in order to keep the polygon count within reason. This makes a USGS 24k quad about 1100x1100 pixels. These look good on the GPS at zoom settings up to about .2 miles but start to become abstract art as you zoom in further. The real trick is HOW to process the raster image before converting to polygons.

[maps4gps]

Is that really 2m orthoimagery from 1930 - 80 years ago?

[Boyd]

Yes, quite a remarkable resource. You can download by setting up a wms entry in globalmapper for

Code Select Expand

http://njwebmap.state.nj.us/njimagery

I suppose you could debate what the true spatial resolution is (maybe 3m?), but that is the resolution it's being served at (according to Globalmapper). It is certainly not as good as the 2007 1 ft/pixel imagery, but considering the age... it ain't bad.

There's also a lot of cool stuff at historic aerials, if you've never looked there. http://www.historicaerials.com/aerials.php?op=home

I purchased a bit of their imagery from the 1950's that covers an area near me with some historic ruins. This was very high quality stuff, probably 1 meter/pixel. Fun website to explore using the free viewer.

[babj615]

There's also a lot of cool stuff at historic aerials, if you've never looked there. http://www.historicaerials.com/aerials.php?op=home

Wow, thanks for that link!

I see many hours of my life slipping away..... quickly....

[Boyd]

When GM displays a DEM, it displays a combination of the elevation data and slope shading it creates from the adjacent elevation values.

I have finished all my map tiles - it keeps getting bigger, and I have to stop somewhere  . So today I revisited the idea of using the text displayed by custom types as elevation labels. Turns out that you *can* do this; the shaded areas appear to only have their brightness altered and not their hue, so they can still be matched. But again, "it's all about the palette" and mine was chosen for visual effect, not elevation accuracy. I generated this with GlobalMapper and Filemaker.




Just as I was starting to refine this, it hit me that my approach was completely wrong. GlobalMapper can calculate all kinds of elevation and slope statistics for any polygon if you also load DEM on another layer. So that's what I did, then exported as a shapefile where Filemaker massaged the data.




This gets imported back into GlobalMapper and the elevation becomes the name for the polygons. Note that, if desired, I could also have included other statistics on the map such as the range of elevations in each polygon, its average slope, etc. What is actually displayed is the result of how I format the data for export in Filemaker. In the examples below I have just rounded off the values GlobalMapper calculated for average elevation.





Notice how this is actually very high resolution - since it was generated from the 1/9 arc second LIDAR DEM




The "jaggy" polygons above are the result of me simplifying them with a 4 meter threshold. No elevation data is available in the "Swiss cheese" holes. These are part of my data compression scheme, where polygons have been removed to allow the background to show through. With a lower resolution map (one with less polygons), this wouldn't have been necessary.

Now on the GPS, I don't think I'd display the elevations on the map because it will be very cluttered. But if I use invisible text in the custom type file, I think the elevation will be displayed when clicking or mousing over the map.

But I am going to save this technique for my *next* map, since I don't want to recompile the 80 some quads in the current project right now.  I should be on track to put this map online sometime this week.

[Boyd]

Have been testing how well elevation can be estimated by palette colors: I analyzed over 125,000 polygons (15 quads) with filemaker to compare my estimated elevation range for each color with actual DEM data.

For example, the table below shows 2509 polygons are filled with a dark blue/green color coded as custom type 0x20. I have chosen a range of 20-35 feet, which is correct 87% of the time.

The true range of elevations represented by that color is actually 2-42 feet, but I'm settling for  being right 87% of the time, in order to narrow down the range of elevations. Kind of an interesting study in statistics... look at 0x32 for example; the actual range of elevations in the 2502 polygons analyzed is 8-38 feet, but 89% (2227) fall within the 20-25 ft range. As you can see, some colors are better at predicting elevation than others.

Might tweak this just a bit more, but it's basically what I'll use, so I can upload version 1 this weekend. The next version will have much more accurate elevation data.

[Boyd]

A universal version of this map is now available for download here: http://www.gpsfiledepot.com/maps/view/276

[Boyd]

Now that the first version is finished, I'm wondering whether SRTM would have been a better choice of imagery for the areas without LIDAR coverage: http://eros.usgs.gov/#/Find_Data/Products_and_Data_Available/SRTM

Resolution is only 1 arc second as compared to NED 1/3 arc second data, but it has a "look" that's more similar to LIDAR. In the examples below, the images on the top show a full USGS quad and the images below show a full size (pixel for pixel) view of both kinds of data after pre-processing with my techniques. Maybe blending both together might produce something that looks more like LIDAR.



BTW, getting back to some discussion way back at the beginning of this thread, have you seen the USGS Center for LIDAR Information Coordination and Knowledge website (CLICK)? Some cool stuff.  http://lidar.cr.usgs.gov/

[maps4gps]

Once again, nice work Boyd. 
Your are approaching what cartographers call a shaded hypsometric map.  Shading to give the impression of relief and hypsometry by using colors to denote elevation ranges. 



Some observations on your recent posts.

Mp_type is showning na accuracy of 284%.  Whatever is causing this, may also be effecting the accuracy values for other my_type.

Resolution is only 1 arc second as compared to NED 1/3 arc second data, but it has a "look" that's more similar to LIDAR.

1. You started this thread using 1/9 arc second LIDAR data; did I miss where you changed to 1/3 arc second? 
2. If have seen the professionals state the 95% limits of vertical accuracy of SRTM data is 13.6 meters.  Which is more important, the 'look' or the accuracy of the data? 

How do all these colors affect the land-use displays you were doing?  Or are they not intended to be used together?

[Boyd]

Thanks!  That's a big word I hadn't heard before (hypsometric). Yes, I noticed the impossible 284% number but I believe it has to do with bogus data from polygons with no elevation values. I think the others are correct, but should check the function I wrote to analyze this because that shouldn't be happening. There's clearly a flaw in the logic somewhere.

But that was just sort of a statistical exercise I got interested in, and a way to get some handle on how accurately the color represented elevation. As I mentioned above, my next update will have very accurate elevation data averaged for every polygon on the map from the source DEM. The current version only has 37 palette colors, so it can only represent 37 different ranges of elevation - not very accurate by any measure.

Yeah, the SRTM data is not very high resolution, but didn't realize the vertical res was so poor. Have just been playing with a test quad however and I used Globalmapper's function to combine terrain layers to average the SRTM and NED 1/3. Visually, it looks a lot more like LIDAR, but of course it really isn't. Among other things, SRTM is what they call "first return" data that bounces off whatever the highest object is (as I understand it) whereas LIDAR shows "bare earth".

LIDAR data is 1/9 arc second, but I don't have it for the whole map so I used NED 1/3 arc second data in those areas. The SRTM data would only be used to augment the 1/3 arc second DEM.

For this particular project, I am definitely emphasizing "look" over accuracy. After a career as an artist/designer, that's just the way I'm wired up. Really, this project is largely about ways to convert raster imagery (any kind) to Garmin's vector format. The whole map covers almost 5,000 square miles and I'm not sure whether anyone has created a Garmin vector map from raster imagery on a comparable scale before.

I did some early experiments with combining landcover data - like forest shading - with the shaded terrain. It seemed kind of confusing from a visual standpoint, so I decided to just let this map concentrate on the terrain.

[Boyd]

I am going to use my averaged SRTM and NED 1/3 data for future projects where NED 1/9 isn't available. There could be issues with accuracy, but the SRTM just "feels" better to me. This comparison shows Mays Landing, near where I live. The ground is definitely "bumpy" here and not as smooth as NED 1/3 would imply.

[Boyd]

Have gone back to the beginning of this thread and started playing with grayscale imagery again, to see if I can create more detailed maps. It's a mixed bag... using a small number of shades of gray produces fewer polygons however they get huge and complex. This is a bigger problem than producing lots of little polygons because it can't be compiled by either cgpsmapper or mkgmap.

To make a long story short, I can get a little better detail with grayscale and may be able to improve slightly more, but I'm really up against the limits of how complex a Garmin vector map can be while still performing acceptably on a GPS.

In the example below, I have only used 4 shades of gray. I'm pretty surprised (and happy) that GlobalMapper and Mapsource render these exactly the same.

But there's another completely unrelated thing that I find interesting here. Even though map was made with Lat/Lon WGS 84 and Mapsource is also set that way, I have to set GlobalMapper to State Plane NAD 27 to get the two programs to match on the computer screen.

[maps4gps]

Was the source NJ LIDAR data in State Plane 27?
Are you using a template .mp file and telling GM to output the data in SP27?

[Boyd]

I think everything is fine with my map. Mapsource just doesn't display it in the correct proportions like Globalmapper. But it's all coming back to me now.... this was discussed at length in Garmin's forums as well as other place. Go to the Garmin forum and search for "projection" in the Mapsource forum.

This gives a clue as to what I'm seeing.... from one of the developers. Garmin made some kind of change to this in an earlier version, causing circles to become ellipses on the screen and this upset a number of people. 

https://forums.garmin.com/showthread.php?t=7049

A few remarks about the projection issue: our ultimate goal is to fix the projection for all regions (just like 6.13.7). We are not quite there yet, so we use band-aids trying to fix the projection for the majority of our users.

For small map products we now (as of 6.16.0.2) use the center of the map as a reference point. So there will be no distortion at the center. The further North or South from the center you go the worse the distortion will get. This works fairly well for most of our products.

For larger products this is not ideal. For Canada especially this did not work so we put in a special fix that should make Canada-only maps less distorted where it matters (in the South).

We still allow users to change the projection angle in the registry. We decided not to openly provide a user interface for this, but made it a bit easier to change the angle without having to meddle directly in the registry. You now can hit Ctrl-Alt-A in MapSource to set the projection angle for the current map product. You will need to restart MapSource for this to have any effect.

Changes to the projection angle that you made in 6.16.0.2 will not be used (the registry location for the angle has changed). You will have to set the angle again. This was done so that the projection angle adjustment can be shared amongst Garmin applications (BaseCamp 3.0.x & MapSource 6.16.x will now both use the angle you set).

So, Mapsource is using the center of the map for projection and, if I'm not mistaken, that would look pretty much the same as State Plane.

[Boyd]

Have gone back to the beginning of this thread and started playing with grayscale imagery again, to see if I can create more detailed maps.

Believe it or not, ten years have passed, I'm still here and still doing "Experiments with LIDAR"!  Still working with grayscale imagery too, but with 1-meter resolution DEM in 3d with my own web app. Garmin devices and software just can't support the maps I want to make anymore.

https://boydsmaps.com/#13.00/40.357081/-77.046754/mbx3dmidatl/69.60/70.20



My web app is free, there's no advertising, no registration and it works on any device.



This has been my most ambitious map-making project yet, ten million map tiles (over 260gb), covering a portion of the US Mid-Atlantic region. Am planning to expand the map to a much larger area later this summer. This is the current coverage



There's also a 2d version of the map which loads faster and uses less data, since there's no 3d elevation data.

https://boydsmaps.com/#13.00/40.991210/-75.132460/midatlidar/0.00/0.00

Still using Globalmapper for almost everything, finally upgraded to version 23 this year. My app uses the open source Maptalks API however the 3d maps use Mapbox GL/JS inside a Maptalks container. Has been quite an education, needed to figure most things out for myself as there is not much documentation on building 3d maps from your own data and hosting them yourself.

I put all the DEM together in 32-bit floating point TIFF files which are imported into the Mapbox rasterio command-line program, then exported in Mapbox RGB elevation format. I wrote my own code to read elevation from these files and show it in a realtime "heads up" display on the map.

As a companion project, I used NASA ASTER 1 arc-second DEM to make a lower-resolution 3d map covering all of North America. Used USGS National Landcover Dataset and ESA Worldcover data (outside the US) with my own pseudo natural color pallete to produce a more "artistic" 3d view, along with atmospheric effects.

https://boydsmaps.com/#12.80/44.099767/-71.572680/mbx3dnalc/48.60/76.80

This was more of an "art project" for fun, but the elevation data is used in the same way as the other map, providing a realtime elevation display anywhere in North America as you move the map around. Still a big project, but "only" about 4 million tiles / 100gb.



There are over 200 other maps available in my app, from servers in many different US states, Canada and even Switzerland. Also OpenTopo, USGS National Map, live weather forecasts and radar plus a number of maps from HERE (the company that provides data to Garmin for City Navigator). I support waypoint import/export in Garmin-compatible GPX files, plus my own format that can be customized and imported/exported in Excel. Not supporting tracks yet, but will get around to it eventually.

Anyway... that's what I've been up to. How did you spend the last ten years?    Is anybody from this thread still here? Quite a nostalgia trip revisiting it now, I do miss the old days.