Welcome, Guest. Please login or register.
Did you miss your activation email?

Login with username, password and session length
Forums Search:  


Author Topic: Experiments with LIDAR  (Read 22655 times)

Boyd

  • Expert Advisor
  • *****
  • Posts: 2840
  • Karma: 27
    • View Profile
Experiments with LIDAR
« on: December 27, 2011, 09:15:55 PM »
We have pretty extensive coverage in Southern New Jersey with USGS NED 1/9 arc second LIDAR DEM, so I'm trying to find a way to use this in a map. It's especially nice around here where everything is so flat because it reveals a lot of surface detail in the forest.

It would be very easy to make a garmin "custom map" (.kmz raster image) from this data, but it's pretty limiting both in size and in the number of devices that can use it. So my thought was to vectorize it using Globalmapper's function to create polygons based on RGB values.

I have all the data, so for starters I cropped it to the outline of the map I want to make, then exported it as gridded 8 bit grayscale TIFF's that are ~500x500 pixels. Globalmapper converts these into anywhere between 50,000 to 60,000 polygons when using a match distance of 16. Smaller values for match distance create way too many polygons.

Here's a comparison of the original TIFF to the vector image. Globalmapper really does a great job - it's hard to tell them apart until you zoom way in.







Next I export a shapefile and use Filemaker Pro to do some processing of the .dbf, parsing the RGB values and mapping them to MP_TYPES in a custom .typ file. I decided to go this route instead of using a program like MOAGU or Mapwel because it gives me full control over everything.

I then import the modified shapefile back into Globalmapper, combine with other vector features like roads, then export an .mp file to compile with cgpsmapper. My current prototype breaks a USGS 24k quad into 8x8=64 tiles, each of which is ~1MB. So far so good. Here's what it looks like in Globalmapper




And the compiled map in Mapsource looks surprisingly good - bet ya didn't think Mapsource could do this!  ;)




But then the fun begins... there are two problems on the GPS. First, it practically brings the unit to its knees due to all the little polygons (I tried on a Nuvi 205 and Montana 600). But worse, the GPS doesn't seem to render the image with as much precision as Mapsource, even with detail set to max. Basically, it's a mess. Kind of an interesting abstract art image, but nothing like the original.




So I'm going to need to re-think this and do more experimentation.  :)

Anyway, it's an interesting although time consuming and sometimes frustrating project that I wanted to share. I have also recently refined my technique for processing the Landsat NLCD 2006 landcover data in a similar fashion and that is working out really well - but that source imagery is about 10 meters pixel. I'll post some details about this in another thread soon.

Hope everyone is enjoying the holidays - here's wishing a happy New Year to all! :)
« Last Edit: December 27, 2011, 09:21:41 PM by Boyd »

-Oz-

  • Map Maker!
  • Administrator
  • Expert Advisor
  • *****
  • Posts: 1477
  • Karma: 19
    • View Profile
    • GPSFileDepot
Re: Experiments with LIDAR
« Reply #1 on: December 28, 2011, 08:10:12 AM »
All I can say is wow.  I bet Garmin never thought they'd need more processing power in the GPS units.
Dan Blomberg
Administrator - GPSFileDepot
GPS Units: Garmin Dakota 20, Garmin GPSMap 60csx, Nuvi 255W, Nuvi 250W, ForeRunner 110
See/Download My Maps!

Boyd

  • Expert Advisor
  • *****
  • Posts: 2840
  • Karma: 27
    • View Profile
Re: Experiments with LIDAR
« Reply #2 on: December 28, 2011, 08:14:05 AM »
Thanks! I'm very impressed that Mapsource can cope with this so well. Once I got the tile size right, the map works very well in Mapsource and scrolls/zooms quickly. I guess this is because it uses the graphics card to help. When I open the same map in nRoute it works, but it's slow motion as each row of pixels is drawn. At the resolution used in the examples above, one USGS 24k quad consists of over 3 million polygons!

Am doing some more testing now with lower resolution images and will post the results later today.

Seldom

  • Expert Advisor
  • *****
  • Posts: 1803
  • Karma: 18
    • View Profile
Re: Experiments with LIDAR
« Reply #3 on: December 28, 2011, 08:30:00 AM »
I'll watch this with interest, but it's a pity nobody knows how to turn this into DEM data.  The sexy thing about LIDAR is the ability of GM to view it in 3D.  Does your data use a 1 foot grid or something smaller?  Have you tried using it to generate contours?

jbensman

  • Expert Advisor
  • *****
  • Posts: 421
  • Karma: 7
    • View Profile
Re: Experiments with LIDAR
« Reply #4 on: December 28, 2011, 09:15:05 AM »
What are you trying to accomplish with that map?  Why are you doing it that way instead of creating topo lines?  It might be neat to create a transprent overlay with a 2 foot contour interval for flat areas. 

Boyd

  • Expert Advisor
  • *****
  • Posts: 2840
  • Karma: 27
    • View Profile
Re: Experiments with LIDAR
« Reply #5 on: December 28, 2011, 09:25:24 AM »
Yeah, I really wish we knew how to put DEM into Garmin's format. But I wonder is this high res stuff would just choke the GPS anyway? But my technique promises to be the next best thing IMO.

I am mapping each elevation range into one of 72 custom types I've created. As you know, when you "mouse over" a map object (in either mapsource or the GPS), it will display any custom text you include in the type definition. So, if I put the elevation into the custom text for each type definition, it will display elevations. Of course, it can't associate that with track profiles, etc., but still kind of cool. Also, I can create difference custom .typ files with different color schemes that have effects similar to the different shaders available in globalmapper. That makes it easy to change the map appearance without recompiling anything - just change the .typ file.

I'm not sure what you mean by "1 foot grid". I really haven't worked a lot with DEM in the past. I downloaded the LIDAR from USGS and in the examples above I used it in the raw format. The source resolution is ~3 meters/pixel (ie: 10 ft per pixel).

Yes, I have used this same data to generate contours for my most recent map - http://www.gpsfiledepot.com/maps/view/294/ . I experimented with different intervals and posted a bunch of examples here, seeking input from my "audience" :) http://forums.njpinebarrens.com/threads/boyds-map-of-the-pines-beta-available.5103/page-7

I haven't yet looked at any LIDAR from mountainous areas. I'm thinking it might be a bit of a mess for generating contours however.... the "too much information" syndrome perhaps? LIDAR is a really different beast from 1/3 arc second data. But it's great around here where jmost of the elevations fall within a range around10-70 meters.

My techniques are tuned for this "bumpy" kind of data. When I use it with the traditional 1/3 arc second DEM, it looks pretty bad. Smooth gradients aren't so easily vectorized, and you hit the limits of the allowable number of custom types available to represent grayscale in Garmin's format.

Boyd

  • Expert Advisor
  • *****
  • Posts: 2840
  • Karma: 27
    • View Profile
Re: Experiments with LIDAR
« Reply #6 on: December 28, 2011, 09:30:25 AM »
Ah jbensman, you just feel that transparent maps are the answer to everything. ;) Don't get me wrong - your work is terrific and a great contribution. Just not my own style, and variety is the spice of life.

Actually I spent quite awhile playing with different ways to generate contours, but they really don't capture the subtlety of the terrain as well. See all the screenshots in the thread I linked to in the previous post at njpinebarrens.com. All those contour lines get very busy.



For me, "a picture is worth a thousand words". I can just cop out and make a "custom map" (.kmz), but then people with older units can't use it. And if I use the source resolution LIDAR at 3m/pixel, Garmin's format maxes out with a map about 6 miles x 6 miles. The Montana can use bigger .kmz files, but not many people have one of those.

You should certainly try experiments of your own with contouring LIDAR. With your expertise, I don't doubt that you could come up with some nice overlays that could enhance City Navigator or OSM maps.
« Last Edit: December 28, 2011, 09:33:41 AM by Boyd »

jbensman

  • Expert Advisor
  • *****
  • Posts: 421
  • Karma: 7
    • View Profile
Re: Experiments with LIDAR
« Reply #7 on: December 28, 2011, 10:20:57 AM »
I live about a mile from the Mississippi River.  What I think would be nice is LDAR 2 ft contour for the floodplain.   The reason why I think transparent is the only way to reasonably do this is right next to the floodplain there are bluffs and lots of steep topoography.  If there was a 2 ft interval there, the map would be a solid contour line.  That way when you were out on the flodplain you could turn on the detailed 2 ft overlay and turn it off when you leave the floodplain.  The last time I checked they did not have any ldar for my area but I know they are working on getting it.  I've been wanting to try it.

Another issue ldar might help with is lots of the other dem data messes up bluffs.  Instead of showing a bluff, the contour lines will show a slope.  I've spent many hours trying to fix this on lots of my maps. 

My understanding is the ldar gets two data sets, the first reflection is the height of the vegetation and the second is the ground.  It might be interesting to make a map of vegetation height. 

Seldom

  • Expert Advisor
  • *****
  • Posts: 1803
  • Karma: 18
    • View Profile
Re: Experiments with LIDAR
« Reply #8 on: December 28, 2011, 10:30:13 AM »
I'm not sure what you mean by "1 foot grid". I really haven't worked a lot with DEM in the past. I downloaded the LIDAR from USGS and in the examples above I used it in the raw format. The source resolution is ~3 meters/pixel (ie: 10 ft per pixel).
Grid actually may not mean much for LIDAR.  I don't know how a "point cloud" is defined.
I seem to remember generating some contours for an airport using LIDAR, and thought I remembered it had a grid, but maybe I was confusing that with an imagery resolution.

I haven't yet looked at any LIDAR from mountainous areas. I'm thinking it might be a bit of a mess for generating contours however.... the "too much information" syndrome perhaps? LIDAR is a really different beast from 1/3 arc second data. But it's great around here where jmost of the elevations fall within a range around10-70 meters.

A few years ago I downloaded some LIDAR of Mt. St. Helens.  Viewed in 3D in GM you can make out trees at the bottom of the mountain.

jbensman

  • Expert Advisor
  • *****
  • Posts: 421
  • Karma: 7
    • View Profile
Re: Experiments with LIDAR
« Reply #9 on: December 28, 2011, 10:55:50 AM »
If you can make out trees, my understanding of LDAR (which could be wrong) is that would be the wrong data.  My understanding is with LDAR it first gets vegetation height and then it gets ground elevation.  So if its showing trees, would it be vegetation height layer, not the ground elevation layer?

Seldom

  • Expert Advisor
  • *****
  • Posts: 1803
  • Karma: 18
    • View Profile
Re: Experiments with LIDAR
« Reply #10 on: December 28, 2011, 11:05:45 AM »
No doubt the wrong data (or meaningless data anyway).  I was looking at the whole point cloud in a 3D rendering without any way of distinguishing between vegetation and topography.  When I turned off the veg layers the trees went away.  The data has an incredible number of layers available though.

Boyd

  • Expert Advisor
  • *****
  • Posts: 2840
  • Karma: 27
    • View Profile
Re: Experiments with LIDAR
« Reply #11 on: December 28, 2011, 11:36:07 AM »
Just did a whole bunch of comparisons using different settings so that I could mull them over. The "match" settings come from the dialog when you right-click the raster image and choose "create area features from equal values" in the globalmapper layer dialog.

I also tried resampling the original 3m/pixel raster image to both 6m/pixel and 10m/pixel. The number of polygons refers to the whole image which is one tile from a USGS 24k quad exported in an 8x8 grid. These tiles are about 500x500 pixels. If I try this with a 1000x1000 image, globalmapper eats up all the resources on my computer - task mgr showed 96% RAM use and 90% CPU.... I had to reboot! My computer has a 2.8 ghz core 2 duo CPU with 6GB RAM.

In this first series of tests, I used the original 3m resolution data and tried different match settings








Next I resampled the raster image at 6m/pixel and tried different match settings






Same thing, but with the raster image resampled at 10m/pixel






But maybe I'm expecting too much? All those examples show the image at 1:6000 scale (in Globalmapper, View > Zoom to Scale) which is probably equivalent to about the 300 zoom on the GPS. Here's a composite comparing a view of the full tile at 1:24,000 scale. Pretty big difference in the number of polygons, but not as big of a visual difference at this scale




So this gives me some things to think about and try. Somewhere in here there's a compromise that should result in reasonable performance on the GPS unit. My original prototype mapset (shown in the first post of this thread) used match=16 at 3m/pixel and had 56,000 polygons. The example with match=32 at 3m/pixel reduces the polygon count to 15,700 while still retaining the full resolution of the source image. Maybe that's enough?

OTOH, it may be that the GPS itself doesn't calculate with enough precision to do justice to 3m/pixel imagery. As discussed earlier, the .img format (as supported by cgpsmapper) supposedly maxes out at 5m/pixel....

« Last Edit: December 28, 2011, 11:46:36 AM by Boyd »

Seldom

  • Expert Advisor
  • *****
  • Posts: 1803
  • Karma: 18
    • View Profile
Re: Experiments with LIDAR
« Reply #12 on: December 28, 2011, 12:34:26 PM »
OTOH, it may be that the GPS itself doesn't calculate with enough precision to do justice to 3m/pixel imagery. As discussed earlier, the .img format (as supported by cgpsmapper) supposedly maxes out at 5m/pixel....

Get a chance to show my ignorance here.  If I understand correctly at Level 0 the circumference of the Earth's ellipsoid (24902 miles) is divided into 224 (16,777,216)segments.  24902 miles/16777216 = 0.015+/- miles = 7.8370 feet = 2.4 meters.  And cgpsmapper complains strenuously if I try to compile a map with routing nodes closer than 2.4 meters.

jbensman

  • Expert Advisor
  • *****
  • Posts: 421
  • Karma: 7
    • View Profile
Re: Experiments with LIDAR
« Reply #13 on: December 28, 2011, 12:36:40 PM »
The Public Land Survey mapset is transparent and the polygons are invisible.  When you view it on the GPS, you do not see the polygons (but you see the polylines at the boundaries).  When you click on the map, the GPS will identify the label of the invisible polygon.

How about you create a transparent map with invisible polygons.  Since you have the polygons labeled with the elevation, if you then click on the map it should give you the LDAR elevation for that spot.  Since the polygons are invisible, maybe it would not grind the GPS to a halt trying to draw the polygons.  While it would not do other things DEM does, it would be a neat map to have.

Boyd

  • Expert Advisor
  • *****
  • Posts: 2840
  • Karma: 27
    • View Profile
Re: Experiments with LIDAR
« Reply #14 on: December 28, 2011, 01:00:23 PM »
Seldom: yeah, that math always confuses me but what you say makes sense. I thought the error was +/- 2.5 meters, which means any two points could be 5 meters apart. Remember this thread? it is pretty graphic evidence that some rounding or truncation is taking place: http://forums.gpsfiledepot.com/index.php/topic,1335.msg8387.html#msg8387

jbensman: that's a good idea, but I need to follow through on what I started first. Anyway, what fun is it unless you can push something to its breaking point.  ;) But I'm not sure... do you think an invisible polygon is easier to draw than a visible one? Seems like the "heavy lifting" is the math to calculate its proper location, although maybe there's some overhead related to actually drawing the pixels.
« Last Edit: December 28, 2011, 01:02:05 PM by Boyd »