Experiments with LIDAR

[Boyd]

Am planning to expand the map to a much larger area later this summer.

The new, expanded map is now available! This is the new coverage area - over twice the size of the original map, more than 600 gb with almost 22 million map tiles! 




Here's Shenandoah, Virginia in 3d

https://boydsmaps.com/#11.71/38.454567/-78.602919/mbx3dmidatl/40.20/73.20



3d view of the Catskills

https://boydsmaps.com/#14.00/41.998428/-74.386694/mbx3dmidatl/-174.00/72.00

This is how it looks on a phone.



You can view the 3d map in 2d North-up mode if you prefer - just click the compass to toggle between 2d/3d. But there's a separate 2d version of the same map without the elevation data if you aren't interested in 3d views, it loads much faster (note that 3d maps appear in blue text in the map menu)

Here's the 2d map of Martha's Vineyard, for example.

https://boydsmaps.com/#11.71/41.398941/-70.647387/midatlidar/0.00/0.00



I would like to continue expanding this map to cover an even larger area in the future. I still have over 200gb free disk space on the server but need to find the best use for it. Ultimately, I'll need to lease additional disk space to support future expansion.

It's not practical to create a map this large in one piece, so I divide it into smaller blocks of 100gb to 150gb of uncompressed 32-bit floating point DEM data. After processing this, I end up with blocks of a few hundred thousand .png map tiles which are ~1/4 the size of the source files. Then there is a companion set of .png tiles that contain the elevation data in Mapbox RGB format. The elevation tiles only use about half the space of the map tiles.

Before creating the RGB tiles, the geoTIFF DEM source file must be processed with Mapbox rasterio, which is a rather complex command-line program

https://blog.mapbox.com/rasterio-gdal-especially-for-python-programmers-553dabf7dd71

This encodes the DEM data as a .tif raster image which can be exported as .png map tiles. It took awhile to figure out exactly how to do all of this, Mapbox does not really support creating your own RGB DEM, they expect you to use their own pre-packaged DEM. But their DEM is about 5-meter resolution and I wanted 1-meter DEM. Using their own content can also result in usage charges but it's free to use their API with your own data. But you're pretty much on your own, I could find very little information about how to implement all of this locally but eventually figured it out through trial and error.

I create all the the tiles with Globalmapper using the option to only export fully-covered tiles, this prevents gaps between different blocks of data when I merge them with the full map. Finally, I zip the tiles, upload them to the server and unzip.

There are so many steps to the process and so many different files, I had to create a spreadsheet to track my progress!

[Boyd]

Believe it or not, ten years have passed, I'm still here and still doing "Experiments with LIDAR"!

Another year has passed... still doing "Experiments with LIDAR"    You can also do some "experiments", just click the button with the blue cube on my site to open the Terrain Viewer. You can also link directly to the viewer, for example:

https://boydsmaps.com/terra/#37.823766/-79.914004/-877/1766/1578/0/292/0/2000/775/-2000/56.99999999999999/2/3d/shader40/0/2/z15



The Terrain Viewer takes a "slice" of the map and exports it as a 3d model where you have many options for controlling the appearance.

There are sliders for the XYZ coordinates and intensity of the "sun" (a directional light source). Observe the effect as you move the sliders, the changing lighting will reveal different details in the terrain. Another slider changes the vertical scale of the terrain (vertical exaggeration) in realtime, which can make a big difference in appearance.

You can choose 6 different map types: aerial imagery from NAIP (the National Agriculture Imagery Program)

https://boydsmaps.com/terra/#40.615801/-74.822234/-1273/1059/-2/0/46/0/1000/437/1014/75/2/3d/ortho/0/0/z15




The grayscale option is just a white surface and the shadows create the shading.

https://boydsmaps.com/terra/#41.460554/-71.944232/-676/454/891/0/250/0/0/465/-1577/52/2/3d/grayscale/0/0/z17



The shaders (5, 10, 20 or 40 colors) use my own code to set the color of each pixel based on its position between the minimum and maximum elevation of the map.

Each map consists of 4.2 million elevation samples in a 2048 x 2048 grid. The spacing between samples determines the map resolution and size, the highest resolution has a sample spacing of 4 feet (1.2 meters) and coverage area of about 1.5 x 1.5 miles (about 2.5 km).

https://boydsmaps.com/terra/#39.251821/-76.735124/0/67689/0/0/314/0/-986/600/-986/45/3/2d/shader20/0/0/z17



The viewer uses webGL with the three.js API and the same RGB-encoded elevation dataset as my main web app. I use the MapTalks API to cover an HTML canvas with RGB tiles, then populate an array using elevation values decoded from each pixel's RGB values. That array is then used to create a 3d mesh. For the color shaders, I iterate through the array, creating another HTML canvas where each pixel is colored according to its elevation. That canvas is then mapped onto the 3d mesh.

The detail available in the terrain viewer revealed some major flaws in my original Mid-Atlantic elevation dataset that were not so obvious before. Took quite a bit of troubleshooting to understand the problem, but eventually I found it. The issue was caused by re-sampling artifacts when I exported the RGB elevation tiles with Globalmapper. Using "nearest neighbor" sampling would have fixed the problem, but that is not an option in Globalmapper. Finally, I discovered that exporting my DEM files at the *precise* resolution for each zoom level would prevent Globalmapper from re-sampling.

I ended up re-processing my entire elevation dataset from almost 2 terabytes of uncompressed TIFF source files, producing 9 million 256x256 map tiles.Took around a week of 24/7 processing but made a huge difference in the quality of the 3d views in both the terrain viewer and the main web app.

My next project is expanding the 3d terrain coverage area to include all of New York, Massachussets, New Hampshire, Vermont and Maine, which should be available later in 2023. I've expanded the server to 2 terabytes of disk, so we'll see how much space is left when this is completed!