Contours show up in BaseCamp, but not on 62stc

[hwstock]

I've had success making maps for some rather small areas in Mexico, near some peaks I wish to climb.

One area has me stumped.  I started working with USGS 1/3 arc-second data for the region around Cerro Pinacate (within 31 to 32 lat, -113 to -114 lon).  I've gone through two processes -- 1) splitting the single usgs DEM into tiles, contouring them under DEM2TOPO, and reassembling the tiles under GPSMapEdit; and 2) using gdal_contour to make shapefiles (without splitting the original USGS dem), which I then import into GPSMapEdit. The files look OK under the map-making programs, and the contours show up just fine in BaseCamp.  But when installed on the 62stc, only linear features (like roads) show up -- no contours at all.

There are some bizarre bits of contouring, which show up when I go through either process.  I'm guessing something is not quite right with this DEM.  I've used the Google Earth overlay in GPSMapEdit, and there is absolutely no real feature to account for these bizarre contours.


Any ideas?   I could breakout the compiler and start processing the DEM; I suspect there must be some oddities.  I tried playing with the -snodata  -- no changes.

[maps4gps]

I get the 'same' thing for the contours for the data in Mexico.  It is the 'quality' of the source date.

The USGS release notes for the data are: 'Elevation data for Mexico are being included in the 1-arc-second layer as of the October 2008 release. The Mexico dataset is a result of collaboration between the U.S. Geological Survey (USGS) and Mexico's National Institute of Statistics and Geography (INEGI). The data were provided and quality control conducted by INEGI. Topographic staff at USGS EROS processed the data to improve edge matching, making the dataset seamless within itself and along the U.S. / Mexico border.'

The USGS data for the 1 degree area is 1/3 arc-sec for the US and 1 arc-sec for Mexico.

Check with Mexico's INEGI to see if any more detailed or reprocessed data is available.

You might also try contouring the SRTM data.  The ground spacing is 3 arc-sec (90m); however, the resulting contours (in none near level areas) were more natural.
I think they are still available from NASA; CGIAR has done void filling (although the download speed is usually very slow); and 'http://www.viewfinderpanoramas.org/'; has done void filling by using data from printed maps and other sources and is a much easier source to download from.

[hwstock]

After lots of neat side trips, I finally figured out the solution!

I changed the map to type 'S' for cGPSmapper (via GPSMapEdit)!  I have no idea why this issue never surfaced before in the 3 other maps I've made by the same process.  Maybe there is something odd about the Garmin world basemap in that region.

Along the way, I dusted off my C programming skills, and went through the flt file with a fine-tooth comb, and made false-color bitmaps of the area around Cerro Pinacate.  When I finally had to concede that there was really nothing wrong with the initial data, I started going over all the mysterious options in the programs that I has treated as black boxes.

[hwstock]

I found the cause of the funky contours in image shown above.  Basically, the original usgs flt data (GridFloat) have bizarre little deviations, almost random in nature, that make adjacent pixels (cells in the x,y array) very slightly different, in areas that are almost flat. For example, 16 pixels in a 4x4 sub grid might look like (meters elevation)

50.005  50.001 50.003 50.002
50.001  50.001 50.005 50.008
50.005  50.001 50.003 50.003
50.001  50.001 50.005 50.007

It looks like the usgs/nasa was trying to add some correction to existing data, maybe to slope large interpolated blocks and smoothly attach the different data sets... but they incorrectly scaled the additions.  These tiny variations wreak havoc with the contouring programs, which earnestly try to thread contours among values like 50.001 and 50.003. In fact, the data for the individual pixels are really not accurate to much better than a meter.

I went to the lower elevation, flatter areas, and did a point-by-point evaluation of the standard deviations and just rounded to even for 1/2 meter (I first fit a plane to each 3x3 pixel section to make sure it was indeed "flat"), discarding all the junk "corrections." All the weird contours went away, and the contours that are left correspond to the terrain in google maps.

I'll start stepping back, effectively check 0.25 meters resolution, etc., but I doubt it will make any difference.

Of course, this was all done by a C program-- rather too much data for a spreadsheet.

I do wonder what the usgs was trying to do, and if they would even care, or change anything, if I pointed out the "error."

[popej]

There is nice algorithm that can perform similar cleaning but in 3D:
http://www.cs.cf.ac.uk/meshfiltering/index_files/Page342.htm

[hwstock]

There is nice algorithm that can perform similar cleaning but in 3D:
http://www.cs.cf.ac.uk/meshfiltering/index_files/Page342.htm

I'm not sure that would have worked in this case -- perversely, the surface was too flat in a few places, but not completely smooth, with a standard deviation of less than 0.1% of the data values, for 9 neighbors.  The problem is really what the contouring algorithm tried to do with such a surface; the fit constraints were probably determined mostly by the variations in the mountainous parts, where there were significant slopes.  Both the dem2topo and gdal_contour routines produced similar results.

[popej]

perversely, the surface was too flat in a few places, but not completely smooth

Yes, denoising algorithms are designed to deal with exactly this problem.

[hwstock]

perversely, the surface was too flat in a few places, but not completely smooth

Yes, denoising algorithms are designed to deal with exactly this problem.

You must have some way to decide which variations are "noise" versus real measures (a rock here, a rock there...). The real problem is in the contour algorithm.

[popej]

You have solved your problem without touching contour algorithm, have you?

I understand, that you have rounded height at flat horizontal regions. Indicated algorithm does similar thing, except is not limited to horizontal surfaces.

[hwstock]

My method is not limited to horizontal surfaces either, and is fairly simple. 

The main problem is that the Z data are really not accurate to more than 1 meter, and almost everything else in the flt file is random junk.  I tried stepping through forced floating point precisions of 0.125, 0.25, 0.5 and 1 meter. Only 1 meter removes all the junk, but even 0.125 meter is better than with no precision control. A very small number of the bizarre contours remain, but by 0.5 meters forced precision, they are not too obnoxius.

I check each 3x3 block by fitting a plane to the cells (normal equations are very simple), and look for sd relative to the plane (chi-sqrd sort of) being larger than the gradient of the plane; areas that have gradients above the sd are deemed fine (they never have odd contours anyway, based on visual inspection).

It's an area with sand dunes and lava flows, so at first I thought the odd contours were real (perhaps reflecting individual dunes or ridges in the flows).  But they merely reflect the contour algorithms trying to thread contours through areas with very small differences in reported (gridfloat) elevation; I've seen those areas in aerial and ground photos, and there is rarely any correspondence between reality and the contours. 

Maybe the data started off as being good to just one meter, then the usgs post-processed to add corrections (perhaps for a new geoid/datum), leaving very small deviations.  For example, in this area the geoid change added about 1 m to the local elevations; the amount added changes very slightly over ten miles or so, so the corrections might locally be (purely made up!) 1.001, 1.0015, 1.002, 1.0025, 1.003... etc. The exactness is meant for monumented sites, whose accuracy is established.  But  if you apply that precision in a correction to all the data (mainly derived by photogrammetry) in between monumented points, you get a false sense of precision on elevation that are probably known to a few meters of accuracy at best.

[maps4gps]

What do the contours look like for that area if you use the original gridded elevation data file(s) from Mexico's National Institute of Statistics and Geography (INEGI)?

[hwstock]

I never got the INEGI DEMs.  I went through the INEGI site, but never found a link where I could download DEMs -- lots of descriptions of what was contained in the data, but couldn't find the actual data (If I could read Spanish, this would probably be an easier process).  I tried using their interactive map, but is glacially slow to redraw.  I have compared the USGS DEM-derived contours, in the region around Picacho del Diablo, Pico Risco, and Cerro Pescadores, with the maps from 2 commercial vendors who used INEGI sources.  The contours were quite similar, down to the same misregistration.

[hwstock]

I found the slide labeled "Surface-to-Surface LIDAR (BE) – NED (LT4X)" in
http://geology.er.usgs.gov/eespteam/terrainmodeling/Pete's%20publications/usgs_gis04.pdf

to be quite sobering.

[hwstock]

I found the slide labeled "Surface-to-Surface LIDAR (BE) – NED (LT4X)" in
http://geology.er.usgs.gov/eespteam/terrainmodeling/Pete's%20publications/usgs_gis04.pdf

...to be quite sobering.  Such error can shift peak positions (as determined by contouring the DEM) in regions where the positions are not constrained by benchmarks.  What I had thought to be uniform misregistration, may be topography-dependent.



I posted this at the end of another thread, but I think it deserves more notice.

[maps4gps]

...  I have compared the USGS DEM-derived contours, in the region around Picacho del Diablo, Pico Risco, and Cerro Pescadores, with the maps from 2 commercial vendors who used INEGI sources.  The contours were quite similar, down to the same misregistration.

I take this to confirm USGS's release note saying 'The data were provided and quality control conducted by INEGI. Topographic staff at USGS EROS processed the data to improve edge matching, making the dataset seamless within itself and along the U.S. / Mexico border.' which I included in my post on June 28th.