I've had success making maps for some rather small areas in Mexico, near some peaks I wish to climb.
One area has me stumped. I started working with USGS 1/3 arc-second data for the region around Cerro Pinacate (within 31 to 32 lat, -113 to -114 lon). I've gone through two processes -- 1) splitting the single usgs DEM into tiles, contouring them under DEM2TOPO, and reassembling the tiles under GPSMapEdit; and 2) using gdal_contour to make shapefiles (without splitting the original USGS dem), which I then import into GPSMapEdit. The files look OK under the map-making programs, and the contours show up just fine in BaseCamp. But when installed on the 62stc, only linear features (like roads) show up -- no contours at all.
There are some bizarre bits of contouring, which show up when I go through either process. I'm guessing something is not quite right with this DEM. I've used the Google Earth overlay in GPSMapEdit, and there is absolutely no real feature to account for these bizarre contours.
(http://hwstock.org/img/Bizarre.jpg)
Any ideas? I could breakout the compiler and start processing the DEM; I suspect there must be some oddities. I tried playing with the -snodata -- no changes.
I get the 'same' thing for the contours for the data in Mexico. It is the 'quality' of the source date.
The USGS release notes for the data are: 'Elevation data for Mexico are being included in the 1-arc-second layer as of the October 2008 release. The Mexico dataset is a result of collaboration between the U.S. Geological Survey (USGS) and Mexico's National Institute of Statistics and Geography (INEGI). The data were provided and quality control conducted by INEGI. Topographic staff at USGS EROS processed the data to improve edge matching, making the dataset seamless within itself and along the U.S. / Mexico border.'
The USGS data for the 1 degree area is 1/3 arc-sec for the US and 1 arc-sec for Mexico.
Check with Mexico's INEGI to see if any more detailed or reprocessed data is available.
You might also try contouring the SRTM data. The ground spacing is 3 arc-sec (90m); however, the resulting contours (in none near level areas) were more natural.
I think they are still available from NASA; CGIAR has done void filling (although the download speed is usually very slow); and 'http://www.viewfinderpanoramas.org/' has done void filling by using data from printed maps and other sources and is a much easier source to download from.
After lots of neat side trips, I finally figured out the solution!
I changed the map to type 'S' for cGPSmapper (via GPSMapEdit)! I have no idea why this issue never surfaced before in the 3 other maps I've made by the same process. Maybe there is something odd about the Garmin world basemap in that region.
Along the way, I dusted off my C programming skills, and went through the flt file with a fine-tooth comb, and made false-color bitmaps of the area around Cerro Pinacate. When I finally had to concede that there was really nothing wrong with the initial data, I started going over all the mysterious options in the programs that I has treated as black boxes.
I found the cause of the funky contours in image shown above. Basically, the original usgs flt data (GridFloat) have bizarre little deviations, almost random in nature, that make adjacent pixels (cells in the x,y array) very slightly different, in areas that are almost flat. For example, 16 pixels in a 4x4 sub grid might look like (meters elevation)
50.005 50.001 50.003 50.002
50.001 50.001 50.005 50.008
50.005 50.001 50.003 50.003
50.001 50.001 50.005 50.007
It looks like the usgs/nasa was trying to add some correction to existing data, maybe to slope large interpolated blocks and smoothly attach the different data sets... but they incorrectly scaled the additions. These tiny variations wreak havoc with the contouring programs, which earnestly try to thread contours among values like 50.001 and 50.003. In fact, the data for the individual pixels are really not accurate to much better than a meter.
I went to the lower elevation, flatter areas, and did a point-by-point evaluation of the standard deviations and just rounded to even for 1/2 meter (I first fit a plane to each 3x3 pixel section to make sure it was indeed "flat"), discarding all the junk "corrections." All the weird contours went away, and the contours that are left correspond to the terrain in google maps.
I'll start stepping back, effectively check 0.25 meters resolution, etc., but I doubt it will make any difference.
Of course, this was all done by a C program-- rather too much data for a spreadsheet.
I do wonder what the usgs was trying to do, and if they would even care, or change anything, if I pointed out the "error."
There is nice algorithm that can perform similar cleaning but in 3D:
http://www.cs.cf.ac.uk/meshfiltering/index_files/Page342.htm
Quote from: popej on August 16, 2013, 03:41:24 PM
There is nice algorithm that can perform similar cleaning but in 3D:
http://www.cs.cf.ac.uk/meshfiltering/index_files/Page342.htm
I'm not sure that would have worked in this case -- perversely, the surface was too flat in a few places, but not completely smooth, with a standard deviation of less than 0.1% of the data values, for 9 neighbors. The problem is really what the contouring algorithm tried to do with such a surface; the fit constraints were probably determined mostly by the variations in the mountainous parts, where there were significant slopes. Both the dem2topo and gdal_contour routines produced similar results.
Quote from: hwstock on August 16, 2013, 04:30:26 PMperversely, the surface was too flat in a few places, but not completely smooth
Yes, denoising algorithms are designed to deal with exactly this problem.
Quote from: popej on August 16, 2013, 06:07:34 PM
Quote from: hwstock on August 16, 2013, 04:30:26 PMperversely, the surface was too flat in a few places, but not completely smooth
Yes, denoising algorithms are designed to deal with exactly this problem.
You must have some way to decide which variations are "noise" versus real measures (a rock here, a rock there...). The real problem is in the contour algorithm.
You have solved your problem without touching contour algorithm, have you?
I understand, that you have rounded height at flat horizontal regions. Indicated algorithm does similar thing, except is not limited to horizontal surfaces.
My method is not limited to horizontal surfaces either, and is fairly simple.
The main problem is that the Z data are really not accurate to more than 1 meter, and almost everything else in the flt file is random junk. I tried stepping through forced floating point precisions of 0.125, 0.25, 0.5 and 1 meter. Only 1 meter removes all the junk, but even 0.125 meter is better than with no precision control. A very small number of the bizarre contours remain, but by 0.5 meters forced precision, they are not too obnoxius.
I check each 3x3 block by fitting a plane to the cells (normal equations are very simple), and look for sd relative to the plane (chi-sqrd sort of) being larger than the gradient of the plane; areas that have gradients above the sd are deemed fine (they never have odd contours anyway, based on visual inspection).
It's an area with sand dunes and lava flows, so at first I thought the odd contours were real (perhaps reflecting individual dunes or ridges in the flows). But they merely reflect the contour algorithms trying to thread contours through areas with very small differences in reported (gridfloat) elevation; I've seen those areas in aerial and ground photos, and there is rarely any correspondence between reality and the contours.
Maybe the data started off as being good to just one meter, then the usgs post-processed to add corrections (perhaps for a new geoid/datum), leaving very small deviations. For example, in this area the geoid change added about 1 m to the local elevations; the amount added changes very slightly over ten miles or so, so the corrections might locally be (purely made up!) 1.001, 1.0015, 1.002, 1.0025, 1.003... etc. The exactness is meant for monumented sites, whose accuracy is established. But if you apply that precision in a correction to all the data (mainly derived by photogrammetry) in between monumented points, you get a false sense of precision on elevation that are probably known to a few meters of accuracy at best.
What do the contours look like for that area if you use the original gridded elevation data file(s) from Mexico's National Institute of Statistics and Geography (INEGI)?
I never got the INEGI DEMs. I went through the INEGI site, but never found a link where I could download DEMs -- lots of descriptions of what was contained in the data, but couldn't find the actual data (If I could read Spanish, this would probably be an easier process). I tried using their interactive map, but is glacially slow to redraw. I have compared the USGS DEM-derived contours, in the region around Picacho del Diablo, Pico Risco, and Cerro Pescadores, with the maps from 2 commercial vendors who used INEGI sources. The contours were quite similar, down to the same misregistration.
I found the slide labeled "Surface-to-Surface LIDAR (BE) – NED (LT4X)" in
http://geology.er.usgs.gov/eespteam/terrainmodeling/Pete's%20publications/usgs_gis04.pdf
to be quite sobering.
I found the slide labeled "Surface-to-Surface LIDAR (BE) – NED (LT4X)" in
http://geology.er.usgs.gov/eespteam/terrainmodeling/Pete's%20publications/usgs_gis04.pdf
...to be quite sobering. Such error can shift peak positions (as determined by contouring the DEM) in regions where the positions are not constrained by benchmarks. What I had thought to be uniform misregistration, may be topography-dependent.
I posted this at the end of another thread, but I think it deserves more notice.
Quote from: hwstock on September 01, 2013, 07:09:31 PM
... I have compared the USGS DEM-derived contours, in the region around Picacho del Diablo, Pico Risco, and Cerro Pescadores, with the maps from 2 commercial vendors who used INEGI sources. The contours were quite similar, down to the same misregistration.
I take this to confirm USGS's release note saying 'The data were provided and quality control conducted by INEGI. Topographic staff at USGS EROS processed the data to improve edge matching, making the dataset seamless within itself and along the U.S. / Mexico border.' which I included in my post on June 28th.
Quote from: maps4gps on September 02, 2013, 08:56:53 AM
Quote from: hwstock on September 01, 2013, 07:09:31 PM
... I have compared the USGS DEM-derived contours, in the region around Picacho del Diablo, Pico Risco, and Cerro Pescadores, with the maps from 2 commercial vendors who used INEGI sources. The contours were quite similar, down to the same misregistration.
I take this to confirm USGS's release note saying 'The data were provided and quality control conducted by INEGI. Topographic staff at USGS EROS processed the data to improve edge matching, making the dataset seamless within itself and along the U.S. / Mexico border.' which I included in my post on June 28th.
I remember that post. That was part of my motivation to use the USGS data, as it seemed much easier to access. If the USGS took the data, regridded it, and went to a new geoid model, that could explain how basically 1 m data got a fine "correction" that causes the odd contours in mainly flat areas.
It's hard to say what is the test of similarity here; qualitatively, the contours look similar, but little changes in the contouring software change the appearance. However, INEGI topo maps (which I have available as TIFFS) show more detail (perhaps not real) than my USGS-based contours (or the two INEGI-derived commercial Garmin maps) show.
In any case, my efforts to make a more accurate map are getting a bit like putting lipstick on a pig, especially given the USGS LIDAR vs. NED comparison.
That was done in 2004, 9 years ago. More recent info on LIDAR indicates that it is not as accurate as one believed. The RMSE for ASTAR is considerably more than for SRTM.
My understanding is that INEGI created 1 arc sec gridded elevation data from their 1:50,000 paper maps. One file per printed topo sheet; if I remember correctly these are 20x30 minute in size. USGS took these files and put them into their standard 1x1 degree structure. I expect the edge matching was along the original file boundaries; although I remember an instance where there was a difference of 150 meters (yes meters) along the boundary.
USGS's agreement was to make the data provided by INEGI more readily available and not 'correct' it. Likewise for the data on Canada from RNCAN.
I'm not saying LIDAR is absolutely correct; I'm pointing out the large and systematic differences between LIDAR and NED (which , I'd hazard, are mainly errors in NED), and what that could mean for contouring the DEMs. We earlier discussed why so many DEM chunks seem to show a shift of 50 meters or more; that particular figure shows a possible reason... and also indicates why it may be a futile task to quibble over a few meters. I am biased to "correcting" the DEM so the mountain peaks are aligned; but if the "real" correction is slope- and aspect-dependent, no simple affine mapping will get it right for the entire region.
Maybe we are quibbling over what "correct" means. If you regrid data and change the reference for sea level, you will invariably end up with z values that don't exactly match the original data set. Going to a new geoid involves some long range, substantial corrections, and some lower amplitude, shorter-range corrections.
For heuristic purposes, try a 2D example. Suppose the original grid has z to just 1 meter, and looks like this:
^
|z
4o
3ooo
2oooooo
1ooooooooooooo
123456789ABCD ->x
...and you must regrid to an x-value that is offset by 0.01. Linear interpolation among the z values will invariably make the new values non-integral -- in fact they will be off-integral by small fractions of a meter. That's just the grid "correction," not a geoid "correction." I guess it might be better to call these adjustments; no one is saying the original data were "wrong."
This article is from 2006:
http://www.ngs.noaa.gov/CORS/Articles/OPUSMexico.pdf
The article talks about constant improvements in the geoid, but I have no idea how one would track what the reference status was when the data were transferred to noaa/usgs. Around here in NV, the geoid change (since surveying) caused shifts of about +1 meter or more for benchmarks -- but there are some areas that "lost" elevation.
I think it is time to get an authoritative opinion on the issue; as you mentioned in one of your earlier posts, ask USGS. The telephone contacts at ned.usgs.gov have me send in issues to
[email protected]
I guess we differ on that. I don't think it is an issue they can "solve" -- it's a natural outcome of data being adjusted so there appears to be more precision than there really is. Few people will ever care; it's just a case of caveat emptor. And we aren't even buying... we can just warn other people of the pitfalls. The data are as good as they advertise.