GPSFileDepot.com
 

News:

Welcome to GPSFileDepot!

Main Menu
Menu

Show posts

This section allows you to view all posts made by this member. Note that you can only see posts made in areas you currently have access to.

Show posts Menu

Messages - hwstock

#1
Thanks for trying so hard.  I find errors all the time in usgs data.
#2
This look like the same sort of error I found 2 years back for the NV maps near Desatoya Twins

Below is a screenshot for ExpertGPS of tracks that I know go to the summit of Humphreys Peak; check the dark blue track:
http://hwstock.org/err/realhump.jpg

Now here is the screen shot for the same track on the Garmin 62sc:
http://hwstock.org/err/62stc.jpg

The caltopo map contours are displaced by about 300' in x-y



I haven't figured out what caused the error.  The contours for nearby Mt Emerson are right on the spot; that's on the next topo sheet (24k) south.
#3
I guess we differ on that.  I don't think it is an issue they can "solve" -- it's a natural outcome of data being adjusted so there appears to be more precision than there really is.  Few people will ever care; it's just a case of caveat emptor.  And we aren't even buying... we can just warn other people of the pitfalls.  The data are as good as they advertise.
#4
Maybe we are quibbling over what "correct" means.  If you regrid data and change the reference for sea level, you will invariably end up with z values that don't exactly match the original data set.  Going to a new geoid involves some long range, substantial corrections, and some lower amplitude, shorter-range corrections.

For heuristic purposes, try a 2D example.  Suppose the original grid has z to just 1 meter, and looks like this:
^
|z

4o
3ooo
2oooooo
1ooooooooooooo
  123456789ABCD ->x

...and you must regrid to an x-value that is offset by 0.01. Linear interpolation among the z values will invariably make the new values non-integral -- in fact they will be off-integral by small fractions of a meter. That's just the grid "correction," not a geoid "correction."  I guess it might be better to call these adjustments; no one is saying the original data were "wrong."

This article is from 2006:
http://www.ngs.noaa.gov/CORS/Articles/OPUSMexico.pdf
The article talks about constant improvements in the geoid, but I have no idea how one would track what the reference status was when the data were transferred to noaa/usgs.  Around here in NV, the geoid change (since surveying) caused shifts of about +1 meter or more for benchmarks -- but there are some areas that "lost" elevation.
#5
Map Making Support / Re: Errors in DEMS
September 02, 2013, 10:16:19 AM
I'm not saying LIDAR is absolutely correct; I'm pointing out the large and systematic differences between LIDAR and NED (which , I'd hazard, are mainly errors in NED), and what that could mean for contouring the DEMs.  We earlier discussed why so many DEM chunks seem to show a shift of 50 meters or more; that particular figure shows a possible reason... and also indicates why it may be a futile task to quibble over a few meters.  I am biased to "correcting" the DEM so the mountain peaks are aligned; but if the "real" correction is slope- and aspect-dependent, no simple affine mapping will get it right for the entire region.
#6
Quote from: maps4gps on September 02, 2013, 08:56:53 AM
Quote from: hwstock on September 01, 2013, 07:09:31 PM
...  I have compared the USGS DEM-derived contours, in the region around Picacho del Diablo, Pico Risco, and Cerro Pescadores, with the maps from 2 commercial vendors who used INEGI sources.  The contours were quite similar, down to the same misregistration.

I take this to confirm USGS's release note saying 'The data were provided and quality control conducted by INEGI. Topographic staff at USGS EROS processed the data to improve edge matching, making the dataset seamless within itself and along the U.S. / Mexico border.' which I included in my post on June 28th.

I remember that post. That was part of my motivation to use the USGS data, as it seemed much easier to access.  If the USGS took the data, regridded it, and went to a new geoid model, that could explain how basically 1 m data got a fine "correction"  that causes the odd contours in mainly flat areas.

It's hard to say what is the test of similarity here; qualitatively, the contours look similar, but little changes in the contouring software change the appearance.  However, INEGI topo maps (which I have available as TIFFS) show more detail (perhaps not real) than my USGS-based contours (or the two INEGI-derived commercial Garmin maps) show.

In any case, my efforts to make a more accurate map are getting a bit like putting lipstick on a pig, especially given the USGS LIDAR vs. NED comparison.
#7
Map Making Support / Errors in DEMS
September 02, 2013, 07:55:21 AM
I found the slide labeled "Surface-to-Surface LIDAR (BE) – NED (LT4X)" in
http://geology.er.usgs.gov/eespteam/terrainmodeling/Pete's%20publications/usgs_gis04.pdf

...to be quite sobering.  Such error can shift peak positions (as determined by contouring the DEM) in regions where the positions are not constrained by benchmarks.  What I had thought to be uniform misregistration, may be topography-dependent.



I posted this at the end of another thread, but I think it deserves more notice.
#8
I found the slide labeled "Surface-to-Surface LIDAR (BE) – NED (LT4X)" in
http://geology.er.usgs.gov/eespteam/terrainmodeling/Pete's%20publications/usgs_gis04.pdf

to be quite sobering.
#9
I never got the INEGI DEMs.  I went through the INEGI site, but never found a link where I could download DEMs -- lots of descriptions of what was contained in the data, but couldn't find the actual data (If I could read Spanish, this would probably be an easier process).  I tried using their interactive map, but is glacially slow to redraw.  I have compared the USGS DEM-derived contours, in the region around Picacho del Diablo, Pico Risco, and Cerro Pescadores, with the maps from 2 commercial vendors who used INEGI sources.  The contours were quite similar, down to the same misregistration.

#10
My method is not limited to horizontal surfaces either, and is fairly simple. 

The main problem is that the Z data are really not accurate to more than 1 meter, and almost everything else in the flt file is random junk.  I tried stepping through forced floating point precisions of 0.125, 0.25, 0.5 and 1 meter. Only 1 meter removes all the junk, but even 0.125 meter is better than with no precision control. A very small number of the bizarre contours remain, but by 0.5 meters forced precision, they are not too obnoxius.

I check each 3x3 block by fitting a plane to the cells (normal equations are very simple), and look for sd relative to the plane (chi-sqrd sort of) being larger than the gradient of the plane; areas that have gradients above the sd are deemed fine (they never have odd contours anyway, based on visual inspection).

It's an area with sand dunes and lava flows, so at first I thought the odd contours were real (perhaps reflecting individual dunes or ridges in the flows).  But they merely reflect the contour algorithms trying to thread contours through areas with very small differences in reported (gridfloat) elevation; I've seen those areas in aerial and ground photos, and there is rarely any correspondence between reality and the contours. 

Maybe the data started off as being good to just one meter, then the usgs post-processed to add corrections (perhaps for a new geoid/datum), leaving very small deviations.  For example, in this area the geoid change added about 1 m to the local elevations; the amount added changes very slightly over ten miles or so, so the corrections might locally be (purely made up!) 1.001, 1.0015, 1.002, 1.0025, 1.003... etc. The exactness is meant for monumented sites, whose accuracy is established.  But  if you apply that precision in a correction to all the data (mainly derived by photogrammetry) in between monumented points, you get a false sense of precision on elevation that are probably known to a few meters of accuracy at best.

#11
Quote from: popej on August 16, 2013, 06:07:34 PM
Quote from: hwstock on August 16, 2013, 04:30:26 PMperversely, the surface was too flat in a few places, but not completely smooth
Yes, denoising algorithms are designed to deal with exactly this problem.

You must have some way to decide which variations are "noise" versus real measures (a rock here, a rock there...). The real problem is in the contour algorithm.
#12
Quote from: popej on August 16, 2013, 03:41:24 PM
There is nice algorithm that can perform similar cleaning but in 3D:
http://www.cs.cf.ac.uk/meshfiltering/index_files/Page342.htm

I'm not sure that would have worked in this case -- perversely, the surface was too flat in a few places, but not completely smooth, with a standard deviation of less than 0.1% of the data values, for 9 neighbors.  The problem is really what the contouring algorithm tried to do with such a surface; the fit constraints were probably determined mostly by the variations in the mountainous parts, where there were significant slopes.  Both the dem2topo and gdal_contour routines produced similar results.
#13
I found the cause of the funky contours in image shown above.  Basically, the original usgs flt data (GridFloat) have bizarre little deviations, almost random in nature, that make adjacent pixels (cells in the x,y array) very slightly different, in areas that are almost flat. For example, 16 pixels in a 4x4 sub grid might look like (meters elevation)

50.005  50.001 50.003 50.002
50.001  50.001 50.005 50.008
50.005  50.001 50.003 50.003
50.001  50.001 50.005 50.007

It looks like the usgs/nasa was trying to add some correction to existing data, maybe to slope large interpolated blocks and smoothly attach the different data sets... but they incorrectly scaled the additions.  These tiny variations wreak havoc with the contouring programs, which earnestly try to thread contours among values like 50.001 and 50.003. In fact, the data for the individual pixels are really not accurate to much better than a meter.

I went to the lower elevation, flatter areas, and did a point-by-point evaluation of the standard deviations and just rounded to even for 1/2 meter (I first fit a plane to each 3x3 pixel section to make sure it was indeed "flat"), discarding all the junk "corrections." All the weird contours went away, and the contours that are left correspond to the terrain in google maps.

I'll start stepping back, effectively check 0.25 meters resolution, etc., but I doubt it will make any difference.

Of course, this was all done by a C program-- rather too much data for a spreadsheet.

I do wonder what the usgs was trying to do, and if they would even care, or change anything, if I pointed out the "error."
#14
Which product did you get?  The variation from that seller is huge.  For example, Bicimapas sells topo maps through Garmin, and they come on a locked SD card.
#15
There should b e a version of dem2topo out there that has idl built in -- you just have to click the "run idl?" and move on.

Dem2topo takes geotif formal files ( tif extension).  The usgs server now just loads GridFloat (flt), ArcGIS, or image format, so you have to use the scripts I mentioned (or directly use GDAL_translate) to covert from one of those formats to tif.

Both GPSMapEdit and Global Mapper cost money (the latter a lot more) if you want them to save anything useful.  Both also require that you have cGPSmapper installed to make the img file for upload to your GPS.