GPSFileDepot.com
 

GPS Process (GNIS Processing) Fails to Fetch Data (invalid URL)

Started by bacall213, April 01, 2013, 08:05:43 AM

Previous topic - Next topic

bacall213

Hi All,

I've been going through the amazing tutorial on making custom Garmin Maps. I got to the section for GNIS data, but ran into some problems.

The first problem is that GNIS Processing requests an old version of the features file from geonames.usgs.gov (at least for Washington State) and continues despite receiving a "404 Not Found" from the webserver.

The second problem is that GNIS Processing calls pgsql2shp.exe with a "-p<password>" flag. In version 9.2, the password flag is a capital "P" ("-P") and a lowercase "p" ("-p") is used to indicate a specific server port.

Here's a timeline of what I've observed and my workaround. I identified the following through the use of packet captures and Sysinternals Process Monitor.

GNIS Processing Workflow:

1) Start GNIS Processing with Internet
2) Select state and output directory (I'll use Washington State for my example)
3) Click "All At Once"
4) GNIS Processing requests "geonames.usgs.gov/doc/stategaz/WA_Features_20090401.zip"
5) geonames.usgs.gov returns "404 Not Found"
6) GNIS Processing seemingly continues without error. The only noticeable error (aside from the lack of output) is in error.log where it's indicated that "gnis.shp" couldn't be found.

Workaround:

1) Download the latest features file from geonames.usgs.gov (WA_Features_20130210.zip)
2) Rename the zip file to "WA_Features_20090401.zip"
3) Unpack the zip file and rename the .TXT file to "WA_Features_20090401.txt" and place it in the output directory.
4) Stand up an Apache server and serve "/doc/stategaz/WA_Features_20090401.zip"
5) Add line to Windows hosts file pointing to Apache server as "geonames.usgs.gov" (I used a VM that had a separate IP)
6) Confirm the webserver is accessible and the URL is valid
7) Execute GNIS processing as usual. The correct .SHP files are created... if you don't have a password set in GNIS Processing...

GNIS processing appears to pull the .ZIP file off the server, but fails to unpack it in the correct location. I found that I had to put an unpacked version of the file in the output directory for it to read. Since I had a screwy workaround, this may be have been an unintended consequence. I couldn't identify what was going wrong through the use of Process Monitor.

And now for the password issue...

- GNIS Processing allows for the entry of a database password
- The command called is:
>> "C:\Program Files\PostgreSQL\9.2\bin\pgsql2shp.exe" -f "C:\GIS\washington_topo\5 gnis\gnis.shp" -h localhost -u postgres -p<password> postgis gnis
>> postgis is the name of the database
>> gnis is the name of the table
>> <password> is the password I have on the database, without brackets and removed for security reasons
- In PostgresSQL 9.2, the "-p" flag is for port, whereas "-P" is the actual flag that should be used.
- I followed the instructions early on and enabled no password authentication when coming from localhost, so I was able to remove the password from the config to make it work near-flawlessly with my above workaround. Leaving the password in place results in pgsql2shp.exe throwing an error that isn't handled nor exposed.

And finally, the .SHP file generated...

- The column "MP_TYPE" did not exist in several tests I ran. Instead, there was a "TYPE" column.
- I haven't completed the map yet (waiting on water-features data), so I don't know if it will work with the column named "TYPE," but it does diverge from the instruction set which indicates it should be "MP_TYPE."

To summarize:

- GNIS Processing requests an outdated version of the features file from geonames.usgs.gov and doesn't handle the error properly
- GNIS Processing calls pgsql2shp.exe with an invalid flag on PostgresSQL 9.2
- The GNIS data .MP file generated in Part 5 may be invalid once it's all put together

Is the author of GNIS Processing still around on these forums? Could the program be updated? Or, if the author can't, could it be open sourced so I the community could maintain the code base and update it for changes like these?

Thanks,

Brian

-Oz-

Yes, yes, and yes.  It is written in visual basic though which is a pain to work with these days.  When I can get on my other computer I can release the code.
Dan Blomberg
Administrator - GPSFileDepot
GPS Units: Garmin Dakota 20, Garmin GPSMap 60csx, Nuvi 255W, Nuvi 250W, ForeRunner 110, Fenix 2, Tactix Bravo, Foretrex 401
See/Download My Maps!