Development of this software has ceased. Feel free to follow the create a map tutorial.I have been working on a tool that helps me create maps faster than the tutorial I wrote. This tool is definitely not ready for full prime-time use but can be used by many people.
Features:
- Transportation - Download and process the 2009 Census data automatically (2009 data downloaded added in v1.0.21)
- Contours - uses fw-tools to process the contours; also uses postgis; this is free and easy (GeoTiff Files Required)
- GNIS - points of interest; just like the program gnis_process; works until the usgs changes the file names
- water data - automatic parsing of the water data, read the directions in gray for how the folder needs to be organized
- Forest Service - VERY VERY BETA; I wouldn't use it (i do but in debug mode)
- to .img - just point it to the folder with the .mp files and it'll convert them all to .img
Requires:
- VB6 runtime files (if you have an error download these files (http://www.gpsfiledepot.com/tools/vb6-support-Install.zip) and run vb6-support-Install.bat (need to run as an administrator in vista)
- Windows XP/Vista
- PostGIS (and make sure it works on vista, on my two vista computers postgis is failing)
- FWTools
- cgpsmapper
Known bugs:
- Inability to handle some encoding of GNIS files
- PostGIS failures in Vista (fix available in this forum)
- Forest Service processing very confusing
Possible bug:
- May not work with password protected postgresql databases
With all that said: feel free to try it out. I'd highly recommend using it in place of dem2topo since the runtime files are hard to come by.
With that said, the current version is attached.
BETA: 1.0.21
Released: 8 Feb 09, 20:02 Alaska Time
Download: http://www.gpsfiledepot.com/tools/topo_process.zip
Installation:
Just put all the files in the zip file in one directory; don't edit them; run topo_process.exe
New version with some random bug fixes.
Also, see http://forums.gpsfiledepot.com/index.php/topic,101.0.html for a version of pgsql2shp that works on vista.
new version added the ability to download the 2008 roads data.
Just thought I would let you know. Using the previous version when running the transportation it ran all counties in the state. The new version only ran 1 county, I did it several times and the same results.
Cheers-Marc
what state because I just did Arizona and Mississippi and it downloaded all the counties.
Washington. One other thing I noticed is that when you process the transport data in map edit, one checks from field then you do not have the MP_type option as I did when I manually did everything. What would be the field to select then?
Cheers-Marc
Hmmmm, for some reason it does not have any of those either, my field set options are:
STATEFP,COUNTYFP,COUNTYYNS,TLID,TFIDL,TFIDR,MTFCC,FULLNAME,SMD,LFROMADD,LTOGDD,FRROMADD,
and a few at the end of the string which look to be associated with rail and such.
Thanks-
there are numbers in series but not the hex numbers you refer to. example of numbers on 1 line going from left to right:
STATEFP=53, COUNTYTP=001, TLID=181850066, TFIDL=206369161, TFIDR=206369159, MTFCC=S1500.
I re-opened the one that I did manually and it looks right.
When you refer to manually did you click each button (the three) in order? I normally just use the all at once button.
I just checked one I did for Mississippi last night and the furthest column to the right was the MP_TYPE column. I'll check washington out for you now.
I just did an all at once with Washington and also had MP_Type as my last column.
Merry Christmas everyone.
Oz, I did them one at a time, Maybe that is the error, I will run again combine at once.
I used county once and by name another time different results as you would expect.
Newest query
I ran the water body program and only ended up with prj files (area, flowline and waterbody) did not end up with either shp files nor dbf.
I did it twice with both versions same thing.
I did end up with a error txt which read:
C:\washington\water\NHDWaterbody.shp: shape (.shp) or index files (.shx) can not be opened, will just import attribute data.
C:\washington\water\NHDWaterbody.shp: dbf file (.dbf) can not be opened.
Feel like I am being a pain the rear for asking for so much help.
Cheers-Marc
Are you guys working on Washington State 24k? I just started downloading dem data yesterday. Can someone verify no dem data for North of Cape Flattery, correct?
John
Oz, here is a interesting tidbit
I previously (last post) mentioned when I did all the water all I got was a prj file. When I just ran the transportation download, combine and process at once I ended up again with just a prj file in the transport folder. So then I downloaded, and then combined, checked folder all-lines.shp and the rest are there.
Then
I process the files (Postgis) and end up with a empty folder except a prj file. My paths all look correct.
Any ideas? I could just run them the other way as place on the tutorials.
Cheers-Marc
Marc: do you have a password on your database or did you do that trust thing?
That error on water sounds like the shp files don't exist where its looking for them.
I would like to know whats wrong because I sit here running it all day. It definitely DOESN'T work if you have a password for postgis though and I haven't experimented to figure out why. I'm currently have my computer process a lot of contours while the other one downloads the new transportation data so I can update my previously released maps.
what settings are you using on water? (checkboxes)
Oz, I did the trust thing. Enough passwords in my life as it is. Well check boxes, I only see the delete files check box for water and I did not check that since I am on bandwidth probation (I live in a very remote place and only have satellite connection) for a few days for hogging bandwidth I guess.
Oh yes shutle64, I haven't downloaded the north of cape flattery yet so I can not help you. But I will resume as soon as I am out of the bandwidth doghouse, I'll let you know then.
Cheers-Marc
I just went through this:
re-started service, ran water again same thing, except when reading the tosql.bat I saw a error stating it could not fine //bin/shp2pgsql.exe.
Re-installed shp2pgsql.exe (patch) and ran it again. Here are the logs:
tosql.bat:
"C:\Program Files\PostgreSQL\8.3\bin\shp2pgsql.exe" -d "C:\washington\water\NHDWaterbody.shp" NHD_Waterbody postgis > "C:\washington\water\nhd_waterbody.sql" 2> "C:\Documents and Settings\Marc\Desktop\topo_process.1.0.15\error.log"
Error.log:
Shapefile type: PolygonZ
Postgis type: MULTIPOLYGON[4]
again all I have are prj files.
driving me to drink, at least I wasn't while I did all this.
Cheers-Marc
I finished download dem today. It started at 1:30pm PST on the contours. I'm using the Topo Beta.
glad it worked for you shutle.
Hmm, it appears that its having a hard time reading it.
Does c:\washington\water\NHDWaterbody.shp exist? Does that .sql file ever come into existence.
I found some Washington websites for Water data.
http://fortress.wa.gov/dnr/app1/dataweb/dmmatrix.html
Can we use .e00 files?
John
Can we use the ftp for water data if we had a list of numbers to download, it could be automated?
ftp://nhdftp.usgs.gov/SubRegions/High/
John
Quote from: -Oz- on December 25, 2008, 09:17:10 PM
Does c:\washington\water\NHDWaterbody.shp exist? Does that .sql file ever come into existence.
No, it never has shown up. ???
Quote from: shutle64 on December 26, 2008, 06:25:20 AM
Can we use the ftp for water data if we had a list of numbers to download, it could be automated?
ftp://nhdftp.usgs.gov/SubRegions/High/
John
I will definitely look into this this weekend; has potential.
Quote from: deepspace on December 26, 2008, 08:02:46 AM
Quote from: -Oz- on December 25, 2008, 09:17:10 PM
Does c:\washington\water\NHDWaterbody.shp exist? Does that .sql file ever come into existence.
No, it never has shown up. ???
Does NHDWaterbody.shp exist? Because it appears that the program doesn't think it exists.
How is your folder structured? It has to be \water\NHDXXXXX\ and there should be quite a few NHDXXXX folders inside the \water\ folder.
C:\washington\water\nhdxxxxx
there are 52 folders with nhxxxx inside the water folder
I downloaded them 100 quad at a time
Well, There must be something in the data base I need to work with. I manually ran them all, but when I did the final stage I ended up with shape files from only the Olympic peninsula map I finished. Even though I ran the current shape files. Maybe re-install the txt file in the bin folder or delete altogether?
On the dem, I need to run this with Combine contours into one file? I tried without and I had a bunch of prj files and it kept starting over coutour.shp
John
Quote from: shutle64 on December 26, 2008, 07:19:21 PM
On the dem, I need to run this with Combine contours into one file? I tried without and I had a bunch of prj files and it kept starting over coutour.shp
John
I have never actually run it with combine contours. What it should do is make the contour.shp then when it doesn't combine it'll export one with the folder/file name (######.shp).
It sounds like its having a hard time loading or unloading the files to the postgis database.
Quote from: shutle64 on December 26, 2008, 05:59:44 AM
I found some Washington websites for Water data.
http://fortress.wa.gov/dnr/app1/dataweb/dmmatrix.html
Can we use .e00 files?
John
Global Mapper can open them and ArcGIS/ArcMap I don't believe GPSMapEdit can. They're similar to shapefiles.
Quote from: deepspace on December 26, 2008, 06:33:19 PM
Well, There must be something in the data base I need to work with. I manually ran them all, but when I did the final stage I ended up with shape files from only the Olympic peninsula map I finished. Even though I ran the current shape files. Maybe re-install the txt file in the bin folder or delete altogether?
What txt file in the bin folder? Topo process just reads the directory thats been loaded and goes through it one by one. Is it pointing to the proper folder?
I can add "Pauses" to the program and you can run that and it'll stop at each change (combine water, pause, create sql file, pause, load sql into database, pause, process data, pause, export sql, pause) then you can see how far it gets by checking for what files exist at each time versus what should exist. Would you be interested in trying it out to see whats up?
I'll be around a lot tomorrow to work on stuff like this so it would be a good time.
The user name was missing on the settings page. I added it back in and clicked save settings, and started topo without combining files. It's spitting out ####.prj and ####.dbf plus contour.shp
John
the nhd query's which are extracted to the bin folder as per the water data page of the tutorials. They remain in the bin folder:
area.sql.txt, waterbody ect.
Ok. It's working for topo, it making shape files. Is it normal for it not make a shape file for certain quads, or do I need to have one for every quad. The very first quad there wasn't a shape file created.
John
I found a list of sub-basins for washington at this website.
http://cfpub.epa.gov/surf/locate/hucperstate_search.cfm?statepostal=WA
There is 73 total for washington. You can look up other states there too.
John
Quote from: shutle64 on December 27, 2008, 09:27:49 AM
Ok. It's working for topo, it making shape files. Is it normal for it not make a shape file for certain quads, or do I need to have one for every quad. The very first quad there wasn't a shape file created.
John
Glad its working but for each folder that has a tif file (make sure it has a tif not arcgrid) you should get a shape file, dbf, prj for each folder.
Quote from: deepspace on December 27, 2008, 08:04:07 AM
the nhd query's which are extracted to the bin folder as per the water data page of the tutorials. They remain in the bin folder:
area.sql.txt, waterbody ect.
I'm confused, this program (at the start of this thread) is very much separate from the tutorial. Are you using the program or following the tutorial; that would totally change how I answer your questions.
Quote from: shutle64 on December 27, 2008, 09:53:54 AM
I found a list of sub-basins for washington at this website.
http://cfpub.epa.gov/surf/locate/hucperstate_search.cfm?statepostal=WA
There is 73 total for washington. You can look up other states there too.
John
That is useful for figuring out which subbasin I might be missing but the ftp only has sub-regions.
Just ran across the page to download national watershed shape files.
http://www.epa.gov/waters/data/downloads.html#national
Just curious if the detail is just as good. I just started downloading the 2 files to try out.
John
Let me know if they're any good. The stupid NHD servers are holding up my next two maps so while I wait I'm updating the roads in my other ones.
Quote from: -Oz- on December 27, 2008, 10:45:52 AM
I'm confused, this program (at the start of this thread) is very much separate from the tutorial. Are you using the program or following the tutorial; that would totally change how I answer your questions.
I have been using the beta program. I just decided to try it the old way since all I was doing was spinning my wheels. I have used the water and transport no less than a dozen times now only to always end up with just a prj file in each folder.
My question was since it did not work using the method from the tutorial yesterday (which had worked before) maybe the 2 issues are some what connected via possibly the txt files I mentioned or something else. Since the result of running the sql final yesterday afternoon after combing all the statewide files resulted in a shape file that was the same one created from my Olympic peninsula map (not the whole state as I had combined) it seems there is some conflict some where.
After all if no one else is having a issue it must be my machine
Cheers-Marc
Quote from: shutle64 on December 27, 2008, 10:51:27 AM
Just ran across the page to download national watershed shape files.
http://www.epa.gov/waters/data/downloads.html#national
Just curious if the detail is just as good. I just started downloading the 2 files to try out.
John
Attaching a picture of what in the shapefiles. It's looks incomplete for certain states. Gpsmapedit would not open for the biggest file, I had to use arcExplorer.
John
John
Topo process uses sql files in the directory where topo process is located. Most of those files have the extension .sql. It loads the directory files "on the fly". in topo_process' directory there should be a settings.ini file. Can you zip that up and attach it to your next post. That may help give me an idea of what is happening.
If you're doing it the old way it should just work just like olympia. If its having issues here you are not alone. Make sure that you have the postgreSQL data filled out correctly (username and all that). Then hit SAVE SETTINGS.
Please post your settings.ini so I can take a look.
I also downloaded what I thought would be high-res data for all of Ohio from the nhd ftp server but its a mdb database so I have to figure out how to open it I guess.
here are the settings:
[Software]
fwtools=C:\Program Files\FWTools2.2.8\bin
postgis=C:\Program Files\PostgreSQL\8.3\bin
cgpsmapper=C:\washington\cgpsmapper.exe
[PostGIS]
host=localhost
database=postgis
[User]
trans=C:\Documents and Settings\Marc\Desktop\topo_process.1.0.15
nhd=C:\washington\water
I also unzipped it directly to the Washington folder to give it a try. Here are those settings:
[Software]
fwtools=C:\Program Files\FWTools2.2.8\bin
postgis=C:\Program Files\PostgreSQL\8.3\bin
cgpsmapper=C:\Program Files\cGPSmapper\cgpsmapper.exe
[User]
dem=C:\washington\dem\78523481
interval=50
nhd=C:\washington\water
trans=C:\washington\transport
gnis=C:\washington\gnis
mp=C:\washington\dem\62841820
[PostGIS]
host=localhost
database=postgis
Cheers-Marc
Marc
The immediate thing that jumps out at me is you're missing a postgreSQL username. It should be postgres. Go and add that then click "save settings"
The reason that it worked one time and gave you olympia is because that is the data that is still in postgresql; once it loads and the new stuff it'll delete that stuff.
Once the username is saved it'll be at the bottom of the settings.ini file and should read user=postgres
Changed the settings:
[Software]
fwtools=C:\Program Files\FWTools2.2.8\bin
postgis=C:\Program Files\PostgreSQL\8.3\bin
cgpsmapper=C:\washington\cgpsmapper.exe
[PostGIS]
host=localhost
database=postgis
user=postgis
[User]
trans=C:\Documents and Settings\Marc\Desktop\topo_process.1.0.15
nhd=C:\washington\water
ran it twice I still have nothing. I think its best to maybe give up.
Quote from: deepspace on December 28, 2008, 12:26:36 PM
Changed the settings:
[Software]
fwtools=C:\Program Files\FWTools2.2.8\bin
postgis=C:\Program Files\PostgreSQL\8.3\bin
cgpsmapper=C:\washington\cgpsmapper.exe
[PostGIS]
host=localhost
database=postgis
user=postgis
[User]
trans=C:\Documents and Settings\Marc\Desktop\topo_process.1.0.15
nhd=C:\washington\water
ran it twice I still have nothing. I think its best to maybe give up.
user=postgres The username it typed in wrong.
i believe shutle caught your error.
use this as your settings.ini file:
[Software]
fwtools=C:\Program Files\FWTools2.2.8\bin
postgis=C:\Program Files\PostgreSQL\8.3\bin
cgpsmapper=C:\washington\cgpsmapper.exe
[PostGIS]
host=localhost
database=postgis
user=postgres
Sorry for not replying in a timely fashion, I fell behind the 8 ball on something. It makes total sense and will post (I am sure) a mission accomplished Tuesday. Thank you all for bearing with me as I muddles through this!
Cheers-Marc
Thank you all for your help things appear to have worked except I still have no M_TYPE selections. In water I have length and area plus a few others in transportation. I have area and length options in water. Best choices?
Thanks-Marc
can you zip up the smalled shapefile you end up getting (should be the area one I assume) and attach it to you post so I can take a look?
When it runs the commands in postgis it should be adding the mp_type column and then filling it in appropriately.
not sure what you mean by small, my areas is 66 megs, the dbf is 565 k for area.
Cheers-Marc
well just attach the .dbf (zip it first so it'll take it).
Then I can know.
Happy new year. Here is the dbf.
Cheers-marc
Well deepspace, you are correct. It appears that the area.sql file (should be located in the topo process directory) isn't being run. If it were some of the items you have would have been deleted and there would have been an MP_TYPE column. Do you have topo process in a completely separate directory from the map files? If not I'd suggest you redownload it and make it in a separate directory.
Secondly, in the topo_process directory can you open up area.sql in notepad and make sure it looks like this:
ALTER TABLE nhd_area ADD COLUMN mp_type character varying(50);
ALTER TABLE nhd_area ALTER COLUMN mp_type SET STORAGE EXTENDED;
ALTER TABLE nhd_area RENAME COLUMN gnis_name TO name;
DELETE FROM nhd_area WHERE fcode = '53700' or fcode = '30700' or fcode = '31800' or fcode = '34300' or fcode = '34305' or fcode = '34306' or fcode = '36400' or fcode = '37300' or fcode = '56800' or fcode = '43100' or fcode = '45400' or fcode = '45401' or fcode = '45402' or fcode = '45403' or fcode = '45404' or fcode = '46100' or fcode = '48500' or fcode = '40300' or fcode = '40307' or fcode = '40308' or fcode = '40309';
UPDATE nhd_area SET mp_type = '0x28' WHERE fcode = '31200' or fcode='44500';
UPDATE nhd_area SET mp_type = '0x49' WHERE fcode = '33600' or fcode = '33601' or fcode = '33602';
UPDATE nhd_area SET mp_type = '0x3b' WHERE fcode = '36200' or fcode='39800' or fcode='45500';
UPDATE nhd_area SET mp_type = '0x49' WHERE fcode = '46000' or fcode = '46006';
UPDATE nhd_area SET mp_type = '0x46' WHERE (fcode = 46000 or fcode = 46006) and areasqkm >= 15;
UPDATE nhd_area SET mp_type = '0x47' WHERE (fcode = 46000 or fcode = 46006) and areasqkm >= 10 and areasqkm < 15;
UPDATE nhd_area SET mp_type = '0x48' WHERE (fcode = 46000 or fcode = 46006) and areasqkm >= 2 and areasqkm < 10;
UPDATE nhd_area SET mp_type = '0x4c' WHERE fcode = '46003' or fcode='48400';
DELETE FROM nhd_area WHERE mp_type IS NULL;
Let me know if it does or doesn't look like that and we'll work from there.
Oz, I ran it from the desktop going folder to export to being: C:\washington\water\ so I will make the other changes and try another run in the morning. So what are the changes I need to make to end up with a MP_type in transportation.
Cheers-Marc
your transportation isn't changing either?
Do you have the latest version of postgis? Does the data ever appear to get loaded (check with pgadmin iii when it says it loading it). It sounds like the data isn't being processed at all.
No transportation does not work either. With version 1.5 I still do not get all the counties. So I am still using v 1.3 which will get all the counties from the 2007 data base.
I am using v 8.3
Cheers-Marc
Quote from: -Oz- on January 01, 2009, 01:28:03 PM
Well deepspace, you are correct. It appears that the area.sql file (should be located in the topo process directory) isn't being run. If it were some of the items you have would have been deleted and there would have been an MP_TYPE column. Do you have topo process in a completely separate directory from the map files? If not I'd suggest you redownload it and make it in a separate directory.
Secondly, in the topo_process directory can you open up area.sql in notepad and make sure it looks like this:
ALTER TABLE nhd_area ADD COLUMN mp_type character varying(50);
ALTER TABLE nhd_area ALTER COLUMN mp_type SET STORAGE EXTENDED;
ALTER TABLE nhd_area RENAME COLUMN gnis_name TO name;
DELETE FROM nhd_area WHERE fcode = '53700' or fcode = '30700' or fcode = '31800' or fcode = '34300' or fcode = '34305' or fcode = '34306' or fcode = '36400' or fcode = '37300' or fcode = '56800' or fcode = '43100' or fcode = '45400' or fcode = '45401' or fcode = '45402' or fcode = '45403' or fcode = '45404' or fcode = '46100' or fcode = '48500' or fcode = '40300' or fcode = '40307' or fcode = '40308' or fcode = '40309';
UPDATE nhd_area SET mp_type = '0x28' WHERE fcode = '31200' or fcode='44500';
UPDATE nhd_area SET mp_type = '0x49' WHERE fcode = '33600' or fcode = '33601' or fcode = '33602';
UPDATE nhd_area SET mp_type = '0x3b' WHERE fcode = '36200' or fcode='39800' or fcode='45500';
UPDATE nhd_area SET mp_type = '0x49' WHERE fcode = '46000' or fcode = '46006';
UPDATE nhd_area SET mp_type = '0x46' WHERE (fcode = 46000 or fcode = 46006) and areasqkm >= 15;
UPDATE nhd_area SET mp_type = '0x47' WHERE (fcode = 46000 or fcode = 46006) and areasqkm >= 10 and areasqkm < 15;
UPDATE nhd_area SET mp_type = '0x48' WHERE (fcode = 46000 or fcode = 46006) and areasqkm >= 2 and areasqkm < 10;
UPDATE nhd_area SET mp_type = '0x4c' WHERE fcode = '46003' or fcode='48400';
DELETE FROM nhd_area WHERE mp_type IS NULL;
Let me know if it does or doesn't look like that and we'll work from there.
Here is area sql un edited by me, looks fine:
ALTER TABLE nhd_area ADD COLUMN mp_type character varying(50);
ALTER TABLE nhd_area ALTER COLUMN mp_type SET STORAGE EXTENDED;
ALTER TABLE nhd_area RENAME COLUMN gnis_name TO name;
DELETE FROM nhd_area WHERE fcode = '53700' or fcode = '30700' or fcode = '31800' or fcode = '34300' or fcode = '34305' or fcode = '34306' or fcode = '36400' or fcode = '37300' or fcode = '56800' or fcode = '43100' or fcode = '45400' or fcode = '45401' or fcode = '45402' or fcode = '45403' or fcode = '45404' or fcode = '46100' or fcode = '48500' or fcode = '40300' or fcode = '40307' or fcode = '40308' or fcode = '40309';
UPDATE nhd_area SET mp_type = '0x28' WHERE fcode = '31200' or fcode='44500';
UPDATE nhd_area SET mp_type = '0x49' WHERE fcode = '33600' or fcode = '33601' or fcode = '33602';
UPDATE nhd_area SET mp_type = '0x3b' WHERE fcode = '36200' or fcode='39800' or fcode='45500';
UPDATE nhd_area SET mp_type = '0x49' WHERE fcode = '46000' or fcode = '46006';
UPDATE nhd_area SET mp_type = '0x46' WHERE (fcode = 46000 or fcode = 46006) and areasqkm >= 15;
UPDATE nhd_area SET mp_type = '0x47' WHERE (fcode = 46000 or fcode = 46006) and areasqkm >= 10 and areasqkm < 15;
UPDATE nhd_area SET mp_type = '0x48' WHERE (fcode = 46000 or fcode = 46006) and areasqkm >= 2 and areasqkm < 10;
UPDATE nhd_area SET mp_type = '0x4c' WHERE fcode = '46003' or fcode='48400';
DELETE FROM nhd_area WHERE mp_type IS NULL;
that does look right. Does it appear to ever load the data? (hve to watch in pgadmin iii).
I have not watched it in admin, I will and let you know.
Thanks-Marc
First off, thanks for this great program. It has really streamlined how I work with the data.
I have one question. I haven't been able to find an answer searching around. How do you recommend processing the contour shapefiles to eventual processing into polish and then img?
Quote from: aussieskier on January 04, 2009, 02:41:39 PM
First off, thanks for this great program. It has really streamlined how I work with the data.
I have one question. I haven't been able to find an answer searching around. How do you recommend processing the contour shapefiles to eventual processing into polish and then img?
Glad its working for you. To "finish" it I just follow the tutorial online except that I use Topo Process wherever possible to automate/streamline. The important steps are 8 and 9: http://www.gpsfiledepot.com/tutorials/how-to-create-garmin-topo-maps---part-8---combining-data/
I ran the transpotation with admin opened up. Made sure all the connections and settings where corrrect. While nothing ever did come up in the sql panel if one watched the statistics one saw postgise open and wait while the transfer happened (couple og hours) then when the shape file was being exported that postgise line had a dialog line (which I did not write down}
So then I opened up the new shape file and once again I di not come up with a MP_TYPE option.
Maybe I need to delete all previous sgl data?
Cheers-Marc
Quote from: -Oz- on January 04, 2009, 03:42:16 PM
Quote from: aussieskier on January 04, 2009, 02:41:39 PM
First off, thanks for this great program. It has really streamlined how I work with the data.
I have one question. I haven't been able to find an answer searching around. How do you recommend processing the contour shapefiles to eventual processing into polish and then img?
Glad its working for you. To "finish" it I just follow the tutorial online except that I use Topo Process wherever possible to automate/streamline. The important steps are 8 and 9: http://www.gpsfiledepot.com/tutorials/how-to-create-garmin-topo-maps---part-8---combining-data/
It works a lot cleaner than the batch file I had cobbled together :).
I am running into memory issues with the contours, do you just convert each shape file to an mp file and then combine those all into one? ( I am working on an Oregon map) I guess I am trying to figure out the best way to split it into smaller pieces since I can't really load it all up together easily.
Also, is anyone else having issues with GNIS data? It is like the names got changed again or something. I run it and it doesn't seem to download anything and I don't get any output.
yea, i have a new version, its posted at the front of this thread it fixes the gnis issues.
Great!
Working perfect for me :)
On GPSmapedit I have found easier to convert all the .shp files to .mp and then only load the data needed to complete one 30x60 quad. GPSmapedit for some reason can load larger files in polish format. Also try closing the program and then reopening it if you run into a memory issue, for some reason this seems to free up "room" for larger files. Closing the file in mapedit doesn't help. I think it may have a memory leak.
Quote from: deepspace on January 04, 2009, 05:37:22 PM
I ran the transpotation with admin opened up. Made sure all the connections and settings where corrrect. While nothing ever did come up in the sql panel if one watched the statistics one saw postgise open and wait while the transfer happened (couple og hours) then when the shape file was being exported that postgise line had a dialog line (which I did not write down}
So then I opened up the new shape file and once again I di not come up with a MP_TYPE option.
Maybe I need to delete all previous sgl data?
Cheers-Marc
delete the previous data; then tell me if it exports anything. I think it won't.
I know I've asked this before; are you using a password for your postgresql username?
Quote from: -Oz- on January 05, 2009, 08:48:23 PM
I know I've asked this before; are you using a password for your postgresql username?
No password. I am going to delete everything. Thanks-Marc
Which everything? Topo Process or PostgreSQL?
I'm very curious and definitely want to help you make it work. I can give you some dos commands this afternoon to run to see why it isn't working.
Quote from: -Oz- on January 06, 2009, 08:31:21 AM
Which everything? Topo Process or PostgreSQL?
Frankly I was going to delete and re-install everything. Something is corrupt somewhere. I was able to build a map manually but since then have just spun my wheels.
I'll wait to see the dos runs.
Cheers-Marc
Wanted to drop a big thanks for this. Working on (another!) WA map and this has been pretty great in regards to letting me easily process large amounts of data.
Also, not sure if shutle64 ever got any word about North of Cape Flattery, but I came up with with no DEM data for it also.
Marc - Just run any of the dos commands from the tutorial (make sure its the none -p version). See if any of them work and let me know.
OZ, I did not ever have a issue with the manual runs, I could not create the same tables.
SOOOO....
I completely deleted everything, cleaned registry both with jvtools and regedit.
re-installed everything and ran transportation today and everything is fine. Ended up with the MP_TYPE needed to go forward but interesting it took twice as long (7 hours) and file is twice as large.
One thing I did was to attempt to do things one stage at a time, ie: download, then come back and combine, then comeback and process the current set up only allows combine at once, no return to operation.
But what the hey it is a great tool even if I had some issues getting it to work.
A question regarding the contour *shp files. If we're using GPSMapEdit, I assume we need to convert the files to *.mp files. Is there an easier way to do this other than importing it into GPSMapEdit? And if not, when importing, I need to select a line style for the polyline. Any help with what to select would be greatly appreciated :)
Really glad its working for you Marc.
If you've already processed them there should be an MP_TYPE column, that'll be the line type.
Thanks, appreciate you pointing that out :) I expected them to take a while to save to *.mp files, but no time at all.
I tried to run the beta. When I clicked on it, the GNIS box that asks for Internet appeared. I clicked on it and got Run-time error '339': Component 'MSINET.OCX' or one of its dependencies not correctly registered: a file is missing or invalid.
Help?!
Tsnake: Thats because it needs vb6 dependencies (stupid language if you can't include them easily); download from here: http://www.gpsfiledepot.com/tools/vbrun60sp6.exe
if that doesn't fix it that means that is in a different set of files and I'll post it.
Nope,
I installed the vb6 thing and it worked. I just used the gnis part and it was great. I will try the rest of them once I download more data.
Tsnake
It would be nice if the topo process could rename the MP files after the .img file has been generated. This would allow us to kill and restart the process.
This is by far the slowest and the most painful part of the entire process.
it could rename them, what exactly are you suggesting?
i think i'm having a hard time understanding. The program could rename, move, etc?
The idea is to be able to restart the process in case it was interrupted. You could rename the xxxxxxxx.MP file to xxxxxxxx.MP_
savage: your wish is my command, v1.0.18
Thank you. I will try it later this week-end.
Question: Does the Topo Process program download data? If so, does it work if you've already completed the downloading? Thanks!
Lalita
It does download some data.
Will it work if you're downloaded it; that depends. The transportation data - no; it needs to download it on its own. GNIS - no; water - yes (doesn't download that); dem - yes (doesn't download that).
Thanks for the info. I'm not particularly computer-talented, so work best with a step by step tutorial. Your program sounds great, but I probably better stick with the tutorial method.
I'm kind of leery of steps 8 and 9. I have an older computer, and ony about 2 G of RAM so am not at all sure it can even handle it. I am ever so slowly following the tutorial.
it'll just take awhile. I did it on my laptop with 1gb ram. Now it has 2gb and its better.
What software are you using; in global mapper you can export the contours and then export the rest and combine them in dos and that got rid of a lot of my memory errors.
All this program does it automate some of the steps btw.
I don't have GM. I have dem2topo, FWTools, Postgresql, GPSMapedit, & cGPSmapper. I also use MapWindow GIS if I need to reproject shape files into different datums/coordinate systems.
I'm currently still in step 1 (processing the dem data), but have already downloaded the water and transportation. My most ambitious map project to date had been a transparent map of some trails, so this should be rather interesting. Glad to hear it can be done with the 2G of ram.
Thank you so much for all of your help and patience. I feel a bit like an idiot trying this, but the worst thing that will happen is I won't end up with a completed map. Best case I'll have my state completed and posted for all the world to suffer enjoy. :P
Lalita
hehe. We're here to help.
Which state are you working on? And using this program here you can skip dem2topo; it can do it for you once you download all the tif's, its more automated that way.
I'm working on Missouri, and am close to halfway through the dem data. I'll probably just stick with dem2topo because my husband and I share a computer, so I can select a smaller group of files to run during the night or while we're off to work.
new version due to gnis updating the file names
So I may have made a error (again) I just finished another set of contours but did not check the combine the contours box because the computer was going to process some images I intended to take tonight. But its cloudy, soooo now I have a bunch of shape files of which do not have the contours separated (ie: 20, 100, 200 and so forth) if I try to open them in GPS Mapper it asks of course what contour they are. Of course I do not want to choose 1 if it hase them all. Is they a way to combine post process for dummies like me?
I was also wondering also would it no be a good idea to offer the contours as a transparent map which could be turned off and on at will.
Cheers-Marc
maps4gps actually tired to release contours that were transparent. I like the idea sometimes but I feel true topos have good water data and POIs. Also, I think a lot of novices don't understand the transparent thing. So i'm not sure.
Now, what do you mean it asks what contour they are? The program should have built in the MP_TYPE. I don't really understand what you have so what you need to do. I never combine all my images; I keep them in 100k quads and then just load them all up at once to export them.
The contour thing was a mistype on my part. I see the error. How do the 100 quad's get combined?
I combine them in global mapper or gpsmapedit. Really though; it doesn't normally affect me because the 100k contour quads match the 100k image files.
If they were combined in the program it loads all of it into one table in postgis; then processed it all; then exports it all at once. This would likely fail on a fat32 drive since it would be more than 4gb.
Hmm, trying to figure this part out. The mount Olympus area is over 1 gig in a 20 foot contour. I need to split it to make it into the image file, smaller ones combined later. I have determined with my computer and via information from Stan I need to keep my file size to about 115 megs (he has a newer version coming which hasn't the memory issues).
I was thinking I would be able to take a bunch of .img files use maptoolset to create 1 complete .img of them all. Maybe it does not work that way.
I still have not figured out how to precisely split a quad either. But I can say this much the quads look good so far.
Quote from: -Oz- on January 17, 2009, 06:17:01 PM
Tsnake: Thats because it needs vb6 dependencies (stupid language if you can't include them easily); download from here: http://www.gpsfiledepot.com/tools/vbrun60sp6.exe
if that doesn't fix it that means that is in a different set of files and I'll post it.
Dan,
That didn't do it for me, I still get the same error as Tsnake but 'component comdlg32.ocx'
can you send the different files?
Thanks
Sure, files: http://www.gpsfiledepot.com/vb6-support-Install.zip
still getting the same error, does it matter where i save the extracted files?
I believe you need to save the ocx files to c:\windows\system32
I copied all files to system32 except scrrun.dll (no permission to copy or rename), i still get the same error when I run topo_process.exe from the program files folder
Sorry, i'm dumb! You have to make a little tutorial for everything so I have a chance at doing it right.
That error is common; I really should write in a different language without so many microsoft imposed issues.
See if this article helps you: http://windowsxp.mvps.org/comdlg32.htm
OK, I followed the instructions in the article for Vista and it went smoothly. I repeated the process for all the files that came with vb6-support-install.
I still get the same error message. Are there any other steps that aren't in that article that i need to perform? I'm probably overlooking a step.
Thanks for your time.
there should be a .bat file included in the zip. Right click on that and click "run as administrator" or something similar (sorry, only have my winxp computer on this trip).
That should copy/register them. Tell me if that still fails.
Are you really still getting the com32 error even after that tutorial?
I ran the vb6-support-install.bat using the steps you described. The command window opened and 6 of the files were copied. However, scrrun.dll failed - Access is denied
I am pretty sure I had tried to run this file multiple times before.
Even now I still get the com32 error
does the file it mentions in the error actually exist in c:\windows\system32 ?
yes it exists in C:\Windows\System32. I have tried to rename it, change permissions, etc but access is denied
thanks
i hate vb...
I'll have to keep thinking...
Well, you can take ownership of the file and then replace it, but if it is a windows file and you replace it with a non-windows file, you might end up in a situation where Vista will claim that "You have made an unauthorized change to your computer..."
Anyways, to change the ownership, open my computer and navigate to c:\windows\system32. Right click on the file you want to replace, select properties. Click on the security tab and then click on the Advanced button. In the new windows that pops up, click on the owner tab. It will list that the current owner is "TrustedInstaller." Click on the "Edit" button below. Another window will show up which will look similar to the previous one. It will list Current Owner: Trusted Installer. From the "Change owner to: " box, select administrators and click Apply. Click OK, to close this window and keep clicking OK until all the windows are closed.
Again, right click the file and select properties. Click the Security tab. Select Administrators from the "Group or user names" list and click "Edit". In the window that comes up, select administrators and then in the "Allow" list in "Permissions for Administrators" check the "Full Control" checkbox.
That should hopefully do it. Just be cautious that if you change a file owned by "TrustedInstaller", your windows installation might run into issues with "You have made an unauthorized change to windows..."
I had a very interesting thing happen which I would like to resolve. I noticed looking at the water base map I had created a error in 1 quad. I re-downloaded the quad, unzipped the file and ran topo process (newest version) on the lower left corner it said there was a error: that there was no meta data (the folder I assume) but it continued to run. At the end of the run it created a file from the last run I had done. Absolutely nothing to do with the files in the folder I ran it from. I deleted all those files and unzipped it again. Opened the shape file to make sure (again) it was the correct quad. And ran topo again. The exact same thing happened, no meta and final was a area 200 miles away.
I am assuming I need to clear a data base to rectify this. Question is which one?
Thanks-Marc
Open pgAdmin III and delete all the nhd_... tables (delete/drop).
I didn't think it required metadata. If you get a chance can you post exactly what it said or tell me how I could get the quad.
Quote from: dhisum_dhisum on March 12, 2009, 01:17:03 PM
Well, you can take ownership of the file and then replace it, but if it is a windows file and you replace it with a non-windows file, you might end up in a situation where Vista will claim that "You have made an unauthorized change to your computer..."
Anyways, to change the ownership, open my computer and navigate to c:\windows\system32. Right click on the file you want to replace, select properties. Click on the security tab and then click on the Advanced button. In the new windows that pops up, click on the owner tab. It will list that the current owner is "TrustedInstaller." Click on the "Edit" button below. Another window will show up which will look similar to the previous one. It will list Current Owner: Trusted Installer. From the "Change owner to: " box, select administrators and click Apply. Click OK, to close this window and keep clicking OK until all the windows are closed.
Again, right click the file and select properties. Click the Security tab. Select Administrators from the "Group or user names" list and click "Edit". In the window that comes up, select administrators and then in the "Allow" list in "Permissions for Administrators" check the "Full Control" checkbox.
That should hopefully do it. Just be cautious that if you change a file owned by "TrustedInstaller", your windows installation might run into issues with "You have made an unauthorized change to windows..."
I haven't tried that yet, sounds kinda scary. Anybody else think I should give this a try?
Quote from: -Oz- on March 10, 2009, 12:02:17 AM
there should be a .bat file included in the zip. Right click on that and click "run as administrator" or something similar (sorry, only have my winxp computer on this trip).
That should copy/register them. Tell me if that still fails.
Are you really still getting the com32 error even after that tutorial?
I am running Bitdefender Antivirus which usually warns when the Windows registry is being edited. Could that have something to do with it?
Quote from: -Oz- on March 18, 2009, 08:56:05 PM
Open pgAdmin III and delete all the nhd_... tables (delete/drop).
I didn't think it required metadata. If you get a chance can you post exactly what it said or tell me how I could get the quad.
Oz, Sorry I have been gone. I will run it again with out deleting the data base so I can re-create the error. But it is something which pops up on the lower left hand corner which has no effect on the operation finishing the process.
I did something to make the patch which ended up in poor results. I opened the shape files in Global Mapper> exported vector .mp file > choose the box to use existing format > used a existing .mp file with zoom and dictionary the rest of the map is> created the mp file. It will not run on cgpsmapper to convert to >IMG file. cgpsmapper opens and spends its time ( I waited a hour twice) just reading the dictionary. Never to change. Soo do Not Do As I have Done.
Cheers-Marc
Running the version ending in .19 and trying to use the GNIS processing which seems to work but no file output? I do have an error log in my folder containing;
C:\Garmin\Maps\gnis.shp: shape (.shp) or index files (.shx) can not be opened, will just import attribute data.
C:\Garmin\Maps\gnis.shp: dbf file (.dbf) can not be opened.
Any ideas?
Quote from: -Oz- on January 17, 2009, 06:17:01 PM
Tsnake: Thats because it needs vb6 dependencies (stupid language if you can't include them easily); download from here: http://www.gpsfiledepot.com/tools/vbrun60sp6.exe
if that doesn't fix it that means that is in a different set of files and I'll post it.
I finally got my new computer and am going to finish my project. I had the 339 run-time error and have this file to install but its not clear as to where is should be installed. When asked for a location what do you suggest?
Thanks-Marc
i will fix gnis processing when I get vb6 here (i'm in Afghanistan currently).
You need the vb6 support files; install that vbrun60sp6.exe wherever it wants and elsewhere in this thread are some vb support files in a zip; just run the included .bat file.
What is the latest version of topo_proces? I just installed 1.0.19 on my Vista machine and I am getting the 339 error but I have not been able to fix that yet. I am going through this thread now.
---------------------------
download_process
---------------------------
Run-time error '339':
Component 'MSINET.OCX' or one of its dependencies not correctly registered: a file is missing or invalid
---------------------------
OK
---------------------------
Running VBRun does not seem to fix the problem.
I am getting ready to upgrade the New England topo map.
savage, you definitely need the vb6 files linked to here: http://www.gpsfiledepot.com/vb6-support-Install.zip
Run as admin the .bat file
This is really frustrating. This is not solving the problem, but I may not have installed it correctly.
I hate Vista! Why don't you develop in Java instead ;)
Oz, seeing your program is great... You know what would also be a GREEEAAT add on for gpsmapedit !!! The ability to open and bulk edit numerous img files at time, and bulk update such as properties, then have the program save in bulk in the polish format and then to img.
Quote from: savage on August 18, 2009, 03:32:02 AM
This is really frustrating. This is not solving the problem, but I may not have installed it correctly.
I hate Vista! Why don't you develop in Java instead ;)
I hate vb actually; I just never learned Java... I was thinking of trying to learn python now so that it can be cross-platform. I can't wait to try and install this on windows 7 :)
Hi Oz,
Were you out of touch for a while? If you want to rewrite it and are ready to learn something new then I suggest Java. Python is fine by me, but if you do it in Java, I will help you.
Sounds like Java it will be, is there a good compiler to start with?
Yea, I spent the last half a year in Afghanistan.
Welcome back!
At work we use NetBeans as a development environment (www.NetBeans.org). You probably want Java SE, but you can download everything if you want to use it for PHP and Web stuff.
You can also look at Eclipse (www.eclipse.org). It is very similar to NetBeans.
Both are equally good.
Good evening, after working fall thru early winter with my map I have decided to re-build from scratch. I found queer little things that are not up to snuff. I have started gathering the data already. Question is: I see no reference to a newer version other than a query. Is there one?
Thanks-Marc
I haven't made any updates yet; the only thing that doesn't work is the GNIS data; just need to install the vb program on my new laptop
Thanks for the reply Oz, shall continue with out supervision.
Any reason to think that postgis 8.4 will be a problem?
I just downloaded and installed FW tools, Postgresql 8.4 installed postgis data base and made a transport run, doing so separately.
Extract and download: OK no problems (copied files to separate location so I would not have to do it again)
Combine files: OK all lines result, copied again
postgis: resulted in just a transportation.prj file only
I seem to re-call this from before
here is the error:
Shapefile type: Arc
Postgis type: MULTILINESTRING[2]
here is the settings.inc:
[Software]
fwtools=C:\Program Files\FWTools2.4.6\bin
postgis=C:\Program Files\PostgreSQL\8.4\bin
cgpsmapper=C:\Program Files\cgpsmapper.exe
[User]
dem=C:\washington\contours\51725823
interval=20
trans=C:\washington\transport
gnis=C:\washington\gnis
nhd=C:\washington\nhd
[PostGIS]
host=localhost
database=postgis
user=postgres
pass=kimmei
Thanks-
Alright; I'll be installing the latest postgresql and postgis this weekend so i can test then.
Good day all, Oz any luck ferreting out issues with new Post? Other some of the layout ones.
Cheers-Marc
Good Morning, I just processed my nhd data. I watched my folder as the topo process created water body.shp and so forth. All went well until it combined everything, when it was done what was left once more was just prj files. Thoughts?
error log:
Shapefile type: PolygonZ
Postgis type: MULTIPOLYGON[4]
My windows 7 computer started crashing so I had to fix that and now try to install it.
If you look in the database with pgadmin III you shoudl be able to see if its ever loading data into the database.
I do not see that is connects, opened admin and watched also same thing. I do not see a error in the settings, postgis 1.4 for PostgreSQL 8.4 was installed via the stack builder.
[Software]
fwtools=C:\Program Files\FWTools2.4.6\bin
postgis=C:\Program Files\PostgreSQL\8.4\bin
cgpsmapper=C:\Program Files\cgpsmapper.exe
[User]
dem=C:\washington\contours\51725823
interval=20
trans=C:\washington\transport
gnis=C:\washington\gnis
nhd=C:\washington\nhd
[PostGIS]
host=localhost
database=postgis
user=postgres
pass=kimmei
I found the answer. In the 8.3 version you open the pg_hba.conf file from start>all programs>postgresql X.X>Configurations Files
There is not such thing with 8.4.
one needs to open explore/program files/postgresqls/8.4/data/pg_hba.conf and change the settings to trust as done with 8.3
Good to know, glad you figured it out because I couldn't get it since I did that and didn't realize I had done it :)
I'll modify the tutorial.
2 quick questions:
A: would it be possible to have a check box that allows the computer to shut down once its finished a process batch? Say as a example having converted a batch of NED files.
B: any one noticed a getting some .bat files during a convert? I ended up with 3 while converting to .img this weekend.
Cheers-
it creates some bat files to do stuff...
I am getting ready to start java programming of it because I hate getting vb to work on windows 7.
Quote from: -Oz- on February 23, 2010, 08:00:05 PM
it creates some bat files to do stuff...
I am getting ready to start java programming of it because I hate getting vb to work on windows 7.
Let me know if you need help to get started.
I was re-doing a few quads things where going as normal but all of a sudden when a quad is finished converting from mp to .img there is not .img file sitting in the folder. Th original .mp was re-named but nothing else. The first 2 in the folder went normally. I re-did and the same thing. Any body else have this happen? I did 88 with no issues at all.
Thanks
When I have this happen it means that the cgpsmapper has errored. Normally an RGN table size error. Solutions are hard to come by but most times I just split the map into smaller segments to avoid the error. My Hawaii topo is a great example it has that issue all the time.
updated v 1.0.21 (still in vb, sorry, working on it)
Update for the latest GNIS
Update for 2009 census roads
Hello,
I am getting ready to update my NE topo map. Anything new with the topo process? I just installed it on a Windows 7 machine and still get the missing Msinet.ocx?
Thanks.
still need those vb modules; I know the zip file is linked to and if you run the .bat as an admin it will install them. I am currently TDY to Ohio so while I'm here I'm actually working on it in Java. Its slow going for now :)
What bat files are you talking about? Maybe I installed the wrong zip file?
these are the vb libraries that are required: http://www.gpsfiledepot.com/tools/vb6-support-Install.zip
They were pretty much included in xp...
Make sure you have the latest Topo Process too so that you get the most recent census data
I tried the bat file but I am still getting the same problem. I will wait for the Java version ;)
i'll probably come asking for your help pretty soon :)
Just FYI. I recently used ver 1.0.21 for the transportation files and discovered that files are now named 2009 while topo process still expects 2008. A simple work around is to download the files, and manually rename before continuing.
Also if you only have a single contour file, don't check "Combine..." as everything gets deleted except for 2 .prj files.
Quite a handy tool though.
crap, vb is only on my pc at home so i can't fix that from where I am right now