Hello, I am a newbie. I downloaded the DEM data for the entire state of washington and have finally after a lot of work generated 40ft contours of the whole state in the Global Mapper gmp format. Since GlobalMapper would cop out when I would give it the whole state at once, I did it chunk by chunk. Today I selected all Washington state subbasins and provided the email address to download the files in the morning but have not received anything so far. Any ideas how long it usually takes?
Thanks
-Dhisum
Anywhere from 2hrs to 48 hours; if its been longer than 48 hours resubmit the request.
Large orders seem to arrive around 4am.
Thanks. The data has mostly arrived. I will muck with this and get back to you folks with results.
excellent
When I requested the NHD data it took them 5 days to get back to me for Washington State. I just finished a map of the state with 200 foot contours, and am generating one with 40ft contours right now. I found with the contour data exporting in smaller grids helps as parts of the north cascades will eat up all your system memory and global mapper will throw an error and exit. Good luck.
will either of you two be releasing the map anytime soon?
I know a lot of people are looking for washington, esp since even the "better" one isn't free.
I plan on releasing the 40 foot contour map and then compressing and putting on one of my ftp servers the raw data so others can customize their own. I have been tweaking some of the POI and NHD data, as well as checking the map for accuracy. All I am waiting on now is the re-processed NHD data. I think that either I have horrible luck with the software in the tutorial, or Washington State makes some bigger shape files than in the tutorial because I'm always running into weird out of memory errors. For instance, the combined flowline shapefile is just shy of 2 gigs. I don't know what is normal, but I had postgre die a couple of times before it successfully exported all of the data to a shape file. I expect to have the 40 foot contour map done, if not today (barring no more weird errors) than by the end of the week. I already uploaded the 200 foot contour to my garmin and went to a couple USGS markers around here to test it, and it seems ok.
In case you have this problem, when exporting the processed data from the nhd_flowline table I had to break the data into quarters and export individually. I made 4 other tables, and put about a quarter of the records in each, and exported to 4 different shp files. There were about 4 million records after processing the data, and after exporting about 3.1 million I would get a fwrite(). Breaking it up into quarters worked well.
Well, I've already been planning on doing Oregon, so sure. Though, I already have Washington done as well, I just went back to add some data that was missing. I noticed some peaks missing in the POI and some of the names in the waterbodies were names that don't match up with my USGS paper maps. It doesn't seem to like duplicate names either in the POI download, as there were several peaks that share a name, but only one gets included in the download, like Diamond Peak.
Well, if the 40ft map is already done, then there is not much point me doing it as well, but unless I see it out in the wild to be downloadable for free, I will continue. I have finished all the data preparation steps and currently GM is humming along generating the MP files. When that finishes, I will start cgpsmapper and see how that does. I could not find the National Park information and I skipped making the county boundaries, but other than that the rest of the data is there. As someone else pointed out, the Cascade Mountains generate huge contour file but lucky me but I have a good machine so it should be ok. Initially I started with 20ft contours but GM kept on crashing, so I finally switched over to 40ft.
If the MP file generation finishes today, I will start the cgpsmapper process tomorrow morning and hopefully by next week early, the maps will be ready.
Meh, I've always been planning on putting it up for free download, but I'm not going to rush my build and not include data, data that I use for mountains that I hike and climb. I never said stop building your map, just said that I had already done Washington, because you had suggested (or implied, whatever) that me and the OP stop and focus on Oregon. All well and good, but I don't appreciate the comment that I wouldn't share the map or that I would sell it even after I said I would put it up along with the raw data on one of my FTP servers in a previous post on this thread.
So yeah, do whatever, I really don't care. I'm going to keep making maps of areas and mountains that I climb, and I'm going to make sure the data I need is there. If it is useful to others, great.
I was going to do Oregon as well; I only have some of the data done.
With that said; if someone else will do it then I can move to somewhere else. I will also be trying our Mapwel with this one.
Thanks for the explanation, and I apologize for getting snapish. As far as not releasing it, not to worry, I'm converting some shape files to GMP files to reduce size during processing, but after that I'm going to process it and wrap it up in a executable.
As far as coordinating with Oregon, if people are interested we could divide some of the work and I could create some accounts on my FTP server to help transfer files around. I dunno how people would want to divvy the work. I don't really care, but if people are interested in that let me know.
I would be interested in oregon as well. We could coordinate. We could divide the work between dem, nhd and transportation data. I cannot however host the data. Would that work for you Krellor? or anybody else? By the way, for the site moderator, the washington state xml file is incorrect in that the URLs for the NED data are all the exact same except for the first one. I had to recreate the URLs by hand to get the right data from the seamless server.
I meant the excel file not XML. Guess I am feeling sleepy :)
I should have remembered that. In the excel file, when you download it, if in the lower right corner of the dem link cell, hover your mouse over that corner and it will make a cross. Now, click and drag to all of the lower cells, and it will make the links for you. This is because those cells are created using a excel formula based on cells in the previous cells. Dragging that lower right corner down copies that formula down, but is smart enough to make sure it uses the correct cells.
As far as coordinating the data, I can host it. I'll go ahead and make a folder structure like the tutorial has and put it in a directory. Go ahead and email me at krellor at shadowsprite dot com and I will send a username and password that has access to the directory along with instructions to connect. Once we do that we can take it from there.
I have split the state into 4 quadrants to deal with memory issues 3 of which are finished. The first run does not have contours (essentially a road and water ways map) the second does. Things should be finished this week. I sold one of my ccd cameras and purchased a copy of global mapper. I am toying with the idea of re-doing it all using global mapper just to be able to dump the made with gprmapper on the lower left corner. What do you think?
I am also wondering does any one know what the default levels are for the Americas base map?
Cheers-Marc
Oh yes I forgot I can host also
Cheers-Marc
So, here is what I am thinking. I will start downloading and processing the NHD and transportation data for Oregon. Someone else can take up DEM data. After I am done processing the data, I can upload the data to Krellor or deepspace's hosting sites. There is not much point uploading the raw data as that would be a bandwidth killer for not much benefit. Does that sound reasonable?
That sounds good. I replied to your site message with the info to connect to the FTP directory.
I myself will finish my Washington projects completely first. I just sent 2 of the maps with installers to some friends for reviews.
What I do not like at this point is when you zoom out at some point you loose any features (as opposed to the base map) and to have any features you need to turn off the custom map.
Cheers-Marc
Thanks Krellor, got your mail. So I will start on the NHD and the transport data for Oregon. And yeah, WOOHOO!! Last night, GM exported all the cgpsmapper files and I started cgpsmapper on converting the mp files to the img files. chug, chug, chug it goes and it has advanced to a large degree, meaning a lot of files are done. I tested one of the IMG files and it works beautifully in map source. At this rate, seems like the WA map will be ready before the week is over :)
Did you have any trouble exporting the whole state to IMG files? I didn't have trouble with the 200 for contour, but now with 40 foot I am running out of memory. I think like deepspace I will be breaking the state into quarters, and then gridding and exporting those.
EDIT: I didn't mean img files, I meant mp files.
Krellor, earlier, when I was trying to load just all the contours gmp files in GM it would cop out with "Unknow error" = Out of Memory errors. But when I started creating the final catalog to export everything to mp files, it surprisingly did load everything. I kid you not, my heart was literally in my mouth all the time as the dem files were huge and even the processes nhd files were a few gigs. It did however load everything and did export them properly as well. Right now cgpsmapper has been running since almost midnight last. I divided the task between two parallel cgpsmapper runs and except for 3 files, everything else has been processed. One of the cgpsmappers has been stuck at 18% for its last 800MB mp file for a while now while the other is I think on the last file in its batch as well. So, maybe if gods are happy, I can have this ready by EOD friday?
Wow, a lots happened since I last looked.
An 800mb mp file; how large is the area? My 300mb files took around 8hrs (only could get 1/4 of the processor for some reason).
I would definitely not waste bandwidth uploading the basic data; process it all or that is just silly.
I also fixed the Washington excel file.
Good grief, I feel like this map is a comedy of errors. So, I finally get everything broken up, cgpsmapper churns out the image files, and mapsettoolkit does it's thing and installs them in mapsource. I'm felling good, and then I start to notice that some little tributaries here and there are missing on the map. I noticed because I was checking areas that I frequent and have USGS maps of. So I go back and load it all up in global mapper and it looks fine. I check cgpsmapper error logs, and it had a bunch of "Not enough node" errors. After some searching and testing I think that the problem is how I broke up the flowline data. That even though it is all displaying correctly in Global Mapper, that the polygons aren't quite right. So, I've gone back to the flowline data to try and re-process it. Very aggravating.
My home internet connection has been having some issues, so if I go offline for a few days don't worry. Hopefully I'll finish with the processing by then. Luckily I kept all of the raw downloaded data so I don't have to download it again. It is grustrating having everything right except the flowlines. Bah. Good luck on all of your map efforts.
once you figure out what caused these issues please let us know so we all can learn from it. Especially if its the postgresql part that is deleting what you need.
WOOHOO! Finally, after a lot of sweat, blood and a sprinkle of pixie dust, the WA map is finished. I will start uploading it immediately. Hopefully other people can use it. If there is a problem, let me know and I will see if I can do something about it.
Oz,
800MB file was not even the biggest one I had, that one was 1.1GB. That is not the one took most time, the 800MB file did. It took almost 20 hours to process. And yes CGPSMapper was stuck at 25% CPU for some reason. I attributed it to the fact that I have a Quad Core system and that probably CGPSMapper is not multi-processor aware.
Anyways, as I said, I think I finished the map. I created the installer, checked it on mapsource and am currently uploading the file to GPSDepot. As of this writing it is 45% done and says it will be done 100% in next 30 minutes.
I hope it helps and Thanks for writing the tutorial. It is pretty thorough and fabulous. Now on to the Oregon map...
Glad its uploading.
Yea, its definitly not multi-processor aware. i try to run multiple of it and hope it'll split between my two processors but last time it did not.
The map has been uploaded. Enjoy people!
Hi,
Just checking out the Washington 24K Topo. I found a few things.
1. Highway 101 is in the water on the west coast by N47 42.772 W124 25.011
2. The strait to the Puget Sound is missing the bodies of water. The Columbia river disappears at Interstate 90. It comes up Tan like the land color.
3. There is funny looking contour errors out in the sea on west coast. I saw the same thing when I was processing Dem files too. It looks like needs crop to the State borders.
4. The Interstate Flags are missing.
Overall, the contours look good. The streams look good until it runs into a body of water, since the body of water and streams are two different colors, the streams show though on the lake as well, but I noticed that on the Idaho and Nevada topo as well. I haven't tried it on the gps yet, just looking at it with Mapsource.
John
Quote from: shutle64 on January 24, 2009, 12:48:59 PM
Overall, the contours look good. The streams look good until it runs into a body of water, since the body of water and streams are two different colors, the streams show though on the lake as well, but I noticed that on the Idaho and Nevada topo as well. I haven't tried it on the gps yet, just looking at it with Mapsource.
That is standard (the stream thing) with the NHD data because of how they make the data.
check out: N47 46 10.5 W122 50 58.9
Shoreline but no water...
I uploaded a mac version of your map and added text/screenshots.
Also, did you include the GNIS POIs?
Quote from: -Oz- on January 24, 2009, 05:43:27 PM
Quote
That is standard (the stream thing) with the NHD data because of how they make the data.
I was just curious if we could change the body of water and the stream to the same color, the stream lines would not show through the body of water, it would blend in the background color of the body of water.
Quote from: shutle64 on January 25, 2009, 06:37:13 PM
Quote from: -Oz- on January 24, 2009, 05:43:27 PM
Quote
That is standard (the stream thing) with the NHD data because of how they make the data.
I was just curious if we could change the body of water and the stream to the same color, the stream lines would not show through the body of water, it would blend in the background color of the body of water.
You'd have to use custom types. Not a bad idea really.
Just in case this helps someone else out, the issues I have been having exporting the NHD flowline data from postgre is actually because of a memory leak in the pgsql2shp binary. I have looked at the source code, ran the process with my debugger attached, etc... and that thing throws a heap corruption for every single row it exports. I have given this info to the postgre and postgis folks and re-compiled postgis from source.
Seriously though, I'm at the point where if anyone has the already processed flowline shp file I'd be willing to find a way to compensate anyone who could upload it to my ftp server. I could give you some free ftp storage or something. I don't know why you all are able to export the data without all of these problems, but the memory leak is there, so either you are crunching less data so you don't run out, or you have more RAM than me or something. I still can't believe how much code I've had to look at so far for this map, lol. The only thing my map needs is the completed flowline data. :) Anyway, however it works out I'll probably take a break for a week, and start on Oregon sometime the first week of February.
krellor; if you have the data uploaded somewhere I'll download the flowline data; process it, and then put it up on the server for you...
Oz, I did put the gnis data in there. It is really weird. When I look at the gnis data in GM, it seems like most of the mid washington is missing data. Eastern and western washington seems to have it. The same gets reflected in the final map. Krellor, I will try and get the Washington NHD processed data uploaded to your server tonight as I still have it.
Currently uploading a single file nhd.zip to the ftp server Krellor provided. Hope this helps.
Krellor, the processed nhd data has been uploaded to your server. I created a separate directory for Washington and placed it there. Hope that helps.
Thanks for the Data upload. Oz was also kind enough to help me out with that, so thanks to both of you. Now I have cGPSmapper churning away, so we will see how it comes out.
outstandin
Hi, I've been using the wa topo 24k built from this thread. I was wondering if there are any plans to make a version that includes the POI data found in like Topo 2008 or Topo national parks 24K. I like the 24K detail but it's missing the mountain peaks, trailheads, picnic areas, etc. Could somebody explain how to recompile the maps with the POI data if I already have a csv to add to each tile? what settings do I use with cgpsmapper and global mapper?/gpsmapedit
I've been using this map in conjunction with northwest trails and really love it.
thanks
As you already have the data in csv format, why not use Garmin's POI loader? You could then add/change any info very easily at any time.
If you wanted the info as a map layer, create one (or more) as an overlay 'mapset'.
The method you are asking asking about would be a lot more work and would involve copywrite issues.
I made an overlay map but it doesn't seem to work right. I try to overlay my map and NW trails and NW trails doesn't show up.. can you only overlay 1 map? I set it to transparent when I compiled it in cgpsmapper
ok. since apparently you are limited to 1 map overlay, I combined NW trails (low res line style) with my POI map and recompiled.. it worked! now I have POI's showing trailheads & summits as well as the nw trails on the 24K topo wa map!
I have never tried a map with two overlay mapsets, and do not remember it being mentioned before. You have enlightened us all. Thanks.
Quote from: spiiiguy on April 08, 2009, 10:11:23 AM
ok. since apparently you are limited to 1 map overlay, I combined NW trails (low res line style) with my POI map and recompiled.. it worked! now I have POI's showing trailheads & summits as well as the nw trails on the 24K topo wa map!
would you mind uploading this combined overlay?
I am in washington, and would love to have this
you could put it here: http://idisk.mac.com/rlreif-Public?view=web
thanks in advance
can't upload it. not free data. all I can do is give a screenshot.
(http://img11.imageshack.us/img11/6407/picnic.jpg)
hopefully there is a WA 2.0 topo 24k eventually that has the POI data, or NW trails with free POI data
Why not use the NW topo map by John at http://www.gpsfiledepot.com/maps/view/134/
It has a large number of POIs. Personally that is what I use. Maybe we should just take out the washington 24k topo and have that page redirect to the NW Topo.
wow guess so. I figured the wa topo was still in development so I was trying to add on to it. I can only fit 8 map tiles of that NW one before I'm out of space! :o
not free data?
where did you get it? garmin topo 2008?
I have that map, i guess I didnt know I could get the POI data out of it and use it elsewhere
yeah, topo 2008
gpsmapedit and global mapper can take a full mapset and pair it down by deleting certain MP_TYPE's. So you can rip out all polygons & polylines
I downloaded that NW map though, it has some good and bad aspects. all the POI's are generic POI type instead of specific icons, other than that the map detail is even higher than the 40ft WA topo map
then I guess the NW topo with NW Trails would be good enough
Quote from: spiiiguy on April 08, 2009, 04:36:15 PM
the map detail is even higher than the 40ft WA topo map
What is detail for one purpose/person can be clutter for another. How do you define higher detail?