Generating 24K maps for entire washington -- a little help

[krellor]

I should have remembered that.  In the excel file, when you download it, if in the lower right corner of the dem link cell, hover your mouse over that corner and it will make a cross.  Now, click and drag to all of the lower cells, and it will make the links for you.  This is because those cells are created using a excel formula based on cells in the previous cells.  Dragging that lower right corner down copies that formula down, but is smart enough to make sure it uses the correct cells.

As far as coordinating the data, I can host it.  I'll go ahead and make a folder structure like the tutorial has and put it in a directory.  Go ahead and email me at krellor at shadowsprite dot com and I will send a username and password that has access to the directory along with instructions to connect.  Once we do that we can take it from there.

[deepspace]

I have split the state into 4 quadrants to deal with memory issues 3 of which are finished. The first run does not have contours (essentially a road and water ways map)  the second does. Things should be finished this week. I sold one of my ccd cameras and purchased a copy of global mapper. I am toying with the idea of re-doing it all using global mapper just to be able to dump the made with gprmapper on the lower left corner. What do you think?
I am also wondering does any one know what the default levels are for the Americas base map?
Cheers-Marc

[deepspace]

Oh yes I forgot I can host also
Cheers-Marc

[dhisum_dhisum]

So, here is what I am thinking. I will start downloading and processing the NHD and transportation data for Oregon. Someone else can take up DEM data. After I am done processing the data, I can upload the data to Krellor or deepspace's hosting sites. There is not much point uploading the raw data as that would be a bandwidth killer for not much benefit. Does that sound reasonable?

[krellor]

That sounds good.  I replied to your site message with the info to connect to the FTP directory.

[deepspace]

I myself will finish my Washington projects completely first. I just sent 2 of the maps with installers to some friends for reviews.
What I do not like at this point is when you zoom out at some point you loose any features (as opposed to the base map) and to have any features you need to turn off the custom map.
Cheers-Marc

[dhisum_dhisum]

Thanks Krellor, got your mail. So I will start on the NHD and the transport data for Oregon. And yeah, WOOHOO!! Last night, GM exported all the cgpsmapper files and I started cgpsmapper on converting the mp files to the img files. chug, chug, chug it goes and it has advanced to a large degree, meaning a lot of files are done. I tested one of the IMG files and it works beautifully in map source. At this rate, seems like the WA map will be ready before the week is over

[krellor]

Did you have any trouble exporting the whole state to IMG files?  I didn't have trouble with the 200 for contour, but now with 40 foot I am running out of memory.  I think like deepspace I will be breaking the state into quarters, and then gridding and exporting those.

EDIT:  I didn't mean img files, I meant mp files.

[dhisum_dhisum]

Krellor, earlier, when I was trying to load just all the contours gmp files in GM it would cop out with "Unknow error" = Out of Memory errors. But when I started creating the final catalog to export everything to mp files, it surprisingly did load everything. I kid  you not, my heart was literally in my mouth all the time as the dem files were huge and even the processes nhd files were a few gigs. It did however load everything and did export them properly as well. Right now cgpsmapper has been running since almost midnight last. I divided the task between two parallel cgpsmapper runs and except for 3 files, everything else has been processed. One of the cgpsmappers has been stuck at 18% for its last 800MB mp file for a while now while the other is I think on the last file in its batch as well. So, maybe if gods are happy, I can have this ready by EOD friday?

[-Oz-]

Wow, a lots happened since I last looked.

An 800mb mp file; how large is the area?  My 300mb files took around 8hrs (only could get 1/4 of the processor for some reason).

I would definitely not waste bandwidth uploading the basic data; process it all or that is just silly.

I also fixed the Washington excel file.

[krellor]

Good grief, I feel like this map is a comedy of errors.  So, I finally get everything broken up, cgpsmapper churns out the image files, and mapsettoolkit does it's thing and installs them in mapsource.  I'm felling good, and then I start to notice that some little tributaries here and there are missing on the map.  I noticed because I was checking areas that I frequent and have USGS maps of.  So I go back and load it all up in global mapper and it looks fine.  I check cgpsmapper error logs, and it had a bunch of "Not enough node" errors.  After some searching and testing I think that the problem is how I broke up the flowline data.  That even though it is all displaying correctly in Global Mapper, that the polygons aren't quite right.  So, I've gone back to the flowline data to try and re-process it.  Very aggravating.

My home internet connection has been having some issues, so if I go offline for a few days don't worry.  Hopefully I'll finish with the processing by then.  Luckily I kept all of the raw downloaded data so I don't have to download it again.  It is grustrating having everything right except the flowlines.  Bah.  Good luck on all of your map efforts.

[-Oz-]

once you figure out what caused these issues please let us know so we all can learn from it.  Especially if its the postgresql part that is deleting what you need.

[dhisum_dhisum]

WOOHOO! Finally, after a lot of sweat, blood and a sprinkle of pixie dust, the WA map is finished. I will start uploading it immediately. Hopefully other people can use it. If there is a problem, let me know and I will see if I can do something about it.

[dhisum_dhisum]

Oz,

800MB file was not even the biggest one I had, that one was 1.1GB. That is not the one took most time, the 800MB file did. It took almost 20 hours to process. And yes CGPSMapper was stuck  at 25% CPU for some reason. I attributed it to the fact that I have a Quad Core system and that probably CGPSMapper is not multi-processor aware.

Anyways, as I said, I think I finished the map. I created the installer, checked it on mapsource and am currently uploading the file to GPSDepot. As of this writing it is 45% done and says it will be done 100% in next 30 minutes.

I hope it helps and Thanks for writing the tutorial. It is pretty thorough and fabulous. Now on to the Oregon map...

[-Oz-]

Glad its uploading.

Yea, its definitly not multi-processor aware.  i try to run multiple of it and hope it'll split between my two processors but last time it did not.