I have been compiling the IMG files for a topo map of the 48 States. The Southeast, Southcentral and Southwest are done. The Northwest is running now. The support and help from the site has been great. However I see 2 problems, The first is ruining the IMG files is taking to long, we want the maps now!!. The other problem is this is only ver 1.0 of the map. The map is missing state and local data and needs formatting cleanup.
So what are we going to do about it?
To give it a name we could call it the "48 State Group Project"
To solve the compile time problem I am looking for volunteers to help compile the IMG files. To help you will need a fast computer and you will need to buy the shareware version of cgpsmapper (30$). The shareware version of cgpsmapper and has support for shapefiles and having cgpsmapper read the shapefiles is how my workflow runs. We will need a file dump to upload and download files.
To make ver 2.0 of the map the need to for volunteers to fine the local and state date and send the data to a common place to load in to the map.
The workflow I use is very automated and uses postgrsql to store the data so there is a need for 2 or 3 volunteers with a fast computer and 500 GB of free disk space to setup a copy of the database.
Everybody need to say Thank You to Oz setting up this side and showing what can be done.
Well.....
My computer is not fast, but for MO I have already downloaded all the dem, water, transportation, federal lands, borders, and trail data. I can convert shape files to mp files for further processing. Basically, I can go through the first seven steps of the tutorial, and then upload for final combining and compiling. Would taking the data that far be useful?
Lalita
Well.....
If your computer is at least not slow get the software and we can try. I have all of the data ready to compile. The processing I do is not the same steps as the tutorial. The tutorial showed me what to do but I am lazy and wanted to run the 48 states so I automated the processing. When you are a map nerd and a computer nerd working with 200 GB of data in a database is fun. As my job is GIS Analyst I hope I good at it.
I also work to automate the process way more than the tutorial.
Now, the key to doing this right is also to make sure we get good contour lines (florida in 5ft like i'm working on now) and good local data. For example for florida I found a wealth of information including some ocean markers; bathymetric data, etc so I"m including all of that in the map I'm working on on top of the normal data.
Now, my other issue is I live in one of the other two states :-P
John,
Do I really need to set up the database to run cgpsmapper on the shapefiles? My machine is free in the nights, so I can run the job nightly. It would help if I understood the workflow a little better. Do you have scripts that pull the data out of the database and then run cgpsmapper on them?
I have already done the Northeast (MA, NH, VT, ME) in one single map . I am sure we can come up with a 48 state solution. Just need a lot of CPU power!
Quote from: -Oz- on February 10, 2009, 07:16:16 PM
Now, my other issue is I live in one of the other two states :-P
And I will be in the other next week. OK should is be the 50 State Group Project.
The current run of the what I am calling ver 1.0 is a one size fit all. The need to mach the contours and other local date to the map is the key to do good maps. I think contours need to be mach at less then the state level. 5 foot contours are good for most of Florida but 5 foot contours are to big for south Florida and the Keys.
Sites with data for Florida
http://www.fgdl.org/ over 350 current and historic GIS layers
http://www.floridadisaster.org/gis/data/index.htm
* Geodetic Control
* Cadastral (property boundaries and ownership)
* Orthoimagery
* Elevation Elevation Contours and Depressions and Bathymetry
* Hydrography (rivers, streams, lakes, etc.)
* Administrative units (boundaries of cities, counties, and special purpose districts)
* Transportation (features such as roads, bridges, railroads, ports, and airports)
dhisum_dhisum
For just getting ver 1.0 of the map done the database is not needed. I would zipup all files needed to run the maps. Yes I have scripts that pull the data out of the database and then run cgpsmapper. So I setup the scripts that pull the data and then zip it up ready to up load. The person would need to downeloade, unzip and start a bat file to run the maps. And you will need to buy the shareware version of cgpsmapper (30$). If there was a shapefile to MP converter the free version of cgpsmapper would do.
savage
I think a 16 computer Linux cluster in the back room would be good.
The entire process for the New England topo map was not too bad except for cgpsmapper. It took 2 weeks of a reasonably good computer running 100% of the time just to do cgpsmapper.
If I had access to a Linux cluster (16 boxes) maybe these 2 weeks could have been 1 or 2 days but that still leaves us with more than 1 month of the Linux cluster to do the entire map.
Is there a way around cgpsmapper. Maybe we should start an Open Source project to develop a faster version of cgpsmapper.
Of 3524 img's to make 1764 are done after 36 days. So at the current rate of 49 img's a day another 36 days to be done. So with a total of 72 days of work a 16 computer cluster would take 4.5 days. May be a 8 computer cluster would work. My computer is 1.5 years old with a AMD 64X2 5600+ CPU.
I do not know how you get 49 img/day. When I did my maps, a single img could take almost a day of cpu. My img are 0.5x1.
Is there any tricks to make cgpsmapper run faster, beside upgrading my computers?
My img are 0.5x0.5. What is the CPU on your computers? Contour lines add to the time in a big way. In a test run of all data but contour lines the 48 states ran in 4.5 days, with 20 foot contour lines 72 days.
I am using a 2.2 GHz, with 1 GB of RAM. This is a computer I bought from my former company. We used it to run computer chip simulation (those are *huge* programs). This is why when I see cgpsmapper taking 24 hours of CPU to process a single img, I think there must be a gross inefficiency somewhere.
I am certainly due for an upgrade!
The new computers are faster. My 1.5 year old 1000$ computer will databases about 2 times faster then the 4 year old 5000$ when new computer I use at work to run ESRI's ArcGIS.
Can you test making the img's 0.5x0.5 vs. 0.5x1. I think cgpsmapper dose not scale well to larger img. I see img's at 2 mb taking 20 minutes and img's at 5 mb taking 120 minutes. Six times the time for 2.5 times the size.
Yes, I will try 0.5x0.5 in my next run. I agree there is something that grows exponentially in cgpsmapper.
How does anyone suggest varying the size of the contours across the US? Doing 20ft everywhere won't work near the ocean and won't work in the rocky mountains...
I wanted to jump back in here, not to steer the conversation away from contours, to say that my poor old computer does not have near 500G free disk space, more like 40G, but I will still be happy to share any of MO data. I can burn it to DVDs and forward it on.
Now back to our regularly scheduled contour interval program......
Lalita
I agree we need to vary the contours within the US and even within a state. The coastal area of Maine is pretty flat but when you move West and North West it becomes steeper.
I'd assume we would vary by 100k quad since its easy to do but I'm not sure how to figure out what to do; a database could store it easily enough and process accordingly with fwtools since its easy to change the contour interval on the command line; but I have no automated way to decide the intervals.
Varing the CI by 100k is what I did in the transparent contour mapset. My previous post(s) on the CI used on the 24Ks was to use that info to help determine an appropriate CI.
I have lots of trail data for lots of states. For several states I gathered all I could find online. For IL and MO I have lots and lots that I have GPSed. You are welcome to use any of my data-I can send mp files.
First, John, you are My HERO and I do not say this lightly. Even from the size of the map, one can see that this is almost twice the size of the previously published chunks (SE, SC or even SW USA) which shows how difficult it was to create this entire section.
Now something interesting that I noted in the NW Topo maps. In the top corner, it does not show the ocean in blue color, the ocean is shown the same color as land although shorelines and contours are just fine. Additionally, the columbia river at the intersection of I-90 vanishes at the dam although the contours are there. It then reappears at the other dam. I guess there is not much water in the area between two dams? Addtionally, the puget sound ocean to the left of seattle again does not show in blue color and instead shows up the color of land. The issue is interesting because these exact problems are present in even the WA topo map that I created. Makes me think that either USGS data itself has problems there or the tools we are using screw up on these areas.
Either ways, Thanks a lot!!
or you don't have an ocean file?
For florida I found that the NHD area files contained some ocean but I had to draw large swaths of ocean by hand to cover to the edge of my 100K area and include my depth data.
Next time there will be a ocean file. The water data is NHD and only NHD.
dhisum_dhisum The Columbia river is not a river to vanish. Something is off with the data need to look at that.
Oz, is there a step by step guide on how I can manually create the oceans file. Or better yet, is there a source from which I can get the ocean file? John, how do I go about getting the data from you so I can additionally help out? I have now enough space and computing power and the right cgpsmapper version to do the processing?
Which state do you want ocean for?
To begin with I just want washington state as I have data for it. I would appreciate if I could also know how to get/create the data myself.
I know I am new, so i can not help at all (mainly because I am still new to all the terminology - but I am studying). I just want to cheer you all on. This is an amazing project you all are on, and I encourage oyu all the way. When I have read up enough and feel there is anyway to contribute, I would love to help with what I can. Good Luck!
I noticed that in N38 15 11.8 W109 51 47.8 area of Utah that the contour lines at White crack are off anywhere from three to nine feet. To me that is pretty darn accurate, but in case you wish perfection, I thought I'd point those out to be adjusted. This project is great. I hope I can learn fast enough to help out.