48 State Group Project

[John_M]

I have been compiling the IMG files for a topo map of the 48 States. The Southeast, Southcentral and Southwest are done. The Northwest is running now. The support and help from the site has been great. However I see 2 problems, The first is ruining the IMG files is taking to long, we want the maps now!!. The other problem is this is only ver 1.0 of the map. The map is missing state and local data and needs formatting cleanup.

So what are we going to do about it?

To give it a name we could call it the "48 State Group Project"

To solve the compile time problem I am looking for volunteers to help compile the IMG files. To help you will need a fast computer and you will need to buy the shareware version of cgpsmapper (30$). The shareware version of cgpsmapper and has support for shapefiles and having cgpsmapper read the shapefiles is how my workflow runs. We will need a file dump to upload and download files.     

To make ver 2.0 of the map the need to for volunteers to fine the local and state date and send the data to a common place to load in to the map.

The workflow I use is very automated and uses postgrsql to store the data so there is a need for 2 or 3 volunteers with a fast computer and 500 GB of free disk space to setup a copy of the database.

Everybody need to say Thank You to Oz setting up this side and showing what can be done.

[n2stitch]

Well.....

My computer is not fast, but for MO I have already downloaded all the dem, water, transportation, federal lands, borders, and trail data.  I can convert shape files to mp files for further processing.  Basically, I can go through the first seven steps of the tutorial, and then upload for final combining and compiling.  Would taking the data that far be useful?

Lalita

[John_M]

Well.....

If your computer is at least not slow get the software and we can try. I have all of the data ready to compile.  The processing I do is not the same steps as the tutorial. The tutorial showed me what to do but I am lazy and wanted to run the 48 states so I automated the processing.  When you are a map nerd and a computer nerd working  with 200 GB of data in a database is fun. As my job is GIS Analyst I hope I good at it.

[-Oz-]

I also work to automate the process way more than the tutorial.

Now, the key to doing this right is also to make sure we get good contour lines (florida in 5ft like i'm working on now) and good local data.  For example for florida I found a wealth of information including some ocean markers; bathymetric data, etc so I"m including all of that in the map I'm working on on top of the normal data.

Now, my other issue is I live in one of the other two states :-P

[dhisum_dhisum]

John,

Do I really need to set up the database to run cgpsmapper on the shapefiles? My machine is free in the nights, so I can run the job nightly. It would help if I understood the workflow a little better. Do you have scripts that pull the data out of the database and then run cgpsmapper on them?

[savage]

I have already done the Northeast (MA, NH, VT, ME) in one single map . I am sure we can come up with a 48 state solution. Just need a lot of CPU power!

[John_M]

Now, my other issue is I live in one of the other two states :-P

And I will be in the other next week. OK should is be the 50 State Group Project.

The current run of the what I am calling ver 1.0 is a one size fit all.  The need to mach the contours and other local date to the map is the key to do good maps. I think contours  need to be mach at less then the state level. 5 foot contours are good for most of Florida but 5 foot contours are to big for south Florida and the Keys.

Sites with data for Florida
http://www.fgdl.org/      over 350 current and historic GIS layers

http://www.floridadisaster.org/gis/data/index.htm
    * Geodetic Control
    * Cadastral (property boundaries and ownership)
    * Orthoimagery
    * Elevation  Elevation Contours and Depressions and Bathymetry
    * Hydrography (rivers, streams, lakes, etc.)
    * Administrative units (boundaries of cities, counties, and special purpose districts)
    * Transportation (features such as roads, bridges, railroads, ports, and airports)


dhisum_dhisum

For just getting  ver 1.0 of the map done the database is not needed. I would zipup all files needed to run the maps.  Yes  I have scripts that pull the data out of the database and then run cgpsmapper. So I setup the scripts that pull the data and then zip it up ready to up load. The person would need to downeloade, unzip and start a bat file to run the maps. And you will need to buy the shareware version of cgpsmapper (30$).  If there was a shapefile to MP converter  the free version of cgpsmapper would do.

savage

I think a 16 computer Linux cluster in the back room would be good.

[savage]

The entire process for the New England topo map was not too bad except for cgpsmapper. It took 2 weeks of a reasonably good computer running 100% of the time just to do cgpsmapper.

If I had access to a Linux cluster (16 boxes) maybe these 2 weeks could have been 1 or 2 days but that still leaves us with more than 1 month of the Linux cluster to do the entire map.

Is there a way around cgpsmapper. Maybe we should start an Open Source project to develop a faster version of cgpsmapper.

[John_M]

Of 3524 img's to make 1764 are done after 36 days. So at the current rate of 49 img's a day another 36 days to be done. So with a total of 72 days of work a 16 computer cluster would take 4.5 days. May be a 8 computer cluster would work. My computer is  1.5 years old with a AMD 64X2 5600+ CPU.

[savage]

I do not know how you get 49 img/day. When I did my maps, a single img could take almost a day of cpu. My img are 0.5x1.
Is there any tricks to make cgpsmapper run faster, beside upgrading my computers?

[John_M]

My img are 0.5x0.5.  What is the CPU on your computers?  Contour lines add to the time in a big way. In a test run of all data but contour lines the 48 states ran in 4.5 days, with 20 foot contour lines 72 days.

[savage]

I am using a 2.2 GHz, with 1 GB of RAM. This is a computer I bought from my former company. We used it to run computer chip simulation (those are *huge* programs). This is why when I see cgpsmapper taking 24 hours of CPU to process a single img, I think there must be a gross inefficiency somewhere.

I am certainly due for an upgrade!

[John_M]

The new computers are faster. My 1.5 year old 1000$ computer will databases about 2 times faster then the  4 year old 5000$ when new computer I use at work to run ESRI's ArcGIS.

Can you test making the img's  0.5x0.5 vs. 0.5x1. I think cgpsmapper dose not scale well to larger img. I see img's at 2 mb taking 20 minutes and  img's at 5 mb taking 120 minutes. Six times the time for 2.5 times the size.   

[savage]

Yes, I will try 0.5x0.5 in my next run. I agree there is something that grows exponentially in cgpsmapper.

[-Oz-]

How does anyone suggest varying the size of the contours across the US?  Doing 20ft everywhere won't work near the ocean and won't work in the rocky mountains...