map sizes

[deepspace]

Good Day, There have been some posts about this but no real answers. I am about to finish (12 quads to go) my Washington state map. I am ending up with huge file sizes (Mount Olympus quad 1.4 gig) which I have been splitting in half. Last time I ran gpsmapper I had to quarter the mountain regions to get gpsmapper to handle them. This I believe lead to some seam issues (sudden gap in contours in 2 places). Question is: anybody really figured out what max size is for gpsmapper? I ask because last time it would run for hours them come up with an error which Stan from gpsmapper told me was file size (he advised splitting the maps the way I did). He mentioned then that a new version was coming which would no longer be size weird. At this point it will take my whole life to run any way since there will be over 70 mp files to run.
Thanks-Marc

[maps4gps]

Mount Olympus 100k quad?  

What file type is 1.4Gb; shape, mp ?

What program is being used to create such a huge file?

What contour interval are you using, 5 foot or less?

What bit-level are you using?

Not sure what the max file size is for cgpsmapper; however, each doubling of .mp file size takes 2.5 to 3 times as long to process, so you a killing yourself by creating such huge files.  

[deepspace]

100k yes, 20 foot contours, made with topo_process built the the header from this site. The nhd files for the area was 1/2 gig alone. The 1.4 is the mp file.
I made one map already in the same manner tremendous details resulted.
Thanks!

[deepspace]

I just finished the Robison Mountain quad for contours. It alone at 20 foot contours is 870 megs.

[deepspace]

Thanks I'll have to break the bigger mountain areas smaller.
When I did it before I ended up with 2 places in the Olympics that had about 100 yards of no contours, (but the water and roads where intact) which I blame on myself and sloppy splitting, but just incase I have been getting fresh data for the whole project.
 The USGS  server is on the frits right now I haven't been able to get the last 12 quads any way

[Seldom]

The USGS  server is on the frits right now I haven't been able to get the last 12 quads any way

If you are doing this quad by quad, and you are doing areas that aren't likely to be terraformed, you could always try geocomm.com.

[maps4gps]

I just finished the Robison Mountain quad for contours. It alone at 20 foot contours is 870 megs.

I would expect the .mp file for Robinson Mountain with 20 ft contours to be 142Mb.  I have not used TopoProcess as I was using GlobalMapper before Oz 'developed' TopoProcess.  TP must be doing something a lot different than GM.

In contouring with GM, I have not gone above a 15' quad size (50k footprint).  A 30' quad size takes considerable longer than 4 times a 15' quad and GM has 'out of memory errors' (3Gb RAM) in some mountainous areas being overcontoured. 

BTW/FYI - In most mountainous areas the USGS 24k topos use a 40', 50' or even 80' contour interval.  Most NED/DEM gridded elevation data is derived from these contour lines.  Hence using a CI of 20' is simply mathematically creating lines between those which are known to describe the land surface - they look nice, but there is no actual data for their placement and they will clutter the GPSr display unless you plan for them.

Were the missing ocean polygons (and shoreline lines) included since a year ago?  The NHD people did not rate it a high priority when I asked them about it last Spring.

BTW/FYI - I do not know how TopoProcess does its contouring; however, if you are contouring with exact boundaries, the location of where a contour line intersects the common boundary can differ if each of the grids does not extend past the 'trimed' quad.

[-Oz-]

TopoProcess uses gdal so it just takes the tif images and uses gdal's contouring to convert them; then it loads it into postgis and spits back out a shapefile with the proper MP_TYPE.  Something else is taking the shp and turning it into contours.

[maps4gps]

The USGS  server is on the frits right now I haven't been able to get the last 12 quads any way

They have a message saying they were doing maintainence on Thursday.

[deepspace]

Well, I did not like the map built with GM, I have it on my machine. Nor did I like the one available for download here.
The unzipped file alone is 223 megs for Robinson Mountain if you get 142 something got tossed out.  it seems to me at least.
Thanks-

[maps4gps]

Would you mind explaining what you did not like about the map built with GM; if there is a 'better' program for contouring I would be interested in knowing about it.

Which of the two available at gpsfiledepot are you refering to?  Why not like; some detailed comments might help all of us make better mapsets?

The 142Mb was an estimate from the size of the shape files using the default settings in GM.  With the adjustment in  the 'smoothness' setting that I normally make; my filesizes are about 50% larger - so my estimate of 142 * 1.5 = 213 is not that different from your 223.  I was interested it what TopoProcess might use a default setting, as it seems those who use it are getting very large files. 

I wonder how much difference there would be in the final .img file sizes.

[maps4gps]

For comparison did two test contouring the NED (seamless server) 1/3 arc sec (10meter) with GlobalMapper for the Robinson Mtn 100k quad.
1.  8 15 min quads default smoothness
2. 32 7-1/2 quads with my normal smoothness setting ( 1/2 default)
Opened the smaller quads and output a combined 30x60 quad for each.
Opened each combined shape file and output an .mp file (minor contours at 23 bit level, intermediate (1/5) at 21 bit level, and major ( 1/10) at 19 bit level)

Time to load NED, contour, write shape files -  61 minutes     /   61 minutes
Total .shp file size                                      -  83Mb             /   126.6Mb
.mp file size                                              -  132 Mb           /   202 Mb
cgpsmapper data:  line segments                -  39473            /   57619
                            processed                     -  70076            /   82877
                            split                              -  7403              /   11565
                            not imported                  -  106               /   263
                            regions                         -  18 & 2            /   25 & 2
                            time to compile              -  221.90 sec     /   347.52 sec
                            .img file size                  -  8.9 Mb           /   9.8 Mb

What values do you get for this quad?


GM makes .mp files with coordinates given to 7 decimals of a degree; how many are in your file?   

Note: pure contour line files are processed by cgpsmapper much quicker than combined (full topo) data.

[deepspace]

Would you mind explaining what you did not like about the map built with GM; if there is a 'better' program for contouring I would be interested in knowing about it.

both on the map I built (Olympic peninsula only) and the one I downloaded from this site there where errors as to where things where. One logging road was off several hundred feet and a road abandoned in 1987 did not even show up. When I ran the very same data in topo_process things where right. Could very well have been me though.

Time to load NED, contour, write shape files -  61 minutes     /   61 minutes
Total .shp file size                                      -  83Mb             /   126.6Mb
.mp file size                                              -  132 Mb           /   202 Mb
cgpsmapper data:  line segments                -  39473            /   57619
                            processed                     -  70076            /   82877
                            split                              -  7403              /   11565
                            not imported                  -  106               /   263
                            regions                         -  18 & 2            /   25 & 2
                            time to compile              -  221.90 sec     /   347.52 sec
                            .img file size                  -  8.9 Mb           /   9.8 Mb

What values do you get for this quad?

I have not created the .img files yet. I also did not time the process (create shape then .mp) but I do not think it took a hour. If I get this one little detail worked out today I will create the .img file and post the results.
I have one quad left to convert from shape to .mp, (Mount Baker, shape file is 956 megs) I keep getting the virtual memory error. No matter how many times I re-boot and close all applications. If I could just get it open once to split it...... 

[deepspace]

maps4gps, while I am not sure this is relevant here is what I came up with running Adam_e quad, I split into 2 sections (which I think is .5x.5, correct me if I am wrong) I have everything set to compile alphabetically. so its not the Robinson Mount you did.
Command closed before I could copy, but the mp file containing  all data (water, transport, gnis, contours ect.)  run time was approx 3.15 hours, original file .mp size = 230 megs output= 4.5 meg.
On a side note, I can open the quad described above (Mount Baker, shape file is 956 megs) with no problems in both Global mapper, and Mapwell.
I fully admit Global Mapper is far easier, I may have made a error in going the route I did looking at you process times.
Cheers!

[maps4gps]

Remember I just used cgpsmapper on the contour lines.  Including area, points and street lines can greatly increase the compile times. 
What I think I see are just too big spatial areas per file.  If that is all TopoProcess allows, or you need to limit the number of quads/tiles in the mapset, then there is no choice; otherwise I would recommend using smaller sizes.