cGPSMapper Taking a long time

[savage]

My Maine map is still being compiled. I have been stuck in the cgpsmapper for more than 9 hours at the 1% mark. I went to 30% in less than 15 minutes!

Should we contact the developer of cgpsmapper? I will be happy to give them my MP file.

[savage]

True, but I am not going to pay for a software that is extremely slow to the point of being unusable.

[-Oz-]

i got a reply once but i'd bet that is the way it is.

i should contact the makers of mapwel and see if they can give me a deal to let me try it out.  cgpsmapper is just so expensive.

[maps4gps]

   In my rememberance and understanding - The version of cgpsmapper I first used in Jan 2007 'processed' at a more uniform rate.  The version of mid-2007 has a lag in the 'processing' at the start but completes the total processing faster.  I am guessing that it is reading the entire file and setting up indexes which speed up later processing at the higher 'zoom' bit levels. 

   What I have not been able to understand is the recent number of people trying to use cgpsmapper to process monster sized .mp files.  Could someone please explain the reason (besides less files to cover an area) for using these large sized .mp files? 

   Has mapwel actually decifered the Garmin coding, or does it just invoke cgpsmapper to produce the .img files?

[erik.the.awful]

I have heard form a guy producing maps with mapwel using scanned raster images of actual 7.5min quad maps and he reported it worked really well and could produce a quad in about an hour. The major issue he had was that it worked too well and was difficult to render on the gps units requiring 20-30 sec to draw. The other issue is it is tied to a specific GPS unit and I think even their offering of unlimited gps program (which naturally costs more) allows you to upload to any GPS but still requires the individual units to be logged in with the program....

[erik.the.awful]

   In my rememberance and understanding - The version of cgpsmapper I first used in Jan 2007 'processed' at a more uniform rate.  The version of mid-2007 has a lag in the 'processing' at the start but completes the total processing faster.  I am guessing that it is reading the entire file and setting up indexes which speed up later processing at the higher 'zoom' bit levels. 

   What I have not been able to understand is the recent number of people trying to use cgpsmapper to process monster sized .mp files.  Could someone please explain the reason (besides less files to cover an area) for using these large sized .mp files? 

   Has mapwel actually decifered the Garmin coding, or does it just invoke cgpsmapper to produce the .img files?

I'm actually not sure why I have monster .mp files. I really think that the topoprocess beta program created larger files for the DEM than was needed. The other issue I'm having with it is when I do use it to convert to img that it overwrites the output file making every file 0000000.img, so I only process one at a time in a folder then rename it and transfer into my completed directory. My monster files are only covering one 30x60 quad (1x.5 degrees) and are compressing down to the right size range 10-12mb each, perhaps when it is using POSTgis to produce the contours it is doubling up on some data. I always tell gpsMapedit to remove duplicate objects before I save my final polish format. It has to be related that cgpsmapper takes so long to run but reaches the right finished size and that topoprocess is exporting extra large contour data.
-Erik   

[maps4gps]

I have not used topoprocess as I have GM and my own procedures for processing, so maybe I suggest will not work.

I run cgpsmapper from a .bat file in a command window - the output .img file has the same name as the input .mp file.  If you have a final .mp file (with everything in one file, contour, hydro, etc),
try running cgpsmapper on it in a command window.

I remember OZ? mentioning adding .mp files together.  cgpsmapper processes points, then lines, then polygons for each bit level.  If an .mp file has these mixed, perhaps cgpsmapper has to spend a lot of time rearranging the data before/during the processing.  I observe that 'on average' a file twice the size takes 2.5/3/3+ the time to process.

Again, please, someone tell me why a 10-12Mb .img is the right size?    I have not heard anything about an optimum size for an .img file, however someone recently posted about Garmin files being in the 400-500K ? size (perhaps because earlier GPSr only had 24Mb internal memory).  I would expect large files to slow performance of the GPSr.  If Garmin only supports a 2Gb gmapsupp.img file with slightly over 2000 component files, about 1Mb per component .img file, why 10Mb file?
I just made some 30x60's using the new Census data which were this size for metro areas - I may test mixing quad/tile sizes to even out the variation in .img file sizes.  I would appreciate it if someone could tell my why large .img sizes are desirable (if they are?)

Give me some specifics and I will create and process one of your areas and let you know what I get.    30x60 area is?   10m NED?  Contour interval?  Other data included as NHD hydro or Census hydro, Census roads,etc, GNIS, etc.?

[maps4gps]

I stand with foot in mouth - sort of.  I was going to process a 30x60 along the NC-TN border, but seem to have deleted the modified processing routines I made just before the holidays.  However, in looking at the detailed level files I made last summer and stored on an external hard drive, 250-300Mb of .mp files in that area is about right.  I used 12x12 minute 'quads' because some test I ran showed that for a 1x1 degree area, 25 12x12 .mp 'compiled' to .img files in about 40% of the time needed for 16 15x15 files.  A 1x2 degree quad in that area took 4 hrs for the 50 12x12' files to compile - about 1 hr per 30x60 area.   Perhaps 2.5 ?? hrs total for 16 15x15' sized files.  If you can specify output file size in topoprocess, run a test at 15x15 or 15x30 and tell us the results.

Again I will ask for info on why these large file sizes are of such interest??

BTW - an hour in mapwel for per 7&1/2 is 32+- hrs for the 32 7&1/2's in a 30x60 quad.  I doubt if any of the features in the pseudo vector output had attribute tags without a lot of extra manual work.

[-Oz-]

The reason I go for <8mb of file size is because some GPS units have 8mb of memory. The reason I go for 1degree x 1/2 is that it seems to make since from a 100k topo perspective.  The reason I dont want a bunch of 600kb tiles is then I hit the segment limit (2025) before I have used up all my memory.  And since I load lots of different maps on and view/hide what i need at each time the segment limit concerns me more than the size.

I believe the resolution on topoprocess (and thus fwtools) is a bit higher than what I was originally using in global mapper thus the super large .mp files.  They compress nicely just take a long long time.

I would get the "super" mapwel license if it did everything cgpsmapper did for less than $100.

Erik: open one of your .mp files in notepad++ and see if you have an id in the header at the top; if not that is the issue, if its blank then it would use the id 000000 and thus they would overwrite each other.  global mapper loads an id in for you when it exports.

[maps4gps]

If maps in the GPS world are like paper maps, your area of interest is usually in the corner of 4 quads, therefore <2Mb file sizes would be more appropriate for GPSr units with 8Mb of memory.  Yes/No  ?  ?? 

I will have to take some time to run a test on the performance of a 10Mb 30x60 file versus 8 15x15 for the same area.

[-Oz-]

I guess you would probably want more than one section loaded but I normally don't want to un-automate the system so unless there's something weird (Hawaii) I just leave it 60x30.  Most times that makes img sizes between 1-3mb which works great only some really detailed quads get up to 6-8mb.

[maps4gps]

I have not included Hawaii nor Alaska yet because of: 1. the nice map you have with contours of Hawaii, 2. the number of 'slivers' of land in so many 30x60's, 3. the NED data at 2 arc sec (60m) in Alaska and how this affects the appropriate contour interval and what, if any, consideration should  be given to the 2:1 at 60N to 3:1 for northern Alaska that convergence of the meridians has on the grid spacing.

I have been working with standard areas too.  A number of the 30x60 quads in the east with large metro areas have been in the 8-11Mb - for just the planimetric data using the hydro in Census (which on-average is about 60% of what is in the NHD).  Then add contours at '24K scale'.

At the other end is if anything special should be done with quads that have little data because much of their area is ocean/Great Lakes and/or non-US land area.  In the 12x12' test quads I did last summer in areas like that, some of the .img files were larger than the source .mp files.

Anyone try using different sized 'quads' to narrow the range of .img file sizes?

[-Oz-]

Anyone try using different sized 'quads' to narrow the range of .img file sizes?

The only time i did it is when the quad was too big.  Are you thinking of some sort of automation?

And good to know about Alaska, I live here now so I'm gonna at least make a decent sized map of the area I live/hike around.  The state is just too large.

[maps4gps]

A program which would check a master file containing info on area extent, file names, etc. then build the GM script file to process the data files.  Somewhat of an extension of the program I created in late Dec and then deleted to process NED files at different contour intervals and area sizes.  It was becoming too involved to keep up with the new NED data being released by USGS every two months.  File names might be a problem as I now include the corner lon-lat in the name.  Also what might be an appropriate division for the planimetric data (roads etc) probably would not be for the contours.  The biggest issues I see would be the very large files in metro areas and the very small files were only a small portion of the quad area was US land.

I was wondering when you would get to Alaska.  NHD can be very large in areas with huge numbers of waterbodies(NW).  I think AK was 2 or 2&1/2 Gb out of 20Gb for the entire NHD.
AK large? - Only slightly over 3 times the size of TX; and most of it you need an airplane to get to.