I'm running CGPmapper to convert my first .mp to an .img file to test it out, it is the 30x60 min section around my house, Commerce GA quad to be exact. The mp file was around 303 mb I set it to run last night and has been running for the last 14 hours.... Is this normal?
....Says it has 160433 elements to process....
One other small thing I embeded the dictionary file in the borders before I trimmed it to fit the small quad and I had to change the zoom settings on the header to 0=0 1=1 2=2 etc b/c cgpsmapper wouldn't start converting saying that the zooms had to be ascending. They are ascending now 1-5 and everything looked good in gpsmapedit. Any thoughts on this would be appreciated.
-Erik
I do not know the limit for an .mp file size, but most of mine have been well under 30Mb. With that size you must have contoured in 10 or even 5 foot intervals. I have had over 200,000 elements. The compile/convert time is related to the types of elements and how complex the area is. It also seems Vista sometimes cleans house during the processing. A number of smaller files takes much less time to process than one very large file. The gmapsupp.img file in limited to 2048? or 2052? files and the Garmin units only support a 2Gb memory card, so there does not seem to be much reason to create super large files. The larger files might also slow the drawing speed on the GPSr.
How did you get the files down that small? ???
I am using the topo process Beta program and following the tutorial directions as they apply to that program as best I can. I processed the files with 20 foot contours and added the transportation, POI, borders, NHD data, land use and some hiking trails. The damn NHD file was over 1.7GB for the 40 30x60 quads I processed. The DEM data produced .mp files ranging from 150-660 mb per quad. The run I am doing now was only for one 30x60 quad so I could see how it works thats only .5X1 degree. It should be a much smaller file I think but with all the layers saved as one .mp file by GPSmapedit that quad is 300mb. It has been running like 16 hours now and is doing SOMETHING but I know not what...
Please tell me how to pare these files down some, I can't fit all 40 DEMs plus the NHD data in GPSmapedit I followed the tutorial directions to a T for downloading but clearly I have larger than needed files.....
-Erik
BTW I am using XP if it matters
On average cGPSMapper produces 1 .img per hour on my machine using the same parameters suggested in the tutorial. But I have also seen .img taking more than 5 hours!
Is the 300mb file size for a finished polish file quad okay? It looks great in GPSmapedit all the data lines up with where it is supposed to be. How small will that be in .img format?
Perhaps I have said too much already since I use GlobalMapper and not Oz's topo process Beta nor tutorial directions (developed my methods for GM before Oz published his methods.
I have GM making .mp files of smaller areas, therefore the .mp file size is smaller. Not sure how you could get 300mb with 20 ft contours for your area. These file sizes are way larger than anything I have produced, so Oz or someone using his methods could advise you better.
Polish size to .img size depends on a lot of factors; without contours: I have seen only 30% (10Mb > 3+Mb) for metro areas (all those 2 point roads with attributes) to less than 4% for areas with few roads and hydro (1Mb > 40Kb). Contours are usually around 5-8%. Also depends on the bit-levels you are using and how many features are defined on how many levels. Time to process depends on your computer speed - and I think becomes much slower if cgpsmapper needs to use hard disk space for temp files rather than RAM.
You'll have to wait for it to finish the entire 300mb to know how big it'll be. Sometimes I have 80mb files that end up at 3mb img files and sometimes i have 70mb files that end up at 6mb files; depends on the number of elements I believe.
My NHD file for nevada is around 2gb in total so that is pretty normal.
And I would say for a file that large it is very normal to run for 14hrs. Expect that many elements to take somet ime.
What I will say though is I have never had a file over 150mb.
Which method have you used to build the other files; mainly, which file is huge? I'm assuming the contours data; did you use the topo process beta; gpsmapedit; or global mapper?
Well I just split off a 7.5 min section without changing anything and it ran in about 15min. I guess I will let it run at least 48 hours to see if it finishes the quad.
Most of my contours are from 150 to 600 mb each, depending on the relief of the area, mainly the NC/TN border ones are large. It should be noted that the actual printed 7.5 min series is in 20ft contours for this area so I selected it to build 20' contours
My methdology was as follows
Downloaded data, used Topoprocess to build DEM data (40 quads, all seperate) and converted to.mp, used topoprocess to build NHD data in 4 sets. Combined NHD data after trimming to fit my area, combined trimmed NHD data into one .mp with Gpsmapedit. Used topoprocess to retrieve other datas. Combined and trimmed with gpsmapedit. Added trail data by converting gpx trails to polylines and labeling as 0x16 in gpsmapedit. Combined all data except DEM and NHD into one trimmed .mp file. Loaded large NHD file, trimmed out a section to fit the commerce 60x30 quad, then loaded DEM for commerce then the other combined data. Trimmed to fit that quad. Saved as commerce.mp (and fixed the zoom levels to 0=0, etc). Ran topoprocess to convert the commerce.mp to a img file. Still running...
I had to use this method to get around the small load limit of gpsmapedit. For some reason it can load larger polish files than shapefiles and strangly enough sometimes restarting the program seems to free up more "room" to load up files. Weird, I think it has a memory leak....
I also need more RAM as I only have 1gb of DDR2.
Thanks for all the help, any other suggestions would be appreciated.
-Erik
Are you sure that the 300mb only covers 1 degree X 1/2 degree? Like during this loading unloading process did it grow larger than that?
Another option you have is next time to grid to the same area always then use dos to combine the .mp files (since they are just text); just have one have a header and the rest don't and it should work.
I started doing the above that way I could keep the contours seperate from everything else since they caused me the most memory/crash issues.
Your steps don't sound bad though. That file is just very very large; i've been processing Arizona for almost a day straight now. Since cgpsmapper doesn't take advantage of dual core cpu I just run it twice so that both CPUs are in use; something to know if you have a dual core.
I have an idea (probably a bad one but we are amongst friends here) why cgpsmapper is so slow.
This morning I looked at my computer and cgpsmapper was running but not making any progress for more than 8 hours. I wanted to kill it. I went to the DOS CMD window to check it out and suddenly the % of completion that was stuck at 2 is now incrementing nicely, currently at 10%.
Is it possible something could be blocking the DOS output? Maybe when I checked it out last night before going to bed?
If you're using vista it is definitely possible that something is blocking it.
On the other hand; I've noticed that its really slow in the beginning and then just cruises after a certain point (a varying point mind you).
;D ;D ;D ;D ;D ;D ;D ;D ;D ;D
It finally got the quad done, it took 27 hours :o but it compressed that 300mb behemoth down to a 10.2 mb file. Now processing the next quad. One down 39 to go....
The map looks great!!!!
Thanks Oz and Savage for all the help Can't wait to finish the set so I can upload it all.
-Erik
Can't wait to see it.
Quite laughable but I now have multiple 300+mb segments for Nevada. Hopefully it won't take quite as long.
Quote from: erik.the.awful on January 03, 2009, 01:05:53 PM
;D ;D ;D ;D ;D ;D ;D ;D ;D ;D
It finally got the quad done, it took 27 hours :o but it compressed that 300mb behemoth down to a 10.2 mb file. Now processing the next quad. One down 39 to go....
The map looks great!!!!
Thanks Oz and Savage for all the help Can't wait to finish the set so I can upload it all.
-Erik
Good job and I have Maine still running. Hopefully it should be ready this week-end unless something unexpected happens.
My Maine map is still being compiled. I have been stuck in the cgpsmapper for more than 9 hours at the 1% mark. I went to 30% in less than 15 minutes!
Should we contact the developer of cgpsmapper? I will be happy to give them my MP file.
True, but I am not going to pay for a software that is extremely slow to the point of being unusable.
i got a reply once but i'd bet that is the way it is.
i should contact the makers of mapwel and see if they can give me a deal to let me try it out. cgpsmapper is just so expensive.
In my rememberance and understanding - The version of cgpsmapper I first used in Jan 2007 'processed' at a more uniform rate. The version of mid-2007 has a lag in the 'processing' at the start but completes the total processing faster. I am guessing that it is reading the entire file and setting up indexes which speed up later processing at the higher 'zoom' bit levels.
What I have not been able to understand is the recent number of people trying to use cgpsmapper to process monster sized .mp files. Could someone please explain the reason (besides less files to cover an area) for using these large sized .mp files?
Has mapwel actually decifered the Garmin coding, or does it just invoke cgpsmapper to produce the .img files?
I have heard form a guy producing maps with mapwel using scanned raster images of actual 7.5min quad maps and he reported it worked really well and could produce a quad in about an hour. The major issue he had was that it worked too well and was difficult to render on the gps units requiring 20-30 sec to draw. The other issue is it is tied to a specific GPS unit and I think even their offering of unlimited gps program (which naturally costs more) allows you to upload to any GPS but still requires the individual units to be logged in with the program....
Quote from: maps4gps on January 05, 2009, 07:07:03 AM
In my rememberance and understanding - The version of cgpsmapper I first used in Jan 2007 'processed' at a more uniform rate. The version of mid-2007 has a lag in the 'processing' at the start but completes the total processing faster. I am guessing that it is reading the entire file and setting up indexes which speed up later processing at the higher 'zoom' bit levels.
What I have not been able to understand is the recent number of people trying to use cgpsmapper to process monster sized .mp files. Could someone please explain the reason (besides less files to cover an area) for using these large sized .mp files?
Has mapwel actually decifered the Garmin coding, or does it just invoke cgpsmapper to produce the .img files?
I'm actually not sure why I have monster .mp files. I really think that the topoprocess beta program created larger files for the DEM than was needed. The other issue I'm having with it is when I do use it to convert to img that it overwrites the output file making every file 0000000.img, so I only process one at a time in a folder then rename it and transfer into my completed directory. My monster files are only covering one 30x60 quad (1x.5 degrees) and are compressing down to the right size range 10-12mb each, perhaps when it is using POSTgis to produce the contours it is doubling up on some data. I always tell gpsMapedit to remove duplicate objects before I save my final polish format. It has to be related that cgpsmapper takes so long to run but reaches the right finished size and that topoprocess is exporting extra large contour data.
-Erik
I have not used topoprocess as I have GM and my own procedures for processing, so maybe I suggest will not work.
I run cgpsmapper from a .bat file in a command window - the output .img file has the same name as the input .mp file. If you have a final .mp file (with everything in one file, contour, hydro, etc),
try running cgpsmapper on it in a command window.
I remember OZ? mentioning adding .mp files together. cgpsmapper processes points, then lines, then polygons for each bit level. If an .mp file has these mixed, perhaps cgpsmapper has to spend a lot of time rearranging the data before/during the processing. I observe that 'on average' a file twice the size takes 2.5/3/3+ the time to process.
Again, please, someone tell me why a 10-12Mb .img is the right size? I have not heard anything about an optimum size for an .img file, however someone recently posted about Garmin files being in the 400-500K ? size (perhaps because earlier GPSr only had 24Mb internal memory). I would expect large files to slow performance of the GPSr. If Garmin only supports a 2Gb gmapsupp.img file with slightly over 2000 component files, about 1Mb per component .img file, why 10Mb file?
I just made some 30x60's using the new Census data which were this size for metro areas - I may test mixing quad/tile sizes to even out the variation in .img file sizes. I would appreciate it if someone could tell my why large .img sizes are desirable (if they are?)
Give me some specifics and I will create and process one of your areas and let you know what I get. 30x60 area is? 10m NED? Contour interval? Other data included as NHD hydro or Census hydro, Census roads,etc, GNIS, etc.?
I stand with foot in mouth - sort of. I was going to process a 30x60 along the NC-TN border, but seem to have deleted the modified processing routines I made just before the holidays. However, in looking at the detailed level files I made last summer and stored on an external hard drive, 250-300Mb of .mp files in that area is about right. I used 12x12 minute 'quads' because some test I ran showed that for a 1x1 degree area, 25 12x12 .mp 'compiled' to .img files in about 40% of the time needed for 16 15x15 files. A 1x2 degree quad in that area took 4 hrs for the 50 12x12' files to compile - about 1 hr per 30x60 area. Perhaps 2.5 ?? hrs total for 16 15x15' sized files. If you can specify output file size in topoprocess, run a test at 15x15 or 15x30 and tell us the results.
Again I will ask for info on why these large file sizes are of such interest??
BTW - an hour in mapwel for per 7&1/2 is 32+- hrs for the 32 7&1/2's in a 30x60 quad. I doubt if any of the features in the pseudo vector output had attribute tags without a lot of extra manual work.
The reason I go for <8mb of file size is because some GPS units have 8mb of memory. The reason I go for 1degree x 1/2 is that it seems to make since from a 100k topo perspective. The reason I dont want a bunch of 600kb tiles is then I hit the segment limit (2025) before I have used up all my memory. And since I load lots of different maps on and view/hide what i need at each time the segment limit concerns me more than the size.
I believe the resolution on topoprocess (and thus fwtools) is a bit higher than what I was originally using in global mapper thus the super large .mp files. They compress nicely just take a long long time.
I would get the "super" mapwel license if it did everything cgpsmapper did for less than $100.
Erik: open one of your .mp files in notepad++ and see if you have an id in the header at the top; if not that is the issue, if its blank then it would use the id 000000 and thus they would overwrite each other. global mapper loads an id in for you when it exports.
If maps in the GPS world are like paper maps, your area of interest is usually in the corner of 4 quads, therefore <2Mb file sizes would be more appropriate for GPSr units with 8Mb of memory. Yes/No ? ??
I will have to take some time to run a test on the performance of a 10Mb 30x60 file versus 8 15x15 for the same area.
I guess you would probably want more than one section loaded but I normally don't want to un-automate the system so unless there's something weird (Hawaii) I just leave it 60x30. Most times that makes img sizes between 1-3mb which works great only some really detailed quads get up to 6-8mb.
I have not included Hawaii nor Alaska yet because of: 1. the nice map you have with contours of Hawaii, 2. the number of 'slivers' of land in so many 30x60's, 3. the NED data at 2 arc sec (60m) in Alaska and how this affects the appropriate contour interval and what, if any, consideration should be given to the 2:1 at 60N to 3:1 for northern Alaska that convergence of the meridians has on the grid spacing.
I have been working with standard areas too. A number of the 30x60 quads in the east with large metro areas have been in the 8-11Mb - for just the planimetric data using the hydro in Census (which on-average is about 60% of what is in the NHD). Then add contours at '24K scale'.
At the other end is if anything special should be done with quads that have little data because much of their area is ocean/Great Lakes and/or non-US land area. In the 12x12' test quads I did last summer in areas like that, some of the .img files were larger than the source .mp files.
Anyone try using different sized 'quads' to narrow the range of .img file sizes?
Quote from: maps4gps on January 06, 2009, 09:41:07 AMAnyone try using different sized 'quads' to narrow the range of .img file sizes?
The only time i did it is when the quad was too big. Are you thinking of some sort of automation?
And good to know about Alaska, I live here now so I'm gonna at least make a decent sized map of the area I live/hike around. The state is just too large.
A program which would check a master file containing info on area extent, file names, etc. then build the GM script file to process the data files. Somewhat of an extension of the program I created in late Dec and then deleted to process NED files at different contour intervals and area sizes. It was becoming too involved to keep up with the new NED data being released by USGS every two months. File names might be a problem as I now include the corner lon-lat in the name. Also what might be an appropriate division for the planimetric data (roads etc) probably would not be for the contours. The biggest issues I see would be the very large files in metro areas and the very small files were only a small portion of the quad area was US land.
I was wondering when you would get to Alaska. NHD can be very large in areas with huge numbers of waterbodies(NW). I think AK was 2 or 2&1/2 Gb out of 20Gb for the entire NHD.
AK large? - Only slightly over 3 times the size of TX; and most of it you need an airplane to get to.