I posted elsewhere about using Filemaker Pro to process New Jersey landcover data for a project I'm working on. I want to expand this map to neighboring states, but there just isn't any high resolution landcover data available for Pennsylvania or New York. The only thing I could find was 250k historical USGS LULC from the 70's and 80's.
But the National Landcover Database has relatively high res data (30m per pixel) from 2001 in raster format where the colors correspond to codes (see: http://www.mrlc.gov/). The problem is, how to get the raster data into shapefiles without a really tedious workflow or purchasing expensive software? I found a way to do this using Mapwel, and I thought I'd post here in case it helps someone else. If you know a better way to do this, please share!
First, I downloaded the NLCD imagery from the link on the site above. I then opened the file and exported small tiles (~1500x1500 pixels) as GeoTiff files. These can be imported into Mapwel with one click.
(http://stephencreek.com/gpsfd/nlcd/raster1.jpg)
I then used the magic wand (trace) tool and clicked on each of the different colors to vectorize them. There are several settings to play with in the magic wand control panel, and some trial and error is needed to get the best results. But after that, it's pretty straightforward - I just clicked on each of the colors in the image (there are about 16), gave them names and let the software trace the outlines. I named the areas with the hex codes of the custom types I would use later in Globalmapper.
(http://stephencreek.com/gpsfd/nlcd/vector1.jpg)
The results were then exported as a .mp file and opened in Globalmapper to massage the data. In Globalmapper I searched for each of the names given to the polygons and assigned the correct MP_TYPE values to them. Seems like there's probably an easier way to do this, but I didn't find it.
(http://stephencreek.com/gpsfd/nlcd/gm1.jpg)
After getting all the correct MP_TYPE's assigned, I combined the landcover with the other layers of my map, exported as a .mp file and compiled with cgpsmapper. My custom type file is the secret sauce that makes it all work in Mapsource. :)
(http://stephencreek.com/gpsfd/nlcd/mapsource1.jpg)
This technique should work with any kind of raster data that uses color codes to represent values. If you just want to produce simple, green forest shading, that would be easy to do from the NLCD since you'd only need to vector three different colors (for Deciduous, Evergreen and Mixed forest).
Mapwel also has a one-click auto-vectorizer and I played with that also. It creates a lot of polygons, and I didn't like the pixellation so much. The Magic Wand tool has a setting to create curved polygons.
I have the advanced version of Mapwel, but I believe the free download may do all of this as well. The free version won't send a fully processed map to a GPS, but it will let you save files on your computer. I'm not certain whether it allows the export of .mp files however. Regardless, you can experiment with the magic wand tool in the free version and see if it looks useful.
Here are a couple screenshots from the Nuvi 3790 along with the North America DEM file.
(http://stephencreek.com/gpsfd/nlcd/nlcd3790.jpg)
Quote from: Boyd on December 23, 2010, 08:13:12 PM
First, I downloaded the NLCD imagery from the link on the site above. I then opened the file and exported small tiles (~1500x1500 pixels) as GeoTiff files. These can be imported into Mapwel with one click.
Boyd, what program did you export small tiles, Global Mapper? Why the 1500 pixel file size? Did you have to pick each island color with the magic wand, or could you select all islands of a given color?
I did this with the National Landcover database and Photoshop and Illustrator. In doing the Photoshop work, I lost the georeferencing of the TIF, which I then had to re-reference using GM.
Since Mapwell is a lot cheaper than the Adobe products and maintains the georeference, it looks like a nice option.
Yes, the tiles are part of my Globalmapper source files that I have built layer-by-layer over the past few years. It was a pretty arbitrary decision originally - I drew a rectangle around the state of New Jersey with a little extra space, then exported it as a 5x7 grid which made tiles that are about 20 miles wide x 25 miles high. The NLCD tiff's for these tiles are 1270x1212 pixels. Some mental arithmetic made me think this was about right for 30 meter/pixel data. :)
I hadn't used Mapwel in a long time, so I upgraded to the newest version (which is free to registered users). It's pretty impressive how fast it can process raster images. Download the free trial and give it a try.
There are a number of settings and I find I have to play with them for each image to get the best results. Tolerance determines how precisely it will match colors (0 would be an exact match). Simplicity determines how many sides the polygon will have (0 is the max). You can choose to draw either curves or straight lines through the pixels. You can select tracing just the outline or outline with islands. Using the "Add Similar" setting, it will select all pixels in of the desired color. So you have to do that for each color in the image's pallette. Like I said, that is only about 16 for the NLCD (a few of those are specific to alaska too :) )
(http://stephencreek.com/gpsfd/nlcd/settings1.PNG)
I found a way to automate things. After creating the .mp file in Mapwel, I open it in Globalmapper then export it as a shapefile. I import the .dbf file into Filemaker, and assign the Area Name I gave each polygon in Mapwel to the MP_TYPE attribute, discarding the value that Mapwel had chosen. I also use a relationship in my database to create some other fields. The modified .dbf and shapefile is opened again in Globalmapper and exported to an .mp file for compilation by cgpsmapper.
Have been working on this some more, and it definitely takes a lot of trial and error to get the best results. At .7 miles zoom in mapsource, it generally looks pretty good now
(http://stephencreek.com/gpsfd/nlcd/valleyforge.PNG)
.5 miles zoom is also pretty good
(http://stephencreek.com/gpsfd/nlcd/adamandeve.PNG)
(http://stephencreek.com/gpsfd/nlcd/downingtown.PNG)
At .2 miles, it starts gettig pretty rough.
(http://stephencreek.com/gpsfd/nlcd/downingtown2.PNG)
There may be better ways to do this, but eventually you run into the quality limits of the landsat data. Anyway, it's been a good project for me and I hope to post a map soon now.
Very impressive.
Thanks. For one of my 20x25 mile tiles, this technique generated 140,000 polygons! Still seems to work fine though. :)
Wow, looks like the new version of GlobalMapper (released on 1/13) can now do this as well. This looks like it would do the same thing that I was doing with Mapwel. Damn, now I am going to be tempted to upgrade. >:(
QuoteGlobal Mapper Software LLC is excited to announce the availability of our next release, Global Mapper v12.01
Added option to create area features from equal colors/elevations in any loaded raster/elevation layer by right-clicking on it in the Control Center and selecting the new option to do so. This is useful for things like extracting vector coverages from something like a land use/cover raster image, like NLCD data.
Glad somebody out there reads the fine print. I'll have to try this one out.
I gave in (no self control ;D) and upgraded. BTW, if you do, look at the banner ad that runs on the GlobalMapper forums. It offers a 10% discount on the upgrade cost, saving $14.
This new feature is very cool but requires some experimentation to understand the settings. I wish they gave you a bit more control, but it's still pretty impressive. I used about a 2 mile x 2 mile section of 1/9 arc second DEM to experiment with. Using the default detail setting, it generated something like 1.9 million polygons - basically one square for every pixel!
But playing around with a smaller test file I was able to get a better feel for it. You could easily use this feature to create pseudo-DEM for a Garmin map. Have you ever seen this article? http://www.cgpsmapper.com/download/Creating%20custom%20types%20to%20represent%20elevation%20data.pdf
That article uses a time consuming technique to generate polygons for different elevations. You could use this new Globalmapper feature to do the same thing with just a few mouse clicks. I used these settings for example
(http://stephencreek.com/gpsfd/DEM02.PNG)
And I created this. I have selected one of the polygons to show what they look like
(http://stephencreek.com/gpsfd/DEM01.PNG)
Now I used a setting that wouldn't make too many polygons - this created only 348 for an area measuring 1.1 miles x 1.4 miles. Of course, this is a waste of the good 1/9 arc second DEM, but I was just playing around. Using a smaller value for "maximum match distance", such as 0.10, would capture more detail and creates a lot more polygons - over 23,000 in this case!
(http://stephencreek.com/gpsfd/DEM03.PNG)
Here's the lower resolution version with the addition of contour lines. All you would need to do is convert each ELEVATION value to a custom type to turn this into a Garmin map - another job for FileMaker :)
(http://stephencreek.com/gpsfd/DEM04.PNG)
Cool stuff, I can think of lots of things to try with this. :)
(http://imghst.gpsfiledepot.com/1295201217.jpg)
Biggest tile I've been able to vectorize. All "wooded areas" converted to the same semi-transparent green.
One thing I've noticed was the selection by COLOR attribute is pretty fussy. GM drops leading "0"s and replaces them with spaces so a text search by COLOR for RGB(028,102,051) needs to be expressed as "RGB(space space 28, space 102, space space 51 space)"
Yeah, I don't like the COLOR attribute either. They need to beef up that dialog box a bit, and let you create some custom attributes. Looks like the ELEVATION attribute will be easier to manipulate, since it's just a height in meters.
What was the source file in your example? NLCD? I need to play with this some more, still not sure but I think I may like Mapwel more for vectorizing NLCD data. Globalmapper can vectorize color images that look just like the originals, but at the expense of creating huge numbers of polygons. From my quick tests, it looked like it would create a square for every pixel in the source image when you set the match distance to zero. That gets out of hand really quickly.
To make it look best in Globalmapper, you can edit all the polygons and set their borders to Null.
Source data is NLCD. Pale green color is my default color for all wooded areas in GM. When GM does the conversion it sets the color of the polygon to match the COLOR attribute so the only way you can differentiate the base raster from the output vector is the black lines indicating polygon edges.
I don't think the individual pixels issue is a concern with NLCD, because they get combined to make large polygons. The workspace file for the image above was about 35 Meg. When I added 2.5 more tiles it bumped up to 95 Meg. My standard tile is about 30 miles square with pixels at 30m grid.
FWIW, when I did my attribute edit for the COLOR, I set the GM type to "wooded area", set the display type to default for wooded area, and added a "Covertype" attribute which I set to "coniferous", "deciduous", or "mixed", based on the NLCD color chart.
It's not really a raster file if it can be converted to a vector graphic. Raster data are pure bitmaps, much like what comes out of a digital camera. Vector data is all about lines, points and areas - with area fill properties, etc.
Many image (raster) programs claim to be able to trace raster graphics, few do it very well, especially for high resolution bit mapped graphics that contain data on the meter or less resolution level.
Quote from: Hillbilly61 on January 17, 2011, 04:42:37 PM
It's not really a raster file if it can be converted to a vector graphic.
Nonsense. Read this whole thread. There are various ways to convert raster data to shapefiles. The results will vary in quality. This new feature of Globalmapper can do it quite accurately but produces mountains of data. Basically, you can create a square for every pixel in the source image.
The method I described using Mapwel earlier in the thread is less faithful as it creates shapes that are more generalized.
Yes, clearly there are trade-offs when converting a raster image to vector format, and it works better for certain kinds of data. But your quote above is simply not true.
I have attached a small example of how this Globalmapper feature works. In this case, the vectorized version looks exactly the same as the original .tif file. The attachment is a ZIPped folder containing the original source raster image (test.tif) and the shapfile that I exported (test.shp, test.dbf, test.shx, test.prj).
The source image is very small - only 117 x 112 pixels, but it should be enough to give you an idea. You can do the same thing with much larger images, but the file size will grow exponentially into millions of polygons.
At 117 x 112, the source image contains about 13,000 pixels. The exported shapefile has about 7,000 polygons in it, so Globalmapper was able to achieve some savings by grouping adjacent pixels of the same color together. I used the highest quality setting to create this example.
The source .tif file was cropped from NLCD data in the vicinity of Wilmington, DE.
Boyd,
Have you seen a way to script this yet? I just posted the question on the GM Yahoo group?
No I haven't, but Globalmapper scripting is something I just haven't had the time to dabble in at all...
Do some testing a use caution. I found this not to be 100% accurate. It was not hard to find 'areas' were GM miss classified the polygons. Anywhere from one or two pixels in size to one area about 1 by 1/4 mile in size. There was a line around most of these areas but the color in GM was the same (the large area was medium tan bordered by cream, and included dark green and medium red pixels). Except for processing a very small 'quad' area around this area; the size of the area/quad processed did not seam to matter; even with a very small sized quad/area there were some small miss classified/labeled areas.
Are you talking about the National Landcover Dataset? There's an interesting article here: https://www.e-education.psu.edu/natureofgeoinfo/c8_p19.html
There's a lot of variation in the colors of each category, and I spent awhile dealing with that when I developed the technique with Mapwel that I described at the beginning of this thread. From the article in the link above:
QuoteThe USGS hired private sector vendors to assess the classification accuracy of the NLCD 92 by checking randomly sampled pixels against manually interpreted aerial photographs. Results from the first four completed regions suggested that the likelihood that a given pixel is correctly classified ranges from only 38 to 62 percent. Much of the classification error was found to occur among the Level II classes that make up the various Level I classes, and some classes were much more error-prone than others.
USGS encourages users to aggregate the data into 3 x 3 or 5 x 5 pixel blocks (in other words, to decrease spatial resolution from 30 meters to 90 or 150 meters), or to aggregate the 21 Level II classes into the nine Level I classes. Even in the current era of high-resolution satellite imaging and sophisticated image processing techniques, there is still no cheap and easy way to produce detailed, accurate geographic data.
The 2001 NLCD. For the 2001 data, USGS used different classification algorithms in different areas which were 'tweaked' to give more accurate results in each region; I think the value was 70, or 80 %, but was not able to find the reference yesterday.
Refer to the article you provided the link for. In the data set provide, each of the classification catagories is assigned and displayed with a specific RGB value. Three years ago when I check using Photoshop, I found some of the values differed by a number or two (from the values given on a USGS site); however they were consistent.
I think the 2001 data has a lot of variation in it..... and I just spent many hours staring at it and clicking on individual pixels using the mapwel feature that selects all other pixels of the same value. Off the top of my head, I'd say that each category probably has from 5 to 10 different RGB values. It seems to vary depending on the color of the adjacent pixels. In other words, if a red pixel is next to a green pixel there can be a range of intermediate colors between red and green. But if a red pixel is next to a yellow pixel, there's a different range of colors.
Your experience and analysis may be different though. If you develop your own techniques that work well for this, I will certainly be interested in hearing about them.
If anyone's interested I've attached a draft script that defines areas, assigns types and attributes, and exports them as shape files, using Mike's Do Loop example. The syntax of color numbers with leading zeros is fussy. You need to replace the leading zero with a space, so there are two spaces in front of all 2 digit RGB values.
To make it work you need to use the latest build of GM12.01.
FWIW and this may only confuse the issue.
The NLCD data uses a single RGB value for each classification type.
In GM9 the pixels may show a slightly different value depending on the values of the adjacent pixels and may even change values as the tool is moved across a pixel.
In GM12 the values appear to be constant.
In PhotoShop, selecting a polygon with the dropper tool appears to 'flag' all with the same classification.
In MapWell, Boyd reports 'blends' of colors. I could not remember how to get the RGB value of a pixel.
I conclude that both GM9 and MapWell are changing the pixel values for some reason and the user has no control over this.
GM12 and PhotoShop are using the RGB values as given in the source data.
Very interesting.... that could certainly explain it. Will have to take a closer look at that.
Nope, that isn't it. I can clearly see the variation when I open the file and zoom way in in GM 12.01 - see for yourself in this screenshot. After downloading the area 13 landcover file, I used Globalmapper 11 to crop it to the boundaries of each of my map tiles, then exported that as a 24-bit RGB GeoTIFF. Could something have gone wrong there?
(http://stephencreek.com/gpsfd/nlcd/TIFF.PNG)
maps4gps, do you have a link to a site that shows the RGB assignment for each NLCD landcover type? Right now I'm assigning them by eye, assuming that Dark Green (028,102,051) is Evergreen, etc.
All I have been able to find is this, and it's a .jpg so I don't think the colors are accurate: http://www.mrlc.gov/downloadfile.php?file=NLCD2001_Colour_Classification_Update.jpg
(http://imghst.gpsfiledepot.com/1295482190.jpg)
Closeup of Cedar City, Utah. No ghosting here, but it sure looks like Boyd has it. There appear to be 4 red to pink settings:
Light pink: Developed Open Space
Dark pink: Developed Low Intensity
Light red: Developed Medium Intensity
Dark red: Developed High Intensity
Quote from: Boyd on January 19, 2011, 04:13:26 PM
All I have been able to find is this, and it's a .jpg so I don't think the colors are accurate: http://www.mrlc.gov/downloadfile.php?file=NLCD2001_Colour_Classification_Update.jpg
That's the basis for my eyeball judgment, but the numbers must be some sort of classification because RGB requires three comma delimited values.
OK, something odd is happening, so I have to check my original downloaded file. If I use the NLCD viewer there's a world of difference. These screenshots don't quite match in coverage and projection, but here's what the viewer looks like:
(http://stephencreek.com/gpsfd/nlcd/viewer.png)
And this is the same vicinity in Globalmapper:
(http://stephencreek.com/gpsfd/nlcd/globalmapper.png)
I'm pretty sure I used the maximum quality in all my imports/exports but maybe I goofed. Or maybe globalmapper is doing some compression?
This is really puzzling. I opened the sourcefile that I downloaded in both GM 11 and GM 12 and it looks clean (like the viewer screenshot above). I then tried exporting sections of it and they also look clean.
But somehow I suspect compression must be the culprit. I am going to want to re-do a lot of stuff now and get it right. How frustrating! >:(
I got my original IMG (graphic) and TIF files from the Seamless Server perhaps a year ago. Zooming tight to both of them shows well defined square pixels. Is it possible you got a JPG or JPG2000?
Boyd,
Looks like that second file may have been reprojected. Earlier today I was thinking it might have something to do with reprojecting the source in Abers to Geographic; however, it check out OK. Something is not right with yours. A compression algorthym not specifically intended to preserve all the original values could produce such results.
When I downloaded the data a few years ago, it was via the Seamless server and defining 2x6 degree areas. Did not expect to make you day, but certainly did not want to break it.
Seldom_sn,
I last looked at the data about 2 1/2 years ago. I will see if I can find that list - I do remember the values GM gave could be a few units different, but nothing close to another classification type so there was no confustion with which was which.
I did not have much luck yesterday finding the webpages on NLCD data I remember from years ago. Not even a reference to the 2007 data that was in the works then.
You know, I think I know how this happened. I believe I resampled the images at somewhat lower resolution after reading that article I linked to earlier
QuoteUSGS encourages users to aggregate the data into 3 x 3 or 5 x 5 pixel blocks (in other words, to decrease spatial resolution from 30 meters to 90 or 150 meters)
But that was in reference to the 1992 imagery, so I shouldn't have done it with the 2001 imagery which is obviously a lot cleaner. Resampling would certainly create a problem like the one I'm seeing.
Aaargh >:( Well, thanks guys for being persistent in pointing out this problem. Now I'll need to do a bunch of stuff again, but should get much better results.
[edit]I also thought about re-projection being a problem, but after trying it doesn't seem to be. It was definitely the resampling.
Quote from: seldom_sn on January 19, 2011, 04:01:31 PM
maps4gps, do you have a link to a site that shows the RGB assignment for each NLCD landcover type? Right now I'm assigning them by eye, assuming that Dark Green (028,102,051) is Evergreen, etc.
I do not think this is the site I used; however these seem to be about right:
http://www.mmm.ucar.edu/people/duda/files/cat.tbl
Thanks maps4gps, as you say, they are not quite what GM reports, but they're generally only one or two bits off.
Hello,
This program will be very helpful for automatic vectorization:
-> Easy Trace Pro
Free full version 7.99 (http://www.easytrace.com/getfile.php?file=et799)
Video Lessons:
http://www.easytrace.com/getfile.php?file=ETOverviewEng.htm
http://www.easytrace.com/site/english/technology/video_list.html
Cheers,