When it comes to GIS data, if I can't open a file in GlobalMapper then I'm pretty clueless. So maybe somebody can set me straight on how to accomplish this. The search/replace function in Globalmapper is very powerful, but still awkward for processing large amounts of data. I am very experienced in developing databases using FileMaker Pro, so I'd like to use it to massage a large amount of Land Use/Landcover data that I have downloaded from my state's GIS website.
For example, I downloaded the following:
http://www.nj.gov/dep/gis/digidownload/zips/lulc07/w07lu07.zip
and ended up with these files:
w07lu07.xml
w07lu07.txt
w07lu07.shx
w07lu07.shp.xml
w07lu07.shp
w07lu07.sbx
w07lu07.sbn
w07lu07.prj
w07lu07.htm
w07lu07.dbf
revised.doc
I imported w07lu07.dbf into Filemaker with no problem. After massaging the data, I can easily export it back out as a .dbf file. But then what? Normally I would just load the shapefile into Globalmapper (w07lu07.shp) and discard all those other files. How do I associate the modified .dbf file with the shape data in the .shp file? I am going to add a few more fields to the database (such as a Garmin polygon type - eg: 0x50)
My goal is to be able to import a bunch of files like this, run a filemaker script to assign Garmin polygon types to each record, open the file for further editing in GlobalMapper, then export it out as an .mp file to compile in cgpsmapper.
Can somebody point me in the right direction? :)
I don't have a clue, but have you asked Mike?
http://tech.groups.yahoo.com/group/global_mapper/
GM (10 and 11) used to have an "Assign_Type" scripting command that I've used to convert NHD types to MP types. That command has been replaced in GM 12.
Yeah, I know how to assign types in GlobalMapper using the MP_TYPE attribute. But I need to extensively filter the raw LU/LC data in the shapefiles, delete the objects I don't want and then classify the others. This is a lot easier in a database program like Filemaker than working directly with GlobalMapper.
I should probably try to find what I did with my user/login info from the GlobalMapper forums and just ask over there, but thought maybe someone here might help for starters. Thanks.
Here is how I deal with this kind of data with the free MapWindow GIS:
http://www.mapwindow.org/
Drag and drop the shapefile into mapwindow gis.
In the legend unclick the label so nothing is showing (if you don't selections and sorting can take forever)
On the tool bar, click on Attribitube Table Editor
Right Click on the field you want to use and then select sort ASC
Select the data you want, Selection, Export selected features. Then give the exported shapefile a descriptive file name.
Just repeat for each thing you want to export. (Ag Wetlands, then Airports)
Then you end up with a shapefile for each thing you want (AG wet.shp, airport.shp).
I then import them into gpsmapedit. But I do not know how to use globalmapper.
But you can add fields and assign types based on querries. If you are a database person you should be able to easily figure out the querry function and field calculator tools. You can do what you describe, but I think it is easier to just create seperate shapefiles for each thing.
I do what you are describing all the time. I am not familiar with FileMaker Pro. I use dBase for accessing the .dbf files. I copy xxx.dbf to xxx.org for a backup, open xxx.dbf, modify, save as xxx2.dbf, delete xxx.dbf, and rename xxx2.dbf to xxx.dbf. I think the .shp file is linked to the .dbf file by record number; thus the records (.shp and .dbf) need to be 'deleted' using GM.
I 'flag' objects in the dbf file using 'drop' in the mp_type field; then in GM search that field for 'drop', delete those objects and save the file with a new name (else GM corrupts the dbf file). I use the relationship function of dBase to assign mp_type, etc. based on the 'feature code', etc. in the source data.
Two other issues which might be important. 1. GM will consider any field with only numbers as a numeric field - besides being a waste of time and very bad data base design, some of the data may be lost if your software considers a numeric field with a blank in the rightmost position as no/zero data. 2. GM left justifies numeric fields; this may be the UNIX convention, however, it causes problems in a PC world. I have to export as ASCII the .dbf GM creates; modify the structure of the .dbf file to character fields, then import the ASCII file. A real PITA as every time GM save the data in shape file format .dbf file it changes the data back to numeric. I have spent many 100's of hours going back to rebuild the lost data when it is later need for relations, etc.
You can do that kind of manipulation of the dbf file in mapwindow gis without having to back it up or reconstruct it. You just edit it and select save when asked if you want to save it. For example, you can run a querry to select the records you want and then add a type to the field for just those records. It is a really nice program for messing with shapefiles. It will also change projecections.
Good stuff guys - thanks! I will have a look at your ideas and let you know how it works out.
I have to chuckle a bit when I hear about dBase. Back in 1982, a board member for the company where I worked hired me to teach him how to use dBase II on his fancy new CP/M based home computer (with those big old 8" floppy disks). ;D
Quote from: seldom_sn on November 01, 2010, 11:36:28 AMGM (10 and 11) used to have an "Assign_Type" scripting command that I've used to convert NHD types to MP types.
Just looked at this a little closer (I am running GM 11). That does look like a powerful command which I could adapt to do what I want as well. Here's an example from their script reference.
Reclassify Features Based on an Attribute and Display Label
GLOBAL_MAPPER_SCRIPT VERSION=1.00
// Import the file to modify
IMPORT FILENAME="C:\Temp\export test\tiger_wyandotte_sample.gmp"
// Assign the type "railroad" to all features with a CFCC attribute with a value of A41
// and a display label with '74' in it somewhere.
ASSIGN_TYPE LINE_TYPE="RAILROAD" COMPARE_STR="CFCC=A41" COMPARE_STR="<Feature Name>=*74*"
[/quote]
This NHD to MP script will type a 5 degree by 2 degree high res NHD file in around a minute.
I do all this with postgis; that is because I am much happier manipulating a database than anything else.
Also, the shapefile only really needs 4 things (3 are really mandatory) a .shp, .shx, .dbf, and .prj (this is the not mandatory but useful one)
If you do not have a prj or metadata with the projection, is there a way to figure out the projection? I did not think there was so I would consider prj mandatory.
You read the meta data for the file and see what the projection is. Some of the government data still doesn't include the .prj but has an xml of meta data.
Thanks everyone for all the insights - I knew this would be a good place to ask my question. :)
As stated in the beginning, I am pretty clueless about GIS stuff and have been learning as I go. Turns out that I was making this seem harder than it needed to be. A little experimentation shows that when Globalmapper opens a .shp file, it will automatically associate records from a .dbf file of the same name in the same directory. That probably seems obvious to some people, but I just wasn't "getting it" for some reason. :-\
Anyway, it's really easy and effective to do what I want. Simply import the .dbf file that was located in same folder as the .shp file and do whatever you want with its contents. I used a text editor to open the .dbf file and made a list of the field names. Then I created a Filemaker database and defined text fields with those names. Next I imported the .dbf file, with the option of matching field names.
I created a new field named MP_TYPE in FileMaker and used the Find/replace function to insert some of my own hex codes in this field. Finally, I exported only the 4 fields that I wanted to a new .dbf file with the same name.
In Globalmapper, I simply opened the .shp file. I then used the search function and - lo and behold - my modifed data with MP_TYPE attributes was all there. Globalmapper adds three of its own fields: Name, Feature Type and Description. There's an option you can choose in the Control Center for automatically using one of the other attributes as the Name.
Anyway, this was just a test, but it worked. I guess you could do pretty much the same thing using excel, but I like using Filemaker personally. BTW, you can download a 30 day fully functional trial of the program at filemaker.com if you want to play with any of this youself.
Here's a little more info about what I am doing; maybe it will give somebody else an idea of a different way to process map data. I am working with landcover data that is classified using the Anderson Code System where a 4 digit code corresponds with each polygon in the shapefile to identify the landscape type.
FileMaker Pro is a relational database which gives it some very powerful features. Defining a simple relationship can perform the same function as a lot of searching/replacing of data and it's also really fast. The file I have been playing with has ~78,000 records and I can update all of them with changes instantaneously.
My database contains two tables. I created the first with fields for the Anderson Code, the Garmin MP_TYPE hex code, and descriptions of Garmin's standard usage for the type as well as my custom type use.
(http://stephencreek.com/gpsfd/fmp1.jpg)
The second table contains all the fields used in my source .dbf file, as described in the previous post, plus my own MP_TYPE field. I import the source .dbf file into this table.
(http://stephencreek.com/gpsfd/fmp3.jpg)
I have defined a relationship between the Anderson code in the source data and my own table.
(http://stephencreek.com/gpsfd/fmp2.jpg)
The value of MP_TYPE field in each record of the source table is therefore determined by the way I map Anderson codes to Garmin types. I do not make any changes to the source .dbf file directly; it is all handled by the relationship that I defined.
This allows me to experiment with different ways to map Anderson codes to custom types by simply changing values in my own list, and the source file is updated instantly. When I'm ready to send this all to Globalmapper, I simply export the fields I need to a new .dbf file. This strips out all of the data in the source file that I no longer want, making it smaller.
(http://stephencreek.com/gpsfd/fmp4.jpg)
You would not really need FileMaker or Globalmapper to use this kind of workflow. It could be done with any relational database and any program that can read shapefiles. Clear as mud? ;D
And here are a few examples of the beta version of the map I am working using this technique. The Nuvi 3790 is a nice platform for this kind of thing. :)
(http://stephencreek.com/gps/mays_landing.jpg)
Great looking map. I assume the green polygons are some sort of land cover, wooded areas, etc.
Are you getting that from USGS data or is it vectorized in New Jersey?
Thanks. The data comes from the New Jersey Dept of Environment Protection (NJDEP): http://www.state.nj.us/dep/gis/lulc07cshp.html They have done a really nice job on this dataset, which just became available a few months ago. The previous version dates back to 2002, then I there's also a 1995 version.
The data is classified with the Anderson Code system where a four digit number represents a very specific type of landcover, such as "mixed forest with coniferous prevalent and 10-50% crown closure". See: http://www.state.nj.us/dep/gis/digidownload/metadata/lulc07/anderson2007.html
So the challenge is to distill all these codes into a manageable number of Garmin custom types, then create bitmaps to give the "flavor" of what they might look like. This is limited by Garmin's capacity for only two colors and bitmap size of 32x32.
I believe the NJ landcover is 48k - it looks pretty nice. I think it was traced by people from aerial photography - big job there! I guess this is the advantage of living in one of the smallest and most populated states - it is really thorougly mapped!
I would love to find some kind of high resolution landcover data for NY, PA, DE and MD since my map also covers portions of these states. Have not looked since last year, but at that time all I could find was the USGS 250k DLG landcover data.
Took the map out for a road test today, and I continue to be impressed with they way the 3790 renders 3d in realtime. I would really like to see Garmin introduce a rugged version of this hardware with some more advanced menus. That would really blow away everything else on the market IMO. :)
(http://stephencreek.com/gps/lu07/pines07.jpg)
I know this thread is old, but for posterity purposes the following is being thrown out there from the been there/suffered through it department:
* The shapefile format is really a relational database, with the primary relations being between the .dbf (data) and .shp (geometry) file. You can add columns to your heart's content to the .dbf file with an outside editor. This doesn't affect anything. Just save the .dbf as its original name and the shapefile will be "none the wiser."
* DO NOT delete or insert rows in the .dbf file. Doing so will screw up the relationship between the records and the geometry. If the shapefile loads at all, very strange results can be had.
* In terms of using a .dbf editor to modify the .dbf in a shapefile make sure it is dBase III compliant, as the shapefile really uses a dBase .dbf file. The reason for saying this is that some dBase file editors natively save to a dBase IV format. One of the big differences between the two is that a dBase III file does not support null values. Missing or omitted ASCII and numeric data is represented as a blank or a zero in a dBaseIII file. Nulls will make third party readers that closely implement this standard puke if they closely implement the shapefile spec.
Good to know, although I pretty much learned all of this the hard way myself. I finished my first big project using the technique I described above, although I refined it a bit. Filemaker really handles this like a champ. Once I setup the database to import/export as I wanted, it only took about 4 clicks to process huge files.
The end result can be seen here: http://www.gpsfiledepot.com/maps/view/441/
That map has about 900,000 polygons representing landcover (not counting all the other map elements). I am now processing the data differently to make a new map, and created on big file that has all the polygons in it. I then exported it as a shapefile and did the processing on the .dbf file in FileMaker Pro. It had no problem ingesting all 900,000 records and spewing the modified ones back out. Took a few minutes to churn through it, but it worked great. I then opened the modified shapfile in Globalmapper and was able to make further modifications.
So, for the things I'm interested in doing, converting my vector data to shapefiles is a great way to automate my work and process large quantities of data with FileMaker. I even used to to process raster data by first converting it to vector format with Mapwel: http://forums.gpsfiledepot.com/index.php/topic,1693.0.html