Manipulating Shapefile data in FileMaker

[Boyd]

When it comes to GIS data, if I can't open a file in GlobalMapper then I'm pretty clueless. So maybe somebody can set me straight on how to accomplish this. The search/replace function in Globalmapper is very powerful, but still awkward for processing large amounts of data. I am very experienced in developing databases using FileMaker Pro, so I'd like to use it to massage a large amount of Land Use/Landcover data that I have downloaded from my state's GIS website.

For example, I downloaded the following:

http://www.nj.gov/dep/gis/digidownload/zips/lulc07/w07lu07.zip

and ended up with these files:

w07lu07.xml
w07lu07.txt
w07lu07.shx
w07lu07.shp.xml
w07lu07.shp
w07lu07.sbx
w07lu07.sbn
w07lu07.prj
w07lu07.htm
w07lu07.dbf
revised.doc

I imported w07lu07.dbf into Filemaker with no problem. After massaging the data, I can easily export it back out as a .dbf file. But then what? Normally I would just load the shapefile into Globalmapper (w07lu07.shp) and discard all those other files. How do I associate the modified .dbf file with the shape data in the .shp file? I am going to add a few more fields to the database (such as a Garmin polygon type - eg: 0x50)

My goal is to be able to import a bunch of files like this, run a filemaker script to assign Garmin polygon types to each record, open the file for further editing in GlobalMapper, then export it out as an .mp file to compile in cgpsmapper.

Can somebody point me in the right direction?

[Seldom]

I don't have a clue, but have you asked Mike?

http://tech.groups.yahoo.com/group/global_mapper/

GM (10 and 11) used to have an "Assign_Type" scripting command that I've used to convert NHD types to MP types.  That command has been replaced in GM 12.

[Boyd]

Yeah, I know how to assign types in GlobalMapper using the MP_TYPE attribute. But I need to extensively filter the raw LU/LC data in the shapefiles, delete the objects I don't want and then classify the others. This is a lot easier in a database program like Filemaker than working directly with GlobalMapper.

I should probably try to find what I did with my user/login info from the GlobalMapper forums and just ask over there, but thought maybe someone here might help for starters. Thanks.

[jbensman]

Here is how I deal with this kind of data with the free MapWindow GIS:

http://www.mapwindow.org/

Drag and drop the shapefile into mapwindow gis.

In the legend unclick the label so nothing is showing (if you don't selections and sorting can take forever)

On the tool bar, click on Attribitube Table Editor

Right Click on the field you want to use and then select sort ASC

Select the data you want, Selection, Export selected features.  Then give the exported shapefile a descriptive file name.

Just repeat for each thing you want to export. (Ag Wetlands, then Airports)

Then you end up with a shapefile for each thing you want (AG wet.shp, airport.shp).

I then import them into gpsmapedit.  But I do not know how to use globalmapper.

But you can add fields and assign types based on querries.  If you are a database person you should be able to easily figure out the querry function and field calculator tools.  You can do what you describe, but I think it is easier to just create seperate shapefiles for each thing.

[maps4gps]

I do what you are describing all the time.  I am not familiar with FileMaker Pro.  I use dBase for accessing the .dbf files.  I copy xxx.dbf to xxx.org for a backup, open xxx.dbf, modify, save as xxx2.dbf, delete xxx.dbf, and rename xxx2.dbf to xxx.dbf.  I think the .shp file is linked to the .dbf file by record number; thus the records (.shp and .dbf) need to be 'deleted' using GM.  

I 'flag' objects in the dbf file using 'drop' in the mp_type field; then in GM search that field for 'drop', delete those objects and save the file with a new name (else GM corrupts the dbf file).  I use the relationship function of dBase to assign mp_type, etc. based on the 'feature code', etc.  in the source data.

Two other issues which might be important.  1. GM will consider any field with only numbers as a numeric field - besides being a waste of time and very bad data base design, some of the data may be lost if your software considers a numeric field with a blank in the rightmost position as no/zero data.  2. GM left justifies numeric fields; this may be the UNIX convention, however, it causes problems in a PC world.  I have to export as ASCII the .dbf GM creates; modify the structure of the .dbf file to character fields, then import the ASCII file.  A real PITA as every time GM save the data in shape file format .dbf file it changes the data back to numeric.  I have spent many 100's of hours going back to rebuild the lost data when it is later need for relations, etc.

[jbensman]

You can do that kind of manipulation of the dbf file in mapwindow gis without having to back it up or reconstruct it.  You just edit it and select save when asked if you want to save it.  For example, you can run a querry to select the records you want and then add a type to the field for just those records. It is a really nice program for messing with shapefiles.  It will also change projecections. 

[Boyd]

Good stuff guys - thanks! I will have a look at your ideas and let you know how it works out.

I have to chuckle a bit when I hear about dBase.  Back in 1982, a board member for the company where I worked hired me to teach him how to use dBase II on his fancy new CP/M based home computer (with those big old 8" floppy disks). 

[Boyd]

GM (10 and 11) used to have an "Assign_Type" scripting command that I've used to convert NHD types to MP types.

Just looked at this a little closer (I am running GM 11). That does look like a powerful command which I could adapt to do what I want as well. Here's an example from their script reference.

Code Select Expand

Reclassify Features Based on an Attribute and Display Label

GLOBAL_MAPPER_SCRIPT VERSION=1.00

// Import the file to modify
IMPORT FILENAME="C:\Temp\export test\tiger_wyandotte_sample.gmp"

// Assign the type "railroad" to all features with a CFCC attribute with a value of A41
// and a display label with '74' in it somewhere.
ASSIGN_TYPE LINE_TYPE="RAILROAD" COMPARE_STR="CFCC=A41" COMPARE_STR="<Feature Name>=*74*"
[/quote]

[Seldom]

This NHD to MP script will type  a 5 degree by 2 degree high res NHD file in around a minute.

[-Oz-]

I do all this with postgis; that is because I am much happier manipulating a database than anything else.

Also, the shapefile only really needs 4 things (3 are really mandatory) a .shp, .shx, .dbf, and .prj (this is the not mandatory but useful one)

[jbensman]

If you do not have a prj or metadata with the projection, is there a way to figure out the projection?  I did not think there was so I would consider prj mandatory. 

[-Oz-]

You read the meta data for the file and see what the projection is.  Some of the government data still doesn't include the .prj but has an xml of meta data.

[Boyd]

Thanks everyone for all the insights - I knew this would be a good place to ask my question. 

As stated in the beginning, I am pretty clueless about GIS stuff and have been learning as I go. Turns out that I was making this seem harder than it needed to be. A little experimentation shows that when Globalmapper opens a .shp file, it will automatically associate records from a .dbf file of the same name in the same directory. That probably seems obvious to some people, but I just wasn't "getting it" for some reason. 

Anyway, it's really easy and effective to do what I want. Simply import the .dbf file that was located in same folder as the .shp file and do whatever you want with its contents. I used a text editor to open the .dbf file and made a list of the field names. Then I created a Filemaker database and defined text fields with those names. Next I imported the .dbf file, with the option of matching field names.

I created a new field named MP_TYPE in FileMaker and used the Find/replace function to insert some of my own hex codes in this field. Finally, I exported only the 4 fields that I wanted to a new .dbf file with the same name.

In Globalmapper, I simply opened the .shp file. I then used the search function and - lo and behold - my modifed data with MP_TYPE attributes was all there. Globalmapper adds three of its own fields: Name, Feature Type and Description. There's an option you can choose in the Control Center for automatically using one of the other attributes as the Name.

Anyway, this was just a test, but it worked. I guess you could do pretty much the same thing using excel, but I like using Filemaker personally. BTW, you can download a 30 day fully functional trial of the program at filemaker.com if you want to play with any of this youself.

[Boyd]

Here's a little more info about what I am doing; maybe it will give somebody else an idea of a different way to process map data. I am working with landcover data that is classified using the Anderson Code System where a 4 digit code corresponds with each polygon in the shapefile to identify the landscape type.

FileMaker Pro is a relational database which gives it some very powerful features. Defining a simple relationship can perform the same function as a lot of searching/replacing of data and it's also really fast. The file I have been playing with has ~78,000 records and I can update all of them with changes instantaneously.

My database contains two tables. I created the first with fields for the Anderson Code, the Garmin MP_TYPE hex code, and descriptions of Garmin's standard usage for the type as well as my custom type use.



The second table contains all the fields used in my source .dbf file, as described in the previous post, plus my own MP_TYPE field. I import the source .dbf file into this table.



I have defined a relationship between the Anderson code in the source data and my own table.



The value of MP_TYPE field in each record of the source table is therefore determined by the way I map Anderson codes to Garmin types. I do not make any changes to the source .dbf file directly; it is all handled by the relationship that I defined.

This allows me to experiment with different ways to map Anderson codes to custom types by simply changing values in my own list, and the source file is updated instantly. When I'm ready to send this all to Globalmapper, I simply export the fields I need to a new .dbf file. This strips out all of the data in the source file that I no longer want, making it smaller.



You would not really need FileMaker or Globalmapper to use this kind of workflow. It could be done with any relational database and any program that can read shapefiles. Clear as mud?  

[Boyd]

And here are a few examples of the beta version of the map I am working using this technique. The Nuvi 3790 is a nice platform for this kind of thing.