Combining Hydro Data - Error

[n2stitch]

I have been going through and combining the hydro data using the batch files on this site per the tutorial.  The file size for the flowlines is huge - over 2G and I'm not done.  Once the file size went over 2G, I got the attached error.  The areas and waterbody files are still ok.

Now what do I do?  Thanks!

Lalita

[-Oz-]

Looks like this program also has a memory limit of 2gb; my waterbody for florida is 708mb.  What state has so much water?

What you can do is not combine all of them together but half then half.  That way you'll avoid this limit.  You'll just have to process and treat each half separately.

[n2stitch]

Ok, I'll try running half and half.  Missouri is the state I'm processing, but I may have a lot of data for surrounds states since identifying and downloading the subbasins wasn't very easy.  I couldn't tell very well what data I was really asking for when selecting it.  I didn't think MO had that much water!

[n2stitch]

Before I started to do any more processing, I wanted to understand why MO had so much data.  Should have known it was some form of keyboard input error.  Many of my files have overlapping data in them, so the same data keeps being added to the database over and over.

What I did to look at them was open them with GPSMapedit, just assigning a random water type.  Then I added the state.mp file I created so that I could see if the data actually fell within the borders.  By looking at a few of them my problem was clear.

Now I can delete data from within these files, but if I save them from within GPSMapedit where I've already randomly assigned a type, will it cause a problem with the sql and postgis processing?

Lalita

[n2stitch]

What I did instead was break them into 3 batches, instead of running all of it at once.  Basically I could get the shp file created, but mapedit couldn't open it.  Except for a couple of places where it looks like I'm missing some data, hydro is now done.  I'll download that data next week and finish it up.

Lalita

[-Oz-]

Congrats.