Welcome, Guest. Please login or register.

Login with username, password and session length
Forums Search:  


Author Topic: Huge Update to Topo Tutorial: QGIS Support/Faster BAT Files/National Map  (Read 3960 times)

-Oz-

  • Map Maker!
  • Administrator
  • Expert Advisor
  • *****
  • Posts: 1577
  • Karma: 27
    • View Profile
    • GPSFileDepot
All,

I have completely rewritten the How To Create Garmin Topo Maps tutorial. This was a major rewrite and was done as I updated the Arizona Topo map.

Biggest changes:
  • Added support/info for QGIS (it's free and can definitely be used in place of Global Mapper); seemed popular on the forums here so I thought I'd try it out
  • Huge updates with data availability because they've added a lot more to the USGS National Map (especially in contours and water data, much faster)
  • Improved all the "automated" files; for all of them its as simple as set the folder name in the file and then double click and it'll automatically do a lot of the work
  • Included "clipping" as part of the tutorial; now the maps you make will end at the state border; much cleaner
  • Created new software to convert shapefiles to mp files.
  • Removed most of the GPSMapEdit information - with QGIS and Global Mapper there is really no reason to use the non-automated processes that GPSMapEdit required
  • Lots of updates for ease of execution/understanding
Anyway, I post this here for two reasons:
1. If you see any issues/errors please let me know
2. If you are using some of the processes from the previous tutorials (bat files especially) have huge updates that will speed up what you do.
Dan Blomberg
Administrator - GPSFileDepot
GPS Units: Garmin Dakota 20, Garmin GPSMap 60csx, Nuvi 255W, Nuvi 250W, ForeRunner 110, Fenix 2, Tactix Bravo, Foretrex 401
See/Download My Maps!

Longostino

  • Newbie
  • *
  • Posts: 2
  • Karma: 0
    • View Profile
Re: Huge Update to Topo Tutorial: QGIS Support/Faster BAT Files/National Map
« Reply #1 on: February 09, 2016, 03:57:19 PM »
Thank you for all your work.  I am working through this tutorial, the latest version.  Most of it is working for me just fine, even though I am using Ubuntu Studio 14.04 LTS.  This means I have to accept slightly older versions of some of the software.  That is usually okay because I can still find my way around to the parts used in the tutorial.  One problem I had was the .BAT file.  I successfully reworked it into a BASH shell script (mergecontours.bat > mergedcontours.sh).  Now I am having trouble with the script that is supposed to be inserted in QGIS, mp_type, Expression field.  I am using QGIS 2.01-Dufour.  The script as written is invalid.  In fact, based on all the QGIS documentation, it is nothing like the syntax the Field Calculator suggests.  Undaunted, I thought I might rework the script using the suggested CASE WHEN condition THEN result END form.  That lead me to an even more frustrated state, because I am unfamiliar with your written syntax.  I imagine that the if (.... ,....,...) form suggests "if (this condition, return that result, otherwise...)."  If that is the case, it would seem the first case says, "if "FCODE" = '10101' OR "FCODE" = '10104', then return 0x20"  Following the rest of the code in a similar manner, the other conditions return, 0x21, 0x22 or NULL.  Now I am really confused.  Why do we need 15 characters for those results?  And are these supposed to be SPACE,  !, ", and NULL, respectively, or the decimal values, 32, 33, 34.  What seems to be written as conditions would make more sense, translating the result of dividing CONTOURELE by CONTOURINT, making it an integer by the power of 1, 10, or 100, depending on how big the result is.  So, what is the point of the conversion expression?  What is the result supposed to be?

Longostino

  • Newbie
  • *
  • Posts: 2
  • Karma: 0
    • View Profile
Re: Huge Update to Topo Tutorial: QGIS Support/Faster BAT Files/National Map
« Reply #2 on: February 10, 2016, 02:24:28 PM »
i found by reading ahead in the tutorial that the values to be created in MP_TYPE are literally '0x20', '0x21' and '0x22', not their ASCII or control equivalents.  Knowing that here is the code I created based on the original in the tutorial, but using the syntax the QGIS Field Calculator Expression Parser would accept as valid.  I believe I interpreted the logic correctly, and I have results that seem to reflect that interpretation:

CASE
WHEN "FCODE" = '10101' OR "FCODE" = '10104'  THEN '0x20'
WHEN  (  "FCODE"  = '10102' OR  "FCODE"  = '10105' ) AND ( "CONTOURELE" / ( "CONTOURINT" *10 )>100) AND regexp_match(  tostring(  abs(  (  "CONTOURELE"  /  (  "CONTOURINT"  * 10 ) )  / 100 -  toint( ( "CONTOURELE"  /  (  "CONTOURINT"  * 10 )  )  / 100 )  )  ) , '5' ) THEN '0x21'
WHEN  (  "FCODE"  = '10102' OR  "FCODE"  = '10105' ) AND ( "CONTOURELE" / ( "CONTOURINT" *10 )>10 ) AND  regexp_match(  tostring(  abs(  (  "CONTOURELE"  /  (  "CONTOURINT"  * 10 ) )  / 10 -  toint( ( "CONTOURELE"  /  (  "CONTOURINT"  * 10 )  )  / 10 )  )  ) , '5' ) THEN '0x21'
WHEN  (  "FCODE"  = '10102' OR  "FCODE"  = '10105' ) AND  regexp_match(  tostring(  abs(   "CONTOURELE"  /  (  "CONTOURINT"  * 10 )  -  toint( "CONTOURELE"  /  (  "CONTOURINT"  * 10  )  )  )  ) , '5' ) THEN '0x21'
WHEN  (  "FCODE"  = '10102' OR  "FCODE"  = '10105' ) THEN '0x22'
ELSE NULL END

MikeDk

  • Newbie
  • *
  • Posts: 1
  • Karma: 0
    • View Profile
Re: Huge Update to Topo Tutorial: QGIS Support/Faster BAT Files/National Map
« Reply #3 on: February 20, 2016, 01:11:43 AM »
I have tried the tutorial. When I get to step 8 - GridSplitter using QGIS - it it VERY slow. My lines.shp is approx. 1,7 GB and after 8 hours less than 1% is splittet. Is there any way to speed up the process?

-Oz-

  • Map Maker!
  • Administrator
  • Expert Advisor
  • *****
  • Posts: 1577
  • Karma: 27
    • View Profile
    • GPSFileDepot
Re: Huge Update to Topo Tutorial: QGIS Support/Faster BAT Files/National Map
« Reply #4 on: February 21, 2016, 09:08:22 AM »
Does your gridplit look similar to this?:


When I do that it tells me that I will make 36 tiles. How many tiles did it tell you it was going to do?

I just did 3.4gb of lines in ~3 hours. I assume you've gotten rid of all the excess columns at this point (as per the tutorial). Because excess data will slow it down.
Dan Blomberg
Administrator - GPSFileDepot
GPS Units: Garmin Dakota 20, Garmin GPSMap 60csx, Nuvi 255W, Nuvi 250W, ForeRunner 110, Fenix 2, Tactix Bravo, Foretrex 401
See/Download My Maps!

GeoH

  • Newbie
  • *
  • Posts: 2
  • Karma: 0
    • View Profile
In Part 8 - Combining Data, instructions 30, 35 and 40 specify the file prefixes of Lines_, Polygons_, and Points_ respectively.

The batch file ConvertSplit2mp downloaded in Step 1 of subsection Convert to .MP specifies Line, Point and Polygon in the filename fetch and won't run with that discrepancy.

You may also want to note in instructions 30,35 and 40 that the subsequent batch file is case sensitive (e.g. the prefix of "line" will not work while "Line" will). Depending on how you choose the change the batch file.

Thanks for your work on this.  :)

glendeni

  • Mapper
  • ***
  • Posts: 80
  • Karma: 0
    • View Profile
Re: Huge Update to Topo Tutorial: QGIS Support/Faster BAT Files/National Map
« Reply #6 on: January 06, 2018, 12:58:17 AM »
Want to thank you for all the time you put into creating this updated tutorial.  Seeing it motivated me to try comparing contours form the latest one-third arcsec data to the ones I've been using from the old CATOPO datafile for my area which has very complex topography (Big Sur, CA).  Following your directions, it was quite painless, using QGIS 2.18 (admittedly I already had some familiarity with that software).

So I only went through the "Part 2 - Elevation Data - Option 2" section.  For that, my only suggestions would be:

Using The National Map Viewer to Download the Data
() instead of "state" should use "area of interest, e.g. a state"

Processing the data - QGIS
() would help understanding at each step to give a description of what will be created following a "click OK'
 eg 12. "Click "OK" -> Click "OK to create a contour .shp shapefile
() after 8. need to insert: Click "OK"

Also, in QGIS I found that the specified filenames sometimes did not appear in the directory I expected, so had to hunt them down.  That may have been due to directories used in my last project being used instead of the the ones I expected.  Anyway, I think its good practice to click on the  "Select" button to make sure the file will be written to the directory you want.

Thanks again.

Jack
« Last Edit: January 07, 2018, 05:52:52 AM by glendeni »

Aodhfin

  • Jr. Member
  • **
  • Posts: 3
  • Karma: 0
    • View Profile
Re: Huge Update to Topo Tutorial: QGIS Support/Faster BAT Files/National Map
« Reply #7 on: November 03, 2018, 12:41:42 PM »
I am stuck on Elevation: Combine All The Contours Together
I downloaded contours for Mississippi (24 tiles)
Tried to merge using FWTools, hit the 4gb limit.
   (Note: Currently FWTools will crash if the file gets over 4gb; if this occurs you must split your contours into two separate folders and combine each half of the state separately.  For reference: for Arizona the file was only 3gb.)
Did in two chunks, succeeded.
Tutorial unclean on how to  combine each half of the state separately.  Tried to do with both QGIS & ArcMap 10.5 by combining shapefiles.  Neither worked.  Then both programs started giving errors on drawing each half.  Size limit on shapefiles is 2gb?

Help!

 

anything