Where do you get computer-readable track data?

[ApproximateKnowledge]

Long-time lurker around here who finally registered to ask a question. I'm working on a tool for the community that I think will be pretty well-received and have most of the code done, but I'm not sure where to get NHC data in a form that I can easily read via code.

I've seen people posting preliminary best-track data in the HURDAT format, as in:

20120930, 1800, , HU, 37.2N, 38.9W, 80, 978, 110, 90, 70, 80, 50, 40, 40, 40, 30, 25, 20, 25,

That would work well in the interim, since it's very easy to read, but I have no idea where to find it for current storms (only for post-analysis). I'd like the forecast points as well; I was able to find this page of GIS data, but that ends up being a lot _more_ than I need and I'd rather not fool around with a bunch of XML if I don't have to.

Ideally, what I'm looking for is a URL I can have the program access to read all the track data on a particular system so far, including a current forecast, in a code-friendly format. I only need positions, intensities, and optionally the type of cyclone (as in the third HURDAT column), though I can do without that last one.

[Extratropical94]

Long-time lurker around here who finally registered to ask a question. I'm working on a tool for the community that I think will be pretty well-received and have most of the code done, but I'm not sure where to get NHC data in a form that I can easily read via code.

I've seen people posting preliminary best-track data in the HURDAT format, as in:

20120930, 1800, , HU, 37.2N, 38.9W, 80, 978, 110, 90, 70, 80, 50, 40, 40, 40, 30, 25, 20, 25,

That would work well in the interim, since it's very easy to read, but I have no idea where to find it for current storms (only for post-analysis). I'd like the forecast points as well; I was able to find this page of GIS data, but that ends up being a lot _more_ than I need and I'd rather not fool around with a bunch of XML if I don't have to.

Ideally, what I'm looking for is a URL I can have the program access to read all the track data on a particular system so far, including a current forecast, in a code-friendly format. I only need positions, intensities, and optionally the type of cyclone (as in the third HURDAT column), though I can do without that last one.

The NHC FTP server has a folder with best track data of all NATL and EPAC systems within the current year in .dat file format.
It's not exactly the HURDAT2 format where all wind radii are in one line, but there are three separate entries for TS force, 50 knot and HU force wind radii for each 6 hour point. So that would require you to pick the data fields you need for your code from the file. It looks like this

AL, 11, 2017090312, , BEST, 0, 178N, 479W, 100, 969, HU, 34, NEQ, 70, 60, 40, 70, 1013, 220, 15, 120, 0, L, 0, , 0, 0, IRMA, D, 12, NEQ, 400, 120, 100, 360, genesis-num, 023,
AL, 11, 2017090312, , BEST, 0, 178N, 479W, 100, 969, HU, 50, NEQ, 40, 20, 20, 30, 1013, 220, 15, 120, 0, L, 0, , 0, 0, IRMA, D, 12, NEQ, 400, 120, 100, 360, genesis-num, 023,
AL, 11, 2017090312, , BEST, 0, 178N, 479W, 100, 969, HU, 64, NEQ, 20, 10, 10, 15, 1013, 220, 15, 120, 0, L, 0, , 0, 0, IRMA, D, 12, NEQ, 400, 120, 100, 360, genesis-num, 023,

The files are available at http://ftp.nhc.noaa.gov/atcf/btk/

Hope this helps

[talkon]

To add a bit, in case you want best track data for invests or storms in WPAC/SHEM (but not Indian Ocean) they are available at http://ftp.emc.ncep.noaa.gov/wd20vxt/hwrf-init/decks/?C=M;O=D

There are also track files of another format, like this:

11L IRMA 170903 1200 17.8N 47.9W ATL 100 969

which can be found on the Navy website in the respective basin and storm's folders here: https://nrlmry.navy.mil/tcdat/tc17/
and these include every storm including Indian Ocean storms but not invests.

Also, if you want data for just the latest cycle but for all active storms, you can find it here: https://nrlmry.navy.mil/tcdat/sectors/sector_file

[Extratropical94]

To add a bit, in case you want best track data for invests or storms in WPAC/SHEM (but not Indian Ocean) they are available at http://ftp.emc.ncep.noaa.gov/wd20vxt/hwrf-init/decks/?C=M;O=D
[...]

Good you mentioned that, I completely forgot about that EMC link. That also includes all the EPAC and ATL storm files from the NHC FTP site and files from previous years, so it's probably easier to use your link.

[ApproximateKnowledge]

The NHC FTP server has a folder with best track data of all NATL and EPAC systems within the current year in .dat file format.
It's not exactly the HURDAT2 format where all wind radii are in one line, but there are three separate entries for TS force, 50 knot and HU force wind radii for each 6 hour point. So that would require you to pick the data fields you need for your code from the file. It looks like this

AL, 11, 2017090312, , BEST, 0, 178N, 479W, 100, 969, HU, 34, NEQ, 70, 60, 40, 70, 1013, 220, 15, 120, 0, L, 0, , 0, 0, IRMA, D, 12, NEQ, 400, 120, 100, 360, genesis-num, 023,
AL, 11, 2017090312, , BEST, 0, 178N, 479W, 100, 969, HU, 50, NEQ, 40, 20, 20, 30, 1013, 220, 15, 120, 0, L, 0, , 0, 0, IRMA, D, 12, NEQ, 400, 120, 100, 360, genesis-num, 023,
AL, 11, 2017090312, , BEST, 0, 178N, 479W, 100, 969, HU, 64, NEQ, 20, 10, 10, 15, 1013, 220, 15, 120, 0, L, 0, , 0, 0, IRMA, D, 12, NEQ, 400, 120, 100, 360, genesis-num, 023,

The files are available at http://ftp.nhc.noaa.gov/atcf/btk/

Hope this helps

It does! That's a good start for the track data itself (though no forecast cone). Is there documentation for the line format?

Re: other posts - for now I'm only looking at ATL data, but later on I might expand worldwide. Wouldn't take much work to adapt.

[Extratropical94]

[...]

It does! That's a good start for the track data itself (though no forecast cone). Is there documentation for the line format?

Re: other posts - for now I'm only looking at ATL data, but later on I might expand worldwide. Wouldn't take much work to adapt.

There is, have a look at this document

https://www.nrlmry.navy.mil/atcf_web/do ... abdeck.txt

[ApproximateKnowledge]

Okay. That takes care of track data. How about NHC forecast points?

[Extratropical94]

Okay. That takes care of track data. How about NHC forecast points?

As far as I know, the forecast points are only included in the NHC forecast advisories and the discussions. The latter ones are a bit easier to process because all the forecast points and intensities are in one block. If you want forecast data for wind radii too then you'll need to use the forecast advisories though.

Unfortunately I don't know about the structure of the URLs of those products on the NHC page, but there is an alternate way to get them from the NWS FTP server.

The discussions are uploaded here
http://tgftp.nws.noaa.gov/data/raw/wt/wtnt41.knhc.tcd.at1.txt
http://tgftp.nws.noaa.gov/data/raw/wt/wtnt42.knhc.tcd.at2.txt
http://tgftp.nws.noaa.gov/data/raw/wt/wtnt43.knhc.tcd.at3.txt
http://tgftp.nws.noaa.gov/data/raw/wt/wtnt44.knhc.tcd.at4.txt
http://tgftp.nws.noaa.gov/data/raw/wt/wtnt45.knhc.tcd.at5.txt

For each new storm, the header number (41-45) is increased by 1. After 45 the number 41 is used again (for example Franklin = 42, Gert = 43, Harvey = 44, Ten = 45, Irma = 41 and so on).
So your code would need to find out which header to use for which storm.

The forecast advisories are located at
http://tgftp.nws.noaa.gov/data/raw/wt/wtnt21.knhc.tcm.at1.txt
http://tgftp.nws.noaa.gov/data/raw/wt/wtnt22.knhc.tcm.at2.txt
http://tgftp.nws.noaa.gov/data/raw/wt/wtnt23.knhc.tcm.at3.txt
http://tgftp.nws.noaa.gov/data/raw/wt/wtnt24.knhc.tcm.at4.txt
http://tgftp.nws.noaa.gov/data/raw/wt/wtnt25.knhc.tcm.at5.txt

That's a little more complicated but at least the URLs don't change.

If you want to include other data from NHC documents, you can find those by looking up the header of each product on the NHC page (it's that 4 letter-2 digit code at the beginning of every product) and navigating to that document on the NWS FTP server: http://tgftp.nws.noaa.gov/data/raw/

[ApproximateKnowledge]

That's probably workable. I'd rather search for a known string than screw around with XML, and I'm pretty sure I can pull what I need out of the forecast/advisories if I can figure out the structure for URLs. Discussions will be enough if I can't do that. Thanks a ton!

[Extratropical94]

No problem. Good luck with your project.

[ApproximateKnowledge]

Expect to see it around here sometime in the next few weeks. I'm pretty excited about it

[ApproximateKnowledge]

Okay. That takes care of track data. How about NHC forecast points?

As far as I know, the forecast points are only included in the NHC forecast advisories and the discussions. The latter ones are a bit easier to process because all the forecast points and intensities are in one block. If you want forecast data for wind radii too then you'll need to use the forecast advisories though.

Unfortunately I don't know about the structure of the URLs of those products on the NHC page, but there is an alternate way to get them from the NWS FTP server.

The discussions are uploaded here
http://tgftp.nws.noaa.gov/data/raw/wt/wtnt41.knhc.tcd.at1.txt
http://tgftp.nws.noaa.gov/data/raw/wt/wtnt42.knhc.tcd.at2.txt
http://tgftp.nws.noaa.gov/data/raw/wt/wtnt43.knhc.tcd.at3.txt
http://tgftp.nws.noaa.gov/data/raw/wt/wtnt44.knhc.tcd.at4.txt
http://tgftp.nws.noaa.gov/data/raw/wt/wtnt45.knhc.tcd.at5.txt

For each new storm, the header number (41-45) is increased by 1. After 45 the number 41 is used again (for example Franklin = 42, Gert = 43, Harvey = 44, Ten = 45, Irma = 41 and so on).
So your code would need to find out which header to use for which storm.

The forecast advisories are located at
http://tgftp.nws.noaa.gov/data/raw/wt/wtnt21.knhc.tcm.at1.txt
http://tgftp.nws.noaa.gov/data/raw/wt/wtnt22.knhc.tcm.at2.txt
http://tgftp.nws.noaa.gov/data/raw/wt/wtnt23.knhc.tcm.at3.txt
http://tgftp.nws.noaa.gov/data/raw/wt/wtnt24.knhc.tcm.at4.txt
http://tgftp.nws.noaa.gov/data/raw/wt/wtnt25.knhc.tcm.at5.txt

That's a little more complicated but at least the URLs don't change.

If you want to include other data from NHC documents, you can find those by looking up the header of each product on the NHC page (it's that 4 letter-2 digit code at the beginning of every product) and navigating to that document on the NWS FTP server: http://tgftp.nws.noaa.gov/data/raw/

Okay so I've been playing with these and it looks like those are only the _current_ discussions and forecast/advisories. Is there a way to get all of them for a storm?

[Extratropical94]

Okay. That takes care of track data. How about NHC forecast points?

[...]

Okay so I've been playing with these and it looks like those are only the _current_ discussions and forecast/advisories. Is there a way to get all of them for a storm?

I'm not sure if they are archived on the same server. The NHC has an advisory archive on their website though.

http://www.nhc.noaa.gov/archive/2017/al11/al112017.discus.026.shtml?
http://www.nhc.noaa.gov/archive/2017/al11/al112017.fstadv.026.shtml?

Only the storm number (and year) and advisory number have to be changed in the URL, but it might be more complicated to extract the text from all the html code than from a page with only the advisory text on it. (Or maybe it is not, I don't know I'm not a programmer )