NOAA tgftp file request issues
Posted: Sun Aug 16, 2020 11:40 am
So I have a couple of automated file grabbers set up that download various data files from NOAA's fileserver at tgftp.nws.noaa.gov every 10-60 minutes. Usually, I download a list of ~50 individual files sequentially.
I noticed that, at some point yesterday (August 15), the automated scripts stopped updating. I looked into the logs and found that after downloading somewhere between 1 and 5 files, the download request for the subsequent file takes forever. No error messages, disconnects, timeouts or other indicators of what's happening.
So far, I've tried the downloads with wget and curl on both a Windows and a Linux machine, as well as the Python modules requests and urllib3. I've also run those scripts from a different network and remotely on a machine at my university to rule out firewall/network issues, but to no avail. The downloads always start to hang after successfully downloading 1-5 files.
Has anyone else encountered this issue recently?
I noticed that, at some point yesterday (August 15), the automated scripts stopped updating. I looked into the logs and found that after downloading somewhere between 1 and 5 files, the download request for the subsequent file takes forever. No error messages, disconnects, timeouts or other indicators of what's happening.
So far, I've tried the downloads with wget and curl on both a Windows and a Linux machine, as well as the Python modules requests and urllib3. I've also run those scripts from a different network and remotely on a machine at my university to rule out firewall/network issues, but to no avail. The downloads always start to hang after successfully downloading 1-5 files.
Has anyone else encountered this issue recently?