STORM2K • View topic - Verified Accuweather Forecasts vs. NHC - Ophelia

Find the Weather for any
City, State, Zip Code, or ICAO

 
 

It is currently Fri Oct 31, 2014 6:58 am

All times are UTC - 5 hours


Forum rules


The posts in this forum are NOT official forecast and should not be used as such. They are just the opinion of the poster and may or may not be backed by sound meteorological data. They are NOT endorsed by any professional institution or storm2k.org. For official information, please refer to the NHC and NWS products.[td100]



Post new topic Reply to topic  [ 36 posts ]  Go to page 1, 2  Next  Last
Author Message
 Post subject: Verified Accuweather Forecasts vs. NHC - Ophelia
PostPosted: Tue Sep 27, 2005 2:43 pm 
Offline
S2K Supporter
S2K Supporter

Joined: Sat Oct 12, 2002 7:51 pm
Posts: 2658
Location: SE Florida
As folks may know, a lead forecaster for Accuweather who also does forecasting from time to time on FOX news started putting out quasi-verifable forecasts a few storms ago. For a period of 10 days, this forecaster made lat/long forecasts for Ophelia.

I say quasi verifiable...because while lat/long pairs were forecast...sometimes intensity was expressd in knots, sometimes in pressure (MB), sometimes both and sometimes neither.

So while there additional data points, they couldn't be used to verify intensity.

However...I did go back and collect this data, import it into a database, and run verification statistics for same-forecast period forecasts from the NHC...thereby comparing apples to apples.

Here are the 2 data sets...the NHC vs. A specific Accuweather forecaster tracks from the same initial time and same verufy time for all Ophelia forecasts from Accuweather from 9/6/2005 to 9/16/2005.

I will let the numbers...speak...for...themselves.

Code:
Model Name               V Time     Cases       Avg Error
NHC official forecast 024 hours   10 cases 42.05 nm
NHC official forecast 048 hours   9 cases 81.99 nm
NHC official forecast 072 hours   8 cases 99.90 nm
NHC official forecast 096 hours    7 cases 153.97 nm
NHC official forecast 120 hours    6 cases 189.62 nm

Model         
Accuwx 024 hours 10 cases 46.36 nm -10.26% skill vs NHC
Accuwx 048 hours 9 cases 106.22 nm -29.54% skill vs NHC
Accuwx 072 hours 8 cases 177.32 nm -77.50% skill vs NHC
Accuwx 096 hours 6 cases 266.20 nm -72.89% skill vs NHC
Accuwx 120 hours 5 cases 385.77 nm -103.44% skill vs NHC


I have a pretty good feeling Rita's numbers will be similar.

There is of course...some margin for error here since I had to grab data from files that are less than clean. But the forecast error verification vs. best track should be good within a couple of nautical miles.

Data files are available upon request...

MW

_________________
Updating on the twitter now: http://www.twitter.com/@watkinstrack


Top
 Profile  
 
 Post subject:
PostPosted: Tue Sep 27, 2005 2:54 pm 
Offline
Category 5
Category 5
User avatar

Joined: Wed Aug 11, 2004 12:14 pm
Posts: 8672
Location: Fort Lauderdale, FL
Glad you were able to get hold of the numbers, Mike. I've been asking supporters of said forecaster to please post just his forecast numbers (time position and intensity), but to date none have obliged.

This result is more or less what I expected.


Top
 Profile  
 
 Post subject:
PostPosted: Tue Sep 27, 2005 2:56 pm 
Offline
Category 5
Category 5
User avatar

Joined: Tue Aug 16, 2005 8:59 pm
Posts: 1588
Location: St. Petersburg, FL
MWatkins, a question to understand methodology:

The "10 cases" refers to the number of equivalent 24-hour forecast points you looked at, and the error is the avg. of the error in the cases combined.

Would you also be able to look at the best and worst forecast point of all the cases -- for example, the 48-hour point on the fourth day was the best for the NHC, their worst was the 120-hr. forecast on the first day. Just wondering if there are significant outliers, or if the trend holds throughout.

The analysis is very interesting. Thank you. NHC rocks, and they always forecast pressure, winds, extent, and track consistently and always face verification.


Top
 Profile  
 
 Post subject:
PostPosted: Tue Sep 27, 2005 3:06 pm 
Offline
Category 5
Category 5
User avatar

Joined: Wed Aug 11, 2004 12:14 pm
Posts: 8672
Location: Fort Lauderdale, FL
jax wrote:
i wouldn't be surprised is one of the many other 120hr
forcasts issued by both in the many many days involving
ophilia where flipped in the results... or any of the other
storms for that matter... we have all been WAY wrong many
times this year... I can recall AW being WAY closed to NO
than NHC durring the approch of Katrina... Lets not be so
subjective... please...


HUH??? How is comparing the verifications numerically "subjective"?

Quite the opposite, I'd say. Your impression is subjective. Mike's analysis is objective.


Top
 Profile  
 
 Post subject:
PostPosted: Tue Sep 27, 2005 3:07 pm 
Offline
S2K Supporter
S2K Supporter

Joined: Sat Oct 12, 2002 7:51 pm
Posts: 2658
Location: SE Florida
jax wrote:
i wouldn't be surprised is one of the many other 120hr
forcasts issued by both in the many many days involving
ophilia where flipped in the results... or any of the other
storms for that matter... we have all been WAY wrong many
times this year... I can recall AW being WAY closed to NO
than NHC durring the approch of Katrina... Lets not be so
subjective... please...


This is every single forecast for this storm from that company. This is about as objective as you can get. If someone wants to round up the Katrina data for me, because like Jan I have been looking for it for a month now, I will be happy to verify the track out. However...this is the data I have...so I verified it.

Forecast verification...is an objective process. I'm sorry you're not happy with the results for whatever the reason, but this is what it is. If you want to go and collect the data and run it and get a DIFFERENT result...let me know and I will check my process for errors. But it has been checked before and it works.

What is probably hurting verification stats more than anything is the Accu forecast that took Ophelia to New Orleans as a 965MB hurricane.

MW

_________________
Updating on the twitter now: http://www.twitter.com/@watkinstrack


Top
 Profile  
 
 Post subject:
PostPosted: Tue Sep 27, 2005 3:09 pm 
i stand corrected... i misread the graph... i made the error of
assuming you had just picked a single 120hr forcast from each...
i see now that they are compiled. my appologies...


Last edited by jax on Tue Sep 27, 2005 3:09 pm, edited 1 time in total.

Top
  
 
 Post subject:
PostPosted: Tue Sep 27, 2005 3:09 pm 
Offline
Category 5
Category 5
User avatar

Joined: Sat Apr 05, 2003 11:41 pm
Posts: 3627
Location: NOT a State-caster
Still, the value in said forecaster isn't in points. He should know better than to try. But give me a Joe B. "10 days out" discussion of a potential threat over "Tropical Development is not Expected through Next Month" or tracks that need to shift 150 miles or more 2 days prior to a Cat 4 landfall any day. I follow the NHC for short term stuff as they are usually superior and have done an overall really good job this year (exception being Cindy), but Bastardi is clearly better than most of them at articulating upcoming threats be it hype or not.

Thanks for the numbers though.

Steve


Top
 Profile  
 
 Post subject:
PostPosted: Tue Sep 27, 2005 3:21 pm 
Offline
S2K Supporter
S2K Supporter

Joined: Sat Oct 12, 2002 7:51 pm
Posts: 2658
Location: SE Florida
Recurve wrote:
MWatkins, a question to understand methodology:

The "10 cases" refers to the number of equivalent 24-hour forecast points you looked at, and the error is the avg. of the error in the cases combined.


Yes...10 cases reflects the number of forecasts. The total error in nautical miles for that verify time are divided by the total number of cases for that verify time to get the average error.

Quote:
Would you also be able to look at the best and worst forecast point of all the cases -- for example, the 48-hour point on the fourth day was the best for the NHC, their worst was the 120-hr. forecast on the first day. Just wondering if there are significant outliers, or if the trend holds throughout.


That's possible...however...a bad forecast does tend to screw up the verification numbers. This why the NHC tends to keep systems stationary when there is a huge split in the guidance...because jumping on the wrong side of the guidance envelope could create huge track errors. This was the case with the forecast from 9/6 from Accuweather...which had the storm approaching New Orleans. This caused a huge track error. Unfortunately...when comparing 2 samples like this there are going to be samples that diverge at different verify times...this timing makes it difficult to take one track out without comparing it against all other tracks.

For example...if you windshield wiper back and forth between 2 diverging tracks...one side is going to be right and perhaps infer skill...when the other side would be terrible and probably not beat climatology. For example...here are the 72 hour forecast errors for each date the 72 hour forecast verified for accuweather...lots of varience...some good and some really bad forecast totals.

2005091012 410.9193962
2005091112 183.2364056
2005091212 56.26543937
2005091312 104.210085
2005091412 65.99847839
2005091512 172.3875466
2005091612 300.2463867
2005091712 125.3340041

Here are the NHC's for the same V-time:

2005091012 199.4472897
2005091112 54.5670838
2005091212 87.36771702
2005091312 113.0261428
2005091412 92.41210801
2005091512 48.9927628
2005091612 117.0582734
2005091712 86.35202784

Much less variance in the totals.

Quote:
The analysis is very interesting. Thank you. NHC rocks, and they always forecast pressure, winds, extent, and track consistently and always face verification.


Yes...they always keep score.

MW

_________________
Updating on the twitter now: http://www.twitter.com/@watkinstrack


Top
 Profile  
 
 Post subject:
PostPosted: Tue Sep 27, 2005 3:24 pm 
Offline
S2K Supporter
S2K Supporter

Joined: Sat Oct 12, 2002 7:51 pm
Posts: 2658
Location: SE Florida
jax wrote:
i stand corrected... i misread the graph... i made the error of
assuming you had just picked a single 120hr forcast from each...
i see now that they are compiled. my appologies...


No problem...perhaps I could have explained it a little better as well.

MW

_________________
Updating on the twitter now: http://www.twitter.com/@watkinstrack


Top
 Profile  
 
 Post subject:
PostPosted: Tue Sep 27, 2005 3:32 pm 
Offline
Category 5
Category 5
User avatar

Joined: Wed Aug 11, 2004 12:14 pm
Posts: 8672
Location: Fort Lauderdale, FL
Steve wrote:
Still, the value in said forecaster isn't in points. He should know better than to try. But give me a Joe B. "10 days out" discussion of a potential threat over "Tropical Development is not Expected through Next Month" or tracks that need to shift 150 miles or more 2 days prior to a Cat 4 landfall any day. I follow the NHC for short term stuff as they are usually superior and have done an overall really good job this year (exception being Cindy), but Bastardi is clearly better than most of them at articulating upcoming threats be it hype or not.

Thanks for the numbers though.

Steve


Well that's the point, isn't it? I said long ago during one of our periodic JB related eruptions that he does some things very well and I learned a fair amount from his columns and videos in the past. But he's not remotely comparable to the NHC in doing what they do - specific 3 to 5 day track and intensity forecasts. Thus he has no business taking jabs at them and rolling his eyes and in general actively participating in his employer's dishonest campaign to discredit them and the NWS in general.

If he would just cut all that cr*p, I'd have no problem with him at all.


Top
 Profile  
 
 Post subject:
PostPosted: Tue Sep 27, 2005 3:36 pm 
Offline
Tropical Storm
Tropical Storm

Joined: Thu Aug 25, 2005 9:58 pm
Posts: 109
The NHC makes fewer errors because they are wise enough to hold their tongue when they don't know something.

If you throw out the biggest error on each side, my guess is that the NHC would still have a huge lead.


Top
 Profile  
 
 Post subject:
PostPosted: Tue Sep 27, 2005 3:47 pm 
Offline
S2K Supporter
S2K Supporter
User avatar

Joined: Wed Oct 09, 2002 8:45 pm
Posts: 906
Location: Sanford, Florida
As I said in another post I intend to keep the data and the forecasts next year for every storm for both Accuweather and the NHC. At the end of the year I will post the data for each storm for both and let the chips fall where they may. I'm also going to capture every "quote" made by a certain forecaster and track the error in miles if any. If nothing else should give me a good refresher course in Excel.


Top
 Profile  
 
 Post subject:
PostPosted: Tue Sep 27, 2005 3:48 pm 
NHC makes fewer errors because they change the forecast track every 6 hours. As the models jump around, their track does likewise. Obviously, there will be a superior correlation, given the fact that their forecast is not static--this is a huge strength, but likewise, a weakness at the same time---this is one of the reasons that evacuations are difficult, as it is hard to pin down a landfall commitment within 72 hours, if their track keeps changing.

This is not intended as a criticism, as I feel they do a fantastic job, given that most of the computer models they have to work with are utterly worthless. They basically can rely upon the GFDL and the FSU Superensemble--other than that, the tropical models are dated and unreliable.


Top
  
 
 Post subject:
PostPosted: Tue Sep 27, 2005 3:51 pm 
Offline
Storm2k Executive
Storm2k Executive
User avatar

Joined: Wed Jul 06, 2005 12:32 pm
Posts: 7039
Location: The Woodlands, TX
x-y-no wrote:
Glad you were able to get hold of the numbers, Mike. I've been asking supporters of said forecaster to please post just his forecast numbers (time position and intensity), but to date none have obliged.

This result is more or less what I expected.


There is a simple reason for that, unless the posts are on the free Yahoo site, etc., as sometimes posted here, his columns/forecasts are copyrighted material. BIG disclaimer on it.


Top
 Profile  
 
 Post subject:
PostPosted: Tue Sep 27, 2005 3:57 pm 
Offline
Category 5
Category 5
User avatar

Joined: Wed Aug 11, 2004 12:14 pm
Posts: 8672
Location: Fort Lauderdale, FL
jschlitz wrote:
x-y-no wrote:
Glad you were able to get hold of the numbers, Mike. I've been asking supporters of said forecaster to please post just his forecast numbers (time position and intensity), but to date none have obliged.

This result is more or less what I expected.


There is a simple reason for that, unless the posts are on the free Yahoo site, etc., as sometimes posted here, his columns/forecasts are copyrighted material. BIG disclaimer on it.


There is such a thing as "fair use." As long as you do not copy big chunks of his column (and all I'm asking for is the numbers - time, latitude, longitude and intensity - nothing else) and you're not doing it for profit, there's no violation of copyright.


Top
 Profile  
 
 Post subject:
PostPosted: Tue Sep 27, 2005 4:03 pm 
Offline
Storm2k Executive
Storm2k Executive
User avatar

Joined: Wed Jul 06, 2005 12:32 pm
Posts: 7039
Location: The Woodlands, TX
x-y-no wrote:
jschlitz wrote:
x-y-no wrote:
Glad you were able to get hold of the numbers, Mike. I've been asking supporters of said forecaster to please post just his forecast numbers (time position and intensity), but to date none have obliged.

This result is more or less what I expected.


There is a simple reason for that, unless the posts are on the free Yahoo site, etc., as sometimes posted here, his columns/forecasts are copyrighted material. BIG disclaimer on it.


There is such a thing as "fair use." As long as you do not copy big chunks of his column (and all I'm asking for is the numbers - time, latitude, longitude and intensity - nothing else) and you're not doing it for profit, there's no violation of copyright.


I'm not defending the practice; just stating what the disclaimer says and it says "in part or in whole". Why would they allow people to publish "just the points" and not anything else?? What if Sally Sue only wants to see his Teleconnection discussion but nothing about the tropics? That's a very, very gray area from a legal POV so they have a pretty stringent disclaimer.

Like it or not, if you wanna see it, you gotta pay. That's the whole purpose of AccuWx's existence.


Top
 Profile  
 
 Post subject:
PostPosted: Tue Sep 27, 2005 4:07 pm 
Offline
Category 5
Category 5
User avatar

Joined: Wed Aug 11, 2004 12:14 pm
Posts: 8672
Location: Fort Lauderdale, FL
jschlitz wrote:
I'm not defending the practice; just stating what the disclaimer says and it says "in part or in whole". Why would they allow people to publish "just the points" and not anything else?? What if Sally Sue only wants to see his Teleconnection discussion but nothing about the tropics? That's a very, very gray area from a legal POV so they have a pretty stringent disclaimer.

Like it or not, if you wanna see it, you gotta pay. That's the whole purpose of AccuWx's existence.


They can say whatever they want. but they can't enforce it. Copying and publishing small portions of copyrighted material with added commentary or analysis is "fair use" and there's not a thing in the world they can do about it.

EDIT:

Specifically, see: http://www.copyright.gov/fls/fl102.html

Quote:
One of the rights accorded to the owner of copyright is the right to reproduce or to authorize others to reproduce the work in copies or phonorecords. This right is subject to certain limitations found in sections 107 through 118 of the copyright act (title 17, U.S. Code). One of the more important limitations is the doctrine of “fair use.” Although fair use was not mentioned in the previous copyright law, the doctrine has developed through a substantial number of court decisions over the years. This doctrine has been codified in section 107 of the copyright law.

Section 107 contains a list of the various purposes for which the reproduction of a particular work may be considered “fair,” such as criticism, comment, news reporting, teaching, scholarship, and research.


Since our purpose here in wanting these numbers is for research, our use of them is covered under the doctrine.


Top
 Profile  
 
 Post subject:
PostPosted: Tue Sep 27, 2005 4:21 pm 
Offline
Storm2k Forecaster
Storm2k Forecaster
User avatar

Joined: Sat Oct 12, 2002 5:44 pm
Posts: 1287
Location: Lexington, Virginia
MW and all:

Here's some additional Katrina data I've collected on TPC's F/As

Verification time / data set / Error Margin in Miles

12/30/28
24/26/64
36/26/71
48/23/107
72/19/201
96/15/256
120/11/266

OVERALL: 115

If one discounts their early errant Panhandle forecasts, the TPC rendered superior service to anyone willing to listen.

Scott
Mid-Atlantic WX.com
Lexington, VA
http://www.midatlanticwx.com/hurricane.htm


Top
 Profile  
 
 Post subject:
PostPosted: Tue Sep 27, 2005 4:24 pm 
Offline
Storm2k Executive
Storm2k Executive
User avatar

Joined: Wed Jul 06, 2005 12:32 pm
Posts: 7039
Location: The Woodlands, TX
x-y-no wrote:
They can say whatever they want. but they can't enforce it. Copying and publishing small portions of copyrighted material with added commentary or analysis is "fair use" and there's not a thing in the world they can do about it.


It depends. It's one thing if someone occasionally posts a snippet here or there with commentary, that would likely hold-up in court as "fair use". Such cases can be difficult to prove as violating the legal definition of "fair use" and would likely be never pursued.

However, if a paying subscriber posts his forecasts or points on a consistent or daily basis on a free site, that is legally considered abuse of "fair use". In that case, the burden of proof falls upon the defendant to prove they had substantive reason to do so (i.e., academic research, etc.). There are legal exceptions specifically for those cases but posting on an internet message board so people can "see" how accurate they are does not follow those guidelines. In that case, the offender would likely be fined if AccuWx pursued the issue.


Top
 Profile  
 
 Post subject:
PostPosted: Tue Sep 27, 2005 4:35 pm 
Offline
Category 5
Category 5
User avatar

Joined: Wed Aug 11, 2004 12:14 pm
Posts: 8672
Location: Fort Lauderdale, FL
jschlitz wrote:
x-y-no wrote:
They can say whatever they want. but they can't enforce it. Copying and publishing small portions of copyrighted material with added commentary or analysis is "fair use" and there's not a thing in the world they can do about it.


It depends. It's one thing if someone occasionally posts a snippet here or there with commentary, that would likely hold-up in court as "fair use". Such cases can be difficult to prove as violating the legal definition of "fair use" and would likely be never pursued.

However, if a paying subscriber posts his forecasts or points on a consistent or daily basis on a free site, that is legally considered abuse of "fair use". In that case, the burden of proof falls upon the defendant to prove they had substantive reason to do so (i.e., academic research, etc.). There are legal exceptions specifically for those cases but posting on an internet message board so people can "see" how accurate they are does not follow those guidelines. In that case, the offender would likely be fined if AccuWx pursued the issue.


I'm sorry, but your interpretation of the law is just plain wrong.

And even if one somehow managed to ignore the fact that this use fits squarely within the research exception, there's this additional point (again from the copyright office):

Quote:
Copyright protects the particular way an author has expressed himself; it does not extend to any ideas, systems, or factual information conveyed in the work.


All I am asking for is the raw numbers (i.e. factual information or ideas) embodied in the specific forecast points and times. Nothing more.

There is simply not the slightest question this is allowed under copyright law.


Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 36 posts ]  Go to page 1, 2  Next  Last

All times are UTC - 5 hours


Who is online

Users browsing this forum: beoumont, wxman57 and 21 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum

Search for:
Jump to:  
cron
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group