120 Hour Model Stats...Good Reading

[MWatkins]

At last! My 2003 database is complete...all 19 storms with best track data and model guidance has been loaded up. I'm still testing some of the totals...and making sure the great circle calculations are right...but so far so good.

Since we've gone to the 5 day forecasts (and again...this data is preliminary)...I took a look at the error rates of the Official TPC forecast against the models that are generally available to the public. And to be honest...compared to the globals and the other guidance available...the TPC track guidance is AWFUL. Check out these 120 hour verification stats:

MODEL_NUM V_Time Cases Avg_Error
TPC Official 120 111 225
GFDL 120 71 262
BAMM 120 182 425
LBAR 120 154 586
BAMD 120 181 605
A98E 120 182 652

Note that the LBAR drops systems faster than the other guidance...and the models were often run on systems before and after dissipation.

As much bashing as the GFDL takes...compared to the other models at 120...it kicked their you know what.

BTW it looks like the AVN (GFS) was clearly the best global (again)...not ready to print that out yet...and the 120 error is almost identical to the TPC totals.

More to come....

MW

[MWatkins]

Stats in an eaiser to read format:

Models at 120 Hours from Initial Time:

TPC Official 111 Forecasts, 225 NM Error

GFDL 71 Forecasts, 262 NM Error

BAMM 182 Forecasts, 425 NM Error

LBAR 154 Forecasts, 586 NM Error

BAMD 181 Forecasts, 605 NM Error

A98E 182 Forecasts, 652 NM Error

MW

[cycloneye]

Great informatiom Mike that you are sharing with us about how the models performed in the 2003 hurricane season.

[wxman57]

...I took a look at the error rates of the Official TPC forecast against the models that are generally available to the public. And to be honest...compared to the globals and the other guidance available...the TPC track guidance is AWFUL. Check out these 120 hour verification stats:

MODEL_NUM V_Time Cases Avg_Error
TPC Official 120 111 225
GFDL 120 71 262
BAMM 120 182 425
LBAR 120 154 586
BAMD 120 181 605
A98E 120 182 652

BTW it looks like the AVN (GFS) was clearly the best global (again)...not ready to print that out yet...and the 120 error is almost identical to the TPC totals.

Perhaps I'm missing something -- in your stats above, it looks like the TPC track guidance beat any model (225 mile error at 120hrs). How is it "awful" compared to the global land other models? Also, you say it looks like the GFS was the best global model, but I don't see any GFS in the table above.

[MWatkins]

No...sorry about that...the TPC official forecast error rate is actually excellent...the TPC forecast Suite is what is awful. However...since I'm picking up runs that occured before upgrades and after downgrades...this could be contributing to the error rate.

As noted above...I'm not done confirming error rates for the globals...but FYI at least for now the 120 hour error rate for the GFS/AVN Operational model looks to be 224 miles in 84 cases.

MW

[wxman57]

No...sorry about that...the TPC official forecast error rate is actually excellent...the TPC forecast Suite is what is awful. However...since I'm picking up runs that occured before upgrades and after downgrades...this could be contributing to the error rate.

As noted above...I'm not done confirming error rates for the globals...but FYI at least for now the 120 hour error rate for the GFS/AVN Operational model looks to be 224 miles in 84 cases.

MW

Ok, keep up the good work. I'll be interested to see the full results.

[hurricanetrack]

MW:

The GFS was apparently worked on last year and into this year so that it does better in the tropics. I attended a seminar at the '03 National Hurricane Conference and Dr. Pasch mentioned that the GFDL and the GFS were both upgraded or tweaked (however you want to call it).

But- what I don't get is why the GFS would show Fabian or Isabel as a 1008 mb low or a hump on the isobar line instead of a deep tropical cyclone. You even mentioned this on some of your posts. You were upset as to why Fabian was seemingly initialized at 1008 or 1012 mb on the GFS surface maps. I think we were all scratching our heads on that one.

Next April in Orlando, I will ask the panel about how well THEY think the GFS did. We all know that it is just a shot in the dark after 7 or 8 days. But I would think that the 5 day GFS is probably coming along.

I'll ask next April from the NHC folks to see what improvements paid off.

[OtherHD]

Yet more proof that the A98E sucks...interesting stats Mike, thanks!

[wxman57]

MW:

The GFS was apparently worked on last year and into this year so that it does better in the tropics. I attended a seminar at the '03 National Hurricane Conference and Dr. Pasch mentioned that the GFDL and the GFS were both upgraded or tweaked (however you want to call it).

But- what I don't get is why the GFS would show Fabian or Isabel as a 1008 mb low or a hump on the isobar line instead of a deep tropical cyclone. You even mentioned this on some of your posts. You were upset as to why Fabian was seemingly initialized at 1008 or 1012 mb on the GFS surface maps. I think we were all scratching our heads on that one.

Next April in Orlando, I will ask the panel about how well THEY think the GFS did. We all know that it is just a shot in the dark after 7 or 8 days. But I would think that the 5 day GFS is probably coming along.

I'll ask next April from the NHC folks to see what improvements paid off.

I attended the same talks last year in New Orleans. I was expecting more of the GFDL this past season after hearing how improved it was. But it seemed to lose just about every incipient storm.

Perhaps I'll run into you in Orlando. I'll be attending the NHC. Will you be at the 26th conference on tropical meteorology in Miami the first week of May?

[MWatkins]

MW:

The GFS was apparently worked on last year and into this year so that it does better in the tropics. I attended a seminar at the '03 National Hurricane Conference and Dr. Pasch mentioned that the GFDL and the GFS were both upgraded or tweaked (however you want to call it).

But- what I don't get is why the GFS would show Fabian or Isabel as a 1008 mb low or a hump on the isobar line instead of a deep tropical cyclone. You even mentioned this on some of your posts. You were upset as to why Fabian was seemingly initialized at 1008 or 1012 mb on the GFS surface maps. I think we were all scratching our heads on that one.

Next April in Orlando, I will ask the panel about how well THEY think the GFS did. We all know that it is just a shot in the dark after 7 or 8 days. But I would think that the 5 day GFS is probably coming along.

I'll ask next April from the NHC folks to see what improvements paid off.

Yep...I was a little more than upset about the initialization thing...to the point that I thought perhaps it was a conspiricy to take the hurricane out of the model so as not to scare us helpless citizens. I still wonder a little.

To your point the GFS...just like all of the other models...is not so great with intesity. In the case of Izzy...I think the initialization problems were a proverbial blip on the radar screen. Relatively speaking...in terms of a track forecast...the GFS is excellent. And it certinally opt-performed every other model including the hurriane specific GFDL. Unfortunately for intensity...I can't get any data from the GFDL but it seems to either drop a system all together or over-develop it. If anyone has some hard data on GFDL intensity performance I'd love to see it.

Anyway...it's the best we have and it has improved significantly over the last 5 years.

I may end up in orlando next spring. Maybe I'll run into you...

MW