Question: Which Forecasting Models Did the Best in 2004?

[gatorcane]

AVN, NOGAPS, UKMET, GFDL, GFDI, LBAR, BAM, NHC90/NHC91, or
CLIPER?

I can't seem to find this information but I'm curious...will this answer impact which models NHC will rely on more or will they continue the same emphasis as last year?

[LSU2001]

I am not much of a model person but it sure seemed that the canadian model performed best in predicting paths. I know a lot of the other models changed around on path alot. I always waited to see the FSU super ensemble it seemed to perform the best of everything. But I remind you that I am only a rank amature and this may not be the case at all.
Tim

[StormChasr]

All of them had significant errors, inasmuch as it is truly impossible to predict landfalls several days out. There are too many variables.

[MWatkins]

Not counting the consensus models....

The folks at the hurricane center have said a couple of times now that the FSU Superensemble did the best track wise. After that...I believe the GFDL performed the best...followed by the GFS. Wait...I think I have a link for that...lemee check.

Ah...here is the full detail on the model verification for 2004:

http://www.nhc.noaa.gov/verification/pdfs/2004.pdf

MW

[gatorcane]

Thanks Watkins, this is what I was looking for

[Derecho]

The Canadian is generally a horrible joke at tropical forecasting routinely producing weird outlier forecasts well outside tight consensuses of the other models; Isabel is a good example.

Last year for a couple of storms, at a couple of forecast points, it wasn't as completely horrible as it usually is; I think for Charley and Ivan mainly. This I think led Ortt to get on a Hobby Horse talking about how great the Canadian is and therfore the perception that it did well overall took hold; but the verification stats show it wasn't very good relative to the others overall; in the long term I suspect Charley and Ivan will end up being flukes.

Interesting how of the raw models the GFDL is superior yet it's bashed so much; the perception of it is ruined by it running 4 times a day (more oppportunity for a goofy forecast now and then) and the many runs of the GFDL for weak INVESTs that result in wacky tracks.

Unfortunate that the FSU superensemble is the one model completely unavaliable to the public. Interestingly from the linked PDF it notes that the previous NHC forecast is a component of the FSU superensemble, and it actually comes out too late to use to forecast, so we really likely aren't missing as much as we imagine.

However, the GUNA performed almost as well and all the components of it are avaliable to the public (and Stomtrakker can calculate it automatically for you.)

One thing that PDF has ALMOST totally cleared up for me is the nature of the GUNA; I always asssumed it was as straight average, but I chatted with a met student who said a prof of his insisted GUNA was a complex weighted average which would mean the GUNA that Stormtrakker caculates wasn't the "real" GUNA; but from the way GUNA is listed in the PDF and talked about it seems it is in fact a plain average.

[Derek Ortt]

and I can give just as many examples where the GFS is a crying joke.

Ivan was a total mess... carried Ivan as a wave through the Islands and over or <b>EAST</b> of the FL Peninsula. Problem lies in its initialization. Had the same problem with Frances where it was steeirng Frances thorugh a 500mb ridge and I'm still waiting for Jeanne to move out to sea. At UM, we got a laugh out of the GFS last year and in fact, have stopped using it on many research simulations, instead use NOGAPS

[mobilebay]

and I can give just as many examples where the GFS is a crying joke.

Ivan was a total mess... carried Ivan as a wave through the Islands and over or <b>EAST</b> of the FL Peninsula. Problem lies in its initialization. Had the same problem with Frances where it was steeirng Frances thorugh a 500mb ridge and I'm still waiting for Jeanne to move out to sea. At UM, we got a laugh out of the GFS last year and in fact, have stopped using it on many research simulations, instead use NOGAPS

I agree with you Derek, 100%. Just ignore the remark above. I have learned to do just that. I tried to point out last year how good the CMC was doing to only get the same thing. In fact, when the CMC was the ONLY model forecasting a more westward motion with Ivan, I got this remark from him "Outliers rarely verify".

[Steve]

>>and I can give just as many examples where the GFS is a crying joke.

Indeed. GFS last year had storms turning up 40 degrees east of where they ended up. Day after day you could watch waves and early developers going up the 45th, 50th, 55th, 60th, 65th, 70th, 75th, 80th, etc. It was ridiculous. If maybe a day or two out, an argument can be made whether it was good or not, but overall, the model was clueless. One could watch the NHC reliance on such as tracks endlessly shifted further west. One thing I do recall overall last year was the consensus usually started too far east and ended up a bit too far west. Anyone can spin that however they want to or make a case for what consituted success. I ran the models. I watched the outputs. I criticized them and the NHC's reliance when I thought they were off. /the truth is out there.

TPS

[mobilebay]

>>and I can give just as many examples where the GFS is a crying joke.

Indeed. GFS last year had storms turning up 40 degrees east of where they ended up. Day after day you could watch waves and early developers going up the 45th, 50th, 55th, 60th, 65th, 70th, 75th, 80th, etc. It was ridiculous. If maybe a day or two out, an argument can be made whether it was good or not, but overall, the model was clueless. One could watch the NHC reliance on such as tracks endlessly shifted further west. One thing I do recall overall last year was the consensus usually started too far east and ended up a bit too far west. Anyone can spin that however they want to or make a case for what consituted success. I ran the models. I watched the outputs. I criticized them and the NHC's reliance when I thought they were off. /the truth is out there.


TPS

[mobilebay]

>>and I can give just as many examples where the GFS is a crying joke.

Indeed. GFS last year had storms turning up 40 degrees east of where they ended up. Day after day you could watch waves and early developers going up the 45th, 50th, 55th, 60th, 65th, 70th, 75th, 80th, etc. It was ridiculous. If maybe a day or two out, an argument can be made whether it was good or not, but overall, the model was clueless. One could watch the NHC reliance on such as tracks endlessly shifted further west. One thing I do recall overall last year was the consensus usually started too far east and ended up a bit too far west. Anyone can spin that however they want to or make a case for what consituted success. I ran the models. I watched the outputs. I criticized them and the NHC's reliance when I thought they were off. /the truth is out there.
.

TPS

True. One has to only look at the NHC 5 day graphic in motion on Hurricane Ivan to deduce that they heavily relied on the GFS and got burned. The GFS tried to make Nor Easters out of most of the TC's.

[AussieMark]

I remember with Ivan on the 2nd discussion it was mentioned that GFDL had Ivan as a category 4 in 3 days. And this actually happened from memory.

[MWatkins]

I'm not sure what all of this model bashing is about...I suppose it comes down to what you expect from the models. If you want a perfect model that is going to predict every storm 100% accurately...well...the atmosphere is WAY too dynamic. It's not going to happen. And NOGAPS? It had Frances recurving in front of 50W...then 60W...then 70W then going into the Gulf on subsequent runs.

If the GFS was bad, every other model was far worse. The verification statistics don't lie.

The models are just another tool...they aren't perfect and they are all useful as guidance. It's funny we all remember the handfull of really bad runs (especially when storms are starting to develop) but forget about the 17 ot 18 straight runs that nailed Ivan's track from the western Caribbean on.

he GFS and any other models are only as good as the observations going into them. For example...with Ivan...once the models had a hold of Ivan in a relatively data-rich environment they performed quite well. The dropwindsondes...as we know...caused many of the problems we saw with the models jumping hurricanes into ridges etc when we had less data to resolve north of the storms. If someone can figure a way to collect more and better data...perhaps the forecast models will provide something closer to what you may expect. But for now...the models are just another tool...but as long as you understand the limitations they can be useful tools.

MW

[dhweather]

A98E is BY FAR the most accurate model known to man.


I couldn't even keep a straight face typing that.

[x-y-no]

A98E is BY FAR the most accurate model known to man.


I couldn't even keep a straight face typing that.

LOL!! Remember when it had one storm (Frances, IIRC) turning south across Cuba and the Carribean and slamming into Colombia?

[dhweather]

Exactly - I've never seen A98E be remotely close to "right",
especially beyond 24-26 hours.

[skywarn]

When it appeared that Hurricane Ivan was heading straight for the S. E. LA/ New Orleans area our local met, using the Viper model,said that the model was trending for a right turn into the Al/Fl state line. It proved right. I don't know much about this particular model tho.

[Rainband]

Same model said Charley would turn. I agree good model

[caneflyer]

However, the GUNA performed almost as well and all the components of it are avaliable to the public (and Stomtrakker can calculate it automatically for you.)

One thing that PDF has ALMOST totally cleared up for me is the nature of the GUNA; I always asssumed it was as straight average, but I chatted with a met student who said a prof of his insisted GUNA was a complex weighted average which would mean the GUNA that Stormtrakker caculates wasn't the "real" GUNA; but from the way GUNA is listed in the PDF and talked about it seems it is in fact a plain average.

The GUNA is, in fact, a straight average of the lats and lons of its component members. The professor is misinformed, or someone confused GUNA with the FSU superensemble.

[mobilebay]

I'm not sure what all of this model bashing is about...I suppose it comes down to what you expect from the models. If you want a perfect model that is going to predict every storm 100% accurately...well...the atmosphere is WAY too dynamic. It's not going to happen. And NOGAPS? It had Frances recurving in front of 50W...then 60W...then 70W then going into the Gulf on subsequent runs.

If the GFS was bad, every other model was far worse. The verification statistics don't lie.

The models are just another tool...they aren't perfect and they are all useful as guidance. It's funny we all remember the handfull of really bad runs (especially when storms are starting to develop) but forget about the 17 ot 18 straight runs that nailed Ivan's track from the western Caribbean on.

he GFS and any other models are only as good as the observations going into them. For example...with Ivan...once the models had a hold of Ivan in a relatively data-rich environment they performed quite well. The dropwindsondes...as we know...caused many of the problems we saw with the models jumping hurricanes into ridges etc when we had less data to resolve north of the storms. If someone can figure a way to collect more and better data...perhaps the forecast models will provide something closer to what you may expect. But for now...the models are just another tool...but as long as you understand the limitations they can be useful tools.

MW

Mike. I don't intend any disrespect here but the following is a direct quote from you when Ivan was in the Caribbean. "Is the GFS on crack". also, "note to GFS- Ivan is not a NOREASTER". You was in there with the bashing yourself. People on this board remember what you type. You are one of the very best Forecasters on this board but please do not accuse others of things you've done. Thanks.