Model performance for 2004 season

[EDR1222]

Just thought I would see what everyone thought about the model performance this year, especially on the 72 hour forecast. And also, will there be modifications made to them for next year. I am not sure how that would work, just wondering if anyone had some information.

Thanks,

Ed

[weatherwindow]

after a casual look, it appeared that the new 120 hr did perform better than the preseason estimates by the NHC. likely the 72 hour performed well also. with the exception of jeanne, the perponderance of long track storms would yield better performance just by the persistence of the motion. the individual models varied widely by storm. it did seem as though the gfs/avn had some difficulties this year with underforecasting ridge strength. whereas, the nogaps, and to a lesser degree the ukmet, were the better performers for the year....again i am certainly not a pro but, at first glance, this was a good forecasting year for the NHC. modelwise, some very good performances. i am becoming a fan of the FSU super ensemble. i do think that consensus models will be the models of choice in the future.........rich

[MWatkins]

Of course...model verification for this season is going to be very interesting. That said...I think there are some preliminary lessons and questions that remain as takeaways for the NCEP model folks and people like us that use the models.

1. With the exception of Charley, the global models consistently had a right of track bias. Much has been made of this already and I don't think that's a surprise, but for some reason the physics that lead to the persistence of the western Atlantic ridge...and it's failure to subsequently break down as repeatedly indicated...needs to be addressed.

2. Speaking of Charley, the global models did not handle Charley's interaction with the large east coast trough well (at all). Tony Cristaldi believes (and I agree with him) that they did not resolve the relatively small TC Bonnie and it's impact on ridging to the north and immediately east of Charley.

3. As much as the NOGAPS model eventually hit the right trend with Frances and Jeanne...the error rates for this model may not turn out to be all that impressive. The model...with Frances in particular...had it's problems. For example...in 3 concurrent runs...the NOGAPS recurved Frances at 55W, then 65W, then 70W, before finally (and correctly) latching on a FL East Coast solution.

4. The GFS freaked out and turned Frances straight into a 500MB ridge when it was sitting and spinning off the FL coast. Apparently the dropsondes caused some sort of problem with the model that affected the GFS and GFDL guidance. Whatever caused this needs to be addressed as well.

5. Speaking of the GFDL...it was aggressive once again with every single storm in terms of intensity (Earl? Hello?). Although it correctly picked up on the rapid intensification of Ivan and Frances (and Charley I think)...I believe it was a result of the broken clock right twice a day scenario...if rapid intensification is forecast all the time then of course when rapid intensification does occur the model will be correct.

6. The press needs to back off the models a bit unless they plan to explain how they are used. For example...the BAMM model was pointed out in a lot of the graphics we saw in print media...why...I have no clue. The model consistently has track errors of 500 nautical miles plus at 120 hours.

7. We still know very little about intensity forecasting.

MW

[weatherwindow]

super discussion, MW....i have developed a lot of respect for your analysis. ....a question: it seemed as though the consensus models resolved the right bias.....fsu, guns guna.....what is your opinion?

[wxman57]

Although the new 5-day tracks worked ok with Isabel in 2003, I thought they showed much less skill in 2004. The NHC bought into the models right-of-track bias most of the time, and most forecasts beyond 48-72 hours had storms like Ivan recurving much too quickly. What was a threat to south Florida became a threat to Alabama/western FL Panhandle. In my talks next spring, I'll be discussing the long-range forecast errors and why 4 and 5-day forecasts should be used as just general guidance.

[Agua]

Yeah, that was the general tendency: too far north early in the forecast period and too far east late.

[MGC]

Not being a fan of the models, I think they did OK as they got the general landfall areas in several of the storms. The models do have a tendancy to want to recurve way too early though. As I recall in 92 Andrew was according to the models to recurve into the New Orleans area at one point but as we all know went west of NO. In 1998 Georges was forecast to head up the east coast of Florida but ended up hitting Mississippi. This season Ivan was forecast to hit the keys but went well west of the keys.......MGC

[caneflyer]

Verification statistics were presented at the 2004 NOAA Hurricane Conference, held at NHC last week. Of the dynamical models, the lowest errors in 2004 were turned in by the GFDL, with the GFS close behind. The UKMET was third, and the NOGAPS a distant fourth. In fact, the NOGAPS was so bad that it was beaten by some of the BAM models in 2004.

The consensus models performed even better than any of the individual dynamical models. Of these, the FSU superensemble performed the best in 2004. Interestingly, one of the major components of the FSU superensemble as it is now configured is the previous official NHC forecast, so the two are not independent.

Since 1996, none of the dynamical models has been the best performer for more than two years in a row. The GFDL has won for the past two years, so now it's someone else's turn for 2005.

[MWatkins]

super discussion, MW....i have developed a lot of respect for your analysis. ....a question: it seemed as though the consensus models resolved the right bias.....fsu, guns guna.....what is your opinion?

caneflyer noted tonight that the consensus models performed the best with the FSU model leading the way. Overall I believe the model blend did help to correct the right of track bias...the GFS and the GFDL did perform well overall...but we tend to remember Frances, Ivan and Jeanne because of their impact.

Interesting that the Canadian model...especially with Ivan...helped straighten out the CONUS. I betcha if you threw that model out the CONUS would have been significantly right of track. The CMC way bad but it helped to overcorrect the other models.

BTW...thanks...

MW

[caneflyer]

Not sure that I understand your comment about the Canadian model. Did you mean "consensus" rather than "CONUS"?

The Canadian model is not part of any consensus model used by TPC. GUNA is the GFDL, UKMET, NOGAPS, and GFS (AVN). GUNS is the same but without the GFS. I don't believe that the FSU superensemble includes the Canadian model as a component either.

[MWatkins]

Not sure that I understand your comment about the Canadian model. Did you mean "consensus" rather than "CONUS"?

The Canadian model is not part of any consensus model used by TPC. GUNA is the GFDL, UKMET, NOGAPS, and GFS (AVN). GUNS is the same but without the GFS. I don't believe that the FSU superensemble includes the Canadian model as a component either.

I'm 100% sure that the CMC is in the FSU data sample...it's usually the first to hit along with the late GFDL when the FSU track comes together.

You're right on the other point....I did not mean CONUS (which shows how removed I am already at reading the output). I meant to type CONU, which is the global model concensus with the GFDL thrown in. The NHC ACTF documentation is not updated to reflect this as of 4/2004 but the lat/longs are available from the guidance...and I believe the addition of the CMC model is what seperates the CONU from the GUNA guidance.

MW

[MWatkins]

Not sure that I understand your comment about the Canadian model. Did you mean "consensus" rather than "CONUS"?

The Canadian model is not part of any consensus model used by TPC. GUNA is the GFDL, UKMET, NOGAPS, and GFS (AVN). GUNS is the same but without the GFS. I don't believe that the FSU superensemble includes the Canadian model as a component either.

I'm 100% sure that the CMC is in the FSU data sample...it's usually the first to hit along with the late GFDL when the FSU track comes together.

You're right on the other point....I did not mean CONUS (which shows how removed I am already at reading the output). I meant to type CONU, which is the global model concensus with the GFDL thrown in. The NHC ACTF documentation is not updated to reflect this as of 4/2004 but the lat/longs are available from the guidance...and I believe the addition of the CMC model is what seperates the CONU from the GUNA guidance.

MW

Did a little more research...the CONU does not currently include the CMC (at least as of end 2003). CONU includes:

GFS
NOGAPS
GFDL
GFDI
UKMET

And is similar to the GUNA guidance.

However...I'm still sure the CMC is i n the FSU sample.

MW

[Derek Ortt]

I believe the CONU includes GFDN. GFDI is the previous GFDL model interpolated (it comes out prior to the forecast issuance rather than the GFDL which comes out an hour or so after)

[MWatkins]

Yep...well to an extent. I meant to include the Navy version of the GFDL instead of putting up the interpolated GFDL. But as I think about it I'm pretty sure the interpolated version of all 5 members is used to calculate the CONU track.

For example, if it's 3PM EDT (19Z) and the models just finished running...the 18Z NOGAPS and 18Z GFS and GFDL products aren't out yet. But an 18Z CONU track would be available. So the 18Z CONU has to be a combination of the following models:

AVNI: 12Z GFS track interpolated from the 18Z CARQ position estimate
NGPI: 12Z NOGAPS track interpolated from the 18Z CARQ position estimate
UKMI: 12Z UKMET track interpolated from the 18Z CARQ position estimate
GFDI: 12Z GFDL track interpolated from the 18Z CARQ position estimate
GFNI (actually not sure what the ACTF representation of this model is: 12Z GFDN track interpolated from the 18Z CARQ position estimate

I think that all of the interpolated tracks from each run go into CONU which may explain why CONU skill lags behind the GUNA guidance in terms of skill.

MW

[Derecho]

In 1998 Georges was forecast to head up the east coast of Florida but ended up hitting Mississippi.

I'm about 99.9999% certain you're completely wrong about that.

Georges was one of the most well-behaved storms trackwise in all respects; in terms of not deviating much from the forecast track, not really turning at all, and not deviating from the model consensus.

[Derecho]

I suspect the CMC would pretty much ruin any model consensus that included it.

The precise details of the FSU Superensemble are still somewhat of a mystery.