Being Mad at the Forecasters

[Frank2]

People will think they can't be here unless they never disagree with any meteorologists.

But of course that's not true and hopefully they don't think it's true, and is what makes this site unique, in that regular folks like ourselves (well, perhaps I'm in-between - lol) can discuss things with the Pro Mets (some who work as private consultants, others that work for the media or academia) and will receive a reasonable answer, without the "send a question but never receive an answer" outcome that often happens on many media sites...

Frank

[ColinDelia]

What specifically should one criticize in a science which is inherently probabilistic?

If the local forecast is for a 70% chance of rain and it doesnt rain that day should one be critical?I would think certainly not. However what if it only rains 30% of the time over a 5 year period on days in which the chance of rain is greater than 50%. That's a lot more questionable especially if the forecasts are paid for with tax revenues. One should at least research further to see if it is more extensive or just some local variation.

As far as predictions of number of seasonal storms and so on I think it is the same. It is the long term accuracy of the methodology which matters.



Related. I have been really impressed with the NHC and it's reduction in forecast errors over the years. Another excellent read is the NHC's webpages on forecast verification that details the average forecast error over time for various periods (24 hours, 48 hours, etc etc). See

http://www.nhc.noaa.gov/verification/

[KWT]

So we should just give up then effectivly because thats pretty much what your saying...

Also read what Klotzbach said, they have a 2NS error which really isn't that extreme to be honest.

Plus there will always be people like for example Joe B who will publish these sorts of informtation, I'd rather go the route of more people making forecasts then everyone just giving up.

Finally...lets try and leave these sorts of convos until AFTER the season...sounds amazing arrogant to me to say a season isn't going to come off when we are barely at the start of the peak period...its like being 15 mins into a football/soccer match with another 75 mins to go. Anything can happen still!

[MWatkins]

I certainly would like to know how folks calling these forecasts "terrible" have arrived at those conclusions. It's possible that they are counting storms vs. the predicted totals, or even more likely, they are looking at the Atlantic Basin on Aug 13th and assuming the season has finished already.

As I mentioned before, I am happy to discuss the "why" with anyone thinking this season will be slow. Specifically, any analysis is fine as long as it's not "well, there should be more storms by now"!!!! That's not analysis, it's not statistically valid and it doesn't take anything into account other than what's happened so far this year. It would only work if the atmosphere had a memory, which of course it doesn't.

Seriously, as KWT notes in his post, making a call on the 2010 season right now is like watching two players for a baseball team bat in the first inning, ground into 2 outs, then proclaiming the opposing pitcher is going to throw a no-hitter!!!

I suppose it COULD happen, but probability says nope.

In fact, based on data from 2000 to 2009, there is a 90% chance that the CSU forecast of above average Atlantic NTC is going to verify.

I just finished writing up a blog post for the HurricaneTrack.com site...and Mark will be linking to it at some point today.

http://www.hurricanetrack.com/watkins/watkins.htm

I hope this objective analysis, using publicly available data, will shed some light on why I think we need to wait until AFTER the 2010 season is over (or after September) before deciding if the CSU forecast was a bust or not.

The next 47 days will tell us which way this season is headed...that's just how the Atlantic basin works.

MW

[Ikester]

Standard deviation or not, the media often blows these numbers up and sensationalizes them into a three ring circus in the hopes of driving ratings. People want to know whether it's going to be 14 or 23. That's a big difference. NOAA could be completely accurate, but perception of the ones receiving the warnings (the average resident) is what should matter. If not a single storm hits their coast, NOAA was flat wrong in their eyes. It's like going downtown after dark. My mother would always tell me to stay away from there because it was dangerous. In reality, Downtown Houston was one of the safer areas after dark, but perception is what kept folks away. One Katrina size storm is far more memorable than 30 storms in a year. Ask any non-weather enthusiest and ask them if they can tell you anything about the 2005 season. I would be you not a single person could tell you about it. Ask the same person if they remember Hurricane Katrina and you'll have a 100% return of the answer 'yes.' Often times, the seasonal forecasts are only good for driving prices of insurance rates and commodities higher.

[thetruesms]

The CSU hurricane forecasts are almost always fail. This year will likely be no exception. I'm not understanding the support they get from this forum. I'm not trying to be rude, but just pointing out that they are not accurate forecasters. This forum has a lot of great value and information, but it should not spread a false meme to the public that CSU is reliable.

http://minnesota.publicradio.org/collec ... eres.shtml

I'm sorry, but I stopped reading here:

The 2009 forecast was a huge bust for the CSU team. On April 7, 2009, the CSU team issued the spring updated forecast for the 2009 season. They called for near-average activity in 2009 of 12 named storms and 6 hurricanes.

Instead, 2009 produced just 9 named Atlantic storms and only 3 hurricanes. That's the fewest number of Atlantic hurricanes since 1997. No hurricanes made landfall in the United States in 2009.

So you're telling me that a year in which CSU calls for near-average activity, and the result was . . . *gasp* near-average, it was a huge bust? Please. This is exactly what the original post is about. People taking potshots at something they don't understand, and clearly don't even make an effort to understand. It would take about 30 seconds at the NHC site to find the Atlantic's average numbers and see that 2007 is near the climatological average - and while clearly not a victory, it was no terrible defeat, either.

[MWatkins]

First, thanks to the new posters who have jumped in here! Please start adding more, your feedback is appreciated!!!

When you look at how big the world is, and how much processing power can possibly be applied to modeling the physics behind the weather, no weather forecast ever is going to meet the standard that PH sets in his blog. Just because he's a meteorologist doesn't mean his argument is a fair one.

The tropics are vast regions with very little data. Think about the size of the Atlantic and then look at where the buoys, ships and aircraft observations come from. It's at best a drop in the bucket, which means the gaps have to be filled by satellite and interpolated data. For example, one recon mission alone in the environment surrounding a hurricane can improve forecast errors by 30%. If one 6 to 8 hour flight can do that, imagine what more observations from over the ocean could do!!!

Then, you have to remember the atmosphere is a chaotic system. Storms can form close to land, near land, over a track previous traversed by a hurricane etc. There are so many variables that a deterministic forecast counting storms can never be done with 100% accuracy.

The question isn't "are the forecasts perfect?" but should be "are they more skillful than historical averages,a nd if so, can they be reliable within an acceptable statistical range? It's the difference between picking whether or not a team wins a game or not vs. predicting the exact final score. CSU and others are attempting to get the final score right!!! We've already seen they can predict who wins (90% accuracy rate in the last 10 years on more active vs. less active than normal seasons), and they aren't doing bad when attempting to forecast the final score!

Unfortunately, that's exactly what this PH guy did on his NPR blog in the spring...he counted forecast and named number of storms and concluded the CSU forecast was unreliable.

Going to NTC normalizes for these variables...but unfortunately the public and some blogging meteorologists aren't considering these unknowns when looking strictly at numbers of storms.

Furthermore, I would argue that applying the same standard on forecast tracks would create significantly higher perceived error rates if the independent variable predicted was the deterministic, binary yes or no landfall forecast point 48 hours in the future, as opposed to distance from the forecast to actual points over time intervals.

The difference between the two schemes isn't apples and oranges, it's more like apples and beer.

MW

PS...Gray hasn't been the lead forecaster for the CSU product since 2004. Your blog guy needs to do a little more research! And, if you read the link above, I just looked at the August time period. Just like ANY weather forecast, resolution improves as the time between forecast vs. verification gets shorter.

The NHC issues a new forecast every six hours. They don't do it as a coverup to make their forecasts "seem better"!!! Just like at the NHC, CSU updates as more information comes available. Every forecast is widely available on their website for all to see...not just the last one!!!!

[Portastorm]

Hey gang -- if you haven't already noticed, we have been having some trouble (especially in this thread!) with trolls. Spoyn3b appears to be the same person who wrote several notes this morning (which were deleted) which were personal attacks on pro mets. While we do not want to distill debate on the NPR piece or other fact-supported criticism of CSU's forecasts, we will not tolerate trolls. So those posts have been deleted (in case you were wondering what happened).

Thanks,
Portastorm

[thetruesms]

Standard deviation or not, the media often blows these numbers up and sensationalizes them into a three ring circus in the hopes of driving ratings. People want to know whether it's going to be 14 or 23. That's a big difference. NOAA could be completely accurate, but perception of the ones receiving the warnings (the average resident) is what should matter. If not a single storm hits their coast, NOAA was flat wrong in their eyes. It's like going downtown after dark. My mother would always tell me to stay away from there because it was dangerous. In reality, Downtown Houston was one of the safer areas after dark, but perception is what kept folks away. One Katrina size storm is far more memorable than 30 storms in a year. Ask any non-weather enthusiest and ask them if they can tell you anything about the 2005 season. I would be you not a single person could tell you about it. Ask the same person if they remember Hurricane Katrina and you'll have a 100% return of the answer 'yes.' Often times, the seasonal forecasts are only good for driving prices of insurance rates and commodities higher.

This creates quite a conundrum - should we give up on the scientific pursuit of attempting to determine a season's potential ahead of time if there's a potential that it could be misused? There's certainly a reasonable argument that we should not. My views, though, are that the pursuit of accurate seasonal forecasts are ultimately worth it - even if most people don't care to look closely enough at them before speaking their mind. A lot - nearly all - of this work is done by public entities, and even if they did not release anything, a proper request would require the information to be released. And beyond that, publication is the currency of science. There is no point in doing research if you can't tell other scientists about it to examine, correct, and expand upon your work.

[Ikester]

You bring up a good point. Thank God for science, or we'd still be eating moldy bread for headaches. However, the 'Freedom of Information Act of 1966 (Nadar/Johnson) has turned out to be an injustice in some forums because people wanna know but have not the capacity to understand what they are looking at. It's like the spaghetti models. I'm a big fan of them. However, when they are displayed, I am on tv explaining which ones are good and the strengths and limits of each. If people see the lines appear and the Clipper is the only one showing a hit for their area, then they get concerned. In reality, that model is useless. It's just a climo model. But explain that to somebody at a crowded gym where headphones are blaring and the only thing they see is my mouth moving and a model that shows the possibility of a landfall. There are just some things that need not be shown and starting with the seasonal forecast would be a good start.

[HurricaneJoe22]

I think itstead of specific numbers or even a range of numbers, there should just be forecasts from these agencies that are just a summary of the factors they see they will make the season ahead active, average or inactive and just say "We expect an above average amount of storms" or "a below average amount" and state the numbers for an average year. In this case, it gets the message out about what kind of season is expected and is easier for the public to grasp than an exact amount of storms. Just my 2 cents for what it's worth.

[HURAKAN]

The general public doesn't care if we have 1 or 20,000 computer models at our disposal, they just want to know if the hurricane is coming or not, or if it's going to rain or not. That's the bottom line. They're not going to understand why we have so many computer models and that many are garbage.

In Spanish, Meteorology is always referred by the people as "Mentirología." Menti comes from Mentira or lie.

Forecasters because they're trying to predict something that it's not completely understood, will always have that stigma.

[somethingfunny]

For whatever my opnion is worth I've never been a big fan of seasonal forecasts. I don't care if they're accurate or not - I just challenge you to find one way that these forecasts are worthwhile for anything other than an academic exercize.

Even in a world of 100% accuracy, should you really prepare differently for a season forecasted to have 4 named storms as opposed to 28 named storms? Ask the people of Houston, who were spared in 2005 but slammed in 1983. The seasonal forecast should have no impact on your seasonal preparations. It seems to me that running experimental forecasts within the academic community is a valuable endeavour, but releasing them to the public serves no purpose whatsoever. Is there some other value to these forecasts that I'm missing?

[Ikester]

I think itstead of specific numbers or even a range of numbers, there should just be forecasts from these agencies that are just a summary of the factors they see they will make the season ahead active, average or inactive and just say "We expect an above average amount of storms" or "a below average amount" and state the numbers for an average year. In this case, it gets the message out about what kind of season is expected and is easier for the public to grasp than an exact amount of storms. Just my 2 cents for what it's worth.

But then it becomes a moot point when the overall basin is very active like now or when it quiets down and Atlantic anomalies begin to cool and transition to an 'inactive' period for 20 years. Since 1995, nearly every season has been 'above average.' Perhaps averages should be configured another way. An average for active periods and an average for a non-active period.

[Stormhunter27]

Working here at the Weather Network (in Canada) has taught me a few things - one is that the general public wants to know "Is it going to rain and how warm is going to be?" having said that, though, there is a real appetite for learning more about the weather as well. Not everyone is nasty, but there are a few.

i think that explaining ACE and beginning to use it more to explain the hurricane season is not totally out of the question. It may take a few seasons of pushing it on air for people to come to grips with it, but it's such a better way of assessing a given season. Numbers of storms may work for some media, but I think we can begin to use ACE as a measure. It would really save a lot of grief for forecasters.

But then again, I could be totally out to lunch...

[Ikester]

Working here at the Weather Network (in Canada) has taught me a few things - one is that the general public wants to know "Is it going to rain and how warm is going to be?" having said that, though, there is a real appetite for learning more about the weather as well. Not everyone is nasty, but there are a few.

i think that explaining ACE and beginning to use it more to explain the hurricane season is not totally out of the question. It may take a few seasons of pushing it on air for people to come to grips with it, but it's such a better way of assessing a given season. Numbers of storms may work for some media, but I think we can begin to use ACE as a measure. It would really save a lot of grief for forecasters.

But then again, I could be totally out to lunch...

That's quite a task. What goes over the air waves is governed by the ND (News Director). If he doesn't understand what ACE is, then it doesn't go out (or that's been my experience). When making a forecast, I have to remember that people are usually not watching but listening...especially in the mornings. When talking ACE to somebody trying to get ready for work will cost you viewers. We're lucky if people can remember 3 things about the forecast that is presented. To help me, I compare the weather to the in-laws: When are they coming, how long will they be here, how bad is it going to be, and when the heck are they leaving!? If you get those points across, your golden.

[thetruesms]

For whatever my opnion is worth I've never been a big fan of seasonal forecasts. I don't care if they're accurate or not - I just challenge you to find one way that these forecasts are worthwhile for anything other than an academic exercize.

Even in a world of 100% accuracy, should you really prepare differently for a season forecasted to have 4 named storms as opposed to 28 named storms? Ask the people of Houston, who were spared in 2005 but slammed in 1983. The seasonal forecast should have no impact on your seasonal preparations. It seems to me that running experimental forecasts within the academic community is a valuable endeavour, but releasing them to the public serves no purpose whatsoever. Is there some other value to these forecasts that I'm missing?

The "academic exercise" argument always bugs me a little bit. There is nothing in meteorology, no matter how obvious and simple it may be now, that was not originally worthwhile only as an academic exercise. Everything, no matter how elementary has been a series of small steps from complete mystery to understanding. Also, this is a strange crux where what people would like to know coincides with something that is still on the bleeding edge of the field. As a result, we have
1) Researchers making seasonal forecasts, in a research effort to improve these forecasts to where they are operationally reliable year in and year out.
2) To further this research, it must be published to share with other scientists
3) People have a desire for a consistently reliable seasonal forecast, and in looking for one, they are all but certain to find these research efforts
4) Seasonal forecasts essentially go viral

Sure, we can wish that people wouldn't know such things exist until they're as accurate as predictions of tomorrow's high temp, but the genie's out of the bottle now, and you can't shove him back in. All you really can do as a scientist doing this is to explain what you're doing as well as you can, then hope for the best.

I 100% agree with you about the effect on preparations. Every seasonal forecast must always be accompanied by the "it only takes one storm to ruin your life" statement. While we're making baby steps into long-range landfall forecasts, it's quite obvious we're much closer to the total mystery end of the spectrum than the complete understanding end. It's safest to prepare for every season as if there will be a strike - especially since most things involved in preparation have relatively long shelf lives, making it easier to stay prepared.

[Ikester]

I disagree. Death is death. In 2005 Houston was NOT spared but was severely impacted by Hurricane Rita. While the city dodged the winds and surge, we were subject to the largest evacuation in US history when ~3 MILLION PEOPLE FLED the region (That's nearly half of everybody in greater Houston). Over 100 people died during the evacuation as temperatures soared into the hundreds ahead of the storm. Can you imagine the 4th largest city a ghost town? Well it was! Below is a picture of I-45 at The Woodlands Parkway (28 miles north of downtown). Keep in mind that every freeway in the city mirrored this image.

I-45 at Woodlands Parkway:



I-45 at Research Forest Dr.

[mpic]

Ikester, that horrible event in 2005 is why I found Storm2K and why I now leave in plenty of (days) time since. As a result of that, though, the last evac was much better as those that didn't need to leave, stayed...made a big difference.

On the subject, though, what if they broke it down better with the projected numbers? Like 1-2 in July, 6-8 in August , etc. People would know not only that it will be an active season, but stop and think about when it will begin to ramp up. Would be more helpful, maybe, to the people new to the area. Is that even possible at this point?

[JtSmarts]

That's an interesting idea mpic but the problem is that the media would likely still focus on the overall numbers meaning we'd have the same basic problem.