distributed computing

[kevin]

Hello, I have been browsing the boards for over a year and have finally registered.

Since weather forecasting involves the integration of massive data volumes, into exceedingly complex models, I was wondering if it could be a candidate for distributed computing. There seems to be a base for the program with thousands of weather enthusiasts. With professional meteorologists visiting this board and actively posting it seemed natural to come here for advice.

-Kevin Gould

[Aquawind]

Welcome to the board Kevin!

I think the concept has merit. The problem is getting the program written that has the worthyness of such an endeavor.. Right now we have some fantastic computers crunching massive data..I am talking some of the most intense computers in the world are crunching daily..Unfortunately it all doesn't matter if the paramiters are not correct..We don't even understand the data enough to write the models now..It's not the size of the CPU that matters as much as the data and programing using the best facts we have..Heck we have all the facts post system and still cannot concieve it in model data with any reliablilty..

The models..and yes it's multiple programs crunching on on different hardware and with software they modify regularly and still..cannot tell me for sure what the weather tomorrow is going to bring..for sure..

We can combine all the CPUs on the planet..but if we can't understand the processes we cannot write a progam to get the answer..

That's right we have a long way to go..were kinda guessing now..even with the facts we have and know..

Why bother..heck SETI is looking for the smallest of facts..yet we have so many facts before our eyes on this planet it's a blurrr in a test tube..

Fact is though.. as you mention someday it would be nice if we could predict the future..unfortunately we are infants in all this knowledge and technology..keep in mind a PDA now produces the power of what use to be buildings of computing power.and those people were no dummies..think about it the senior veterans of weather learned this in mid-stride..the art of reading satellite data alone is way past the average joe..

Bring it on though..I will gladly donate some CPU time if it has any credibility..

[kevin]

Thank you for your considered reply. I agree with everything you've said about the enormous complexities in modeling the future. It is amazing that our forecasts have any degree of credability considering all the variables.

While modeling the entire atmosphere is far too ambitious, the distributed program might examine an area of meteorology (for instance say, the electrical properties of thunderstorms) or try and replicate an artificial atmosphere and see if we can better understand the dynamics. Since I'm not a meteorologist or a computer scientist I can't produce either the process or the source.

Through discussion though perhaps we can examine the feasibility of such a program and begin to lay the foundations.

[Rainband]

Welcome Kevin. Your truly a bright individual!!

[Aquawind]

I agree we need to do it in small piceces to get some absolute facts and then blend those models/facts for a bigger picture..I am all ears for the dicussion to reality..Maybe after Microsoft punches out Longhorn we can get them to put together some weather models in detail..ahh yes I can see it now we have it all figured out in detail and timing... and Bamm!!..blue screen of death 24 hours before landfall...ugh..

One thing is for sure it will take many minds through colaboration..

[MGC]

Kevin, Good to see you again....MGC

[kevin]

Thank you rainband. You guys run a very nice bbs.

MGC, hello there, do you remember me from the weather.com message board? Just curious.

Looking forward to everyone's continued comments. In case the topic winds down and sinks into the archives, if anyone would like to keep in touch about this, my email is cognosco@muchomail.com . When hurricane season really starts to blow I'm sure everyone will have their hands full post wise.

[MGC]

Yes I remember you from TWC. Have you graduated HS yet? Looking forward to another season of banter?..........MGC

[kevin]

Just graduated, and not quite believing it yet. Starting community college in August, going for the transfer with Associate of Arts route.

And yes, looking forward to all the 'Its coming right at us!' and 'Why hasn't the NHC named blob #97.". There's nothing quite like hurricane season.

[MGC]

Congrats Kevin. Wish I could remember HS. I have one son left to run the gauntlet as he is a senior this year. What are you going to study in JC?..............MGC

[OtherHD]

Wow kevin, great to see you again! I always enjoyed your posts at TWC. I'm hurricanedude over there.

[kevin]

I remember ya hurricanedude! Too cool.

As for college, I'm taking chemistry and calculus along with a humanities course to start off with. I want to major in mathematics because that will give me the greatest options of things to pursue for graduate school. Too many things interest me, but hopefully by then I'll find what really is my niche.

[ColdFront77]

Hi Kevin, welcome to Storm2K. I feel I remember you from TWC as well, although not vividly at the moment. Again welcome!

[cycloneye]

Welcome to storm2k and you will enjoy it fully here.

[kevin]

Thanks coldfront and cycloneye.

[Wthrman13]

Hey Kevin,

Since nobody else mentioned this in this thread so far, I'd thought I'd point out that weather forecasting has utilized distributed computing for many years now.

While modeling the entire atmosphere is far too ambitious, the distributed program might examine an area of meteorology (for instance say, the electrical properties of thunderstorms) or try and replicate an artificial atmosphere and see if we can better understand the dynamics. Since I'm not a meteorologist or a computer scientist I can't produce either the process or the source.

This is exactly what the discipline of Numerical Weather Prediction (NWP) is all about. NWP is not only an important part of modern meteorology, it is the very backbone of it. Several agencies, both government and private, around the world, utilize just the sort of "artificial atmospheres" that you are suggesting to forecast the weather day in and day out.

Now, there are many limitations to such an ambition to accurately reproduce the atmosphere with a computer program. You and Aquawind are absolutely correct in pointing out that one of them is that we just don't have a complete physical understanding of many atmospheric processes, although we are getting better and better at it each day. As an example, two of the notoriously difficult problems in meteorology that we just don't know how to accurately model are turbulence and the microphysical processes of clouds. Both of these, far from being simple side issues, are absolutely critical if we hope to accurately model the atmosphere, especially on the small scales of individual thunderstorm cells. Oh, and don't even get me started on the problem of initializing our atmospheric models by ingesting all the observational data we take of the atmosphere (i.e. surface obs, satellite, upper air soundings, and radar obs). This problem is actually the major focus of my current M.S. research.

Now, as far as distributed computing goes, if you mean distributing a weather model out among the general public in a SETI @home type of system, you might be interested in this website:

http://www.climateprediction.net

This is a group that has actually developed a climate model that runs in real-time on your computer, using idle CPU cycles. What they do is tweak several model parameters randomly, such as the amount of humidity you need in a certain atmospheric grid cell before a cloud forms, or the efficiency at which precipitation is produced from clouds, etc. The idea is to run a whole bunch of these versions of the climate model on thousands of machines to determine the sensitivity to the above parameters and others. It's a pretty good idea, I think.

Now, I should point out that virtually every NWP model of the atmosphere out there runs on distributed computing platforms in the form of multiple processor computers, because running a model in a high-enough resolution to be useful for day to day regional weather forecasting, and having the forecast get done in enough time (i.e. much faster than real-time, otherwise, what would be the point in running the model, if the very weather it's trying to predict is happening outside as, or even before, the model is predicting it!) requires a great deal of computational power that even our fastest single processor machines can't handle it. So, instead, the model utilizes several processors running in parallel on different pieces of the atmosphere. This speeds up the execution time by several orders of magnitude.

As an example, the model I'm using for my research, the ARPS (Advanced Regional Prediction System) is coded in such a way that it can spread itself out among several CPU's in a cluster in this manner. One of my domains covers most of the Central Plains at 3 km horizontal grid resolution, and a 24 hour forecast of a bow echo that I perform on this domain takes about that long (24 hours) to complete, and this is with 100 processors from the local supercomputer crunching it! If I were to try to run the same simulation on my desktop computer, it would probably take at least a month to complete, obviously not conducive for me to graduate any time in the next, say, decade!

So, I apologize for this huge exposition, but you are indeed correct in pointing out how difficult it is to forecast the weather using computers, but, having laid out all these difficulties, the models we do have out there are very sophisticated, and perform incredibly well considering all the hurdles we have to overcome in designing and using them!

Dan

[kevin]

Thanks so much Dan, your information is well beyond what I initially expected to recieve, and I'm glad I started this thread.

I did come across the climateprediction program in my google searches, but thought its catch phrase which mentioned the Day After Tomorrow movie indicated a bit of sensationalism. I will however check out their project.

The problem of weather prediction is enormous, I can definetely see how model initiation would be a significant problem. When I was thinking about the basic necessities of a distributed computing project, I came across the problem that once a seed was inputed and all the information processed what you have is another time frame. 1,2,3,4,5,6... a series. Time progressing in little blocks, which it doesn't. The program would inaccurately predict 2, and there wouldn't be an easy way of correcting the erroneous data and integrating current obs for time 2 to predict 3. Essentially the simulation would have to start over at 2 and predict 2,3,4,5,6,7. And then at 3 predict 3,4,5,6,7,8. Leaving a meteorologist to do what they do best, and interpret the flow of the data, the trends.

Also it doesn't seem very practical to discount observations at .1, .2, .3., .4, .5.... and only accept data taken from .9 or whatever time information is gathered to plug into the alogrithms. Do weather models take in account the past behavior of the systems they analyze?

Thanks very much for the exposition. Hoping that I was lucid enough.

-Kevin

[kevin]

Are bumps allowed on this bbs?

<< a cleverly disguised bump

[ColdFront77]

Are bumps allowed on this bbs?

<< a cleverly disguised bump

Sure they are, Kevin. In my opinion they are more necessary when the thread you have nothing to respond to hasn't gotten buried yet, at least one member may eventually get to that thread, including those such as this one that didn't get a post for awhile, such as this one.

[Wthrman13]

I did come across the climateprediction program in my google searches, but thought its catch phrase which mentioned the Day After Tomorrow movie indicated a bit of sensationalism. I will however check out their project.

Indeed. I hadn't actually been to the website itself in a while. A little surprised they used that catchphrase. I joined before that movie came out.

When I was thinking about the basic necessities of a distributed computing project, I came across the problem that once a seed was inputed and all the information processed what you have is another time frame. 1,2,3,4,5,6... a series. Time progressing in little blocks, which it doesn't. The program would inaccurately predict 2, and there wouldn't be an easy way of correcting the erroneous data and integrating current obs for time 2 to predict 3.

At the risk of getting too long-winded, what you have just described is a fundamental issue in model initializiation and forecasting. Time in the real world indeed doesn't progress in little blocks, and neither do the basic differential equations used to describe atmospheric motions. A computer, however, has finite memory and computational resources, so we have to break these equations down into discrete forms that can be integrated over finite intervals in both time and space, and this naturally introduces error, which accumulates as the model is run forward in time. There are various techniques used to minimize this error, and in some cases entirely eliminate parts of it, but that's a whole other topic.

Anyway, usually what happens in a typical NWP framework is that observational data at some particular initial time is blended with a "first guess" field of the state of the atmosphere at that time. This "first guess" field, or background field, is usually simply a forecast field from a prior run of that same model. The observations adjust the forecast field to make the final field, called the analysis, as close to the real state of the atmosphere at that time as we can get, taking all the error sources into account. These sources of error include the error of the background field because the forecast is of course not perfect, and the error of the observations themselves (which are significant and cannot be neglected). Once we have the analyzed field, we put it into the model to start it up, and the forecast goes from there, typically not bringing in any more observations as it goes. The reason for this is simple, you want your model to predict the state of the atmosphere *before* it happens. If you wait on additional observations, you've pretty much defeated this purpose. One way around this is to have an interval of time during which the model runs and brings in additional observations. This is called an assimilation cycle. After this cycle, the model is run normally. An assimilation cycle can help train the model to accept the incoming data, which is discontinous, in a smoother manner.

Hope this helps,

Dan