Studying Temperature Data Using the Language of Science

What it is climate folks are averaging

This is the proper expression of T_Avg for my Idaho dataset in 1914. Notice....all that little daily variability is there, the changes from day to day are there, the change with seasons, for the entire year. I have one of these for each year, 1914 to 2014. I can give you one of these for 100 January 1sts if I wanted to, or December 25ths, or whatever day you've like. Multiple levels of uncertainty, based on multiple drivers, different levels of uncertainty with each and every example for sometimes the same, sometimes different reasons. All of this nuance and detail is then hidden from you, the average reader, by just taking all these numbers, adding them up, and dividing by the number of data points to create a single point estimate they hope you will understand. Move a SINGLE one of those daily average temps and that answer they just gave you is wrong, because what they DON'T give you is the graph in the post preceeding this one showing how much that single metric moves around naturally because of this built in variability.

It gives the amateur reader the impression of certainty where no such thing exists. An average is just one number, interesting but no different than a median or P43.2, and can be just as misleading if used improperly. The average adult American has one testicle and one breast....has anyone on this forum ever even SEEN such an AVERAGE American? Me neither.

 

yeah, unfortunately, my pc doesn't like these files. So I don't see them.

 

Try these, Mean Annual of Idaho



Uncertainty around annual means of Idaho



Mean Daily Across Multiple Stations in Idaho, 1914

 

thanks, I can see those.

 

Translation: No idea what PeakProphet is talking about.

 

Quick question for those who haven't seen these results yet. What would you EXPECT to happen under the following conditions.

Take, as our current example, 30 temperature stations in Idaho, and their average daily temperature, on one day, pick a day, doesn't matter. June 1, 1955. Now examine all those same temperature stations the next day. Mark each record within the database as to whether or not that station increased in daily average temperature, or decreased, or stayed the same, and measure by how much. So you are now creating some value added data to examine, in this case a measure and count of change, this station, between June 1, 1955 and June 2, 1955, showed a temperature change of +0.5C and the temperature increased.

Now do this exercise for every day, for every station, in the entire dataset. So now you have, for every day, or month, or year, the ability to count and add all these things up. So for example, I can add up all the changes between June 1, 1955, and June 2, 1955, and this value added data set will tell me..something. Same for all of June, 1955. Or all of 1955. Same principle applies.

Here is the common claim within climate science that I am testing, and will test at levels far beyond just Idaho. IF, and this is a conditional test, so IF matters, correlation of stations across a given area (small or large) is taking place, then there will be a statistical bias that can be pulled from the above described data set. For example, I have 30 stations, and whatever this THING is that causes a change in temperature the next day, a storm, clouds, change in the seasons, El Nino, doesn't matter, whatever this THING is, at a scale large enough to matter, it will introduce a bias into my day that I can see with this test.

For example, a large weather system moves in during some winter day, and really throws a curve at all the temperature stations....the average daily temp changes across a wide area from Day 1 to Day 2. This would generate a profile within these counts that would look something like this:

Stations Increasing In Temp: 7
Stations With No Change in Temp: 3
Stations With Decreasing Temps: 20

So out of 100% of my stations, 66% would indicate a drop in temps, from day to day. Only 23% would show an increase, and 10% would show no change. This is a measure of bias, I don't care what this THING was that induced a large area temperature change, I can see it effect loud and clear. On that day, something most likely happened, and it showed up as a statistical bias towards cooler daily average temps.

This example holds true.......IF TEMPERATURE IS ACTUALLY CORRELATED ACROSS THIS AREA. Because this example is intuitive, everyone can understand it, SURE a cold front moving in should cause some majority of my temp changes to move together, at the very least directionally, if not in magnitude as well.

If, however, some front moves in and the counts remain something more like this table:

Stations Increasing In Temp: 14
Stations With No Change in Temp: 2
Stations With Decreasing Temps: 14

then we have temperatures that aren't really correlated across this area. If temperatures are not correlated, then they are considered to be independent. One is as likely to go up on the same day that another in the area goes down. This conditional IF applies to the area in question, so I can run this test at small areas (and might) and big areas (and am) because I am searching for a specific area that matches correlation, a kernel and sill within a semi-variogram. Haven't seen climate folks using those, but they must somewhere. Maybe. I have been continually surprised by what they HAVEN'T done as opposed to what they have.

The results will tend to be more like a coin flip in this case, because this statistical test is set up as just that simple...the answer is either yes, no, or don't count me because I didn't change, which is less likely then the other two. But the other two are going to be similar in size, just like multiple trials with a coin toss.

For those who care, I have used exactly this type of bias detection when trying to determine optimal solutions for well design by trying to normalize out geology, looking for that one thing that delivers a better then average result, in the case of well performance I WANT that bias, I want to know why, and I want to build my wells to fall on the high side of the performance curve.

In climate science, assumptions of correlation are made across vast areas, here we are just trying to see if such correlation even exists, and if we can see it, how strong is it, and at what level.

So this is the test. The question is, what do folks think the result say? Will these temperature changes across the state of Idaho more likely move in conjunction with each other, or are they more likely to be random? A slight bias, in my opinion, would range from 55-60% of the samples leaning one way or the other (doesn't matter when just trying to get this directionally right). I strong bias would be 65-75% leaning one way or the other. Correlations applied at the level of some of the climate science work I've seen assumes as high as 100% bias. And some of that is ASSUMED. A value at any level between 50-55% I am going to call independent or mostly independent, while I have what some folks might think is a bunch of data, a million records is pretty small potatoes in the overall scheme of things, and by the time I'm done I plan on having 10X or 15X more before I hit hardware/software limitations. Then we can REALLY start to test some of these 100% bias claims across the entire country that people are assuming. Assuming being key here. Why assume when we can measure? Long live empirical results!

Anyway, so before I release the results, what do folks think? Any answer on any scale you'd like, I've got results at the daily level, monthly level, and yearly level for a century. Are large scale effects large enough to introduce bias? At a daily level only perhaps? Is there just too much randomness in temperature measurement in general, what with all these temperature shifting and correcting and adjusting going on...over a century? Does the answer change? Across all January 1 1914->January 1,2014? All Januaries across all years? Before or after WWII? Feel free to put qualifiers on your answer..."yes..I think bias will be present...earlier in the 20th century than late". "No...bias will be there but only in winter months because the people forced outside to read the machines didn't want to and made up random numbers", whatever caveats you'd wish.

At this point I have the raw data to look at combinations should you be interested in particulars, but I haven't looked at it all. Just checked to make sure the programs ran correctly, and I can whip out quick point estimates to provide the % answer to particular questions.

I'll let this sit for a day to see if anyone is interested in the question (I don't expect to see those who are so busy running from data at this point that I don't ever expect to see them again) and then I'll begin churning out some graphs and charts, and then anyone interested can request particular time frames or resolutions to see if we can find an effect we know about. Like we could look up the largest snow storms in Idaho and the days they were around, and check those days against the rest of the month, or all other months across the entire century.

 

I would think you would need to see the bias for each month across the whole span of dates. Jan to Jan, Feb to Feb, etc..

 

Congratulations PP, you're only duplicating work from 27 years ago now. For you, that's catching up.

Hansen, J.E., and S. Lebedeff, 1987: Global trends of measured surface air temperature. J. Geophys. Res., 92, 13345-13372, doi:10.1029/JD092iD11p13345.
http://pubs.giss.nasa.gov/abs/ha00700d.html
---
We analyze surface air temperature data from available meteorological stations with principal focus on the period 1880-1985. The temperature changes at mid- and high latitude stations separated by less than 1000 km are shown to be highly correlated;
---

The real scientists talk to each other. And one says "We're thinking of doing this", his colleague can say "We tried something like that, and this and this problem showed up, so to get that right you have to do this and this."

Deniers? They refuse to look at any previous science, or talk to anyone with experience, hence they repeat the mistakes that the real scientists fixed decades ago. Ah, the foibles of ego.

 

Okay. So I can test this idea, are you saying you want to see ALL Januarys, from 1914 to 2014, in sequence? And that your hunch is that there will be a bias of some size, in each and every sequence? For each and every month? I think I can find a way to display this for you and we'll see.

 

Here's a quick question for you, PeakProphet. Do you agree that atmospheric CO2 will retain heat proportional to its concentration?

 

Thanks for the reference. I don't suppose you are the same kind of non-reading, drive by citation dropper that got Poor Debater into so much trouble are you?

PS: After a quick scan, you really should have read that paper before pretending I was replicating that work. Or that they are examining data at near the level of detail that I am. You didn't even look at the size of the confidence intervals they claim did you? Nor how they were generated, right?

But hey…we can certainly check out to see if some of those conclusions you quoted show up in Idaho.

 

So you're asking if I'll spank you as badly as he did?

Go on, scream out your credentials now. You know, that reflex action thing you do when you're flustered.

 

I don't agree with that.

First, it's kind of incorrect to say CO2 itself "retains" heat, as the heat isn't in the CO2. CO2 causes the whole system to retain more heat.

Second, the heat is more proportional to the log of the concentration, not the concentration.

 

Dunno. This thread is focused on uncertainty and variability of temperature data, starting at the most basic levels, trying to build up to see if fundamental assumptions hold true. Quite a different topic. Does the answer to your question help explain why "global" doesn't include all the globe, or for some reason excludes Idaho? If it does, I'll put it on my list for later, because it certainly would be good to know why global isn't really global.

- - - Updated - - -

Did you see the way he refuted the data and examples I've provided? It was awe inspiring!

 

It is not proportionate, it is logrithmic as I understand it, theorized at 1.2 degrees per doubling, to four times would be 2.4 degrees. The hypothesis is that the warming is due to positive feedback effects. The main feedback effect is that of increased evaporation of the oceans leading to an increased greenhouse effect of water vapor which is already 80 to 90% of greenhouse effect.

 

Can you explain why you deliberately left off the corrections for known errors in the station data?

Corrections are necessary, you refuse to make them, hence your results are garbage. Deliberate negligence at that level verges on being data fudging. Good luck convincing the world that it isn't.

 

Any EDA starts with examining what is. You quantify total uncertainty (including known errors), and it provides the baseline. Without a baseline, you can't then quantify improvement. Or lack thereof.

Currently TOBS adjustments are leading to a fine sunny day in Idaho achieving highs of 50C, so either it can REALLY be unexpected hot in Idaho, or my interpretation of the NOAA documentation on TOBS is incorrect.

As for any other "errors", I am still trying to find their effects within the data. Systemic error should introduce bias, and I have only in the past day or two devised a method to locate and quantify it.

Feel free to demonstrate how you have determined the bias of error visible in the data. My results are anchored on direct measurements in part because of the assumptions, aggregations, normalization, correlations, adjustments, interpolations and extrapolations used to alter this data. You have chosen to buy in wholesale to whatever is being fed to you. I choose to think for myself.

 

How about the homogenization of known, well placed, and unchanged stations where the raw temperature data is modified by other stations because it was not heading in the same direction?

 

That would be naught naughty.

M's comment was inherently anti-science, because it supposes that jumping to conclusions comes first, as opposed to studying what you have, and moving forward from a basis founded in empirical study. CORRECT THE DATA!!! they scream...and yet don't have a clue as to the net effect of such error, aren't familiar with how such errors are found, can't determine for themselves IF they should be compensated for, don't compare that compensation to the original variability to see if it improves the uncertainty involved, I mean they don't know a thing about how one approaches a topic from the perspective of a scientist.

But as I mentioned in the OP, we are talking about internet forum experts who really don't know anything about how science works, let alone speak the language.

 

Naughty but it happens.

I know enough to know some of the problems in how the data is handled and the assumptions based on the results and what isn't reported by the media, such as uncertainty. I also have worked in computers longer than some on this forum have been out of diapers and know the capabilities of modeling or better, the limitations.

 

You could look at the station logs and see when they changed the equipment or station locations, both of which causes the baseline to jump to an entirely different spot.

Best I can tell, your method assumes a constant baseline. Thus, it gives bad results.

That's another illustration of why one should study how the science works before jumping to conclusions. Otherwise, you end up repeating the same mistakes that scientists fixed decades ago. Like you're doing now.

 

Absolutely. And in every climate science paper referenced around here, they are happy to mention it. Assumptions of this, normalize that, interpolate between those. But this type of honesty is expected in science. What I find surprising is what is missing, in everything provided by our local swallowers of the hook, the line, and the sinker.

Scientists in the natural sciences understand uncertainty quite well, but it is not an easy subject for Joe Average. Humans crave certainty, and getting the average Joe to understand uncertainty in measurement is no easy task. I have tried, and failed, to do so myself spanning some 20 years now, reporters, politicians, other scientists, my mother,and it has always been an uphill battle.

But never, not once, have I decided that it is better to abandon honesty and make up scary scenarios in frustration. To me, even THINKING that is something other than science, interfering with the very objectivity needed to do the science. Zealotry isn't just a problem with internet experts who think quoting articles they haven't read and couldn't understand if they had, it exists in very, very smart people as well.

 

You still haven't addressed homogenizing good stations.

 

Let me rephrase the question then. Do you agree that CO2 causes Earth's atmosphere to retain thermal energy that would be otherwise lost to space, in logarithmic proportion to its concentration?

 

From what exactly do you think that Idaho is being excluded?