271 replies »

1. Another very well researched and informative piece on the stolen emails. It’s been interesting to watch the divergence between the wing-nut response to the whole thing (conservative talk show hosts gloating about how there’s no longer any scientific evidence for warming) vs. the relatively more subtle and informed shenanigans of McIntyre, Mosher, etc. While it might be tempting for some to believe the ‘relatively’ more plausible voices, relative plausibility has little to do with fact. Thanks for taking the time to pull apart these claims and add a bit more sanity to the online speculation about climate science.

2. Pingback: Tweets that mention Scholars and Rogues » Climate scientists still besieged -- Topsy.com

3. Brian you are in denial plain and simple.

   What you should have done is reconstruct the original paper(s) from their raw data and see where it takes you.

4. Brian, interesting interpretations you have here. You frame the communication between Briffa and Wahl as different from that of Neil Roberts. Yet just as Roberts sent in a suggested revisions (which Overpeck said needed to go through the formal IPCC process) Briffa shared the Chapter 6 Second Order Draft with Wahl who twice sent it back to Briffa with suggestions, something the IPCC considers reviewer comments and should be logged in keeping with the IPCC policy of transparency. Would you care to explain how this differs from the issue that caused Overpeck to send Neil Roberts through the formal IPCC process?

   You seem to realize that Wahl addressed reviewer comments by McIntyre. Don’t you realize that Wahl’s suggestions to the SOD were also related to McIntyre (specifically M&M 2005 GRL) on the basis of the unpublished at the time Wahl and Ammann paper?

   I had hoped you might continue this to include the E-mail with the subject line IPCC & FOI in which Phil Jones said “Can you delete any emails you may have had with Keith re AR4?” You know, the subject above where Briffa and Wahl passed back and forth IPCC drafts and revisions. This is a big part of this story, but with your interpretations there would be no reason for Jones to have sent this E-mail.

   • JimR – in partial answer to your questions, I’ll refer you to the IPCC principles and procedures, specifically Section 1, Annex 1 of Appendix A on page 13 (pdf page – there’s no page numbers on the document that I could find), on the responsibilities of Lead Authors:

     Lead Authors may not necessarily write original text themselves, but they must have the proven ability to develop text that is scientifically, technically and socio-economically sound and that faithfully represents, to the extent that this is possible, contributions by a wide variety of experts.

     Now, without asking questions of IPCC authorities, I can’t claim that this is the last word. However, my interpretation of this is that lead authors are permitted to ask for help writing their responses to comments and writing their drafts. And Overpeck, who was coordinating lead author for Chapter 6, told me in his interview that

     Keith [Briffa] as a Lead Author could consult with any scientist he wanted. There are reasons for this openness, but for me the key reason is that our job was to assess the peer-reviewed literature up to that point as accurately as possible. This requires openness with respect to communication.

     (The link to the IPCC procedures was blank. It has now been fixed.)

5. “So far as we could tell, Schmidt’s claim appears to be accurate. ”
   You fact-checked Schmidt’s claim, did you?

   “But none of these papers were highlighted (or even mentioned) in McIntyre’s submission to Muir-Russell or the House of Commons,”
   Yes they were, for example Briffa et al 1992, Briffa et al 1995, appear in his submissions as well as in the “fair sample” post.

   Sometimes it can be difficult to tell who is moving the goalposts, who are the scholars and who are the rogues, but sometimes it’s easy.

   • PaulM – Schmidt said that McIntyre came up with 5 papers that should have been answered by the Oxburgh inquiry but that those 5 papers did not show in McIntyre’s submissions to the House of Commons or the Muir Russell review. When I checked, it appeared to me that Schmidt’s claim was correct – there were papers mentioned at the links above that McIntyre claimed were key papers that were not in McIntyre’s submissions. Here’s all McIntyre’s references from his House of Commons submission:

     Briffa, K R, 2000. Quaternary Science Reviews, 19(1-5), 87-105.

     Briffa, K R et al, 1992. Climate Dynamics, 7(3), 111-119.

     Briffa, K R et al, 1995. Nature, 376(6536), 156-159.

     Briffa, K R et al, 2001. Journal of Geophysical Research, 106(D3), 2929-2941.

     Grudd, H, 2006. PhD Thesis, Stockholm University, Faculty of Science.

     Grudd, H, 2008. Climate Dynamics, (DOI 10.1007/s00382-007-0358-2).

     International Panel on Climate Change, 2001. Climate Change 2001: The Scientific Basis.

     International Panel on Climate Change, 2007. Climate Change 2007: The Physical Basis.

     Mann, M E, Bradley, R S & Hughes, M K, 1998. Nature, 392, 779-787.

     Melvin, T and K Briffa, 2008 in M K Hughes, H F Diaz, and T W Swetnam, editors. Dendroclimatology: Progress and Prospects. Springer Verlag

     National Research Council, 2006. Surface Temperature Reconstructions for the Last 2,000 Years. Available at: http://books.nap.edu/openbook.php?isbn=0309102251.

     Wahl, E R & Ammann, C M, 2007. Climatic Change, 85(1), 33-69.

     Wegman, E J, Scott, D W & Said, Y H, 2006. Ad Hoc Committee Report on the ” Hockey Stick” Global Climate Reconstruction.(emphasis added to show equivalent papers between this list and the following list)

     Here’s the complete list of papers from the post I reference:

     Aside from CRU activities at IPCC (the sections in AR3, AR4 and AR4 Review Comments), the most prominent CRU articles criticized here are the following nine: Briffa et al 1992 (the Tornetrask chronology and “Briffa bodge”); Briffa et al 1995 (Polar Urals), Briffa 2000 (passim introduction of Yamal, Taimyr); Briffa et al 2002 (the famous cargo cult “assumption”); Mann and Jones 2003; Jones and Mann 2004; Osborn and Briffa 2005; Rutherford et al 2005; Jones et al (1990) on UHI. (emhasis added to show papers that are identical between the two lists)

     If you add up the papers that McIntyre says are key in the second list, you find that there are five of them just like Schmidt said. I neglected to add this detail to the post above and should have. I’ll attach an update to this effect.

6. A nice piece. I’ll return more later with additional comments, but to start I would like to ask the following question:

   In your email to me you stated the following:

   “2. In his testimony before the House of Commons Science and Technology Committee, Phil Jones estimated that he had sent about 1.5 million emails over the period covered in the published CRU emails. That compares to approximately 200 emails that were from Jones in the published emails (via a quick search at eastangliaemails.com). Do you think that not having access to those other emails limits what we can say about the context of the published emails, and why or why not?”

   I provided an analysis of this claim. You changed the subject in our correspondence when I showed that this claim was questionable on its face. Would you please provide your readers with our exchange and your assesment of the truth of Dr. Jones claim. As I wrote I was unconvinced by this claim. rather than assess the truth of this claim, you changed the subject. Can you explain to your readers why you raised the issue, my treatment of the issue, and your final assesment.
   You can post our exchange with your explanation or I can.

   Further, you wrote:
   “Looking closely at the graph shows that the tree ring data was neither replaced nor substituted. The zoomed-in version of IPCC TAR WG1 Figure 2.21 at right shows that the instrument data starts around 1900 (red line, red arrow added) while the tree ring data ends at around 1960 (green line, green arrow added). If the tree ring data after 1960 were simply substituted or replaced as McIntyre and Fuller claim, then the instrument data would have been appended to the end of the tree ring data or the instrument data would be shown in green in order to maximize the potential for misinterpretation. Neither is the case.”

   The TAR is the third Report. We are talking about the FAR. figure 6.10. But I can make the same point with the TAR was with the FAR. You clearly don’t know how the trick works. Let me explain. The tree ring data POST 1960 is truncated. That is step 1. That step is covered in the text of chapter 6 ( more on that later ) The next step is to SMOOTH the data for the graphical presentation. The smoothing algorithm is a 30 year smooth. Whats that mean? For example,
   if you have data from year 1 to year 100, your first data point is year 15. Its value is the combination of the 15 PRIOR YEARS and the 15 Following years ( for illustration only to give you an idea how centered filters work) your LAST year is year 85. This year is the combination of the prior 15 years of the record and the last 15 years. year 86 has no value because there are not 15 following years. So with a record that goes to 1960 your SMOOTH with a 30 year window should only display up to 1945. The problem of end point padding ( what do you draw from year 1945-1960) has extensive literature. So for example, there is extending the means of adjacent values at both ends of the smooth. ( the proceedure used in Ar4 ch06) In the case of Briffa’s curve, this procedure was not used. It was used for all the other curves, but in Briffa’s case it was not used. To fill out the filter, to supply data for 1945-1960, the INSTRUMENT SERIES was used.
   This has been confirmed by replication. So still, after all this time people do not understand the trick because they have not attended to the math.

   1. the series is truncated at 1960.
   2. a smoothing filter ( typically 30 years) is applied.
   3. To compute the final years of the smooth ( half the filter width) the temperature series is used.

   That procedure is the trick. in a nutshell. If you want directions read Jones’ mail.

   I’ll return to the rest later as you get a few other things wrong. The rules against taking comments outside the process were clear. What you failed to note in the briffa, wahl,overpeck communications are the governing IPCC regulations. Briffa contacted Wahl AFTER the official review period was over. Whal’s comments and edits CHANGED the conclusions of the draft.
   the issue at hand was the treatment of McIntyre’s paper. McIntyre was an official reviewer and author of the paper that Briffa was discussing. Wahl was not an official reviewer and had a paper “critical” of McIntyre. Briffa’s idea of being objective was to contact Wahl, AFTER review had closed to get Wahl’s comments on his argument with McIntyre. In short, his idea of fairness was to give Wahl the last word. or rather to use Wahl’s words as the last words. So Wahl’s words enter the record after the last official review. This fundamental unfairness is exactly the thing the process was set up to avoid. in fact, there were official process for handling this exact kind of issue. In any case, Jones and Briffa seem to have taken the rules more seriously than Overpeck. Briffa, felt the need to label the mail confidential and Jones asked people to delete these mails. Ah yes, and they denied Holland his rights when he requested this material. It’s always the cover up. I will put it this way. If it was clear to Jones and Briffa that the communication with Wahl was above board, if it was clear to the players at the time in the moment, then why label mails confidential? Why fight an FOIA request? Why suggest “Keith could say he received nothing” Why ask others to delete mails?
   These were the questions your should have asked Overpeck. You should have asked him?
   1. Was it within the process to contact and author directly AFTER review was finished?
   2. was it within process to pass the comments of reviewers onto people after review was closed?
   3. Was it within process to allow a non reviewer to edit the final draft after review was closed?
   4. You encouraged Briffa to come up with a figure more compelling that the hockey stick. Do you think this was undue pressure when in fact he was writing you and informing you that he thought the science didnt show any more certainty now than it did during the TAR.

   More later. And please post up that Question about Jones the million mail man.

   More later.

   • First, your analysis of to what I asked you is not on-topic for this post so far as I can tell. If you’d care to post this request at last week’s post (link), I’ll happily post the entirety of your response there. I’ll post your summary here however – you said that you regarded Jones’ claim with “some skepticism.” That should be sufficient for whatever purpose you have in mind.

     If you were talking about AR4 instead of the TAR as McIntyre was (and as I thought you were), then I made an error and will run a correction accordingly. Can you and Fuller confirm that the excerpt from Fuller’s website applies to the AR4 and not the TAR as I said above? Regardless, however, the Climate Audit post in question is in fact referring to the TAR, not AR4. McIntyre wrote “The trick was a clever way of tricking the readers of the IPCC 2001 graphic into receiving a false rhetorical impression of the coherency of proxies (emphasis added).” The IPCC 2001 report was the TAR, not AR4.

     Would you care to provide references to the rest of your claims, Steven? Point us to the emails that you believe support your interpretation that Briffa contacted Wahl specifically to counter McIntyre? I searched through much of the email archive looking for it, but I must have missed it, because it’s certainly not in the emails you did provide as references and that I addressed in the post above. In addition, while formulating my response to JimR, I read what I could find regarding the official IPCC processes, and there’s nothing there that supports your claim that Briffa’s contact with Wahl was against IPCC rules. However, you may well have information that I do not, and so if you have references that contradict the documents I found and Overpeck’s comments to me, I’d love to read them.

     You said

     If it was clear to Jones and Briffa that the communication with Wahl was above board, if it was clear to the players at the time in the moment, then why label mails confidential? Why fight an FOIA request? Why suggest “Keith could say he received nothing” Why ask others to delete mails?

     Aren’t you engaging in the same speculation on motives that you said you and Fuller chose not to engage in? You’re drawing conclusions based on insufficient information, as I pointed out in the post linked at the start of this comment, and in most cases, your conclusions are stronger than the evidence justifies.

     My questions to Overpeck were purely to fact-check what he CCed Briffa and what the email meant. He offered some additional detail, but going deeper wasn’t the purpose. However, his comment stands, and it stands contrary to your own.

7. Who to believe regarding process? Mosher, of “Piltdown Mann” fame, with a history of smearing and defaming leading climate scientists, or Overpeck?

   It’s a tough decision. Oh, yes.

8. Brian

   “First, your analysis of to what I asked you is not on-topic for this post so far as I can tell. If you’d care to post this request at last week’s post (link), I’ll happily post the entirety of your response there. I’ll post your summary here however – you said that you regarded Jones’ claim with “some skepticism.” That should be sufficient for whatever purpose you have in mind.”

   Well it is on topic. The topic as far as I can tell is how you regard and investigate the claims made by the “team.” In short, you presented me with Jones’ testimony at parliament. You took that testimony at face value when in fact it is questionable on its face with 30 seconds of calculation. I pointed this out to you. The premise of your question was undermined, yet you chose to ignore that. Did you write Jones and ask him about this? Does it impact your general impression of his truthfulness. Or was this a line of questioning that did not serve your purpose. In short, you refuse to look at all of Jones’ behavior both during and after the incident with a critical eye. Rather, when it appears that he stretches the truth, you hide that fact. Were you forthright in your assessment of Jones you would report your appraisal of his accuracy. This is about your credibility when you are faced with facts that don’t fit your preconceptions. My question to you is what is your assesment of his truthfulness in his comments made to Parliament. If you cannot or will not make a public assessment then I have to wonder if you wish to engage in a truth seeking conversation. You raised the issue of context. You took Jones comments at face value. I illustrated the problem with those statements. You chose to ignore that. You still chose to ignore that. Why? You thought the claim of 1.5 million mails was credible. You did not pause to do a simple calculation. When I did, you moved on. Why? What does it say about your preconceptions? You should be willing to forthrightly comment that you took those words at face value, didnt think twice about them, and used them as a governing context for asking me questions. When I pointed out the flaw in your mental frame, you didn ‘t see how your preconceptions govern your investigation.

   My method is different. Quite simply I question everything. You don’t.

   • And I’ve repeated Jones’ claims exactly….where? Not in this post. Not in the other post either. I didn’t repeat them precisely because they weren’t credible, and you weren’t the only person to say so. Tom Wigley pointed out to me in his communications that suspected Jones was referring to the sum of his sent and received emails, not just to his sent emails. Given your quick analysis and Wigley’s concurrence that it was a likely exaggeration, I didn’t spend any more time on that particular detail and intentionally didn’t mention it when I pointed out the email output of multiple different professions including climate researcher:

     S&R surveyed its own members as well as Tom Wigley to estimate how many emails were sent per year by different occupations. We found that

     approximately 1,500 emails per year sent by the electrical engineer
     approximately 1,100 emails were sent by the home manager
     between 2,500 and 3,500 emails sent by the marketing professional
     about 1,500 emails were sent by the university English professor
     and about 5,500 emails sent by climate scientist Wigley (with another 33,000 received emails).

     If we estimate that the S&R writers surveyed each receive three emails for every email sent, then we get a yearly total of 6,000 emails, 4,400 emails, 10,000 emails, and 6,000 emails respectively for the S&R writers plu a total of about 39,000 emails per year for Wigley. Over the course of 13 years and for a 15-member workgroup (the period of the CRU emails and the size of the CRU), the total for both the electrical engineer and the English professor is 1.17 million emails, 858k emails for the home manager, a minimum of 1.95 million emails for the marketing professional, and 7.51 million emails for Wigley’s.

     Now, I’ll be happy to continue discussion of this particular detail on the post that produced it, namely this one. It is off-topic for this post.

     I didn’t “change the subject” or ignore it, however, as you claim. In fact, I asked you this question in followup:

     3. Even if we assume that Jones’ email estimate is an exaggeration, even 1100 emails represents a very small percentage of the total email output of CRU over the past 20 years. Does it bother you that to date we don’t know who edited the emails, why they did it, and what criteria they used to choose between what emails they published and what emails they deleted?

     It certainly looks to me that this followup is very much in the same line of thinking as my original question, which was “Do you think that not having access to those other emails limits what we can say about the context of the published emails, and why or why not?”

     The “governing context” I used when formulating questions to you and everyone else I interviewed was Tim Osborn’s statement to the House of Commons, CRU28 paragraph3. Jones’ claim about his own email output was a convenient way to make a point – that there are a LOT fewer emails in the public record than CRU produced over the last 13 years, and to ask you about what you felt about that fact. That I reached conclusions that you don’t like is one of the risks you or anyone takes talking to a journalist – sometimes the journalist will fact-check the claims, find that the balance of evidence appears to go against the claim being made, and report that fact.

9. Brian:

   “If you were talking about AR4 instead of the TAR as McIntyre was (and as I thought you were), then I made an error and will run a correction accordingly. Can you and Fuller confirm that the excerpt from Fuller’s website applies to the AR4 and not the TAR as I said above? Regardless, however, the Climate Audit post in question is in fact referring to the TAR, not AR4. McIntyre wrote “The trick was a clever way of tricking the readers of the IPCC 2001 graphic into receiving a false rhetorical impression of the coherency of proxies (emphasis added).” The IPCC 2001 report was the TAR, not AR4.”

   There are three examples of the trick: The TAR the WMO presentation and the FAR. Steve discusses all three, as do we. However, you still miss the salient point. Let me explain what the salient point is NOT.

   1. the point is not that the data was redacted.
   2. the point is not that the redaction has not been covered in the literature.

   I will just take the FAR as an example. In the FAR briffa used a series he has used before in publication. In those publications the decline was SHOWN and then explained. In the FAR Briffa used this same series. He redacted the decline. A reviwer requested that he SHOW the decline and explain it in the text. As was his practice in prior publication. Briffa said no. In the end Briffa decided that he would redact the decline from the graphic, but explain it it 264 words. Briffa had a choice as Lead Author: Show the decline and explain it in the text ( as was his practice) or redact the decline from the graphic and try to explain in words that something was missing from the chart.
   My question to briffa is why? In order to explain this action, this editorial choice, I rely on the only extant texts I know that cover a similar decision made by the team. The decision WRT the WMO chart. I also look at the emails Overpeck is sending to Briffa as he wrote. I will give you the gist of that, you go read the mails yourself.

   1. At the start of the writing of Ar4 Ch06, Overpeck encourages Briffa to create an image more compelling than the Hockey stick ( see mails starting in 2005)
   2. Overpeck encourages briffa to ignore the comments of Rind. Rind has advised Briffa that he thinks all the uncertainties should be handled openly and forthrightly. Overpeck says pay no attention to Rind.
   3. Briffa was uncomfortable taking conclusions beyond the TAR.
   “Peck, you have to consider that since the TAR , there has been a lot of argument re
   “hockey stick” and the real independence of the inputs to most subsequent analyses is
   minimal. True, there have been many different techniques used to aggregate and scale
   data – but the efficacy of these is still far from established. We should be careful not
   to push the conclusions beyond what we can securely justify – and this is not much other
   than a confirmation of the general conclusions of the TAR . We must resist being pushed
   to present the results such that we will be accused of bias – hence no need to attack
   Moberg . Just need to show the “most likely”course of temperatures over the last 1300
   years – which we do well I think. Strong confirmation of TAR is a good result, given
   that we discuss uncertainty and base it on more data. Let us not try to over egg the
   pudding.
   For what it worth , the above comments are my (honestly long considered) views – and I
   would not be happy to go further . Of course this discussion now needs to go to the
   wider Chapter authorship, but do not let Susan (or Mike) push you (us) beyond where we
   know is right.
   —
   Professor Keith Briffa,”

   4. Overpeck and others pushed Wahl’s paper through the publishing process so that Briffa would have something to counter McIntyre’s questioning of the certainty of the hockey stick. As they wrote, the paper will make Keiths job easier.

   In short, based on the text we have one can raise a plausible argument that goes like this. Briffa, who had been critical of Mann ( those mails i will GLADLY post) was pressured by Overpeck to create an image more ‘compelling’ than the hockey stick. Rind suggested that Briffa should not gloss over the uncertainties in reconstructions. Overpeck told Briffa to disregard Rind. I month prior to the “publication” of the Wahl paper, Briffa was still expressing his reluctance to take an agressive line on the certainty behind reconstructions. The deadlines for papers were extended so that Wahls submission could be considered. That paper had references to another paper that would not be published for a year. Briffa had a choice of whether to show the decline and explain it or redac the decline and explain it. I would argue that this decision, absent any other information, can best be explained by his desire to please overpeck.

   More later..

10. Brian,

    can you explain what Jones meant when he talked about appending the temperature series?
    I explain the proceedure in the book; pg 153 and pg 154: I explain appending the temperature series, smoothing, and then presenting the smoothed result up to 1960. There are thes steps to the trick as Jones understood better than you.

    1. redact the post 1960 data.
    2. append a temperature series.
    3. Smooth to 1960 ( using temp out to 1975)
    4. Show the smoothed line out to 1960.

    Thats the trick ( in one form )

    We discuss the WMO incident and the TAR and the FAR. The proceedure is roughly the same in all three.

    So, your opinion on Splicing and smoothing? Which is what Jones said he did, which is what the record shows he did? or shall we discuss 1.5 million mails in 13 years. dodgy maths..

11. Brian,(6) you’ve pointed out that lead authors “may not necessarily write original text themselves” which is true because there are contributing authors as described on the next page of the IPCC procedures. The scenario you have come up with is contrary to the transparency of the IPCC with outsiders confidentially receiving drafts and suggesting revisions.

    • My responses may be a little slower in coming as I’m traveling. However, rest assured everyone, that I will get to your comments.

      JimR – again, I refer you to Overpeck’s comment that Briffa was free to talk to anyone he wanted. That doesn’t mean just contributing authors – anyone.

      Steven – I have a number of detailed responses that I don’t have the time to write at this exact moment, but for now I’ll simply ask two things. First, instead of providing “read every email starting in 2005” as support for your argument, provide precise references. I’ve done so, and I expect the same courtesy from you and everyone else. I’m certain you have the references available. Second, please stop referring to the Fourth Assessment Report as the “FAR.” According to the IPCC, the “FAR” is the First Assessment Report released in 1991 – they transitioned away from using letter notations for the fourth report because of the ambiguity that was going to occur. The correct notation for the Fourth Assessment Report is “AR4.” After all, “FAR” could refer to the first, fourth, or fifth report.

12. Steve M #13

    Hasn’t Brian already given his opinion of splice-and-smooth in the original post?

    If the scientists had actually substituted or replaced the tree ring proxy data with instrument data, then McIntyre and Fuller would have a valid claim of fraudulent behavior by Phil Jones et al.

13. I’ve also seen 4AR.

14. Brian, sure I agree that Briffa was free to talk to anyone he wanted. Sending confidential IPCC drafts and receiving revisions from someone outside the process is an entirely different matter.

15. It will be interesting to see how this develops. Brian, your quantitative meta analysis trying to invalidate criticism based on small sample size is ludicrous, and you should drop it quickly before you embarrass yourself.

    The more interesting thing for me is to see if this is an attempt at revisionist history or if you are trying to have an honest discussion. Jury’s out on this one.

    At any rate, it has fulfilled one purpose, as in the future people at Deltoid or Desmogblog will be able to casually link here and tell their readers that the arguments we advance were decisively debunked as they Gish Gallop through their alternate universe. And it really doesn’t matter to them how this conversation ends. It’ll end up being posted on a site as an answer to skeptical questions, and it really doesn’t matter to them how this conversation ends, either.

    So the remainder of this thread can actually be honest conversation, if you like. Do you think Jones, Briffa, Overpeck, Wrigley and Mann acted in the best traditions of science, as shown by these emails? Remember that Mosher and I do not argue against either climate science or recent temperature rises in our book. Our claim is that a group of scientists succumbed to pressures both internal and external and acted improperly to first protect their prior claims and reputation and second to preserve a chain of argument for IPCC and other publications. We both stand by our point.

    The emails prove it. They’re online. Your argument from quantity is mere hand-waving–I laughed when I first read it. Your argument from quality fails at the first reading of the emails.

16. For a very good and readable review of the climategate emails:

    Click to access climategate-emails.pdf

    After reading this, you’ll never doubt their intensions anymore. They were up to no good.

17. Very interesting exchange on many issues. Not a scientist myself but work with numbers and statistics daily and as such am mostly curious up to know about the implied question posed by Steven Mosher in his first post on June 8th; namely if the blog-holder accepts the description of “the trick” and still have no issues with it. The implied question was ignored in the following answer but was stated in explicit form later the same day by S.Mosher (June 8, 2010 at 5:00 pm). Such padding to me seems…. curious. Especially when dealing with a proxy-series. Looking forward to the answer (and possible enlightenment).

    Another point that especially drew my attention is the red herring on asking Steven Mosher: “Point us to the emails that you believe support your interpretation that Briffa contacted Wahl specifically to counter McIntyre?” rather than address the fact that after review period outside comments was taken in and text was changed which was not made available to official reviewers or included in the the IPCC review archive. Isn’t this exactly circumventing a transparent review process? Whether this was done to counter McIntyre specifically is a separate issue. Both issues were raised by S. Mosher, one was responded to, the other one ignored.
    Could I in that context ask what part of the text was changed after the Wahl comments? Is it possibly it mostly was the text concerning the issues raised by a certain official reviewer? And irregardless of the existence of an e-mail would that not imply why Wahl was contacted? (Pure speculation I know, confess to not knowing exactly how much text was changed and how many issues were involved. Thus meant as a completely honest question).

18. @Brian,

    While we are cleaning up the Mosher FAR/AR4, that December, 2010 CA article can use some adjustment 😉

19. If only the panels investigating the witch hunter claims had more witch hunters, then it would be truly objective. This is a particularly absurd dismissal of one of the investigators:

    “Lisa Graumlich is a coauthor with MBH’s Malcolm Hughes ”

    She’s a co-author with someone who co-authored with someone at CRU. What idiocy. Michael Mann has co-authored with James Annan, who’s co-authored with Roger Pielke Sr.. Pielke Sr. is obviously can’t be trusted to critique Michael Mann. Neither can Anthony Watts, who has co-authored with Pielke Sr. (hey at least we’re in agreement there!)

20. Brian, congratulations on a fine peice of detailed investigative reporting. Highly valuable work in the context of the mis-representations you clarified in this peice.

21. Steven, you say the WMO chart uses similar methodology. However, didn’t that one actually append the temperatures directly?

    This is the chart that Jones was referring to in his e-mails.
    There is one other e-mail that provides context here.
    From Mick Kelly on Oct 26, 2008

    Yeah, it wasn’t so much 1998 and all that that I was concerned about, used
    to dealing with that, but the possibility that we might be going through a
    longer – 10 year – period of relatively stable temperatures beyond what you
    might expect from La Nina etc.

    Speculation, but if I see this as a possibility then others might also.
    Anyway, I’ll maybe cut the last few points off the filtered curve before I
    give the talk again as that’s trending down as a result of the end effects
    and the recent cold-ish years.

22. OK, the insults are getting a little out of hand. I’m going to ask the Admin to take them out, and I’m asking all of you to tone them down, read the comment policy, and stay on topic here. There’s a lot of information to talk about here and arguments that can be made without falling back on insults.

    • What Brian said. Stick to issues, and personal attacks will be deleted and repeat offenders will be dealt with.

23. You still seem confused about the false Schmidt comment, and are making the wrong comparison. Schmidt’s statement was about papers Mcintyre said that Oxburgh should have included.

    Some of the papers that McIntyre said in his blog post that Oxburgh should have included were listed in McIntyre’s House of Commons submission.

    Again it is you who is trying to move goalposts, not Mcintyre.

24. I’d like to know if Brian now understands how the “trick” was done, and if that changes his assessment of whether the hockey stick graph was fraudulent.

25. Mosher, you claim:

    “In the case of Briffa’s curve, this procedure was not used. It was used for all the other curves, but in Briffa’s case it was not used. To fill out the filter, to supply data for 1945-1960, the INSTRUMENT SERIES was used. This has been confirmed by replication. So still, after all this time people do not understand the trick because they have not attended to the math.”

    But you provide no citation to the evidence on this, nor any description of the magnitude of the effect. Is the different choice of truncations actually in any way evident in the figure? As far as I’m aware this procedural choice has essentially zero visual impact – and your claim that this is what Briffa in fact did is not based on any documentation from Briffa or IPCC, but on “replication” – i.e. you are guessing. Perhaps you are right, but what could *possibly* be the motive? There is no substantial “decline” to “hide”. It would have been extra work for no purpose, if that’s really what Briffa did – what on earth could be the point?

    And why are you blowing this up into some huge, enormous conspiracy when at worst it involves perhaps a couple of pixels in one figure of one IPCC report???

26. I apologize for how long it’s taking me to respond. I should be able to get a response out in the next day or two.

27. Pingback: Desmogging Desmog’s Trick – Part 1 « Climate Audit

28. Brian

    Perhaps you might like to ASK Jones what he did instead of trying to come up with your own version of what he might have done. You might like to then publish an article confirming McIntyre’s assertions.

29. Hmm, it appears the other Steve(McIntyre) directly answered my question. The WMO chart that Phil used did not do the trick in the way Steve Mosher described. The WMO chart appended instrumental temperatures directly. Brian Angliss has statedthat this is fraudulent behavior.

30. >>”There is no substantial “decline” to “hide”. It would have been extra work for no purpose, if that’s really what Briffa did – what on earth could be the point?”<<

    Arthur, the decline they are trying to hide is the well-known "divergence problem," where temperature estimates from tree ring data move in the opposite direction from actual temperature measurements in the 20th century. This obviously represents a problem if you want to claim that tree ring data provides an accurate estimate of temperatures.

    One way to deal with this in temperature reconstructions would be to simply discard the tree ring data. However, this tree ring data was a key component in claiming that there was no Medieval Warm Period, which you need to believe in order to claim that temperatures today are unprecedented in human history.

    What happened instead was the tree ring data was kept in the temperature reconstruction, but steps were taken to "hide the decline" in the 20th century, probably to avoid questions about what the tree ring data actually measures (if anything). Thus, they smoothed (combined) the later years of the tree ring data with the actual temperature data in order to "hide the decline" in the tree ring data (NOT the temperature data as many have erroneously claimed).

    It's not just "a few pixels in a graph" — the "trick to hide the decline" calls the entire temperature reconstruction record into question.

31. Brian: There is no doubt that instrumental records were pasted onto proxies. The principals themselves state this. Not on any FAR or TAR, but in the WMO charts.

    From CRU:

    One particular, illegally obtained, email relates to the preparation of a figure for the WMO Statement on the Status of the Global Climate in 1999. This email referred to a “trick” of adding recent instrumental data to the end of temperature reconstructions that were based on proxy data.

    In the email in question Phil Jones had written

    I’ve just completed Mike’s Nature trick of adding in the real temps to each series for the last 20 years (i.e. from 1981 onwards) and from 1961 for Keith’s to hide the decline.

    Royal Society Research Professor at the University of East Anglia, Andrew Watson, explained this in an article in The Times
    
    Jones is talking about a line on a graph for the cover of a World Meteorological Organisation report, published in 2000, which shows the results of different attempts to reconstruct temperature over the past 1,000 years. The line represents one particular attempt, using tree-ring data for temperature. The method agrees with actual measurements before about 1960, but diverges from them after that — for reasons only partly understood, discussed in the literature. The tree-ring measure declines, but the actual temperatures after 1960 go up. They draw the line to follow the tree-ring reconstruction up to 1960 and the measured temperature after that.
    

    Time reported a confirmation of this description

    According to PSU’s Mann, that statistical “trick” that Jones refers to in one e-mail — which has been trumpeted by skeptics — simply referred to the replacing of proxy temperature data from tree rings in recent years with more accurate data from air temperatures.

    All the above I happily stole from Henry at:

    Desmogging Desmog’s Trick – Part 1

    Not looking so good, Brian.

32. I’m afraid you have stumbled out of your element Mr. Angliss. Truth is a difficult adversary. The scholar and the rogue are quite distinguishable here.

33. Brian,
    At this point, it is standard practice for climate scientists/activists to declare that it “doesn’t matter” and that they are “moving on”

    • And at this point, it’s standard practice for the admin to remind everybody that if you have something meaningful to contribute, you should do so. Empty snark that’s all attitude and zero substance is frowned upon. I’m seeing a lot of comments here that could just as easily come from six year-old C students who lack strong parenting.

      Many thanks for helping us keep the average IQ level in here as high as possible.

34. Skip Smith, there is so much wrong in your comment, I hardly know where to begin. You seem to have entered a parallel universe – one in which unfortunately there seem to be a large number of others as well…

    First, in responding to my comment, I was referring to Steven Mosher’s argument above not about a WMO cover graphic, but about one figure in the 2007 IPCC report. Mosher appears to be claiming that use of instrumental data rather than tree-ring data to determine the end-point smoothing for the period 1945-1960 (under a 30-year smooth) makes some sort of significant difference to that portion of Briffa’s curve in that image. But the choice of end-point smoothing on that scale (15 years out of 1000) would be essentially invisible – use of a tree-ring continuation would perhaps drop the end point very slightly, but that end point was already mixed in with a lot of other curves in the overall “spaghetti”, and hardly a major issue. Unless Mosher has some images showing that his argument actually makes any substantive difference to the figure, from what I’ve seen it’s the tiniest possible nit, and one based on speculation on what the authors did, not on any real evidence that I’ve seen. Again, unless he has some substantive citation to prove his point.

    So what I don’t understand is what could the possible motive of the authors have been to go through the intricate process Mosher describes to move a few pixels on that one image? Anybody have an explanation? It makes no sense to me.

    Skip Smith claims:”this tree ring data was a key component in claiming that there was no Medieval Warm Period,” – but who claims such a thing? The MWP is discussed in the IPCC reports – or do you have some citation where they say “there was no Medieval Warm Period”? Many different lines of proxy evidence – including tree rings! – show some degree of warming around 1000 to 1200 AD. There’s also a great degree of uncertainty about it, since we don’t have much worldwide coverage and there’s intrinsinc uncertainty and discrepancies with the proxies in the first place.

    Smith then says: “which you need to believe in order to claim that temperatures today are unprecedented in human history.” – but nobody says that either. The central issue is the projection of future warming, not present temperatures, which are certainly still lower than temperatures 8000 years ago at the start of the Holocene, if not during MWP times.

    Smith concludes:”It’s not just “a few pixels in a graph” — the “trick to hide the decline” calls the entire temperature reconstruction record into question.” – but why? This makes no logical sense.

35. Arthur, based on your aggressive and slightly insulting tone my hopes for a rational discussion are low, but I’ll start by providing you with a link that makes exactly the graphical pre-post “trick” comparison you are asking for, and also presents a number of emails from the authors discussing the motives for undertaking the “trick”:

    IPCC and the “Trick”

36. Skip Smith, the McIntyre post you point to is talking about precisely the 2001 TAR graphic that Brian Angliss highlights in this very article. Nowhere in McIntyre’s article is there any mention of the “hiding process” that Steven Mosher claimed above. Angliss stated that this was clearly not a case where the scientists “substituted or replaced the tree ring proxy data with instrument data”, and as far as I can tell Angliss is perfectly correct.

    Yes, we know the Briffa data was truncated in 1960 in that graph. The reasons for this have been discussed ad nauseum. The question at hand is whether there was any instance in the IPCC reports where the scientists “substituted or replaced the tree ring proxy data with instrument data”. Mosher claims that this graph (and a similar one in AR4) is such an instance due to the effects of smoothing on the 1945-1960 portion of the curve. This claim is not supported by the McIntyre article you refer to. Mosher has not provided a separate reference. I claim that it would make such an imperceptible impact on the graph that I can think of no possible motive for the scientists to do this. Show me a graph that compares the Mosher-claimed smoothing with the standard end-point-mean smoothing (or even smoothing using the continuation of the tree-ring data past 1960) and we’ll see how much difference it would actually make in the graph; I sincerely doubt it would be even visible except at high magnification.

37. #41 Arthur, as Steve Mohser explained, the TAR graph also substituted instrumental temperatures for the 1961-1974 period that was used in the smoothing. The value in the graph for any year is a 30 year average fifteen or so years forward and fifteen or so years behind that point.

38. MikeN, I can read what Steven Mosher claimed. What I want to see is (A) any proof that his claim is true, and (B) evidence that the different choices for smoothing make any visible difference to the graph that would justify this sort of conspiracy/malfeasance claim.

    (A) could be proved by showing an explicit quote from the “climategate” emails or some other source by the scientists themselves that they actually did what Mosher claimed, but that doesn’t seem to be there. Or actually showing the published graph and alternative smoothing choices proving there’s a significant difference, and that the published one only matches the one Mosher claimed would show both (A) and (B). Should be simple enough – show us the evidence.

39. Skip Smith writes: “[..] the “trick to hide the decline” calls the entire temperature reconstruction record into question.”

    I would propose clarification: It is specifically the decline that calls into question tree rings’ contributions to the temperature reconstruction record. It is the trick to hide the decline (or, since we’re being granular about it, the numerous instances and variations thereof) which calls into question the integrity of the scientists who exert control over that record.

    @ Arthur, the decline in the tree ring proxy temperature estimates has been discussed and well documented by Briffa himself elsewhere – not just in Climategate emails but peer-reviewed literature. I’m unsure what could possibly convince you of its significant and dramatic departure from observed temperatures. If you won’t believe Keith himself, why would you believe anything posted by Steve Mosher, Steve McIntyre, MikeN?

    I think willingness to engage on the topic is commendable, but I really do think you have an obligation to familiarise yourself with the public record at the very least.

40. Mike’s Nature trick

    
    Scroll down and you can see a difference between the two for Mike’s Nature trick. Not sure what the effect is for the WMO graph that Phil Jones used, but then again he used a different trick, as his graph goes all the way forward with instrumental temperatures, which Brian has said is fraudulent.

41. Pingback: New Dogs; Old Tricks: | Watts Up With That?

42. Steve — Mining Executive. Is your last name McCarthy. Why do you guys not stick to thepoint instead of taking cheapshots at each other. OhBTW are your mutterings must be useless untilthey are blessed by aclimate scientist?

43. OK, time to start chewing my way through some of the comments.

    intrepid – Oops. Thanks for the catch on the 2010 error. It’s fixed now.

    Steve (#11): In the post in question, McIntyre appears to be only talking about the TAR, as I pointed out in my comment #9 above. Furthermore, the “trick” he refers to in the original post he referred me to (this post) is from the TAR – you can tell from the graphs in McIntyre’s post. Given that both posts clearly refer to the TAR and do not appear to focus on either the AR4 or the WMO, let’s start with what I actually critiqued, rather than something that I didn’t (the similar WMO or AR4 figures).

    Figure 2.21 from the TAR WG1 report refers to data from Briffa 2000. I looked into Briffa 2000, and the Figure 5 (reproduced below – click to see a larger version) appears to closely match the curve reproduced in the TAR (also reproduced below – click to see a larger version), although without seeing the data that generated both curves, I can’t be more certain than “appears to.” Either way, however, there does not appear to be a glaring discrepancy between the two lines. If you have references that show there is a glaring discrepancy, I’d be interested in reading them.

    Briffa 2000
    

    TAR
    

    Furthermore, the smoothing algorithm for Briffa 2000 is not, as you said in comment #7 above, a “30 year smooth.” According to the caption for Figure 5 in Briffa 2000, it’s “normalised values smoothed with a 50-year low-pass filter.” According to the caption for the TAR image, the data was smoothed “with a 40-year Hamming-weights lowpass filter, with boundary constraints imposed by padding the series with its mean values during the first and last 25 years.” It’s possible that this is incorrect, but if so, it’s upon you to provide evidence of it. Padding data with the mean is entirely reasonable and supportable

    I realize that you were probably waiting for me to respond (and I do greatly appreciate your patience), but I hope you don’t mind if I don’t take your word for it that the procedure you describe is accurate. Provide me the references please. And referring me to 140+ emails in the archives isn’t good enough – given you wrote the book, you should be able to pull out the links easily enough.

    Mosher again (#13): My opinion is that the appropriateness of appending temperature data at the end of a dataset should be investigated. My own personal experience as an EE with knowledge of digital signal processing (a specialized form of discrete data processing) is that there are precious few types of endpoint padding that can’t be justified. The important thing is to ensure that people understand that you padded the endpoints, what you padded them with, and how the padding affects your data and conclusions.

    At this point, the WMO email by Jones has been looked at specifically by multiple investigations, and none of them to date have found any misconduct with respect to the “trick” and “hide the decline.” Given the problems with drawing conclusions from incomplete records such as the CRU emails, the results of the various inquiries and investigations should be given more weight and considered more complete than your own.

    JimR (#18): I provided a reference that said, as Overpeck did, that Briffa was free to get help in writing his part of IPCC AR4 WG1 Chapter 6. Do you have a reference that disproves mine? There may well be one, but until one is produced, I’m going to stick with what I was able to dig up out of the IPCC WG1 policies and procedures.

44. Fuller (#19): Stating that my conclusions in the other post are invalid doesn’t actually make them invalid. I could state that the sky was a beautiful shade paisley, but that doesn’t make it so. I produced a paper that, put into the context of the CRU emails, shows that the work you and Mosher did is insufficient to draw broad conclusions on most points. If you wish to rebut that argument, go ahead and do so in the comments on the other post. If it’s so ludicrous, then it should be easy enough for you to disprove. Thus far you haven’t even tried.

    You asked:

    Do you think Jones, Briffa, Overpeck, Wrigley and Mann acted in the best traditions of science, as shown by these emails?

    No, I don’t. But that doesn’t mean I think they committed scientific misconduct either. Scientists are people just like everyone else, and that’s something that you seem to be unwilling to acknowledge here. Scientists make mistakes, or make bad judgment calls that they later regret, and so on. Only in the most extreme cases does this make them poor scientists. You don’t need to be a saint to do good science. Suggesting that you have to be perfect to do good science is a fallacy, and arguing against the work of a scientist by way of questioning the ethics of the scientist is the very definition of an ad hominem fallacy.

    You also said:

    Remember that Mosher and I do not argue against either climate science or recent temperature rises in our book.

    But this is a major inconsistency in your own arguments, as I pointed out on the other post (and which you didn’t answer). You excerpted from your own book that

    We note that discussion of Climategate, as the scandal has been dubbed, is full of defenders of these scientists who characterize the emails as a ‘tempest in a teapot,’ saying that ‘boys will be boys’ and that the science isn’t affected. We don’t think any of those statements are true. (emphasis mine)

    Mosher, in his interview with me, said both “As we argued, the mails do not and cannot change the science” and yet, above, Mosher describes a hypothetical procedure for smoothing tree ring data that does affect the science. You can’t have it both ways – either the science was affected, or it wasn’t. Choose one or clarify what you mean by “the science.”

    Mikkel (#20): There are all sorts of ways to pad data in order to make it usable. What ways are right and appropriate depend on the circumstances and, in many cases, will be judgment calls. In my experience as an EE with DSP experience, the most common padding by far is zero-padding, but there can be cases when alternating 0/1 patterns or one-padding is preferred. The important thing is to document what padding was done and why.

    It’s not a red herring to request information that someone else should have in support of their argument. Arguing by assertion doesn’t hold much water here at S&R, I’m afraid.

45. Pingback: Desmogging Desmog’s Tricks – Part 2 « Climate Audit

46. Brian,

    I provided a reference that said, as Overpeck did, that Briffa was free to get help in writing his part of IPCC AR4 WG1 Chapter 6. Do you have a reference that disproves mine? There may well be one, but until one is produced, I’m going to stick with what I was able to dig up out of the IPCC WG1 policies and procedures.

    But that’s not what your reference said. You reference the IPCC policies and procedures which say that a lead author “may not necessarily write original text themselves” and leap to the conclusion that they can get help from anyone to write this. But that is moving outside the IPCC process by sending confidential IPCC drafts to people who are not actually involved in the IPCC. assessment and receiving revisions from that outsider. That’s well beyond Overpeck’s “ could consult with any scientist he wanted“. You are making faulty assumptions in attempting to defend the improper actions illustrated in the Climategate E-mails.

    • JimR – maybe you’re right and I made an inappropriate assumption. Can you provide me the link to the IPCC process or procedure or FAQ or official WG1 guidance that disproves my assumption? If one exists I’ll happily run a correction.

47. Brian, unfortunately your grasp of the details is very inadequate and you’ve gotten mixed up between different versions of the trick. You say:

    Steve (#11): In the post in question, McIntyre appears to be only talking about the TAR, as I pointed out in my comment #9 above.

    This is incorrect as readers familiar with the emails or with Climate Audit know. The post in question discussed two different graphics – IPCC TAR and WMO 1999. You missed the segue – which was clear to CA readers – but not to you. The post was an immediate response to the COmmons Select Committee report that day and the segue might have been clearer.

    The post started with a reference to my submission to the COmmittee which discussed the IPCC TAR diagram and its version of the trick to hide the decline.

    The post then discussed the UEA submission to the Committee which discussed the trick email and the WMO 1999 diagram and its different version of the trick to hide the decline. I then commented on the Committee’s findings on the UEA submission (which pertain to the WMO 1999 diagram, not the TAR diagram.)

    By conflating these different tricks, you’ve gotten totally mixed up and are making accusations that are simply untrue (as are your other allegations, though for different reasons,)

    • Steve McIntyre – If you notice, I issued a correction above based on your clarification of the TAR vs. WMO issue and the note you added to the related post at Climate Audit. I apologize for the error.

48. Angliss, the argument from quantity you make in your previous post is so poorly constructed that it does not even rise to the level of wrong.

    1. The model you choose to examine is incorrectly selected. The emails are the exact opposite of a bug that is being actively sought for removal. The emails related to ethical and professional misconduct are being actively concealed. Your argument that behaviour surrounding the emails can be compared to emails surrounding a software bug fails.

    2. Your estimates of the quantity of emails is, as Mosher notes, not realistic (unless you include spam) and not relevant. The quantity of emails is not realistic. It does not correspond with levels of traffic reported by similar professionals in other contexts, and I would refer to the authors and ask if multiple addressees were counted as separate emails (which a server log analysis would mistakenly do). It is not relevant. People engaged in wrong doing live most of their lives and conduct most of their interactions exactly as do the innocent. Hence the ubiquitous phrase, ‘he was a quiet man. I never would have thought…’ Indeed, the more intelligent the ‘malefactor’ the harder the attempt to look and act like the innocent.

    If you were attempting an honest investigation, you would have mapped the remits of the investigations done so far and charted their results, instead of saying that they had largely exonerated CRU and the individuals whose behaviour they looked at. Serious criticism of institutional and personal behaviour was noted. Serious allegations have yet to be investigated. You make no note of what is falling through the cracks.

    Your argument that insufficient data is available for analysis is simply innumerate, and is essentially refuted every time a poll of 1,000 people is extrapolated to correctly predict an election. If you were to assume a total of 50 million emails involving the subjects of the controversy you would only need a sample of 666 emails to be able to make statistically significant statements at a 99% level of confidence with a confidence level of +/- 5%.

    Every one of your arguments fails both logically and mathematically. It is obvious that you reached your conclusion and began to build a model that would provide you with data needed to support your conclusion.

    That this level of Weird Science is serving to distract so many people in 2010 shows several truths: First, the political arguments of those who believe catastrophic global warming is the likeliest outcome of our emissions of CO2 are getting progressively weaker. Second, attempts to construct an electronic record rebutting criticism from Steve McIntyre and Mosher and myself have so far avoided a thorough examination of what we have written and the sources we have used. It is clear that you have not read much at McIntyre’s site, Mosher and my book, and it seems evident that you have not read all of the emails.

    You found a cute little trick. You’re riding that pony for all the links and exposure you can get. But you are wrong on every count.

    • Fuller (#51): If you wish to prove that my prior analysis was wrong, simply stating that it is isn’t a great place to start. Prove it.

      While the emails aren’t bugs, they do represent an electronic record. And according to Aranda and Venolia, electronic records are “deeply unreliable,” as I pointed out in the other post. This is especially true of records that are likely filtered and/or analyzed using electronic methods instead of humans asking questions of other humans. The CRU emails very likely qualify, a point that Mosher claims you and he made in your book. The graphs are entirely clear – the electronic records almost always got the number and names of people involved wrong and almost always got the number and type of key events wrong, and in nearly every case, the errors were not discovered until the paper’s authors sat down with the people involved in the electronic record and talked to them. Let’s not forget that

      the paper found that the electronic records including the email records were missing or had incorrect data, failed to include events that were critical to solving the bug, didn’t describe structural issues and problems related to group dynamics and internal company politics, and had very little explanation of why things were done. For example, the authors found that the steps required to reproduce a bug, the list of corrective actions taken to try and fix the problem, and the root cause were often missing from the electronic records. The bugs often had lifespans that started in advance of the official record or ended either far before or well after the bug was actually declared “fixed.” The authors found that the officially responsible person (ie the bug’s “owner”) was not the person actually responsible for fixing the bug 34% of the time and were totally unrelated 11% of the time. Furthermore, in 7% of the bugs, all of the people listed in the electronic record had no relationship at all to the bug.

      At best, Tom, you and Mosher got half the story by talking to McIntyre et al. But given the findings of Aranda and Venolia, you’ve probably got much less than that. I can certainly appreciate that you don’t want to believe it, but the quantitative and qualitative findings of Aranda and Venolia can’t be dismissed by assertion – they have to be disproven, and thus far you’ve not even come close.

      The estimates of the quantity of Jones’ emails (which I didn’t use in my post, if you recall) are not realistic. But the quantity of emails for the electrical engineer (me in my day job, FWIW), the marketing professional, the english professor, the home manager, and the climate scientist (Tom Wigley) are all based on estimates from the people involved. If you have proof that those numbers don’t “correspond with levels of traffic reported by similar professionals in other contexts,” provide it please. I’d be interested in knowing if my own email totals were high or low compared to other EEs.

      You’re mathematical argument is the worst you’ve made yet. 666 emails would be necessary if they were randomly selected. You and Mosher make the point that they aren’t randomly selected – they were selected from the hundreds of thousands to millions of emails based on certain search criteria. Therefore, it’s not possible to make statistically-valid claims about the emails.

49. JimR: “which say that a lead author “may not necessarily write original text themselves” and leap to the conclusion that they can get help from anyone to write this. […] That’s well beyond Overpeck’s “ could consult with any scientist he wanted“. “

    But, couldn’t that mean the result is a blank sheet of paper or, even worse, completely wrong? Was that a possible intention on the part of the IPCC? I doubt it. If a lead author decides they require help from an expert, it beggars belief to think that they would be shackled to a bench in the IPCC’s dungeon and could only ask other inmates. My bolding…

    “Appendix A to the Principles Governing IPCC Work

    4.2.4 Review
    Three principles governing the review should be borne in mind. First, the best possible scientific and technical advice should be included so that the IPCC Reports represent the latest scientific, technical and socio-economic findings and are as comprehensive as possible.
    […]”

    Click to access ipcc-principles-appendix-a.pdf

    If there’s any doubt then it’s reasonable for someone with responsibility to adhere to the spirit of the aims of the report, if a rule is not explicitly stated. Given the aims of the IPCC are to assess scientific information relevant to human-induced climate change, and WG1 is to assess scientific aspects of the climate system and climate change, I fail to see how anyone being discussed acted inappropriately.

50. Tom Fuller: “2. Your estimates of the quantity of emails is, as Mosher notes, not realistic (unless you include spam) and not relevant. The quantity of emails is not realistic.”

    March 1st, House of Commons Inquiry, Examination of Witnesses (Question Numbers 140-159) …
    “Professor Jones: You have to realise that you have only seen a tenth of 1% of my emails in this respect.”
    http://www.publications.parliament.uk/pa/cm200910/cmselect/cmsctech/387b/38714.htm

    You really should provide hard evidence to prove Jones is mistaken, rather than anecdotal, Mr Fuller.

51. Your argument that insufficient data is available for analysis is simply innumerate, and is essentially refuted every time a poll of 1,000 people is extrapolated to correctly predict an election.

    Tom Fuller: go read a bit about statistical sampling. You’re wrong, but as usual you don’t have the educational background to understand why you’re wrong.

52. Brian Angliss>

    I have little of technical merit to add, but I think it’s worth noting that Steve McIntyre and you are on the same side here, even if you currently have different views. You both just want to work out what’s really going on. He is certainly not pursuing an agenda beyond discovering the truth, and my impression is that nor are you.

    A combative atmosphere is prevalent in these discussions, but there are plenty of reasonable people out there who are capable of discussing matters in a mature, polite manner. We’ll all benefit if you and Steve work together instead of butting heads.

    If you and Steve disagree on something, I’m sure he’ll be perfectly happy to discuss it with you. Standard operating procedure in climate-science blogging is to write a polemic, but it doesn’t help anything. I don’t know about you, but I’m not interested in being persuaded of something; I want it to be true.

53. Gasp! Are we actually going to see an informative and civil back and forth discussion on a climate blog? Stay tuned…

54. I responded over at CA, if you want to continue there.

55. I certainly haven’t wasted any money on the Fuller / Mosher book, nor I have I read all the CRU mails; however I have read enough of the increasingly farfetched defences of their actions and allegations by our brave and ‘honourable’ authors to begin to feel a little sorry for them. Thanks to the simple journalistic precaution of actually checking facts and talking to the parties concerned for the whole context, taken by Brian and others, nearly all their claims of malfeasance, conspiracy, rulebreaking and dishonesty have shown to be either inconsequential, exaggerated or in most cases simply wrong. Over on WUWT, Mr Mosher is reduced to parsing a press interview with Phil Jones, trying to morph the journalistic interpetation of a few short quotes into evidence of Jones lying. All very boring, but in the comments Mr Mosher says of Mr Angliss…

    Since I think you are a lazy journalist and lazy thinker who couldnt fact check whether the sun came up today, …. . Now Brian has already furnished a polite, substantive and responsive reply over there, but Mr Mosher’s remark produced a belly laugh from me, as it reminded me of his own often less than exemplary approach to the tedious task of fact checking.

    Back in February in an attempt to demonstrate that the Climate scientists corrupted the IPCC process, the science journal process, the statistical process, and the FOIA process to silence one man– Stephan McIntyre , Mosher accused Jones of falsifying the acceptance date of a paper saying that he ‘even suggested changing the dates on papers to hide the misdeed’

    But it emerges that Jones was actually making a joke about a printer’s error. It further emerges that this error had been discussed in a CA post, a post that Mosher himself referenced in a WUWT article. So it would be interesting to know whether Mosher was smearing Jones based on poor reading comprehension, a faulty memory or dishonesty. To Mr Mosher’s credit he has acknowledged the blunder, and yet the piece remains online. Seems to me the proper journalistic [dare I say ‘honourable’?] thing – here would surely be to correct the error and apologise for the mistake (and the many others) to his readers and to the Professor before throwing out accusations of sloppiness in others. No?

    Sources:
    http://biggovernment.com/smosher/2010/02/01/leake-and-the-london-times-climate-scientists-thwarted-foia/

    When Was Wahl and Ammann 2007 "Accepted"?

    
    http://www.eastangliaemails.com/emails.php?eid=816&filename=1189722851.txt

56. Here’s what I put there.

    Angliss, here are my assumptions about how your project was born. Feel free to correct any misimpressions that I have. They are not really relevant to the results, which I will return to.

    Assumptions

    1. It appears to me that you wanted to rebut statements made by McIntyre, Mosher and myself. (I am not criticizing you if that is the case–although acknowledging it in your blog posts would have been helpful as well as refreshing.)

    2. You (again, naturally, and this is also not a criticism) did not want to wade through 1,000 posts at CA, plunk down $16.99 for a book by people you don’t like just to pick through it for a couple of blog posts, and you didn’t want to read the commentary surrounding existing chronological versions of the emails and didn’t have time to plough through the raw collection. (Again, for a couple of blog posts I don’t have a problem with this.)

    3. You encountered the study referenced above and immediately made a connection between their findings and what could be said about the Climategate emails.

    It’s what happened next that is important. From reading your two blog posts on the issue and the comments made subsequently here and on your weblog, you refer only to a previously published study regarding software bugs and selected passages from the book Climategate, as well as correspondence with McIntyre and Overpeck. You have arranged them to form a narrative that fits your worldview and the only original work on your part was the creation of your estimates of email volumes for the period covered by the Climategate emails and your estimates of the percentage of the total provided by the Climategate emails.

    What I don’t see any evidence of:

    1. Examining the work you are criticizing.
    2. Evaluating the suitability of the Microsoft study to the Climategate model, or looking for parallel studies or analysis.
    3. Attempting to reason through why the behaviour of the Climategate scientists might be different from software programmers.
    4. Understanding how quantitative and qualitative data sets should be described, analyzed and reported on.

    The problem as it seems to me is that you are trying to paint a faux-scientific patina on what are speculative and kinda fun blog posts. Which is okay until you start slamming people who also use electronic publishing tools.

    I don’t think you did your homework. I think it shows.

57. MikeN (comment #45 here) – thanks for the link – I assume you’re referring to the graph preceded by the text “UC’s figure (violet original, green without “Mike’s Nature trick”)” with a prominent “1998”, an upside-down y-axis from 50 to 250 and an x axis from 400 to 520?

    The preceding text links back to an earlier ClimateAudit post where a commenter “UC” did something that doesn’t seem to be well explained (and the original figures seem to have vanished). In any case, the curves in this figure do not appear to be examining the 1960-truncation issue because (trying to estimate from the curve shape ignoring the very odd x axis) it appears they extend roughly to 1980, not 1960. Also note the differences between the violet and green curves seem to extend over 40 or 50 years, not the 15 years that truncation differences would be limited to under the technique Mosher claims was used.

    Also it’s not clear what the green “non-trick” curve here is – is it smoothed according to the full Briffa data, or is it smoothed according to the mean-padding technique referred to in the IPCC graph itself? Given that it doesn’t seem to stop at 1960, it doesn’t look the right comparison at all.

    Even so, given all that, the difference between the violet and green curves here is very minimal – neither of them goes outside the bounds of natural variation of the superimposed yearly curves. And this particular figure seems to be limited to just 200 of the full 1000 year record, so aside from the factor of 3 or so expansion in the x-axis extent of the discrepancy, there’s another 5-fold expansion of it due to the change in time period. So the real effect on the original graph, even if this is a valid comparison which seems doubtful, would have been very small. What possible reason could the scientists have had for tweaking their graphs so imperceptibly, other than perhaps a certain sloppiness of technique? This hardly looks like any intent to deceive anybody!

58. You missed the segue – which was clear to CA readers – but not to you.

    Ah, clarity. Punctuation like that ain’t gonna help.

59. Brian, thank you for acknowledging that a CA criticism had been made incorrectly. I hope that you will also be convinced on the other issues. 🙂

60. Tom:

    ‘You have arranged them to form a narrative that fits your worldview…’

    Oh the irony Tom!!

    This is exactly what you and Mosher did with your book.

61. Brian:

    It’s a good article, but I think your conclusion about Wahl and Briffa and “potential ethical misconduct vs. professional courtesy” is incorrect. At the time of the emails (June-July 2006), the Wahl paper had not been accepted by the journal, so Wahl absolutely should not have shared anything about that paper with Briffa, especially in light of the IPCC WG1 rule that unpublished articles not be used. Wahl cannot assume that his paper will be accepted in time and go ahead and give his permission for parts of it to be paraphrased. Now, I don’t know if that qualifies as scientific misconduct or not, but surely it should have set off some kind of internal alarm like, “Hey, is this the right thing to do?”

    Bill

62. Brian,

    I must agree with PaulM that you are being disingenuous in discussing the overlap between the House of Commons submission and the blog post.

    In your update, you say:

    “…If you add up the papers that McIntyre says are key in the second list, you find that there are five “key” papers that McIntyre neglected to mention in his submission to the House of Commons, just like Schmidt said…”

    But that is not what Schmidt said, is it?. His words as quoted by you are:

    “…Oxburgh ‘had not looked at the right papers’ and [McIntyre] came up with 5 papers they hadn’t looked at. But none of these papers were highlighted (or even mentioned) in McIntyre’s submission to Muir-Russell or the House of Commons, nor were they mentioned in Andrew Montford’s submissions…”

    A Paul M pointed out, McIntyre came up with 9 papers. A correct comment by Schmidt would have been: “came up with 9 papers they hadn’t looked at. But only four of these papers were highlighted”

    An incorrect analysis would be to say “just like Schmidt said”.

    Time for another apology?

63. #63, I don’t have any additional knowledge of the details beyond what’s in those posts. Keep in mind the allegation is that they are ‘hiding the decline’, so a shortening of the X-axis is irrelevant. Hiding a downturn at the end is key, as it hides a decline in the quality of the proxy, which calls into question whether the proxy is useful in previous time periods.

64. #69 – so even if the “decline” or non-decline was essentially imperceptible or within the noise of the “spaghetti”, that is still an indictment? I don’t get it – it doesn’t match the behavior of any scientists I’ve ever met (and I have met thousands). Scientists care much more about the opinions of their peers than the general public, so the mentality would have been to get things justifiably as technically accurate as possible while communicating the general state of the knowledge. Tweaking the tiny tail end of one graph for the motive you and Steve Mosher have asserted (to “hide the decline in the quality of the proxy which calls into question whether the proxy is useful in previous time periods”) seems just totally unbelievable. Yes, they hid the post-1960 data by removing it from the graph – but they had other reasons for thinking the proxy was useful in previous time periods, so it was justified in that light to their peers. And that should be all that mattered. Choosing a different smoothing technique to tweak a tiny portion (less than 1.5% at the end) of one graph in the middle of dozens of others seems really a very strange accusation.

    But Mosher here in #7 made a very specific claim, stating to Brian Angliss: “You clearly don’t know how the trick works. Let me explain.” Either Mosher is right about “the trick” or he is wrong. So far we have seen no explicit evidence that he is right, and to me there’s a lot of circumstantial evidence that his claim makes no sense whatsoever. Let’s see the evidence for Mosher’s claim, or hear a retraction of it. Because if such a specific detailed claim of behavior by the scientists is flat-out wrong, it leads one to doubt ever other word Steven Mosher has uttered in this discussion.

65. Arthur Smith said: “Choosing a different smoothing technique to tweak a tiny portion (less than 1.5% at the end) of one graph in the middle of dozens of others seems really a very strange accusation.”

    I would assume you arrived at the 1.5% figure by taking 15 years (1960 to 1975) and dividing by 1000 years. But Briffa’s graph doesn’t extend for 1000 years … it starts at 1400, which is 600 years. So that’s 2.5% instead of 1.5%. And according to the original paper, it seems that 30 years (1960 to 1990) was deleted … so that’s now 5.0%. But it’s worse than that…

    The proxy data has to be calibrated and validated by actual temperature records. Now, I’m not going to quibble about exact methods or exact dates, but a naive guess would be: you take a sub-section of known temperature records, calibrate the proxy data to those records, then ‘predict’ another sub-section of known temperature records and see how well the proxies perform. If it is good enough, then you can say something substantial about hundreds of years in the past that do not have temperature records.

    I won’t quibble because: a) I don’t care about the *exact* details, and b) it seems to me that 30 years of “divergence” (i.e., proxy-derived temperatures that *don’t* match the known temperature records) is a substantial portion of the ~150 (1850 to 2000) years of known temperatures. And it’s an even higher portion of whatever sub-set of that data was used for testing the prediction. 30 years out of 150 is 20%, and if half of that 150 years is used for testing, then 40% of the testing data (30 years of divergence out of 75 years of data) doesn’t work. So now we’re talking about something on the order of 20-40%. But I suspect it’s even worse than that…

    I may be reading it wrong, but it appears that Briffa didn’t set aside temperature data for testing, but rather simply calibrated the proxies on the bulk of the best available temperature data, from ~1880 to ~1960. Then *assumed* that this calibration produced good results, and predicted the actual temperatures back to 1400. Then mentioned the “divergence” problem, showing that the proxy-derived prediction doesn’t work from 1960 to 1990, and speculated that other factors affect the proxies during this period, but – miraculously – don’t affect the predictions back to 1400.

    The reason I don’t care about the exact details? Because others far more competent than I have already analyzed this stuff in striking detail, and have pointed out that this divergence is quite enough to question the use of Briffa’s proxies. And Michael Mann (and others) have also acknowledged (secretly, until the CRU emails came out) that it’s a problem … as Mann said, “This [the divergence issue in Briffa’s data] is the problem we all picked up on (everyone in the room at IPCC was in agreement that this was a problem…”

    Beyond that, it is quite clear that the IPCC reports and the WMO graph use a few variations on the trick to hide this problem. The IPCC in particular simply deleted the divergent data to present a stronger narrative, thus giving the impression that proxies work just fine.

    Back to your original point Arthur, merely the fact that those involved spent so much time tweaking the Briffa portion of the spaghetti graph shows that it was perceived – by their peers – as a very important issue. Beyond that, it shows the sorts of acrobatics that they are capable of. If they did this in the semi-transparent forum of the IPCC, then what are these authors capable of when preparing their individual papers? What sorts of rationalizations and tweaking of data did these authors do by themselves?

    Even worse, the CRU emails uncover a disturbing tendency to accept “good” results with little push-back, and focus like a laser on “bad” results that don’t fit the narrative. That is much more of a problem than “1.5%” of a single graph, I’m afraid.

66. @Tom Fuller:
    “It will be interesting to see how this develops. … your … analysis trying to invalidate criticism … is ludicrous, and you should drop it quickly before you embarrass yourself. ”

    Too late, Tom.

67. Tom (#62)

    Assumption #1: incorrect. I’ve had experiences reading McIntyre’s work, and I’ve found that some of it is very good while some of it is not. That I found statements that I felt needed rebuttal is entirely coincidental – the first post on how much context is in the CRU emails was the entire point of my contacting Mosher and McIntyre. I didn’t even find you until about 85% of the way through writing the first piece, when I was fact-checking something Mosher said you and he wrote in the book – that’s when I came across the excerpts on your website. In the process of writing the first piece, however, I found a significant number of what I consider to be inaccurate, inconsistent, or misleading statements by Mosher and McIntyre. It was after I did so that I decided to run a companion piece to the original where I addressed the statements that I found. I have enough material that I haven’t used yet (they were more of a “grab bag” and didn’t really fit the main themes of the second piece, so I cut them out to save over 1000 words) to run at least one more piece.

    Assumption #2: partially correct. I’ve read every email to which I’ve been provided specific reference, as well as a number of others. Probably about 10% of them to date, although I’m slowly working through the others (I have the complete archive on my computer, although I find eastangliaemails.com is more user-friendly). You’re correct, however, that I don’t have the time to read 5+ years of Climate Audit posts – I’m an electrical engineer who works in aerospace at a 40+ hour per week job. I admit to being tempted to acquire your book, however, although I’ll wait until after the Muir Russel review has posted their results – I expect that their review will overwhelm anything you’ve done, even if they agree 100% with everything you said.

    3: Incorrect. I encountered the paper well after I started this project, about 2 weeks into doing research (it took me about a month of research and interviews and another week of writing ) when I was pointed to it by someone whom I’d asked for research assistance with a note that it might be applicable to what I was working on. I’ve been interested in this ever since I realized that my own personal experience as an EE informed me that your approach was fundamentally flawed (shortly after the House of Commons report was released, when I first read Tim Osborn’s quote), and the paper provided a non-subjective way to illustrate that point.

    You’re correct that I arranged the quotes into a narrative, but that’s what journalism is – taking the available facts and creating a narrative out of them. However, I interviewed more than just Overpeck and McIntyre – I contacted Eugene Wahl for comment (he refused multiple requests), Keith Briffa (who responded but asked not to be quoted), Martin Vermeer, Steven Mosher, Gavin Schmidt, Tom Wigley, and a few others. The difference is that, while I had an idea where the paper was going, it was guided by my research rather than my personal opinions. In fact, Tom Wigley and Mosher’s responses sent my original idea careening while reading the paper sent my work off in a new, but related, direction.

68. >>”However, I interviewed more than just Overpeck and McIntyre”<<

    But you didn't really interview McIntyre, did you? You sent him an email for comment, he pointed you to a blog post, and you didn't read it.

69. Hi Brian,

    Thanks for the clarification. If you decide to proceed with this subject and would like an electronic version of our book, I will be happy to provide it to you.

70. Hi Brian, I posted up a response at my place of business–feel free to react.

    Also, did Steve or I ever mention in all of this that part of the reason we wrote our book was to umm, provide context for the emails? Just so you know, when you say that the emails don’t provide adequate context…

    I hope the context we provide is useful. But just as a thought experiment, could you tell us anything that might be adequate context for these?

    Phil Jones: And don’t leave stuff lying around on anonymous download sites—you never know who is trawling them. McIntyre and McKitrick have been after the Climatic Research Unit … data for years. If they ever hear there is a Freedom of Information Act now in the United Kingdom, I think I’ll delete the file rather than send it to anyone.

    Keith Briffa: I believe that the recent warmth was probably matched about 1000 years ago.

    Malcolm Hughes: I tried to imply in my e-mail, but will now say it directly, that although a direct carbon dioxide effect is still the best candidate to explain this effect, it is far from proven. In any case, the relevant point is that there is no meaningful correlation with local temperature.

    Tom Wigley: I have just read the M&M stuff critcizing MBH. A lot of it seems valid to me. At the very least MBH is a very sloppy piece of work — an opinion I have held for some time.

    Tom Wigley: A word of warning. I would be careful about using other, independent paleoclimatology … work as supporting your work. I am attaching my version of a comparison of the bulk of these other results. Although these all show the “hockey stick” shape, the differences between them prior to 1850 make me very nervous. If I were on the greenhouse deniers’ side, I would be inclined to focus on the wide range of paleoclimatology results and the differences between them as an argument for dismissing them all.

    Phil Jones: Can you delete any emails you may have had with Keith regarding the latest Intergovernmental Panel on Climate Change report? Keith will do likewise.

    Let your imagination run wild!

71. Tom (#76) I hope you’ll forgive me for not responding to your error and fallacy filled post, Tom. I did, however, take note to add it to the list of misinterpretations and errors that I was going to deal with next. Thanks for providing so much additional material.

    The number of possible contexts for the email quotes you’ve provided is nearly infinite, and I won’t bother to play this particular game. The Briffa and Hughes quotes alone are so easily countered that I’m surprised you bothered to include them.

    The fact you did, however, suggests again that you’re being inconsistent (a charge that you haven’t yet responded to) – you claim in your book that the emails don’t change the science, yet these two emails would appear to change the science. Choose one claim and stick with it, please.

    Skip (#74): First off, I did read it, and I concluded that it had been well discussed elsewhere that I could reference (namely Deep Climate’s blog) and that I had nothing new to add. So I read other posts of McIntyre’s in his “Climategate” category that I felt applied to my questions.

    I interviewed McIntyre as much as I was able – his responses were VERY brief and didn’t actually answer the questions that I asked. Here’s one example (this communication was on the record, and I said so in the process of asking him my questions).

    I’ve placed some emails into context e.g. the trick email analysis here http://climateaudit.org/2009/12/10/ipcc-and-the-trick/. In this particular case, it seems to me that the email looks worse placed into context. Other situations are similar.

    If Osborn wishes to argue that the emails are mitigated by context, then he should provide the other emails that demonstrate the mitigation. Until they do so, I don’t see any reason to take seriously the idea that additional emails would show mitigation or why it is unreasonable to proceed on the record that is available.

    —–Original Message—–
    From: Brian Angliss [mailto:angliss@scholarsandrogues.com]
    Sent: May-10-10 4:27 PM
    To: smcintyre25@yahoo.ca
    Subject: Two questions about the context of the CRU emails

    Mr. McIntyre,

    I’m a climate/energy writer and blogger at the website scholarsandrogues.com, and I’m writing a piece on the CRU emails. I’m hoping that you would be willing to answer a couple of questions for me, on the record:

    1. Tim Osborn of CRU was quoted in the House of Commons investigation report as saying that it was “impossible to draw firm conclusions from the hacked documents and emails. They do not represent the complete record, and they are not a random selection from the complete record.” How much context do you feel the CRU emails have, and what do you think would be required to put the published emails in their proper context if there is insufficient context at present?

    2. In his testimony before the House of Commons Science and Technology Committee, Phil Jones estimated that he had sent about 1.5 million emails over the period covered in the published CRU emails. That compares to approximately 200 emails that were from Jones in the published emails. Do you think that not having access to those other emails limits what we can say about the context of the published emails, and why or why not?

    Thank you very much for your time.

    Brian Angliss
    Climate/energy writer
    Scholarsandrogues.com/
    Member: Society of Environmental Journalists

    Unfortunately, this is about what I got from my followup questions as well – no direct answers to my specific questions. Given that he had provided such a brief response, I asked what posts he suggest I read for CRU email context, and he suggested the entire “Climategate” category.

    McIntyre’s response contains at least one fallacy, actually, a point that I’ll go into in greater detail later.

72. I put this post on ClimateAudit, but since you didn’t respond to it there I will repost here.

    Brian,
    You admitted an error and struck out a significant portion of your blog posting but the facts are still not as clear to most readers as they could be. You struck out this quote from Steve:

    “There is no valid statistical procedure supporting the substitution of tree ring proxy data going the wrong with instrumental temperature data to create a false rhetorical impression of the coherence of the proxy data.” (emphasis added, source)

    And you struck out this portion you wrote:
    “If the scientists had actually substituted or replaced the tree ring proxy data with instrument data, then McIntyre and Fuller would have a valid claim of fraudulent behavior by Phil Jones et al.”

    I’m glad to see you changed your position when confronted with the facts and struck out this portion of your blog post. Instead of just striking out the error, why not write a blog posting with a positive statement to make things clear for readers who are new to the subject? Something along the lines of:

    “I learned something important from Steve McIntyre and ClimateAudit today. I learned Michael Mann and Phil Jones both redacted the temp reconstruction based on tree-rings and replaced the data with the surface temp record. There is no statistical support for a replacement or substitution of this kind. I now understand now why some skeptics are complaining about fraudulent behavior among climate scientists.”

    I am certain you can write it better than I can.

73. Hi Brian,

    Really? The answers seem clear to me. The answer to 1 is basically that the climategate emails are bad enough by themselves, and look even worse in context. The answer to 2 is basically no, because if there were other emails that demonstrated these emails were taken out of context, they would have been released by now.

    But as I said back on CA, these word parsing games are a worthless waste of time. I’d rather focus on the science.

    Earlier you said “If the scientists had actually substituted or replaced the tree ring proxy data with instrument data, then McIntyre and Fuller would have a valid claim of fraudulent behavior by Phil Jones et al. However, nothing was substituted or replaced.”

    Do you now agree that in some cases the temperature record was substituted for the tree ring proxy data? And if so, do you agree this constitutes fraudulent behavior by Jones at al.?

74. Skip,

    After a few hundred posts I finally learned something mildly interesting from Steve McIntyre and ClimateAudit today. I learned Michael Mann and Phil Jones in a couple of graphs both redacted the temp reconstruction based on tree-rings and replaced the data with the surface temp record. In McIntyre’s lay opinion there is no statistical support for a replacement or substitution of this kind, however the impact on the graphs is barely perceptable and the impact on the science inconsequential so I don’t understand why some skeptics are making themselves look foolish by complaining about fraudulent behavior among climate scientists when such claims demonstrate just how poor is their supply of supporting evidence.

    Fixed that for you!

75. Phil

    Every bolded comment in your post is clearly untrue to anyone with any knowledge of the events in question – why would you even bother?

76. Brian, I am interested in the amended part of your blog as quoted:

    “If the scientists had actually substituted or replaced the tree ring proxy data with instrument data, then McIntyre and Fuller would have a valid claim of fraudulent behavior by Phil Jones et al. However, nothing was substituted or replaced.”

    Can you expand on why you deleted this paragraph, which is a conditional statement by you; IF [condition] THEN [conclusion]

    The condition is “[scientists] substituted or replaced the tree ring proxy data with instrument data”

    The conclusion you drew was “McIntyre and Fuller would have a valid claim of fraudulent behavior by Phil Jones et al.”

    Can you state more clearly why either of both of these elements to your statement are no longer correct, or of relevance to require you to delete them.

    As it stands it appears as if you have discovered that the condition does in fact hold (measured temperatures were substitued in for deleted proxy data), but you want to row back on your conclusion (this makes a case for fraudulent behaviour).

77. >Yes, they hid the post-1960 data by removing it from the graph

    The hide the decline e-mail refers to a WMO presentation where the graph does not stop in 1960. What do you think happened in this one where the graph kept increasing? The allegation is that they grafted on instrumental temperatures.
    You keep talking about 1.5% of the graph, but the length of the X-axis is not important.
    Suppose they had a proxy going back 100000 years, but in the last 100 years kept going down.
    I then edit that to show that it is going up to match recent temperatures. Hey it is just .1% of the graph!

78. Phil Clark,

    It is not just McIntyre’s “lay opinion” that grafting the surface temperature onto a proxy reconstruction is inappropriate. It is also the view of Michael Mann and apparently Brian Angliss (among many others). You may wish to revise your post.

79. Phil Clarke,
    I am the one who suggested Brian write what he learned in a clarifying statement, not Skip. The way you changed my suggestion does not fit well with what Brian wrote and later struck out in this blog post. Read the struck out lines again and you will see that Brian admits:

    “If the scientists had actually substituted or replaced the tree ring proxy data with instrument data, then McIntyre and Fuller would have a valid claim of fraudulent behavior by Phil Jones et al. However, nothing was substituted or replaced.”

    To replace one type of data with another in the same graph is fraudulent behavior. Brian admitted it. You cannot find any statistical textbook that would condone this. Now Brian admits “Jones et al” (which no doubt includes Michael Mann) did it. So it is time Brian issued a clarifying statement and thanked Steve McIntyre for what he learned..

    Also. on ClimateAudit Steve McIntyre asked Brian if there was anything wrong with Steve’s statement when he said:
    “There was no testimony to the Committee (nor has it ever been suggested) that the tree ring data was measured incorrectly or that the data was “erroneous” – the data is what it is. The tree ring data [the Briffa reconstruction in this context] goes down instead of up – but that doesn’t make it “erroneous”.”

    Brian replied “Nope, that statement seems fine to me. It is based entirely in fact to the best of my knowledge.”

    Based on this answer, I have asked Brian to run a second correction on this blog post because he clearly quotes this sentence and then declares “This might be true if the data supported it, but the data clearly doesn’t.”

    Look for this right after the struck out portion of this blog post. I do not know why Brian has not run the second correction yet, but I’m certain he will.

    The problem is that Brian wrote this blog post after talking to climate scientists who did not think Brian would actually check the facts. Unfortunately, he did not check facts until he after he wrote it, so it is a little embarrassing. I think Brian is going through the same process Judith Curry went through. It is a difficult road but the fact Brian was strong enough to admit his mistake and run the first correction is reassuring. He is a smart guy. He will figure out where the facts take him. Give him time.

80. Phil, irrespective of the conclusions reached there would be no need for all the endless tit for tat argument and counter argument if the data was openly available to those that wished to check the conclusions and where errors in data or methodology are found corrections made.
    Skeptics have a right to be skeptical, as media has inflated the confidence of any underlying science.
    By the way which expert statistical opinions back this approach if in your opinion a lay argument (opinion) doesn’t suffice?
    If there are no grounds for taking this approach would it not suggest an approach of getting data to fit the desired outcome?

81. I don’t know if all the software authors had a PhD — or even a masters.

    Let’s look at this in a slightly different manner. The assertion is that people with a PhD who run research projects, write long technical reports and, write grant proposals and routinely make involved technical presentations are in the habit of writing emails wherein they say things they don’t mean and write things that are easily misunderstood. Furthermore they are in the habit of creating technical graphs that can be easily misconstrued because they routinely make a lot of mistakes. Furthermore they are in the habit of contradicting themselves because of these mistakes. (assumption).

    However, I read every email in the file and I did not see a lot of retractions and corrections in the email record — which would be consistent with your assertion [that they get it wrong a significant amount of the time]. Perhaps you can point this out or explain the lack.

    Why would a gang like this be advanced millions of dollars in grant money? Good question?

    Why would people change social and economic policy in the values of trillions of dollars of public money based on their work if they cannot say what they mean? Reasonable question?

    When you assure me that the authors of the software study were studying people at the PhD level, that were in fact the R&D directors then it would be worth my time. Is it so? If they were studying junior programmers and customers then I could see that the study might be what you said — as little as you offered of the basis of the study.

    Hope that’s more clear.

    Now if they want to accept your defense, then I suspect they are about to lose any hope of further grant money.

    I made a similar post on CA which you never answered.

    I am curious if you have a response to these points.

82. #47 Brian Angliss

    Briffa 2000 as published is *not* the series used in the TAR graph. This is shown to be the case in: 1. Deep Climate’s post which you cited. 2. a CA article “Swindle and the IPCC TAR Spaghetti Graph” which was linked to by DC in the post you cited. 3. The emails.

83. There are many, many claims that have been raised here in the comments. I really don’t have time or expertise to address any significant fraction of them. But I would like to get to the bottom of just one thing, that I have repeatedly been asking: is there any proof of the intricate “hide the decline” process that Steven Mosher claimed, in detail here, in comment #7? He explicitly stated this was regarding the IPCC reports, not WMO reports which I personally have never looked at – he said (oddly unfamiliar with standard IPCC terminology) “We are talking about the FAR. figure 6.10. But I can make the same point with the TAR was with the FAR.”

    The question of the percent of the x-axis affected (Mosher’s claimed method would modify only 1.5% of a graph in the 1000-year time period displayed in the figures in question) is directly relevant to the message the graph conveys to those reading it. This is less than a 1/60th portion of the extent of the figure – in itself the time period is almost too short to make a noticeable difference on such a graphic. And then, the difference between the two different smoothing methods over that period is also not very large on the vertical axis – perhaps a few percent there.

    If the purpose of “hide the decline”, which Mosher claims that Brian Angliss didn’t understand, was to make such a tiny change on an imperceptible portion of a graph – my question is just why? Why on earth would the scientists make an unjustifiable change that had essentially zero visual impact? It makes absolutely no sense to me.

    Or if Mosher is really just wrong here but there is some other specific issue that the climategate emails raise that you all think is important, state it as clearly as Steven Mosher did in #7 – and first please agree among yourselves exactly what the problem is – but then, state it clearly, and we’ll see. Once again we just seem to have a whole lot of nothing when we examine any specific claim (like Mosher’s) in detail. And Mosher hasn’t bothered to come back here and explain or provide evidence – I think that’s called a “drive-by”. Not very impressive at all.

    My point is – if you’ve got something, get it right, in detail, in full. Don’t just keep skipping around to some new claim or statement of malfeasance, when each one that’s been examined in detail turns up empty.

84. Being a sceptic is not incompatible with having a sense of humour, boys and girls.

    Ron – if you read the para before the one starting ‘If the scientists had actually …, it is clear that ‘scientists’ means ‘CRU scientists’, which excludes Michael Mann, so where could Brian have learned from McIntyre that Mann conflated proxy and instrumental data, as you stated?

    Notwithstanding Brian’s opinion [which needs to be read in conjunction with the piece he is responding to], it is not necessarily the case that plotting instrumental and proxy data together is necessarily ‘fraudulent’. If fraud was the intention then the fraudsters are hardly likely clearly to label the sources of the data being plotted and discuss the ‘divergence problem’ in the IPCC reports and the literature, which is what happened.

    Fraud is about the most serious allegation one can make against a professional scientist – yet despite the thousands of words expended post ‘ClimateGate’, nobody throwing the word about on discussion boards to my knowledge has levelled a formal accusation of same against any of the academic institutions employing these charlatans. Why would that be?

85. Aurthur, in TAR, the “trick” is a two step process:

    step #1. truncate the low frequency Briffa reconstruction (published as Briffa 2001). This is not sufficient to complete the trick. Briffa displays a graphic with the series truncated at 1960 here (H/t Deep Climate): http://www.ncdc.noaa.gov/paleo/pubs/briffa2001/briffa2001.html
    Note that unlike the TAR graph, the endpoint in this graphic is clearly visible wrt Jones et al (also in TAR).

    step #2: add the insturmental data to the truncated series, smooth, and truncate at 1960 again. This was originally replicated by UC a few years ago. A recent post by Jean S discusses this process and links back to UC. For effect Jean also displays a graphic originally posted by Jones which shows WMO *before* adding the insturmental series for the WMO “trick” – IOW Briffa truncated at 1960 with no further processing of the endpoint. Compare with the “post smoothed/padded” version in TAR where the endpoint appears to merge with Jones et al. Here is the article: http://climateaudit.org/2009/11/20/mike%E2%80%99s-nature-trick/

86. Arthur Smith, you seem to keep on missing the very simple point. Proxies are supposed to track temperature. The only way we can tell if this is the case is by examining their relationship to the instrumental temperature record which only exists since about 1860, From around 1860 the Briffa series has a reasonable relationship to the instrumental record. Thereafter the Briffa series go down when temperature goes up. The natural conclusion is that the Briffa “proxies” are not in fact very good temperature proxies at all. Deleting the series after 1960 deletes precisely that part of the evidence which calls into question the Briffa series as temperature proxies. The pre1860 portion of the graph has no bearing on this divergence problem. As Mosher has already pointed out to you the deletion affects a substantial proportion of the years for which instrumental data is available.

87. Sorry, editing mistake in previous post. I accidentally deleted the sentence which should have said that after about 1950 the Briffa series begins to go down.

88. Despite the thousands of words accusing McIntyre of fraud (“Climate Fraudit”, etc.), nobody throwing the word about on discussion boards to my knowledge has leveled a formal accusation of same against him. Why would that be?

    Hey, this game is fun!

89. Hi, Arthur. You said, “Why on earth would the scientists make an unjustifiable change that had essentially zero visual impact? It makes absolutely no sense to me.”

    I guess you didn’t like my long post in response to yours, so I will try a shorter one.

    It seems that your main issue is that the “1.5%” difference is too small to worry about in a 1000 year graph. This number – 1.5% – is incorrect. Depending on the version of the trick, anywhere from 15 years (1960-1975) to 40 years (1960-2000) of Briffa’s proxy data was removed and/or replaced with instrument data. Further, Briffa’s temperature reconstruction only goes back to 1400, not 1000. Finally, this portion of the data – i.e., the most recent – is a quite substantial portion of the ~150 years of *instrument* temperature data. The instrument record is the only portion used to validate and/or calibrate the proxy data. Therefore, at a minimum, it represents 10% of the record: 15 years out of 150. And that is being quite generous, for all the reasons I stated in post #71.

    You seem to be having difficulty accepting this basic fact, as you have mentioned 1.5% a few times. Perhaps we should straighten out this part before moving on to more complicated information? Can you explain why, in light of the above, you still believe it is only 1.5% of the record?

90. Steve Mosher was referring to IPCC reports, but the original ‘hide the decline’ e-mail refers to a presentation given by Phil Jones, where he copied Mike’s Nature Trick.

    IPCC and the Briffa Deletions

    You can claim this is no big deal, just a small percentage of the graph, but as I posted above the visual presentation is considered important by some of these scientists. Mick Kelly said he was taking off the last few points of his graph to hide a decline in temperatures.

91. THE IDEA OF A FIGURE, IS THAT FIGURES CAN BE MORE COMPELLING AND CONNECT BETTER THAN TEXT.

    Jonathan Overpeck in the Climategate e-mails, and an IPCC lead author.

92. MikeN, mikep, Ted, Layman Lurker – all of those things you say may or may not be true, but please *focus*. The question before us that I am trying to get an answer to is whether or not Steve Mosher in comment #7 was correct in his very specific claims about what the scientists did regarding two IPCC graphs. He claims to be able to provide “context” (as does Fuller) about the climategate emails. He seems to think we should trust his assessments and buy his book. So here is a very clear straightforward claim he made – surely we can figure out, is it true?

    Brian Angliss made a strong statement that “If the scientists had actually substituted or replaced the tree ring proxy data with instrument data, then McIntyre and Fuller would have a valid claim of fraudulent behavior by Phil Jones et al” – and went on to examine the IPCC graphs that seem to have been the basis for those claims. Brian has now crossed that out since McIntyre provided a clarification that his claim applied only to a WMO graph. I don’t know anything about the WMO graph in question; I have never seen or read WMO reports. I am interested in the claim regarding IPCC graphs, in particular because Mosher above made the very specific claim on how the scientists actually did “substitute or replace tree ring with proxy data” in comment #7 here.

    Was Mosher pulling claims out of thin air, or are his specific claims here correct? That is what I want an answer to. Nobody has supplied clear evidence yet on this point.

    The two figures in question are from the 2001 IPCC report (TAR) – figure 2-21, available here:
    
    (a portion of this highlighted by Brian above)

    and from the 2007 IPCC report (AR4) – figure 6.10b, available in the PDF of chapter 6 here:

    Click to access ar4-wg1-chapter6.pdf

    The first figure shows a “Briffa(2000)” curve in green; the second shows a “Briffa et al (2001)” curve in light blue. Both curves do appear to end in 1960, and both curves appear to end with negative slope curling off to a flat line or perhaps a small rise. That slight curling off is exactly what you would expect from the smoothing terminations claimed to be used by the IPCC authors – in the 2001 case it was:

    “All series were smoothed with a 40-year Hamming-weights lowpass filter, with boundary constraints imposed by padding the series with its mean values during the first and last 25 years.”

    and in the 2007 case it was:

    “All series have been smoothed with a Gaussian-weighted filter to remove fluctuations on time scales less than 30 years; smoothed values are obtained up to both ends of each record by extending the records with the mean of the adjacent existing values.”

    In fact if you look at the curves in the 2007 figure, almost every one of them has the same flattening termination behavior.

    Now, I don’t know if “Briffa (2000)” is supposed to be identical to “Briffa et al (2001)” – AR4 does state that the Briffa et al (2001) curve was used in the TAR (and it also has a separate Briffa(2000) curve with a note that it was recalibrated by Briffa in 2004). Examining the two curves between TAR and AR4, there are some quite noticeable differences, but possibly just due to the different smoothing choices. Comparing the AR4 curve with the “Briffa 2001” curve Layman Lurker linked to:

    http://www.ncdc.noaa.gov/paleo/pubs/briffa2001/briffa2001.html

    they seem to match quite well – the Briffa 2001 curve in the AR4 figure also dips well below most of the other curves at the 1960 end, although there is an even deeper dip around there from the (dark blue) Moburg et al 2005 reconstruction so it doesn’t look as much out of place as in the NCDC figure. But look at the numbers – in the NCDC graph the peak in the Briffa curve around 1940 is a little below 0 degrees anomaly while the end of the curve is a little below -0.2 degrees anomaly. In the AR4 graph the Briffa curve also peaks around 1940 at a little below 0, and then ends with the smoothing tail-off at about -0.17 degrees or so.

    So comparing like with like, it looks like the NCDC graph had no smoothing, while the AR4 graph had some sort of smoothing – but why would it not be the smoothing claimed by the IPCC? I don’t see any evidence that the Briffa 2001 curve in the AR4 graph was smoothed with the instrumental numbers. If it was, it must have had an effect of less than the 0.05 degrees (over, in this case, less than 15 years out of 1300) that is the difference in the endpoint between the NCDC and AR4 graphs.

    Now, as to the Briffa 2000 curve in the TAR, it peaks around 1940 at about 0.03 degrees, then drops to end around 1960 around -0.03 degrees. So the end-point seems quite different from the AR4 curve – the data must have been calibrated differently or in some other way have been differently analyzed. But even there, the curve ends in a down-ward slope curling to flat, exactly as expected from the mean value endpoint smoothing that was claimed. If the curling is removed from that end of that curve and you projected the maximum slope downwards in analogy with the AR4/NCDC difference, you would see the end point drop to -0.07 or so. The green curve would be slightly more noticeable than before, but it seems an extremely minor effect.

    The pointers people have provided to ClimateAudit analyses of this point so far are completely beside the point – they do not address Mosher’s claims regarding the IPCC curves. Was Mosher right, or was he blowing smoke? That’s a pretty detailed elaborate claim to make just out of thin air…

93. @ Arthur #98,
    The short answer to your question is Mosher is right. Read his book and you will have more data to examine his claims.

94. @ Phil #90,
    It is pretty clear from the context that “Phil Jones et al” includes Michael Mann. Mann is the “Mike” of “Mike’s Nature trick” so I’m not sure why you are trying to exclude him from the “scientists” Brian is discussing.

    Yes, fraud is about the most serious charge you can make against a scientist, but I do not see how that helps Phil Jones or Michael Mann. There is no justification for blending data from two different sources. Any statistical student would be shocked to see this. Some may call it fraud others may call it academic misconduct, and others may call it simple incompetence. None of them are good. It seems you are pleading for simple incompetence. This is one of the reasons Steve McIntyre does not allow commenters on ClimateAudit to discuss the motivations of scientists.

    Jon Stewart had a pretty funny line on the claim this is a commonly used mathematical technique. “We are just using a common statistical tool to trick you into not knowing about the decline.” Funny stuff! Only it is not a common statistical tool. But it was used to trick people into not knowing about the decline or the unreliability of tree-rings as temperature proxy.

95. Ron – great, you believe he is right. That means the statement in the TAR and AR4 regarding smoothing in those graphs was a lie.

    But why do you believe this? Tell me one piece of clear evidence on this. Was it somewhere in an email where Briffa or Jones outlined their smoothing procedure exactly as Mosher stated? Was it a detailed graphical comparison of the different smoothing techniques?

    If “read his book” will give more data to examine this specific claim – please tell me exactly where in his book this is all explained.

    Remember, the issues here are science, they require precision of thought and statement, clear logic. Vague imprecations are irrelevant, the question is what, specifically, was done that might have been some form of malfeasance, as Mosher is clearly claiming, something specific that the IPCC report states one way but in fact is wrong and has not been admitted or corrected. Vague “read the book” answers are not sufficient to support precise statements of this sort.

96. Jon Stewart had a pretty funny line on the claim this is a commonly used mathematical technique. “We are just using a common statistical tool to trick you into not knowing about the decline.” Funny stuff!

    Yes, it would be funny if temperature had actually declined 1960-2010.

97. … but since it’s hiding the decline in tree ring temperature proxies, it’s not funny?

98. Despite the thousands of words accusing McIntyre of fraud (“Climate Fraudit”, etc.), nobody throwing the word about on discussion boards to my knowledge has leveled a formal accusation of same against him. Why would that be?

    Oooh let me see… Because he is his own employer? Because he follows no Code of Research Conduct? Because he is under no obligation to publish? Because he is a member of no professional body? Because he will never be the subject of a Parliamentary Enquiry? Because he is not subject to the FOIA?

    But good to see evidence of a sense of humour. Perhaps if Climate Audit was run according to professional standards of discourse and courtesy it’s output might drop by a large percentage? We’re discussing graphs [well trying to] but Where, for example is the outrage and scrutiny of stuff like this? Here is McIntyre publically describing Dr Gavin Schmidt as ‘full of crap’ over an arcane disagreement about data availability (it was, of course, McIntyre who had his facts wrong). Either of these would raise a few eyebrows, to say the least, amongst genuine academics.

    Then there was the captioning of a video clip of Professor Mann with ‘Try not to puke’ …. just a few examples of asymmentry. No doubt when McIntyre opens up his email inbox for the last 13 years for public scrutiny, there will be many, many more …. 😉

99. @ dhogaza #102,
    Jon Stewart realized the decline was in the proxy record and not surface temps.

100. @ Arthur #101
     Haven’t you ever just taken someone’s advice on a book before?

101. but since it’s hiding the decline in tree ring temperature proxies, it’s not funny?

     Not particularly, because Stewart’s humor is heavily dependent on him being *right*, and there was no attempt to “trick you”. He (and his writers) are smart enough to know how “trick” is commonly used, indeed, I’m sure every show employs technical tricks to make the show possible.

     Like, for instance, recording three separate interviews with John Oliver to cover the US beating, tying, or losing to England in their World Cup match, a trick of the trade to work around timezone vs. airtime issues.

102. dhogaza #106,
     Stewart was right. The statistical method used is not common. It is completely unknown and unsupported by any statistical text. The only possible reason for this innovative statistical technique is to, in the words of Phil Jones, “hide the decline.”

     Why do you think there is no attempt to “trick you?” Do you think the policymakers have read the peer-reviewed literature which talk about the Divergence Problem? Or do you think it is more likely the policymakers have never heard the term? Do you think the policymakers would have more or less confidence in a temp reconstruction based on tree-ring proxies if they knew the tree-rings have not worked well during the period when we have the best surface temp record? Would that fact inspire confidence in the temp reconstruction or cause doubts and uncertainty?

     dhogaza, I can explain why the Divergence Problem is embarrassing to dendros but I cannot understand it for you.

103. Arthur, let me try and answer your questions as best I can. You appear to have two parts, WMO vs IPCC, and evidence of same.

     As for WMO vs IPCC, Steve Mosher is correct in his statement, and he already explained it.
     The WMO chart which is what Phil Jones used.

     Desmogging Desmog’s Trick – Part 1

     
     Phil Jones directly appended instrumental temperatures, and Brian Angliss has suggested this is fraud. It also appears that when he enquired about it, he was given a deceptive answer.
     Now you are not interested in this part, and are asking about the IPCC charts.
     Instrumental temperatures were not appended, but if you append data from 1961-1990, smooth the data, and then take off the part after 1960, you have still changed part of your graph by removing proxy data and replaced it with instrumental temperatures. Not the same as what Phil Jones did, but still substituting instrumental for proxy data, and apparently done as a trick to hide the decline.

     I don’t know of any evidence from the scientists themselves saying we appended instrumental temperatures to the proxy record. Indeed, at RealClimate, Michael Mann posted that noone in the field has ever done anything like that, and only big-oil funded websites would claim such a thing.
     Now, you have these graphs with descriptions of how they were produced, and when Steve McIntyre took the actual data and produced a graph with those methods, it did not look like the graph produced by the scientists. So this suggests a mistake by the scientists, or a mistake by Steve McI, or that Steve McI is lying about his efforts. Now different people can try the same thing with the data that is available and see for themselves. Steve McIntyre and some others concluded that instrumental temperatures were used in place of the proxy data, though looking at the relevant posts, they have not completely replicated the graphs in question.

104. Phil Clarke, post 90: “Being a sceptic is not incompatible with having a sense of humour, boys and girls.”

     Phil Clarke, post 104: *hysterical hyperventilating over some gentle mocking*

     My apologies if you are being deliberately ironic.

105. dhogaza, I’m glad you see that climate science is equivalent to a comedy show.

106. Arthur: the clear evidence that Mosher is right was, as you suggest, “a detailed graphical comparison of the different smoothing techniques”. Basically, the skeptics at ClimateAudit tried to reproduce each chart in the manner it was claimed to have been constructed, found that this did not produce the shape given, and kept experimenting until they hit upon methodologies that exactly replicated the given curves.

     So one piece of evidence is that the chart as published looks just like the procedure Mosher describes was followed – and doesn’t look like the claimed one.

     Perhaps an even stronger bit of evidence is that we haven’t seen Mosher’s claims “debunked” by any of the usual suspects. If his account were incorrect and there were some innocuous alternative way to generate the same graphics, don’t you think we’d have heard about it by now? Wouldn’t a rebuttal have shown up in gloating posts or comments at Deltoid, DC, RealClimate, Tamino, or all of the above? I think it’s safe to say *if* these claims were false they’d be easy for somebody with access to the data to debunk and it’s also safe to say that if they *could* be debunked that would have been done.

107. Ron Cram:

     dhogaza, I can explain why the Divergence Problem is embarrassing to dendros but I cannot understand it for you.

     There’s nothing embarrassing about it. It’s a mystery to be solved, that’s all.

     Those datasets which demonstrate the problem match about 2/3 of the instrumental record and a variety of other proxies. Evidence is strong that something has changed that has caused the datasets to diverge (after the kind of end-effects of smoothing algorithms mentioned by Arthur are taken into account).

     1. Anthropomorphic change. There’s no denying that humans have made vast changes to the planet, including its atmospheric chemistry, and it’s very possible that the cause is anthropomorphic in origin.

     2. A change that is natural in origin. Most likely here is that rising temps cause something other than temps during the short far-northern growing season to be the limiting factor in growth. Apparently the most likely candidate is water. The implications of a natural origin associated with rising temps is that the time series may underestimate temps during extraordinarily warm periods in the past.

     What isn’t going to be true are claims that trees can not be good temperature proxies. Too much plant physiology to overturn there, sorry.

     Now, where’s the embarrassment?

     My expectation is that the scientists looking into the divergence problem will have a much better idea as to cause in the next decade or so. I also understand that Ron Cram will not accept any solution to the divergence problem.

     Meanwhile, the world warms …

108. anthropogenic, sorry 🙂

109. dhogaza, I’m glad you see that climate science is equivalent to a comedy show.

     Yet, the world warms …

110. … just like it did 1000 years ago.

111. @ Phil Clarke #104

     Steve McIntyre is rarely wrong. When he is wrong and it is pointed out to him, he admits it. Regarding the little dust-up with Gavin, did you bother to read what McIntyre had to say about the incident? He wrote an update on his post stamped 11:33 pm on Sept 4. See http://climateaudit.org/2008/09/03/mann-et-al-2008-proxies/

112. > What isn’t going to be true are claims that trees can not be good temperature proxies.

     What might easily be true is that trees aren’t good *long-term* temperature proxies absent something else to calibrate them against or that they’re only good temperature proxies within a specific temperature range. With “divergence” in the form of a *reversed* temperature/growth relationship as soon as some optimum temperature is exceeded. (In which case any *low* points in your reconstruction might actually represent *high* points in past temperatures!)

     Quote: “it is fundamentally impossible using tree ring data to say that recent decades are warmer than any time in the past n years because temperatures warmer than those of the first half of the twentieth century are likely to be suppressed by the method used for reconstruction.”

     Source, Craig Loehle’s _Climatic Change_ paper: http://icecap.us/images/uploads/Loehle_Divergence_CC.pdf

113. dhogaza #113,

     I’m sorry, but the Divergence Problem is an embarrassment to Phil Jones and Mike Mann or they wouldn’t have tried to hide the decline. It is pretty simple really. There is no reason to hide it if you are not embarrassed by it. Evidence is not strong (as you claim) that something changed, because the Divergence Problem is not universal. Most trees reflect lower temps in the last half of the 20th century but not all. Even the dendros don’t universally buy that hypothesis.

     It is possible mankind is changing the climate, but there is no evidence climate change is outside of natural variation or that mankind’s influence will ever be catastrophic. Look around at all of the climatologists who are becoming more and more outspoken about how the alarmist claims have been overblown.

     Trees have never been proven to be good temperature proxies. Show me the science that demonstrates that dendros can separate out the influence of changing hydration, changing available sunshine, changing natural fertilization from a longer growing season. It has never been done and can never be done. In addition, the ancient trees are especially poor thermometers. All of the older trees are strip bark trees, bristlecone pine or foxtails, etc. Have you ever seen a strip bark tree? Or seen pictures? The trees only grow where there is bark covering, so the trees are not symmetrical. Think about this! Now think about the commonly used pic of Michael Mann next to a cut tree with all of the tree rings looking nice and symmetrical – the exact same ring thickness all the way around the tree. That is not a strip bark tree! After Congress got involved in the Hockey Stick controversy, the NAS panel published a report in which they agreed with Steve McIntyre saying strip bark trees should be “avoided” in temp reconstructions. Well, you can only go back about 400 years before all available trees are strip bark trees. So, even if the dendros could prove their theory that tree rings make good temperature proxies, they would still not be able to go back more than 400 years.

114. Steve McIntyre is rarely wrong

     Perhaps you could give an instance of his being (a) right and (b) his correctness having significant real-world consequences….

     He himself has described paleoclimatology as ‘somewhat tangential’ amongst the larger issue of what is affecting the current climate and the policy response.

115. If he’s (a) wrong and (b) irrelevant, why are you so worked up?

116. Skip, if McIntyre is righ, then why don’t you provide Phil with an example of him being so, instead of changing the subject to Phil’s motives?

117. What’s the point? You and Phil are going to believe what you want to believe no matter what I say.

118. Phil asked a simple question, and your response is to change the subject to his motives, and then mine. Not surprising, but a bit dull these days. Can’t you come up with something a bit more original, like some actual science for a change?

119. It turns out the raw (unsmoothed) data for Briffa et al (2001) can be downloaded from that NCDC page. I’m looking at it now in R; may take me a day or two to put up graphs. Then we’ll see who’s right about all this.

     Glen – the problem is the “skeptics” at climate audit never did exactly replicate those curves, or at least nobody so far here has pointed to a page where that was done. There was a lot of speculation, but if that’s the basis for Mosher’s very strong claims that there is a lie about smoothing in the IPCC reports, he’s on very shaky ground.

120. Show me the science that demonstrates that dendros can separate out the influence of changing hydration, changing available sunshine, changing natural fertilization from a longer growing season. It has never been done and can never be done.

     Yet, carefully selected sites match other proxy data remarkably well, as you yourself point out some match the full instrumental period very well, and those that supposedly bring the entire field down match the instrumental period for about 2/3 of its history.

     As far as “showing you the science”, I’ve done a fair amount of reading and my judgement is that the professionals in the field have a much firmer grasp on their profession than does Ron Cram.

     And, as I said earlier, it is clear that no matter what answer these professionals will come up with, Ron Cram will not accept it.

121. @ Phil Clarke #120
     Are you kidding? You want examples of when Steve has been right and it has been important? How about the Hockey Stick Controversy? Steve was right. The Wegman Report said so. While the NAS Report was more polite to Michael Mann, it agreed with Steve on all major points of science, such as strip bark trees not being reliable temp proxies. And von Storch and Zorita agreed that Mann’s approach resulted in an Artificial Hockey Stick when used with trendless red noise as data. Yep, Steve was right on all major counts.

     But we do not have to go back that far in time. Just read the blog you are commenting on. Brian Angliss claimed Steve was wrong but when he actually checked the facts, he learned he was right. This happens all the time! While do you think Judith Curry has become a Steve McIntyre fan?

     By the way, Brian admitted Steve was right on a second point but has not yet issued his correction to this blog. Why is that? What are you waiting for, Brian? Christmas?

122. dhogaza #126,
     You have not shown me any science which has demonstrated the ability to separate out the contribution of various growth factors 1000 years ago or even 20 years ago. So don’t make this claim that I am unwilling to change my mind when shown the facts. You cannot say that because you have never shown me any facts.

     I am looking for a dendro who can credibly say something like this:
     “In 1000 AD, this tree indicates the local temp avg for the year was x. In 1001 AD, the tree ring is 32% thicker than in year 1000. We know the additional growth came from the following factors. 14% of it came from increased fertilization due to a nomadic herd of gazelle that stayed for a year near this tree and fertilized it with its droppings. 37% of the extra growth came from increased local rainfall. 49% of the growth came from warmer temperatures and a longer than normal growing season. In this particular year, the change in available sunlight did not play a significant factor.”

     Until dendros can do something like that, they are practicing pseudoscience and not science at all. Everyone knows water, fertilizer, sunlight and other factors can be important. There is no way they can separate out all of these factors which contribute to growth. Ignoring these other factors does not make them go away.

123. dhogaza : have you read Craig Loehle’s paper “A mathematical analysis of the divergence
     problem in dendroclimatology”? ( that URL again: http://icecap.us/images/uploads/Loehle_Divergence_CC.pdf ) What did you think of it? Angliss, same question.

     Ron Cram: I think you’re going “a bridge too far”, as McIntyre would put it. Trees can probably be decent “treemometers” in the short term. The problem is that even if you manage to find good ones today, trying to extend the record into the past or the future much beyond the calibration period is speculative at best. Because the fact that a certain tree’s growth is constrained by temperature *now* doesn’t mean that was always true in the past or will always be true in the future.

     Yes, there are problems with “growth pulses” and cherrypicking and strip-bark uneven growth and so on – and that matters to the specific results – but to focus there is to, um, miss the forest for the trees. Trees could, in theory, act as treemometers. If you had a good mechanism for selecting and calibrating the right trees you could, in theory, use them to construct a record that reflected temperature. (Moberg’s sort of approach – using trees for the high-frequency data but relying on other proxies to pin the low-frequency endpoints – might work) The problem is that (a) our mechanisms aren’t very good, (b) the steady-state assumption that “once a treemometer, always a treemometer” is false, and (c) the linear-response-to-temperature assumption is false. So even if you grant for the sake of argument that those trees selected by Briffa were good thermometers during the calibration period, it doesn’t get the conclusion Mann et al would like to argue for. In particular, it doesn’t show current temps are “unprecedented”.

     To prove the assumptions of steady state and linear response were true, you’d need the trees match *new* temperatures outside the calibration set. They failed that test miserably, so they’re clearly not measuring temperatures now. And Mann et al were wrong to hide the evidence of this. But that doesn’t prove these trees weren’t reflecting temperatures at all earlier.

     And dendros are mostly just measuring stuff and trying to figure out what it means – science, not psuedoscience. The fact that “climatologists” make bogus claims about the climate based on tree ring series doesn’t discredit all of dendrochronology any more than reiki practitioners making bogus claims about the implications of high-energy physics discredits physics.

124. Brian,

     Could you please clarify something. Tom Fuller is claiming at Bart Verheggan’s place that you are on the record saying that you have not read the (illegally obtained, my words) emails. Now you have obviously read at the very least a portion of the emails as shown on your analysis here. Fuller seems to be confused and/or distorting; perhaps he is suggesting that you have not read all the emails. If that is true, then he should make that clear.

     Anyhow, even if that were true, that is not relevant b/c it is my understanding that here you were investigating some very specific allegations that did not require you to read all of the illegally obtained emails.

     If you could clarify I’d be very grateful. Thanks.

125. You have not shown me any science which has demonstrated the ability to separate out the contribution of various growth factors 1000 years ago or even 20 years ago

     Liebig’s law of the minimum has been around for nearly 200 years.

126. Ron Cram: I think you’re going “a bridge too far”, as McIntyre would put it. Trees can probably be decent “treemometers” in the short term. The problem is that even if you manage to find good ones today, trying to extend the record into the past or the future much beyond the calibration period is speculative at best. Because the fact that a certain tree’s growth is constrained by temperature *now* doesn’t mean that was always true in the past or will always be true in the future.

     This is why they’re calibrated with a variety of proxies, and why sites near the altitudinal or latitudinal limits of their range which have adequate water and soil nutrients are chosen.

     Despite Ron Cram’s blathering, the fact of the matter is that tree physiologists know that the growth of trees in such areas are limited by the number of sufficiently warm days in summer. It’s not guesswork.

     Anyway, rather than blather why not go read D’Arriga etc? The problems and shortcomings as well as the strengths are laid out quite plainly. They’re not as bad as Glen states, and they do not rise to the level of overturning the work of Briffa et al as Glen claims.

     Having read what some of the experts in the field have to say, once again I’ll state that I believe they have a better handle on their field than Glen, McI, and the other amateurs on the web who are motivated by what they view to be the political consequences of climate science.

127. > Despite Ron Cram’s blathering, the fact of the matter is that tree physiologists know that the growth of trees in such areas are limited by the number of sufficiently warm days in summer. It’s not guesswork.

     And despite yours, we have many examples of trees demonstrating *negative* response to increased temperature in such areas. Including during the first part of the 20th century, not only since the 1960s. So there *is* guesswork in figuring out whether a particular tree was a positive responder in the past or will be in the future. The fact that there’s adequate water and sunlight and soil nutrients *now* doesn’t mean there was in the past or will be in the future on century scales.

     Yes, I’m an amateur on the web – I haven’t (yet) published. But Loehle is an expert on forest ecology who says you can’t use the data to demonstrate current temps are unprecedented. And Briffa is an expert on trees who said (in the climategate emails) he thought recent warming was probably matched by the MWP.

128. Ron – Yep, Steve was right on all major counts

     You missed the ‘significant real-world impact’ half of my question. Here’s why:

     McIntyre and McKitrick (2005a,b) raised further concerns about the details of the Mann et al. (1998) method, principally relating to the independent verification of the reconstruction against 19th-century instrumental temperature data and to the extraction of the dominant modes of variability present in a network of western North American tree ring chronologies, using Principal Components Analysis. The latter may have some theoretical foundation, but Wahl and Amman (2006) also show that the impact on the amplitude of the final reconstruction is very small (~0.05°C; for further discussion of these issues see also Huybers, 2005; McIntyre and McKitrick, 2005c,d; von Storch and Zorita, 2005).

     from IPCC AR4 WG1 Section 6.6.6.1.

     This is not the place to get into whether the political appointee Wegman’s apparently plagarised report or the NAS, who endorsed the broad findings of Mann et al is correct – that’s been done to death elsewhere, besides which the study is over a decade old now and has been largely superceded and confirmed by better ones. No, the problems I have with McIntyre’s approach, more than the dripping sarcasm and the thinly veiled accusations of malpractice – are that he rarely follows through – that is expanding on whether his findings have any effect at all outside the blogosphere by, for example, exposing them to the scrutiny of publication, and also the blatent partiality. In recent times two climate papers with major statistical errors – McLean et al and Douglass et el – have made it through review and into the literature. These should have been meat and drink to a ‘climate auditor’ yet – perhaps because they contradicted the concensus – it was left to Tamino and others to do what McIntyre fails to – work their critques up and submit for publication. Impartial auditors don’t pick and choose like that.

     Click and scroll down to ‘Close Encounters …’ for Ben Santer’s experience of the McIntyre Method then tell me what McIntyre did with the data he went to such lengths to obtain….?

129. Glen #129
     I do not think I am going a bridge too far. I draw a distinction between dendrochronologists who core trees and do basic science from the dendroclimatologists who treat trees like thermometers. You can safely assume when I am disparaging dendros that I have in mind the dendroclimatologists. It is just too long a word to type all the time.

     Trees are not thermometers and I can safely say the science will never progress to the point they can be used that way. There are too many confounding factors. I hope this explains my position a little more clearly. If I was a dendrochronologist I would be very upset at Michael Mann et al for making ridiculous and unfounded claims because it brings discredit, unreasonable expectations and confusion to the field.

130. dhogaza #131,
     Are you serious? Did you even read the article you linked to? Try thinking about this sentence and how it applies to a forest over a 1,000 year period.

     “It is limited to a situation where there are steady state conditions, and factor interactions are tightly controlled.”

131. Glen #133,
     Good response. But don’t expect dhogaza to change his mind even when confronted with the facts.

     • I’ve been away from access to the internet for a few days, but over the next few days I’ll get some responses out to all of the many, many comments and questions above.

132. Phil Clarke wrote:

     scroll down to ‘Close Encounters …’ for Ben Santer’s experience of the McIntyre Method then tell me what McIntyre did with the data he went to such lengths to obtain

     Sure! Here’s what McIntyre did: he and Ross wrote up and submitted a comment to the International Journal of Climatology discussing Santer, Schmidt et al versus Douglass et al. He mentions the comment status here and then gives some more details on the work here. Apparently if you bring the data sets up to date, Santer’s conclusions no longer hold. Funny how often that seems to be the case in this field…

133. This came from a comment Fuller wrote on the other post. Given what he references below was in the comment thread above, I’ve moved it from the other post to this one.

     Angliss, you have addressed the assumptions I have imputed to you. You have not touched on my comments on your results.

     The thrust of your argument is anti-science–you don’t have to read McIntyre’s blog, the leaked emails or our book. I have no problem with that. You can even tell other people they don’t need to read it. But to hijack studies of dubious relevance and try to invent some jiggery-pokery formula saying that what you’re doing is a valid way of discovering the truth is ridiculous.

     I will address your conclusions when I can make the time over the next several days, possibly as long as a week. As I indicated, I’m still traveling and don’t always have the time or internet access (and attempting to thumb out a long response on a smartphone doesn’t work very well) due to other committments.

134. Mapleleaf (#130): I have not read ALL of the emails, that’s true, and I’ve said as much. I’ve read probably 10-15% of them, and all of the ones that I was referred to with specific links. I haven’t read all the emails that Mosher suggested, given that he told me to start reading in 2005 and there are ~140 emails in just 2005 (and yet either Mosher or Fuller pointed out that the emails in question cover a 2-3 day period and have yet to provide me with which 2-3 day period). I’ve also spent a significant amount of time searching them for specific quotes and search terms looking for what I thought was a quote from the emails but wasn’t in the emails anywhere.

     I’ve read all the emails that I used for this particular analysis, and you’re correct that I was strictly limiting myself to specific claims that I fact-checked and found to be wanting.

     Does that answer your question?

135. And despite yours, we have many examples of trees demonstrating *negative* response to increased temperature in such areas

     If the cause is due to the fact that a site dries after a certain threshold is reached – which is one of the hypotheses out there – then it is in fact not such an area as I described. It’s not a site where water availability exceeds the needs of the trees in question at all times.

     But what are the consequences?

     If the threshold can be determined, then those periods in the past in which reconstructions show the same or higher temps as during the period of divergence can be identified. There will be periods of time in which the proxy isn’t useful, and periods in which the proxy is useful.

     What won’t be true is that the proxy isn’t ever useful, which is the rather silly claim being made by the denialsphere.

     And, of course, if the threshold temperature has not been reached earlier in the reconstruction, then the divergence can be entirely ignored. As it can be if it turns out to be due to some anthropomorphic change.

     Of course, it’s entirely possible that no divergence exists and it’s just due to UHI screwing up the nearest thermometer record …

     (couldn’t resist that jab, sorry).

136. Try thinking about this sentence and how it applies to a forest over a 1,000 year period.

     “It is limited to a situation where there are steady state conditions, and factor interactions are tightly controlled.”

     There are places where individual *trees* greater than 1,000 years old have survived, and plenty of places where forests older than that have survived with the same species mix over the entire period (this being one of the things they look for, of course).

     This indicates that local climate has remained within the survival parameters for those species (or individuals), nutrients and water are sufficient, etc.

     Trees are not thermometers and I can safely say the science will never progress to the point they can be used that way.

     If you want to toss a whole bunch of plant physiology unrelated to paleoclimate reconstructions into the toilet, feel free.

     Don’t be too surprised though when scientists continue to ignore y’all. You can insist that they’ll never be able to do what they’ve already done until you’re blue in the face. That won’t change reality.

137. dhozaga 142: You say”If the threshold can be determined, then those periods in the past in which reconstructions show the same or higher temps as during the period of divergence can be identified. There will be periods of time in which the proxy isn’t useful, and periods in which the proxy is useful.

     What won’t be true is that the proxy isn’t ever useful, which is the rather silly claim being made by the denialsphere.”

     I suppose that may be true. But the problem remains: even if we could so identify the potentially “unuseful” periods, we still have no way of discerning temperatures during those, and therefore the significance of current temperatures in relation to them.

     As an aside, this has been the most interesting discussion I have seen on climate science. Minds on either side may not be completely open, but it looks like they are approaching a meeting, thanks all!

     -Paul

138. Glen Raphael ,Loehle’s paper is discussed in Jan Esper & David Frank
     “Divergence pitfalls in tree-ring research” Climatic Change (2009) 94:261–266 if you access to that journal.

139. Bill: Thanks, that Esper&Frank paper is interesting, doesn’t conflict with Loehle and seems to point out additional problems with dhogaza’s claim that the diverging series in question “match[es] the instrumental period for about 2/3 of its history.” For instance, we don’t know if that degree of allegedly-amazing match was really inherent in the underlying data or was merely put there by the detrending step. Or at least I don’t. 🙂 Here’s an ungated copy of Esper&Frank:

     Click to access Esper_2009_CC_PIT.pdf

     rather than blather why not go read D’Arriga etc?

     You mean D’Arrigo, right? Presumably the very same whose papers said: “Therefore we stress that presently available paleoclimatic reconstructions are inadequate for making specific inferences, at hemispheric scales, about MWP warmth relative to the present anthropogenic period and that such comparisons can only still be made at the local/regional scale.”?

140. #124, J Bowers — If McIntyre is (a) wrong and (b) irrelevant, why are you so worked up?

141. It just occurred to me that you didn’t get the joke last time, so I need to explain it.

     If McIntyre was really wrong and irrelevant, you wouldn’t be spending so much time talking about him. Therefore, if you want examples of where he’s right and relevant, just think about the cases where he makes you feel uncomfortable.

     Now, can we PLEASE stop playing these retarded games and get back to the science?

142. Phil Clarke,
     Are you aware that the passage you quote from the AR4 is one of the most contentious para over which much ink has been spilt. It includes the reference to the infamous Wahl and Amman ((2006) paper whose inclusion broke all the IPCC procedural rules, and which has never appeared in the from i which it appears to have been circulated to IPCC reviewers. David Holland has one discussion in the discussion of the AR4 defence of the hockey stick in his Energy and Environment paper which can be found here

     Click to access Holland(2007).pdf

     The wording of the paragraph appeared in the final report without any chance for reviewers to comment further. The statement about the size of the effect it makes seems simply to be wrong or, at best, disingenuous. What matters in the MBH re-construction is the inclusion of the bristlecone pines and related species. If these are excluded the MBH methodology produces a reconstruction which does not have a particularly elevated modern period (and in any case, whether the bristlecones are included or not, the verification stats are abysmal). There can be little question but that this AR$ paragraph is simply wrong.

143. Skip Smith: “If McIntyre was really wrong and irrelevant, you wouldn’t be spending so much time talking about him.”

     It seems to me that when it comes Steve McIntyre, you’re the one who goes off on one and gets all worked up.

144. Arthur, looking forward to your results. I haven’t gotten into R and done this myself. The most I’ve one is charted Kaufmann’s proxies in his Arctic warming paper, and determined that one of the proxies was used upside-down, as it was in Mann ’08 as well.

     One caveat is that Steve McIntyre has written that Briffa’s data was not available until the ClimateGate e-mails were leaked. He said something about the NCDC having either cutoff data or the instrumental data. I’ll compare to what’s in the ClimateGate docs.

145. I suppose that may be true. But the problem remains: even if we could so identify the potentially “unuseful” periods, we still have no way of discerning temperatures during those, and therefore the significance of current temperatures in relation to them.

     Yes, you might end up with holes in the reconstruction. I have no idea why anyone would think that this is particularly significant to the big picture, though. Some people seem to think that if they can prove that parts of the Northern Hemisphere at times reached temperatures similar to those seen today, that this proves AGW false, or proves that it’s harmless. Of course, over coming decades, rising temps are going to make the argument that “it was once as warm in some stand of trees up in northern russia as it is today” irrelevant.

146. Glen:

     “Therefore we stress that presently available paleoclimatic reconstructions are inadequate for making specific inferences, at hemispheric scales, about MWP warmth relative to the present anthropogenic period and that such comparisons can only still be made at the local/regional scale.”

     Yes. So what? Who cares? Obviously some workers in the field disagree with D’Arrigo. There’s uncertainty here. The uncertainty D’Arrigo discusses has nothing to do with whether or not trees can, under proper circumstances, be used as reasonable temperature proxies. Nor does it have anything to do with the “hide the decline” crap we’ll be hearing about a century from now, after the summer ice has been gone from the arctic for a few decades (Watts and Goddard predict ice free summers there in 2060).

147. By the way looking at 6.10b, I think you are right that the smoothing makes no difference. That graph does not have much visual impact. The visual impact of that graph is that temperatures are higher now than in the past, but the proxies themselves are not showing that high value, mainly because they don’t extend that far in the future. This looks to me like a different trick entirely. Whether an individual proxy goes down or up at 1980 has no impact on that graph.

148. Ron Cram (#78, 85, and 127): I agreed with Steve McIntyre that the data was not erroneous. However, I do still believe that his following statement was incorrect – the data as I understand it doesn’t support that tree rings are a bad proxy.

     Skip Smith (#79 and 82): I should have said “might” rather than “would” with regard to whether Jones et al. committed fraud. As I pointed out above, I defer to the people who have done the actual work of understanding the email’s context, namely the three inquiries completed to date. And I look forward to reading the results of the Muir Russell review.

     You repeat an error that McIntyre (and Mosher, and Fuller, et al) made, namely assuming that the scientists involved would naturally release any emails that they have if they had any that showed the released emails in a better light. McIntyre may be right about this, but he could also be wrong. It’s not possible to know without gaining access to the rest of the emails and/or talking to the people involved whether there are or are not exculpatory emails that were not released or had not been filtered into the CRU email record.

     There are many reasons why the victim of a crime might not want to talk to people doing an investigation. It happens all the time when personal, private lives are suddenly thrust into the public sphere by way of paparazzi or police investigations. While not releasing the entire email archive might not have been the best PR for the scientists involved, demanding that the victims of a crime victimize themselves further is unfair at best.

149. Well, MikeN, Glen, and friends, it turns out Steven Mosher was truly making stuff up, pulling claims out of thin air, i.e. lying to us here on this very thread.

     http://arthur.shumwaysmith.com/life/content/steven_mosher_even_fuller_of_it

     You can reproduce the analyses yourselves, I included links to the R code I used for every graph.

     The defense of Mosher’s claims here has been pathetic.

150. The Deleted Portion of the Briffa Reconstruction

151. MikeP: It includes the reference to the infamous Wahl and Amman ((2006) paper whose inclusion broke all the IPCC procedural rules,

     Perhaps you could help us out with exactly which IPCC rules were broken then? Here’s an excerpt from the comments on the 1st order draft:

     Wahl and Ammann 2006 did not meet several publication deadlines. Is it
     fair to use this study when other studies also not meeting publication deadlines
     were not used? It was not accepted by December 13-15. TSU did not have a
     preprint by late February. The version available for review was not the same as the
     accepted verion – in particular, the version made available omitted critical
     information that MBH98 failed cross-validation r2 and CE statistics.
     [Stephen McIntyre (Reviewer’s comment ID #: 309-119)]

     Response: Rejected- the citation is allowed under current rules

     So you seem to know the IPCC rulebook better than they themselves do ….

     It is said that the more fuss is made on a topic the more discomfitted are those making the fuss, the striking thing about this controversy is that is is nearly all about the process, a controversy that can actually be dealt with economically in a few short answers.

     Could this smokesceen be because Wahl and Ammann did indeed prove that incorporating the Wegman/McIntyre ‘corrections’ into the study has, er no ‘real world significant impact’?

     Could be!.

     PC.

     Sources: http://www.realclimate.org/index.php/archives/2006/07/the-missing-piece-at-the-wegman-hearing/
     http://pds.lib.harvard.edu/pds/view/7786989?n=119

152. Bingo!

153. That was a fascinating post, Arthur. Thanks for doing all that work.

154. I should have gotten cautious after seeing the error Steven Mosher made above in describing Jones’ trick. Steve is claiming that this trick is used in AR4, TAR, and WMO. In WMO it is a basic trick of appending instrumental temperatures, there is no deletion after 1960.
     AR4, the Briffa data is not being smoothed with temperature data as Arthur Smith has shown.
     So at best Steven scores 1 out of 3.

155. dhogaza #143,

     Do you really think “tightly controlled” and “within survival poarameters” mean the same thing? I assure you they do not.

     Psuedoscientists who continue to practice their tree thermometry will continue to be ignored.

156. Brian Angliss #155,
     You are not adequately dealing with the issue. In this blog post you disagree with McIntyre’s quote saying the data does not support his statement. But on ClimateAudit, regarding the same quote you say “that statement seems fine to me. It is based entirely in fact to the best of my knowledge.” You cannot have it both ways.

     In comment #155 you say it does not mean the tree rings are a bad proxy. But the issue is the Divergence period from 1960 onwards when temp goes up and tree ring proxies point down. How can that possible be a good proxy?

     You do not have to agree with me that all tree ring proxies are unreliable to see the accuracy of McIntyre’s quote. Please deal with the issue at hand. You owe your readers another correction.

157. But the issue is the Divergence period from 1960 onwards when temp goes up and tree ring proxies point down. How can that possible be a good proxy?

     Because there is no divergence any time prior to 1960, as McIntyre himself has so ably demonstrated.

158. Oops

     http://climateaudit.files.wordpress.com/2009/09/rcs_chronologies_rev2.gif?w=420&h=360

159. 162 Ron Cram: “Psuedoscientists who continue to practice their tree thermometry will continue to be ignored.”

     Only by you, Ron, only by you. But we’ve come to expect that.

160. luminous beauty, I think that graph is both trees. Pretty sure there were no thermometers around 2000 years ago. But who knows

161. joe,

     You’re missing the point. The post 1960 divergence phenomenon is only seen in some tree ring series. There is no divergence between divergent and non-divergent tree ring series before 1960. What does that suggest?

162. The post 1960 divergence phenomenon is only seen in some tree ring series. There is no divergence between divergent and non-divergent tree ring series before 1960. What does that suggest?

     It suggests you weren’t paying attention. 🙂

     Some trees do in fact “diverge” before 1960. The particular trees that were chosen to make up Briffa’s series don’t diverge *much* but even they do diverge a little – you can see periods when the trees and the temperature didn’t switch direction at the same time.

     What this suggest to me is one of two options:

     (a) Out of the corpus of available trees, the ones that happened to match reasonably well were the ones that were picked for inclusion in proxy series. In which case, “divergence” is a natural outcome once there’s no more opportunity to exclude the ones that don’t diverge. There are enough different kinds of patterns in the data that even if trees had no relation to temperature at all it would be possible to *select* a set that seemed to do so. If it stops doing so after the selection step is done, that suggests the match was to some degree spurious – the result of “cherrypicking”.

     (b) or perhaps the trees chosen as proxies were *calibrated* in a way such that they matched the temperature record, whatever it was that they were actually doing at the time. For instance, this could be done during the “detrending” step. And again, if the relationship breaks up later, that suggest the match was to some degree spurious.

     Out-of-sample data after all decisions were made as to how to handle the data is the best kind of out-of-sample data. There’s no way to fudge it. The fact that there’s divergence after 1960 (in some series) seriously undermines the credibility of the claim that those series accurately represent past temperatures. And note that other series diverge in 1980….

163. Glen Raphael says:

     Briffa’s series don’t diverge *much* but even they do diverge a little – you can see periods when the trees and the temperature didn’t switch direction at the same time.

     Notable fluctuations between nearby series are to be expected. These are not comparable to the large late twentieth century divergence. The physiological causal relationship between tree growth and temperature is well understood and selection criteria are well designed to maximize that relationship. Correlation with instrumental records is not an essential proof of that relationship, but is used to calibrate tree ring indices to temperatures. Though the decadal trend of temperature change is not well preserved in divergent series, annual variability is. This would suggest that there is some gradual temporal influence that is repressing the climatic thermal response over the period of interest. Exactly what that influence, or more likely, influences, as evidence suggests that not every divergent series is responding to some universal change in conditions, but various localized effects, is an open question, but very little evidence points to non-linear negatively correlative processes that may have existed in pre-industrial times. Perhaps in some cases, but not all. It is a caution enough to exclude divergent series from tree ring reconstructions, even if their inclusion doesn’t make any significant difference, which it doesn’t, just as exclusion of tree rings from multi-proxy reconstructions does not make any significant difference.

     The arbitrary out-of sample-data that McIntyre chose to replace in an authoritatively selected series without a clue as to what criteria were used in the selection of that data, do not show any significant divergence prior to 1960 as far back as the early 1700s when that data ends. Multi-millenial comparisons, where possible, in Scandinavia and North America show much the same thing. No thermometers involved. That does suggest that divergence is a recent problem and undermines the claim that divergence must mean that tree rings do not represent past temperatures.

     Your suggestions indicate a complete lack of understanding of what the selection criteria for climate sensitive trees are, how it is done and the care taken in normalization and standardization of data in order to preserve low frequency variability. I would suggest a grounding in plant physiology, forest ecology, some good textbooks like Fritts’ “Tree Rings and Climate”, and Speers’ “Fundamentals of Tree Ring Research”, a thorough study of the extant refereed literature and a few years of field and lab work, before you comment on things you dimly understand. Until you do, THIS might be of help.

164. Your response suggests that “divergence”, much like “hockey stick”, is a term in desperate need of a definition more precise than “I know it when I see it.” Absent such a definition, you might want to settle for saying “there is less divergence” or “there is little divergence” or “there is little significant divergence” rather than “there is no divergence” over period X or sample Y.

     Given that “Correlation with instrumental records […] is used to calibrate tree ring indices to temperatures”, the fact that the calibrated indices seem – after calibration, over a period prior to when the calibration was done – to match temperatures really well might say more about how well the calibration step worked than how strong the inherent relationship between tree rings and temperature is.

     As for Dunning-Kruger, one thing I know that I know about tree selection is that it includes a subjective component. Which allows for the possibility of bias creeping in, regardless of how much “care” is taken. Another thing I know is that Briffa, an expert, said in the emails that he thought recent warming was probably matched during the MWP.

     (One last thing I know is that when you’re trying to snarkily accuse other people of being incompetent, it’s likely to backfire if the link URL you copy/paste depends on local cookies and/or contains non-portable session ids. 🙂 )

165. Glen,

     I stand corrected about saying “no divergence”. Wait … there is a difference between divergence and site drift, whether between geographically separate tree stands or highly localized tree stands and geographically and statistically removed thermometer data. I believe both phenomena are well defined in the literature. You are comparing apples to oranges. As I said, the relationship between temperature and tree growth is not dependent on thermometers, it is dependent on known plant physiology. If the calibration step is in strong agreement with the validation step (the period prior to the calibration period), then the calibration is said to be robust. Whatever it is you are arguing, it does not follow. The calibration is a necessary step to convert tree ring indices to degrees. If you know another way, please inform.

     I don’t know what you are saying about subjectivity in selection criteria. The criteria are pretty straightforward. One either follows them, objectively, or one doesn’t, in which case one’s selections will most likely be useless.

     I’d be curious what Briffa would answer today about the MCA. If it is true that MCA temps matched recent warming, doesn’t it suggest, given what we know about climate forcings during that period, that climate sensitivity is at the high end of current understanding?

     Concerning your last comment; unlike yourself and dendrology, I claim zero expertise in IT, so excuse me if I say I have no idea what you are talking about.

166. WillR (#87): I’m trying to figure out how you concluded that I’m saying that the CRU-related scientists get it wrong most of the time. Could you point me to where you think I make that argument?

     Tom Fuller (#140): I chose not to address your conclusions (presumably those in comment #62) because I don’t feel the need to address them, either because they’re bare assertions provided without evidence or because they’ve already been addressed. I especially feel no need to answer them when I’ve asked you to provide a whole slew of references for your other assertions that you have thus far failed to provide (my comments numbered 48, 55, & 77).

167. [These comments were recently posted at Climate Audit in response to a few points made by other commenters there. I’ve published them here in case they get cut due to some violation of CA’s comment policy or other problem I didn’t anticipate]

     Bruce (from his CA comment posted on Jun 14, 2010 at 9:01 AM): Actually, I don’t yet know that Briffa did what you claim. There is significant disagreement on that question, even now nearly a decade after the TAR was published. Furthermore, there remains some question about whether it was the raw instrument data that was used for padding or the mean of the instrument data in the case of the WMO. I’m going to do some more reading on that and see if I can find the data to crunch my own numbers before I say more about it, however. Finally, Arthur Smith effectively disproved this for the AR4 data in a recent post of his here.

     In response to your second claim, the basis is statistical correlation with other proxies that are not subject to the divergence problem.

     more Bruce (from this CA comment): Allow me to address a major error you made, Bruce. You quote that I chose not to address the specific post McIntyre referred me to, which is true. You then claim that it means that I “didn’t read anything Steve wrote.” This is an assumption on your part and entirely incorrect. I read nearly every post on Climategate at CA from December until I posted my piece a couple of weeks ago. Some I read closely, others I skimmed. I also read all three of DC’s pieces deconstructing Steve’s original trick piece and concluded that a) most of DC’s arguments were reasonable and b) I had nothing to add to the record with regard to that particular post.

     Mosher (in response to his CA post here): Thank you for clarifying that you didn’t mean to suggest you’d told me to ask Overpeck anything, as you joked about here.

     You claim that I “softballed” Overpeck, but you fail to recognize an important point – I was asking him a question to fact check what you claimed the emails meant. He disagreed with your interpretation of the email in question, and for the reasons I’ve reported. I was not asking him to explain the sum total of his published emails as you might, but rather I was addressing your very specific claims. Asking questions for fact-checking is different from asking questions for understanding or context or content, and I was doing the former.

168. >Furthermore, there remains some question about whether it was the raw instrument data that was used for padding or the mean of the instrument data in the case of the WMO.

     Why does this make any difference? That is almost certainly not raw data, as the graph looks very smooth. However, it is clearly not Briffa’s data on that chart after a certain point. UEA has released the chart without the data, and the e-mail stated the trick to hide the decline.

169. >>”Skip Smith (#79 and 82): I should have said “might” rather than “would” with regard to whether Jones et al. committed fraud. As I pointed out above, I defer to the people who have done the actual work of understanding the email’s context, namely the three inquiries completed to date. And I look forward to reading the results of the Muir Russell review.”<<

     Brian, never mind what other people think. What is *your* opinion on Jones replacing the tree ring proxy data with instrument data? Is that fraudulent behavior?

170. Below is a comment I posted on ClimateAudit. Arthur Smith has not responded to it there, perhaps he will respond to it here.
     ——

     Arthur Smith, I don’t understand you. You are demanding an apology from Steve McIntyre when you should be offering one. It is clear you are complaining that McIntyre’s writings have misled Mosher. Yet you come here and admit McIntyre is right about all the important points of science. Then you wants to pick a fight with McIntyre rather than criticize Jones and Mann.

     Arthur, let me give you some advice. Quit throwing sand in people’s eyes. Agree with the people who are mostly right. Criticize the people who are mostly wrong. Admit that Mosher is more right than Jones and Mann. Finally, apologize to McIntyre and Mosher for majoring on minor points.

     Arthur, when describing the trick you wrote this:
     “There is nothing scientifically nefarious or “wrong” about this – the “divergence problem” has been extensively discussed in the scientific literature including in the text of the most recent IPCC report. If you have reason to believe a particular collection of tree ring data is a good measure of temperature before 1960 but for some still uncertain reason not after that point, then it’s perfectly legitimate to create a graph using the data you think is reliable…”

     For that bit of writing you need to apologize. It is blatantly untrue and unsupportable. I challenge you find on statistical text that would recommend such an action.

     • Ron, why don’t you take this comment to Arthur’s site and ask him there? You’ll be even more likely to get a response that way.

       FWIW, your demand that Arthur provide a reference to a statistics book is a red herring – statistics books are mathematical toolboxes that show you how to do things and point out the various advantages and disadvantages of each mathematical tool described. Statistics textbooks are not ethics manuals, and math can’t provide ethical guidance any more than the laws of physics can. What you really need is a solid, widely accepted/acclaimed reference or set of references on scientific ethics. The National Academies might have something that works, but a quick search on “ethics” in their publications returns 312 results.

171. luminous beauty #172,

     Please let’s stop the fantasy about known plant physiology and using trees for thermometers. It is science fiction. It is well known that trees at all altitudes have more than one limiting factor and there is no way to separate out the different influences.

     Alternatively, you can post a reference to any paper you think best explains this “science” and the time period in which the science “works.” You realize, of course, that it cannot work on strip bark trees. I await your response.

172. 177 Ron Cram: “Quit throwing sand in people’s eyes. Agree with the people who are mostly right. “

     Ron, what’s gotten a hold of you?! Agree with the consensus and majority!? Are you okay?

173. Ron Cram, while I find it likely that McIntyre’s writings have misled Mosher, that was not something I investigated or came to any conclusion on, and I never stated that in the post. As far as me admitting “McIntyre is right about all the important points of science.” I have not done that either. McIntyre is clearly wrong in his latest post about me. I have not put in any effort to investigate whether McIntyre is right or wrong about any point of science he has posted on in the past. Maybe he’s right, maybe he’s not. He certainly can be confusing. And his latest post on me is most definitely wrong.

174. Hi Brian. I’m curious to see your response to post #176.

     • My opinion is that there are times when a scientist has to make an expert judgment call. Sometimes the call is correct, sometimes not, and they always have to be supported in a fashion that can survive review. If the judgment call survives review, then even if it’s later found to be the wrong call due to new evidence/data, the original judgment call doesn’t qualify as fraud.

       As an engineer, I make such calls nearly every day – should I use this part or that part, should I design for a 3-wire or a 4-wire interface, should I put down a source termination resistor or an endpoint termination network? So long as I justify WHY I chose a given part (the best part as defined by N different trade space metrics), why I chose to use a 3-wire interface instead of a 4-wire (insufficient pins on the backplane connector and no room to use a larger connector), or why I chose a source termination (multiple destinations made endpoint termination problematic at best) and can convince a review team of the validity of my logic, then it’s not questioned even if later there turns out to be a problem. If it turns out there is a problem, then I adjust my personal design processes accordingly to make sure that I don’t make that same mistake again.

       Science advances. People gain expertise and maturity. What was understood and justifiable 10 years ago might not be given the advances since. And in this particular case, I am no longer willing to put my own limited experience over that of the scientists who did the work originally, reviewed the work originally, or re-reviewed the work over the course of the completed inquiries to date.

       I’m not sure I would have done things the way they were done, but I’m not sure that I wouldn’t have either. Putting aside data judged to be erroneous may well be acceptable, and I’ve done it myself in certain circumstances when I felt the situation allowed it. Whether the situation allowed it here is a question I’ll leave to the experts.

175. J Bowers said: 177 Ron Cram: “Quit throwing sand in people’s eyes. Agree with the people who are mostly right. “

     Ron, what’s gotten a hold of you?! Agree with the consensus and majority!? Are you okay?

     These two statements are not equivalent. “Agree with people who are mostly right” is not the same as “agree with the majority of people.” That you believe they are equivalent shows either a poor facility with logic, or a willful misrepresentation. I’ll leave it up to you to let us know which it is.

176. Brian said: “Your demand that Arthur provide a reference to a statistics book is a red herring … Statistics textbooks are not ethics manuals, and math can’t provide ethical guidance any more than the laws of physics can.”

     The point is not primarily one of ethics, but rather of scientific rigor. It is not scientifically rigorous to say, “This method works sometimes, and sometimes it doesn’t. However, I will pretend it works all the time, and throw out the data where it doesn’t work, so that I can say something meaningful about the last 1000 years.” You will not, I wager, find that method in a statistical textbook.

     Ethics come into the equation when one uses methods that are so obviously poor, scientifically. Perhaps it is fraudulent, as you first claimed, and perhaps it is not, as you later amended. Personally, I believe the people involved were smart enough to know that it was unsupportable scientifically, and therefore deceptive.

     Brian, you said that you lack the expertise to judge whether throwing out data that undermines one’s conclusions is acceptable or not. That would be fine, I suppose, if we didn’t have a wealth of information from the CRU emails, which have been widely quoted and read on this subject. It is quite clear that the “hockeyteam” objected to the original graph based on outcome rather than process. It did not produce a clear narrative, it was a problem, it undermined the message. And by the way, here’s a neat trick to wash it away.

     So now it seems you are making two arguments: 1) you no longer consider it to be fraudulent behavior; and 2) you now defer judgment as to whether or not it is scientifically supportable. For (1), I suggest you take a fresh look at the emails. For (2), I think you should defer to statistical experts, such as McIntyre, rather than to those who were clearly looking for a “trick.”

     I should also remind you, Brian, that it was the “review team” who were looking for a trick. So you cannot claim, implicitly, that the “review team” is independent in this matter and that the trick “survive[d] review.”

177. Brian, let me summarize what’s happened so far:

     (1) You said replacing the tree ring proxy data with instrument data would be fraud when you thought Jones et al. didn’t do it.

     (2) Then you learned they did exactly that.

     (3) You then changed your mind to say replacing the tree ring proxy data with instrument data was probably a justifiable judgment call.

     (4) Your credibility was flushed down the toilet.

     • I do realize how it looks, Skip, and the fact that I misspoke initially does nothing to change it. So given I messed up and seriously damaged my own credibility, the only thing to do now is take this down as a painful learning experience and move forward in a way that rebuilds my credibility. And doing that requires two things – that my work going forward is rock-solid, and that I admit errors when I make them.

       I don’t yet know what exactly transpired with respect to the truncation and padding of data, and so I’ll need to research it in greater detail. I hope that I’ll know a lot more when the results of the Muir Russell review are released on July 7. But part of the process will be reading all the CRU emails (something I’d been avoiding given that I have a full-time job and family and didn’t want to spend the time) and probably reading the Climategate book(s) and more of Climate Audit. So be it.

       Time to get started.

178. Best of luck Brian,

     Again, if you need it for your next report, I will send you an electronic copy of our book gratis.

     Do not be discouraged. You are far from the first to err on this subject.

179. Brian, #187. Well said.

180. Brian #180,
     No, Brian, you are wrong. Search any statistical reference about how data is to be handled. You can find information about infilling, about spotting errors in data, etc. But you will never, ever find an exposition about when it is right to throw out data you don’t like acquired from one source and replace it with data from another source. Two different data sources have very different statistical properties. Is it an ethical issue? Of course, but it is a statistical issue first. Arthur needs to apologize for making claims which are blatantly false.

     In fact, Michael Mann wrote in RealClimate that such an action would be wrong and that no climate scientist had ever spliced the temp record onto a proxy reconstruction. Even you, Brian, said it would be fraud to splice the temp record onto a proxy record — that is, until you found out that is exactly what Mann and Jones did. Now you sound like you are trying to defend it. It is indefensible.

     As for my attempt to get Arthur Smith to respond to my statement, I put it here because he announced his results here. I did not bother to post on his site because he will not allow through comments which disagree with him. A number of commenters on ClimateAudit have already made that point so I am not going to waste my time.

181. Arthur #181,
     You write: “Ron Cram, while I find it likely that McIntyre’s writings have misled Mosher, that was not something I investigated or came to any conclusion on, and I never stated that in the post.”

     You say you did not investigate this, but anyone who reads what you wrote will come away with the idea that is what you are trying to say. You attempt to use a clever way of writing in order to slime the good name of Steve McIntyre without actually accusing him of an error. It is low, Arthur, very low. And it is the reason Steve was upset at you and wrote his post.

     You also wrote: “As far as me admitting “McIntyre is right about all the important points of science.” I have not done that either. McIntyre is clearly wrong in his latest post about me.”

     First of all, Steve’s post about you was not about the science. It was about your attack on Steve. When Steve pointed out that he never made the error you claim he made, you wrote on ClimateAudit:
     “In fact, in that post I was generally not interested in ClimateAudit at all, and certainly not interested in any statements made by Steve McIntyre, I have no complaint about what Steve has said other than perhaps he has confused some of his readers (such as Mosher).”

     It is pretty clear to me that you have no complaint about the way Steve has handled the science. But even though that is true, yet somehow McIntyre has misled Mosher. It is an untenable position, Arthur. Steve cannot be both right and misleading.

     And you still owe people an apology for saying truncating proxy data and replacing it with temperature data is not nefarious or “wrong.” It most certainly is both. It is insulting for you to claim it is not wrong in the presence of intelligent beings.

182. Brian #187,

     If you truly want to rebuild your credibility by admitting your mistakes, then you need to begin with issuing another correction as I called for in comment #85 above.

183. Ron, I added the following as another update to the post above.

     As seen in the correction below, I initially claimed that the deletion of data would be fraudulent. I then realized that, based on my own experience as an electrical engineer, this wasn’t necessarily the case and the appropriateness of doing so would depend on a host of factors that I had not considered or explained in the post. It is also clear that I did not conduct sufficient research into this specific issue before posting on it, and as a result I made two errors. The first was misunderstanding the differences between the tree ring proxy graphs for the WMO, TAR, and AR4. The second was not understanding that the WMO graph appears to have been created by padding data from the instrumental record onto a tree ring proxy record. The propriety of this is open for debate, but I should have known about the issue before posting this piece. I appreciate the fact that McIntyre and various commenters below have pointed this out, and I apologize for this error as well.

184. Ron (#190): You said:

     Search any statistical reference about how data is to be handled. You can find information about infilling, about spotting errors in data, etc. But you will never, ever find an exposition about when it is right to throw out data you don’t like acquired from one source and replace it with data from another source. Two different data sources have very different statistical properties. Is it an ethical issue? Of course, but it is a statistical issue first.

     You’ve made two fallacies here, Ron. You’re using an argument to authority (stats textbooks are the last word on data crunching) and you’re making an appeal to ignorance (stats books don’t specifically say it’s OK, so it must be bad). Neither does your argument any good.

     There are times and places where data padding is OK, and its a judgment call as to what the “best” way to pad data is. Whether this is one of those cases or not I haven’t researched sufficiently well to know. But it’s clear that experts are of two minds on this issue – which set of experts you agree with is a matter of opinion and ethics, not of statistics.

185. Brian, I don’t know if this would shorten or lengthen your research on appropriate application of certain statistical techniques, but they are discussed in mind-numbing detail by some knowledgeable people in the comments section of quite a few posts on Climate Audit. I believe Steve Mosher could point you pretty quickly to some.

186. Brian #194,
     You have made two errors. My request for support from a textbook is not an appeal to authority. An appeal to authority would be if I said “Steve McIntyre is an Oxford trained statistician and he says this was wrong.” In such a case, I would be taking Steve McIntyre’s word for it.

     Textbooks exist because they train students to become statisticians. You may have countered my argument by saying “I could find no textbook, but I did find a paper in a statistical journal which made it through the review process and will the idea has been accepted and it will no doubt be in a textbook some day.” That is the way new statistical methods are properly developed. I would even accept a peer-reviewed article in a statistical journal. But no such evidence exists. Why is that? Because all of the statisticians know the method is bogus!

     As for the claim that it is an appeal to ignorance, I am aghast. Statistical texts exists to show people how things are properly done. If you are not following the rules, you are not doing it properly. Again, if you want to innovate a new statistical method, then present the idea to a statistical journal. But I guarantee you the idea will get no where. Truncating and splicing data which has such different statistical properties is an embarrassment to statisticians everywhere. It is completely indefensible.

     You claim there are times and places when data padding is acceptable. Of course! What Mann and Jones did is not padding. it is no where close to padding. It is DIFFERENT DATA from DIFFERENT TYPE OF INSTRUMENT.

     I’m sorry. I’m starting to get emotional. This is not a matter of opinion. Just like 2+2=4 is not a matter of opinion. What Mann and Jones did was an error. Mann knew it was an error and that is why he denied doing it on RealClimate.

     Take the blinders off Brian. They are not becoming. And they do not help your credibility.

187. Brian, I must say that Ron Cram is correct directly above, which is why Mann said that it had never been done with paleoclimatic data.

     • Ron (#196): Before I go any further on this, I’m going to define a couple of terms so as to clear up any ambiguity in my response: “right,” “wrong,” “better,” and “worse” are terms that connote judgments based on experience that can be justified or not depending on the context of the judgment, while “correct” and “incorrect” are terms that apply to fact, such as your “2+2=4” example.

       With that out of the way, here’s why you’re request for textbook information is a red herring, why your appeal to textbooks is both an appeal to authority and an appeal to ignorance.

       First, why your call for a textbook explanation is a red herring. As I said above, textbooks on technical subjects like mathematics and statistics are intended to provide the student the tools by which they can solve problems, not solve the student’s problems for them. The reason for this is obvious – no textbook can possibly anticipate every single problem that the reader will encounter in the course of his or her work. For this reason, textbooks are careful to describe what they’re doing in terms of what is “correct” or “incorrect” instead of in terms of “right” or “wrong.”

       From early math classes, we know that addition and multiplication are essentially identical functions – multiplication is merely multiple additions: 2+2+2+2=8 vs. 2*4=8, for example. Somewhat more advanced math shows that subtraction and division respectively, are also essentially the same – division is multiple subtractions. The next step in complexity is understanding that subtraction and addition are inverse operators, so you can convert addition to subtraction by adding a negative number and you can divide by multiplying by an inverse of a number, as in 10-2=10+(-2)=8 and 8/4=8*(1/4)=2. And the final step in mathematical complexity is realizing that exponentials are merely repeat multiplications, and so are effectively identical to multiplication AND addition (and by extension, root functions are the inverse of the exponential). And you can get all of this from textbooks.

       What the textbook isn’t going to tell you is when to use an exponential (root) instead of multiplication (division), or multiplication (division) instead of addition (subtraction). Any of the substitutions is correct, but which is better is a subjective judgment call. This is directly applicable to data when you have to make a judgment call as to whether to scale your data (multiplication/division) or offset it (addition/subtraction). In my professional studies, this is specifically an issue for whether I need to remove a DC offset in order to focus on the AC component (roughly equivalent to the calculation of an anomaly) or whether I should normalize the data (usually to percentages). Which is better, normalization or offset removal, is a judgment call that I have to make based on my specific requirements at the time. Neither choice is “wrong,” and both are “correct.”

       To use another EE analogy, I can use either node voltages or mesh currents derived from Kirchoff’s Current and Voltage Laws (respectively) to calculate the voltages and currents in a circuit, so they’re both “correct.” Which is “better” depends on the exact circuit, and most realistic circuits are complicated enough that there is no “better,” just personal preference for what the engineer finds most comfortable to use.

       The closest thing to describing right/wrong/better/worse was a DSP book that said basically “this class of problems is usually most easily solved using these types of transforms assuming that you meet these conditions. If you don’t, then you may have to try every type of transform before you find the one that works the best for your problem.” I suppose that statistics textbooks could be different, but I have a hard time believing that.

       Second, why your appeal to textbook authority is an appeal to authority. Appeals to authority aren’t necessarily wrong, and appealing to the authority of a textbook is one of the better ones, but textbooks are ultimately written authors with their own biases and blindspots. I’ve got three noise textbooks on my desk at work, each of which disagrees with the other on some key points, including some equations. It’s not that one is right and the other two are wrong, but rather that they use different methods to derive their equations – one uses one set of simplifications of Maxwell’s Equations, one uses a similar but different set of simplifications, and the third uses empirical data based on a single set of experiments that may or may not apply directly to the issue at hand. All three books are “correct,” but which I use in any given application depends on a judgment call regarding applicability.

       You can’t appeal to the authority of textbooks, Ron, because the experts might not agree on the point you’re appealing about. In this case, I’ve seen enough disagreement among experts even just in the various CRU inquiries, never mind things I’ve read at Tamino’s place and other sites around the web, that your otherwise reasonable appeal to authority doesn’t work here.

       Third, why your appeal to the contents of the textbook is an appeal to ignorance. Fundamentally, an appeal to ignorance is an appeal to missing information. Logically, an appeal to ignorance says one of two things – either “there is no evidence against A, so A must be true,” or “there is no evidence for B, so B must not be true.” Your claim is of the second type – “because statistics textbooks don’t explicitly permit padding one series with data from a different source, it must not be allowed.” If textbooks explicitly said and mostly agreed that padding one data source with data from another, presumably related, source, then that wouldn’t be a fallacy, but I find it very unlikely that they say that. If textbooks say anything on this subject at all, they probably say “be very careful doing this, as it can cause a host of problems and is prone to error unless you’re positive that the padded data source actually is related to the original source.”

       Padding one series of data with data from an unrelated source might well not be allowed, but what about padding data from a related source? Is padding data from one tree with the data from another that’s directly adjacent to it OK? If there’s a high correlation between one source of data and a presumably related source except for one endpoint, is it OK? Is direct correlation OK, or can two degrees of separation between correlation be tolerated (ie proxy to proxy to instrumental record)? How high of a correlation is necessary? Can only a mean be used, or can the raw data be used? And for that matter, is it right or wrong to combine multiple proxies measured different ways into combined datasets by using scaling and offset adjustments? All of these questions are fundamentally judgment calls, and they all relate to accepted standards of statistical data processing and professional ethics. But none of them is mathematically correct or incorrect. They all produce results that are either better or worse, but so long as the mathematics have no errors (sign inversion, missing digits, misplaced decimal points, that sort of thing), the results are correct. Teasing out problems in the answers to these questions and verifying that the mathematics is correct is what we have the adversarial peer review process for.

       I don’t know the answers to any of the questions I proposed, because I’m an EE and not a professional statistician. And that’s why I defer to the experts, who do not appear to have produced a single answer on the subject.

       A critical aside is that we’re talking about the methodology of graphs generated in 1998/1999, 2001, and 2005. It’s clear from the results of the two completed UK inquiries to date that methodologies changed (and, IMO, improved) between 1998 and 2005. The most questionable methodology issues occurred in the earliest graphs, and by 2005, the AR4 graph appears to have been produced in the EXACT way described (as Arthur’s detailed analysis of various other possibilities illustrated). To me, this means that the scientific method and peer review is working just as it’s supposed to – questionable methods were used, problems were identified and publicized, and later versions used more standard methods. Science progressed and the old, worse methods have been replaced with better methods.

       IMO, this means that McIntyre et al (including you) have had a positive impact on improving the science since 1999.

188. Ron Cram says:

     Please let’s stop the fantasy about known plant physiology and using trees for thermometers. It is science fiction. It is well known that trees at all altitudes have more than one limiting factor and there is no way to separate out the different influences

     Upper tree line growth is limited by temperature. It is too cold for trees to grow above tree line. It is the controlling limiting factor in line with Leibig’s Law. Upper tree line growth is not significantly limited by moisture or insects or disease or fire or insolation and so on. In most cases the cold is a minimizing influence on these factors. Most of these factors have distinct markers that are fairly easily discernible with sufficient expertise. That is why there are sub-disciplines in dendrology and forest ecology dealing specifically with these ‘confounding’ factors. In the case of moisture, lower tree line data or from sites where soil hydrology indicates xeric conditions are the controlling limiting factor, additional useful climatic information can be derived that makes the temperature component even more robust.

     I would suggest Hal Fritts’ “Tree Rings and Climate” as an introductory textbook on how different influences are separated, but I would caution, that in the sciences, textbook knowledge is no substitute for field and lab experience. It is unrealistic to expect any single paper to ‘explain the science’ in any field. Papers are devoted to narrow and particular questions within a field and it is presumed the reader has some a priori grasp of the science. However, “Recent unprecedented tree-ring growth in bristlecone pine at the highest elevations and possible causes”, Salzer, et al., 2009 substantially answers your specific concerns about stripped bark vs. whole bark trees. Or it would if you had the intellectual courage to let go of your pre-conceived biases. Judging from the tone and lack of insight apparent in your comments, I have little confidence of that happening.

189. Dear Brian:

     I’m afraid you are not very well-versed in logical fallacies. The appeal to authority fallacy only occurs when the source in question is not an authority on the subject in question. A statistics textbook is an authority on statistical tools and methods. It is perfectly reasonable and legitimate to appeal to such an authority on statistical methods.

     The claim was that there does not exist a statistics textbook that would allow or recommend: 1) calculating a connection between temperatures and proxies; 2) deleting a portion of data that undermines or weakens this connection;; and 3) deducing temperatures from proxies based on the calculation in (1), after performing (2). Even if you could provide a counter-example – which, I note, you have not yet done – this would not be an “appeal to authority” fallacy.

     You also said that Ron committed an “appeal to ignorance fallacy.” This is also not correct. An appeal to ignorance fallacy only occurs in the absence of any other information or evidence regarding the statement in question. Ron didn’t say that the absence of this method “proves” it is wrong … he said it looks wrong, a lot of other people have said it is wrong, *and* there is nothing in any statistical textbook that says it isn’t wrong. The presumption, based on this evidence, is that the method is wrong. Given that, it is perfectly logical to state that there does not exist a statistical textbook that contradicts this evidence.

     Further, the absence of information can sometimes serve as proof, and in this way may also avoid a logical fallacy. Consider the famous “dog that didn’t bark” as a prime example. If this method were such a reasonable and useful one, then a statistical textbook would probably cover it. Therefore, the absence of this method can be viewed as additional evidence that it is wrong.

     I can’t help but note the irony here: appealing to misunderstood logical fallacies is itself a logical fallacy.

     Brian, I worry that you have jumped on these inaccurate interpretations of logical fallacies rather quickly. My opinion is that this shows a marked bias towards one side of the argument. I worry because this would seem to go against your stated goals of taking a fresh (i.e., unbiased) look at the evidence in the papers, blogs, and emails. I don’t see much evidence that you would be able to do so in an unbiased way. But this is only my impression, based on the current exchange.

     Cheers.

190. Ted (#200) I admit that I’m not an expert on fallacies, but I did research these before I labeled them as such. I didn’t jump to these conclusions based on personal bias, but rather my understanding of fallacies based on my research. I could be wrong about them, and am willing to accept that I might be. However, I don’t think your arguments above have proven me wrong yet. Here’s why.

     From what I understand, there are several ways an appeal to authority qualifies as a logical fallacy. The most obvious is what you identified, appealing to a non-expert source as if it were an expert source. Another way, and the way I’m using, is appeal to an authority that is not representative of expertise. Given that textbooks are not likely to say anything about the questionable method for the reasons I indicated above, it would be inappropriate to call textbooks an authority on something that they make no specific mention of. If a significant majority (a threshold that is admittedly open to debate) statistics textbooks generally said that this method was not OK, then appealing to the authority of textbooks on this matter would not qualify as an appeal to authority fallacy as you pointed out. However, in this case, it is my opinion that the But I am willing to concede that I could be wrong – textbooks in general are a less misleading authority than appealing to any single individual expert, for example, something that Ron has taken care not to do.

     As a quick aside here, Ron did not say that the questionable method “looked” wrong as you claim, he said “This is not a matter of opinion. Just like 2+2=4 is not a matter of opinion. What Mann and Jones did was an error.” Given this equivalence (something that is also erroneous in my opinion), he clearly considers this issue to be one of fact, not ethics. In my defined nomenclature, he would consider this to be an issue of “correctness” rather than “right or wrong.” Therefore it’s a provable or disprovable thesis.

     I’d agree with you that the presumption would be that the method is wrong if a) textbooks said anything positive or negative about this and b) experts were not divide on this question. However, expert opinion is divided on this and statistics textbooks almost certainly don’t say anything about this. I’m reasonably certain statistics texts say things about correct data handling methods and correct padding methods, but for the reasons I’ve already described in gory detail above, there won’t be discussion of precisely when or if the questionable method is right or wrong. Furthermore, Ron appealed specifically to the lack of evidence in textbooks as proof of the method being incorrect (as well as wrong), and in that respect it remains an appeal to ignorance.

     I agree that the absence of information can sometimes be considered evidence, but I am unconvinced that it applies in this case. If you look at textbooks in general, they very rarely discuss problems in real world applications where the idealities presented in textbooks cannot be applied directly. I’m confident that nearly every field aspires to an accepted standard of “high correlation” with an r2 > .99, but different scientific fields work to different standards due to real problems with size of datasets, noise, difficulty of reproduction, unusually high percentages of outliers, non-Gaussian distributions, and so on. How to draw conclusions in fields where an r2 of 0.3 to 0.5 is considered “high” is something that is gleaned as a result of working in the field and following the accepted practices of the field, not necessarily the ideal practices described in a textbook. And most textbooks are not written with the difficulties of specific fields in mind – the texts are intended to be general enough that they are broadly applicable to any field that needs access to the textbook. I doubt that statistics textbooks are any different. Given the fact that textbooks are almost certainly written for general audiences rather than for specific fields, and given that real world problems necessitate different “accepted practices” with regard to statistics, the lack of discussion of methodologies similar to that done to the tree-ring proxies a decade ago could be representative of the difference between the ideals described by statistics textbooks and the real-world problems encountered in the application of those ideals to real data.

     Given the fact that the experts are split on the appropriateness of the method in question, it would be just as reasonable (and just as fallacious) for me to claim that the failure of statistics textbooks to mention the method supports the conclusion that it’s OK. In this case, lack of evidence is merely lack of evidence and not evidence in support of any particular claim.

     In this particular case, the utility of the method used is certainly questionable and open to debate, and experts continue to disagree about the appropriateness of what was done. But if the method is considered acceptable within the field of dendroclimatology due to problems with the real datasets that statisticians don’t necessarily appreciate, then it reasonable to consider the expertise of the experts in the field of dendroclimatology as well as statistics experts when discussing this methodology.

     Both you and Ron are neglecting an important point about the method described – the deletion of MXD proxy data that diverged from the temperature record was believed at the time (and is still) to be a result of the proxy losing it’s connection to temperature for an unknown reason. Therefore, the scientists involved specifically felt that the proxy, which had tracked other proxies that did not suffer from this divergence, concluded that there was sufficient reason to suspect that the divergence was spurious for some unknown reason. You’re both either ignoring the intent of the scientists involved in the context of the time, or you’re presuming that the intent was deception even in the face of the scientists’ statements and expert opinion to the contrary.

     Finally, you haven’t readdressed my detailed explanation about why this whole textbook issue is a red herring, however, only the two secondary points I made. Neither secondary fallacy has any direct bearing on whether the issue of textbooks as the right authority (instead of ethics guidelines) is a red herring. Do you have anything additional to say on that topic?

191. Neither secondary fallacy has any direct bearing on whether the issue of textbooks as the right authority (instead of ethics guidelines) is a red herring. Do you have anything additional to say on that topic?

     Thanks, Brian. I believe I addressed that in #185, but I’ll state it a different way. The absence of this method in any statistical text is evidence that the method is a poor one. It is so poor, in fact, it is akin to lying. It is as if you were to say, “If you don’t like a number, add five to it and keep adding five until it is a better number.” That method won’t be found in a textbook either.

     “Throw out data you don’t like and just use the other data” is both wrong and incorrect. That this method doesn’t appear in a textbook attests to it’s incorrectness. That it is used anyway is an ethical breech. But you can’t give evidence towards an ethical breech without showing the incorrectness. Hence, the quote about the statistical textbooks supports the contention that the authors of the various tricks were behaving fraudulently, by supporting the contention that the method is clearly wrong. Therefore, it is not a red herring.

     By the way, you seem to be saying that the ones who performed these tricks are actively defending them. Is this true? I was under the impression that they were either not defending them (such as Michael Mann stating that no dendroclimatologist would do such a thing), or avoiding the issue in various ways … such as by saying McIntyre, Mosher, and others are wrong about the methods; or that they’ve lost the data; or they can’t be bothered to respond to bloggers; or they try to minimize the effects that these wrong methods have; etc. Can you point me to where any of these authors of the WMO and IPCC reports have actually defended these methods lately?

     • Ted and Ron: Let me summarize the debate thus far – I’ve provided significant logical argument for why referring to textbooks is the wrong authority and qualifies as a red herring. I believe that I’ve countered every argument you’ve made, in fact, and yet you continue to assert that you’re right, but without providing actual logical arguments to support your position. So permit me to counter a few of your new arguments.

       Ron said

       Brian, you have not yet come to terms with the fact that if the tree rings can diverge from temp in recent years then it can happen in other centuries as well. If it can happen in other centuries, then it is not a good proxy.

       This is true, but also incorrect. The data shows that the MXD tree rings don’t diverge in prior centuries because they match up with other proxy series such as coral, boreholes, ice cores, etc. It’s as if you said that a car that just crashed on a test track could have crashed halfway down the track even though there is a complete visual record of the car along the entire track and no prior crash was observed. Sure, it could have crashed, but the data shows it didn’t.

       And while it’s not directly tied into this debate, your argument weakens the MWP as well – it could just be a spurious divergence in the trees due to some non-temperature effect.

       Ted said

       But you can’t give evidence towards an ethical breech without showing the incorrectness.

       and Ron said

       Nothing can be ethically wrong if it is not first incorrect.

       This is one of the most wrong things either of you have said yet. In my nomenclature, using the laws of physics to calculate the amount of thermite needed to slag an ex’s car engine is mathematically or physically correct, but it’s sure as hell wrong and unethical to do so (never mind illegal). It is mathematically correct to use padding to fill out a data series as required for a windowing function, and you can choose any padding method you want to and still be correct because followed the mathematical operation shown in the textbook. But the degree of better/worse of choosing a mean from the data itself vs. the mean from a related dataset vs. the mean from an entirely different and totally unrelated dataset (say, grafting the mean of sunspot numbers onto a temperature graph) is a judgment call and a question of ethics.

       Ron said

       Brian did you know that Michael Mann ran the r2 verification statistic on the Hockey Stick and it failed, yet when he published he claimed his method was robust?

       I was not aware of this, but given that r2 isn’t always the best verification method for non-linear data, there could be justification for it. However, I’d be interested in reading this. Can you provide a reference?

       Ron also said

       Ethics come into play when people know they are doing something wrong and do it anyway.

       Um, no. I think you have a very different definition of ethics than most people do. Here’s how Mirriam Webster defines ethics:

       1 plural but sing or plural in constr : the discipline dealing with what is good and bad and with moral duty and obligation

       2 a : a set of moral principles : a theory or system of moral values —often used in plural but singular or plural in construction b plural but sing or plural in constr : the principles of conduct governing an individual or a group c : a guiding philosophy d : a consciousness of moral importance
       3 plural : a set of moral issues or aspects (as rightness)

       No mathematics textbook will ever provide a “set of moral principles” or “principles of conduct governing an individual or a group” or even “a guiding philosophy.” Ethics guidelines provide this, but not math texts.

       Given how different your definition appears to be from the dictionary definition, perhaps this is one of the sources of our disagreement.

192. Brian.
     If you corrected all of the errors in your post, you would have to change the headline to “Climate scientists still beseiged and rightfully so.”

     Ted has it right. If a method is not found it a textbook, then it is not accepted. Can new statistical methods be developed? Absolutely, but there is a process and the method used by Mann and Jones would never, ever stand up to the process.

     Does a textbook really need to say you cannot disregard data that does not fit your theory? Really?

     Brian, you have not yet come to terms with the fact that if the tree rings can diverge from temp in recent years then it can happen in other centuries as well. If it can happen in other centuries, then it is not a good proxy.

     Brian did you know that Michael Mann ran the r2 verification statistic on the Hockey Stick and it failed, yet when he published he claimed his method was robust? Now, is that an incorrect thing to do or it is wrong? Both. A researcher has to report whenever verification tests are failed. I believe most researches understand that a failure to report a failed verification statistic is academic misconduct.

     Marcel Crok wrote an excellent article for a Dutch science mag titled “Kyoto protocol based on flawed statistics.” Have you ever read the English translation of it? You can find it at http://www.uoguelph.ca/~rmckitri/research/Climate_L.pdf

     Let me be perfectly clear. Nothing can be ethically wrong if it is not first incorrect. You seem to be under the impression that truncating data and replacing with a different type of data may be an acceptable course of action but unethical. That is not possible. It can only be unethical if it is first incorrect.

     If it was a similar to padding in some way, there are different ways to do padding. Some ways may be better than others depending on the statistical properties. If someone finds a better way, it does not mean the less optimal way was unethical.

     Ethics come into play when people know they are doing something wrong and do it anyway. Why would Michael Mann pull the trick and then deny it on RealClimate? There is only one reason. I’m certain you can figure it out.

193. 203 Ron Cram: “Ted has it right. If a method is not found it a textbook, then it is not accepted.”

     Where do textbook methods originate from? Not from textbooks.

194. J Bowers #204,
     You need to read the entire thread. I have said more than once that new methods can be developed but the way they are developed is through journal articles in professional statistics journals. Just because a method get into a journal does not mean it will be accepted, but that is the first step.

     You cannot just develop a new method and put it in a earth sciences journal and claim it is validated. Mann and Jones did not even do that. They pulled their tricks without telling people what they did. When Mann was asked about it, he denied it said it was a bogus allegation arising from fossil-fuel funded blogs. Mann knew what he did was wrong. Jones knew what Mann did was wrong when he copied “Mike’s Nature trick” to “hide the decline.”

     Mann, Jones et al do not have the balls to submit what they did to a statistics journal. They would be ashamed to admit it and rightfully so.

195. 
     Brian did you know that Michael Mann ran the r2 verification statistic on the Hockey Stick and it failed, yet when he published he claimed his method was robust?
     

     Ron, you have been set right on this. Memory is cheap nowadays. Get some.

196. Martin #206,

     I don’t buy it. If it was a poor statistical test to run, why did Mann run it? If it was good enough for him to run, why not report the fact it failed the test?

     I don’t know anyone in statistics who cannot understand that point.

197. Martin #206
     BTW, did you know when Mann was specifically asked about the r2 while testifying before Congress that Mann said he never ran the r2?

     This would have been another opportunity for Mann to admit the truth. Instead he claimed it would have been an “improper” thing to do. Totally bogus untruth while under oath.

198. Re. 205 Ron Cram: “You need to read the entire thread. I have said more than once that new methods can be developed but the way they are developed is through journal articles in professional statistics journals. Just because a method get into a journal does not mean it will be accepted, but that is the first step.”

     And Mann et al’s error bars were first published where exactly? Theyr’e the ones praised by some of the world’s leading statisticians for their innovation.

199. J Bowers #209,
     What have you been smoking? Who are these leading statisticians? Can you provide links to this praise you reference? I am only aware of criticisms of Mann’s error bars.

     Mann should never have attempted to innovate new statistical methods. When confronted with his errors, Mann admitted he is no statistician. So, I will be very interested to see this praise you are talking about.

200. Re. Ron Cram: “What have you been smoking? Who are these leading statisticians?”

     Dr Gerald North.
     [video src="http://geotest.tamu.edu/userfiles/216/NorthH264.mp4" /]

     To quote…

     “Michael Mann… was a student of Barry Salzmann… so Michael is kind of a mathematician, statistician, sort of a bit like myself…
     [..]
     But the interesting thing about this (hockey stick) graphic is that… it was really the first one that attempted to put any kind of error bars on the graph. So that’s really very important here, because most of the curves that you’ve seen before this are just a single curve, and so that is really a commendable thing that they did, and they were really the first ones to try to put this together.”

     North is not only a statistician but is Distinguished Professor of Atmospheric Sciences and Oceanography at Texas A&M. How much better can you get?

     I smoke tobacco by the way, and given I’m not nor ever have been an Obama birther I believe I can perhaps say I’m not the one who should be asked that question, don’t you?

201. Richard L Smith, Incoming Director of the Statistical and Applied Mathematical Sciences Institute, supporting Amman’s conclusions.

     Click to access 10_05_11climateSmith.pdf

202. > Instead he claimed it would have been an “improper” thing to do. Totally bogus untruth while under oath.

     Ron (and Brian too), did you read the Wahl-Amman paper linked to? It gives a very good and clear explanation, with pictures, why it is an improper thing to do. Because it is. Ask any statistician. Don’t speculate, find out.

     That Mann had the test in his code proves nothing — I have all kinds of stuff in my code too that I just don’t use when it’s improper. There are proper uses of Pearson r^2 … this just wasn’t one of them.

     Here again.

203. Brian #213,
     You claim you have provided “significant logical argument for why referring to textbooks is the wrong authority and qualifies as a red herring.” Not true. You threw against the wall a lame excuse that an appeal to a statistical authority was a logical fallacy. Statistical authorities are cited for support of statistical choices all the time. If you cannot cite a statistical authority for your choice, then your choice is unsupported and you are wrong. It is that simple, Brian. There is no getting around it.

     If tree rings are unreliable as a proxy, saying they match up with reliable proxies does not improve their worth. What are you trying to say exactly? That tree rings have value because they provide confirmation of other proxies? How valuable is unreliable confirmation?

     Your dictionary definitions of ethics do not prove your point. When I said that nothing can be wrong in the moral sense unless it is first incorrect, I was speaking in the context of science – not murder. Let me give you an ethics lesson. If someone tells an untruth because he believes the untruth, it is not a lie and the inaccurate statement is not unethical. Only when he knows a statement to be an untruth and tells it anyway is it a lie. People will coverup when they tell a lie. People do not coverup if they think they are telling the truth, even if they are misinformed. What was Michael Mann doing when he said he never spliced temp data onto the proxy record?

     Regarding the fact Mann ran the r2 and did not report the fact it failed, I have already provided a citation. It was written about by Marcel Crok. The English language translation is here. http://www.uoguelph.ca/~rmckitri/research/Climate_L.pdf

204. Martin #214,
     Wahl and Ammann are trying to provide their friend with some cover. The paper is a sad piece of work really. Read the main reason they reject r2:

     “Our assessment is that such a rejection of validated performance at the low frequency scale would be a waste of objectively useful information…”

     Geez.

     Steve McIntyre,prompted by Tamino, has written a nice post on the issue of verification. See http://climateaudit.org/2008/03/26/tamino-and-the-magic-flute/

205. Brian,
     BTW, if you want to know how a new statistical method is developed you should study the example of Nicola Scafetta and his development of Diffusion Entropy Analysis. He has published in statistics journals and in a variety of other disciplines showing how his new analytical tool can be used. By the way, he has also published on climate and is a leading skeptic. He has also proposed a phenomenological theory of climate change. Pretty interesting guy and a much better statistician than Michael Mann.

     http://en.wikipedia.org/wiki/Nicola_Scafetta

206. J Bowers #211 and 212,

     Gerald North is no statistician. When he testified before Congress on the Hockey Stick Controversy, he brought along a statistician named Peter Bloomfield. North and Bloomfield testified the NAS panel agreed with the Wegman Report.

     Regarding Richard Smith, I am not familiar with him. I will have to read this report before I comment. Thank you for the link.

207. What was Michael Mann doing when he said he never spliced temp data onto the proxy record?

     There is no evidence that he ever did. All we have is Phil Jones saying he is using ‘Mike’s trick’, presumably truncating anomalous data back to a point where the proxy data and the instrument data intersect, specifically for the WMO graph. Something that Mann wasn’t personally involved with, and hasn’t ever done in any of his publications.

     It isn’t only true that temperature sensitive tree ring chronologies match other proxy series. Not all boreal tree ring series show late 20th century divergence. Comparison of divergent and non-divergent series reveal little differences going back as much as a couple of thousand years.

     Also, divergent series still show temperature sensitivity on semi-annual scales. It is only lower frequency trends that show divergence. This strongly suggests that whatever the cause, it is not a direct non-linear response to elevated temperatures. What indirect responses that have been hypothesized quite likely bear the fingerprint of GHG forcing.

     All these together suggest orthogonal corroborating empirical evidence that divergence is most likely unique to the late 20th century. Preliminary investigations suggest likely anthropogenically forced ecosystem changes in many cases.

208. luminous beauty #219,
     You express an atypical perspective. Why would Phil Jones write what he did and describe perfectly what Mann was later accused of? At the time Jones wrote the email, no one has raised the issue of Mann splicing two divergent data sets. Mann was a recipient of the email. There is no evidence Mann objected to Jones’s characterization of his work. In fact, in the comment on RealClimate Mann wrote that to his knowledge no climate scientist had ever spliced the two data sets. When ClimateGate breaks we find an email from Jones written to Mann (among others) talking about how he spliced the data sets. I’m sorry but the evidence is clear. Mann spliced the data sets and lied about it. All of the people on the Hockey Team knew about the splice and the lie and no one had the balls to stand up and say so.

209. Ron Cram: “There is no evidence Mann objected to Jones’s characterization of his work.”

     There’s no evidence that he didn’t either. Or do you deem 0.1% of all correspondence as being 100%?

     It’s a bit like getting a 1000 page secondhand crime novel to find there’s only one legible page. Not only do you not know who did it, you don’t even know if anyone was murdered, robbed, tickled, if there was a crime, or even if you’re reading a romance novel that had the cover changed by the guy you bought it off. I say the last option.

210. Re. 217 Ron Cram on Scafetta

     Has Scafetta released that code yet? Or do you now agree that others who wish to verify and replicate scientific results should write their own by referring to the original published paper, just like Scafetta does? I look forward to your opinion on Scafetta’s unwillingness to share what contrarians believe to be absolutely vital to the scientific method.

     http://www.realclimate.org/index.php/archives/2009/12/please-show-us-your-code/

211. Ron,

     Why would Mann object?

     The WMO graph is a piece of cover art for a pamphlet meant for public consumption, not a scientific publication. It is unlikely Jones even produced the graph, but rather supplied data for that purpose at the request of some unknown graphic designer who wanted a smooth transition between the various lines. The rationale being; it would be simpler and less confusing to a lay audience than a tangle of disparate end points, and very likely, just because it is more aesthetically pleasing. As cover art.

     Additionally, the three proxy curves were labeled with proper references, so anyone who wished could go to the sources and see that in none of the relevant scientific publications had any splicing been done.

     There was no intent to deceive. The intent was to minimize confusion.

     Your desire is apparently to repudiate those whom you cannot legitimately critique on scientific grounds, but instead with trumped up ethical lapses, for no other sensible reason than the scientific facts threaten your ideological beliefs,

     Your obsessive mania has led you and your tribal cohort across bridge too far.

     It’s time to return to reality.

     • Ron, you say that I’ve not proven that your appeal to statistical authority (as opposed to a statistical ethics authority) is a red herring. I disagree, so let me describe our debate (as well as Ted’s contributions, as you and he are making essentially identical arguments) thus far. The description gets a little complicated below, so I’ve referred to you as “Ron” in order to clarify who’s saying what.

       Ron claimed initially (originally 108, repeated in #177) that statistics textbooks were the authority on whether a statistical method was appropriate or not, claiming that any method that wasn’t discussed in a stat textbook was inappropriate and wrong. In response, I said that Ron was making an argument about ethics while referring to a statistics authority, which I claimed was a logical fallacy (#180), namely a red herring. Upon review, it’s possible that Ron was not attempting to make a claim about ethics. However, the rest of our debate and Ron’s demanding an apology from Arthur suggests that Ron was claiming that stat texts are ethical authorities even though he didn’t directly claim so.

       In response to my claim of a red herring, Ted countered that the issue was not primarily of ethics (#185), but rather an issue of scientific rigor/statistics, and he provided this example:

       This method works sometimes, and sometimes it doesn’t. However, I will pretend it works all the time, and throw out the data where it doesn’t work, so that I can say something meaningful about the last 1000 years.

       In Ron’s response to my claim of a red herring, he said (#190) that no stat text will contain a discussion of replacing data from one source with data from a different source. Ron repeated this point in #196 as well, pointing out that the data sets in question were different and were taken from different types of instruments.

       I have rebutted this argument twice now (#201 and #213) by pointing out that Ron and Ted both appear to be mischaracterizing the method by neglecting to include a critical point, namely that the scientists believed that the data was being influenced by some non-temperature effect that was new to the tree ring record and that the method used related data instead of unrelated data. In response to my rebuttal, Ted repeated his mischaracterization of the method used (#202) but did not address whether the presumption that there was a relationship between the data sources altered his example and why or why not. Ron also has also failed to address whether the presumed relationship alters his argument in any way. I pointed out what I considered a significant flaw in Ron’s and Ted’s assertions that either could have addressed either by arguing that the flaw doesn’t alter the conclusions or by arguing that the flaw was actually incorrect. I see no evidence that either counterargument has been made to date. Instead I see repeated assertions of the same claims. Given this, my criticisms of Ted’s example and Ron’s related claims have not yet been rebutted.

       At the same point Ron was making his claims about the inappropriateness of the truncation and chosen padding method (#190), he also implied that stat texts were the authority of what was OK with respect to data padding, and the claim was made repeatedly (#190, #196, & 203). Ted implied this as well in all three of his comments to date (#185, #200, and #202) by claiming that “The absence of this method in any statistical text is evidence that the method is a poor one” and “The claim was that there does not exist a statistics textbook that would allow or recommend….” These claims were in response to my calling the appeal to stat texts a red herring.

       I responded to Ron’s red herring counterarguments with two different rebuttals. The first was that I asserted that stat texts are not ethical guidelines, but are rather “mathematical toolboxes” (#180). Ron and Ted both responded with a claim that the lack of specific approval for the questionable padding method was evidence of disapproval (#185, #190). My rebuttal (#199) contained three parts. First, I provided support for my initial “toolbox” assertion by pointing out that textbooks on technical subjects such as statistics cannot provide for all real world problems and so focus on providing students the means by which to solve a variety of problems. Second, I gave two detailed examples from electrical engineering that each described two different correct approaches to solving engineering problems and where each had no right answer. In response to these two points of rebuttal, Ted referred to his original comment (#185) and then proceeded to reassert – twice – a point I’d already countered in my response to his claims from #185 – that the lack of evidence was proof of both incorrectness and wrongness. In response to my two points of rebuttal, Ron also reasserted his original claim without additional proof that rebutted either of my two points. As of this point, I see no place in our debate where either my toolbox point or my two examples have been rebutted.

       My second rebuttal to Ron’s red herring counterarguments was that he had made two more logical fallacies in defending the initial red herring fallacy, namely an appeal to misleading authority and an appeal to ignorance. Ron disagreed, saying a) that an appeal to textbooks as experts was not an appeal to misleading authority because he was not referring to the authority of an individual person, b) that he rejected my appeal to ignorance claim because “Statistical texts exists to show people how things are properly done. If you are not following the rules, you are not doing it properly” (#196). Ted agreed with Ron on both points, saying a) that the appeal to misleading authority fallacy only applied if stat texts were not actually authorities on statistical methods (which of course they are), b) saying that the appeal to ignorance only applies if there are no other evidence on the issue being debated, and c) saying that sometimes the absence of evidence is allowed to be considered evidence (#200).

       My response to Ron’s rejections of my claim of argument to misleading authority contained several parts. First, I pointed out that the fallacy can apply to any authority, not just to people. Second, I pointed out that textbooks contain the biases of their authors. And third, I pointed out that three of my own technical textbooks have conflicting equations with regard to noise because the authors derived the equations using different simplifying assumptions and/or empirical data vs. pure theory. My response to Ted’s rejection of this claim was that stat texts are highly unlikely to make specific claims about the methods in questions and so it doesn’t follow that texts can be considered experts on something that they make no mention of. To date, none of the four points I made appear to have been rebutted by either Ron or Ted.

       I did not specifically address Ron’s rejection of my claim of argument to ignorance because his rejection amounted to yet another reassertion of his initial statement, not an argument in and of itself. I did address both of Ted’s rejections, however. First, I agreed that the appeal to ignorance would not be fallacious if either of two conditions were met – if the texts said anything about the rightness/wrongness of the method in question or if the experts were united on the incorrectness of the method in question, but I pointed out that texts are not going to say anything anything and expert opinion is divided on the issue. Second, I agreed that the absence of evidence is sometime acceptable as an evidence of absence, but I argued that this did not apply to this case. My reason was that stat texts, like any other technical textbook, cannot and do not attempt to account for all the real-world problems that different science fields face, and so the absence of evidence could be explained by the difference between ideal stats practices and real world problems as explained by disapproval. And third, I pointed out that the differences between ideal and real-world statistical expertise meant that the expertise of dendroclimatologists should be considered alongside the expertise of professional statisticians. To date, none of these three points appear to have been addressed by either Ron or Ted.

       Unless I’m missing something, it appears that my arguments to date remain largely unrebutted. While Ron has had the opportunity to rebut my most recent claims regarding the fallacies, he has not yet done so. Ted has not responded to the latest round of debate, and so it’s possible that some of my arguments will be rebutted by Ted the next time he shows up. However, the most significant problem is that both Ron and Ted have repeatedly made a single assertion and failed to address my criticisms of that assertion.

       (Back to direct address, since the rest of this comment is much less confusing).

       Ron, you said (#215)

       Statistical authorities are cited for support of statistical choices all the time. If you cannot cite a statistical authority for your choice, then your choice is unsupported and you are wrong. It is that simple, Brian. There is no getting around it.

       This isn’t necessarily the case, actually. I have to be able to refer back to known laws of physics when creating models for my circuits, but I don’t have to refer back to a given textbook every time I create a new model. I do, however, have to explain why it’s a valid model to my fellow engineers. If they disagree with me, then there’s a good chance that I’ll be asked to do it again or to develop a better explanation of why my model is valid. Either way, however, it’s only in the most unusual circumstances that I have to refer back to KCL, KVL, partial differential equations, or what have you. I have, in fact, been asked to walk someone through my calculations once or twice because my results ran counter to the “common knowledge” taught to most engineers.

       You asked what I was trying to say exactly, so I’ll rephrase what I was trying to say.
       1. Non-tree ring proxies correlate well with temperature over the period where there is directly measured temperature to correlate against.
       2. MXD tree ring proxies correlate well with temperature over the period where there is directly measured temperature to correlate against except for the period where the divergence occurs.
       3. MXD tree ring proxies correlate well with non-tree ring proxies over the period prior to the period of directly measured temperature.
       4. Given 1, 2, and 3, it’s reasonable to hypothesize that MXD tree rings would correlate over the entire period of directly measured temperature if there were not an unexplained problem.
       5a. Therefore it’s potentially justified to a) truncate the divergent data, and
       5b. it might be justified to pad the end of the series with temperature data (or it might not – I defer to the experts on this, and the experts are of two minds).

       I concede that someone saying something that is not truthful but who doesn’t know that it’s not the truth is not being unethical. However, you’ve claimed repeatedly that stat textbooks are the authority for truth when I’ve demonstrated repeatedly that they are not. In areas where the the textbooks have nothing to say for whatever reason, the ethics of something becomes gray, not black or white as you seem to be presenting it. And this is clearly a gray area, given the disagreement among the experts about the appropriateness and correctness of the method in question. Remember that there is a difference between the ideal of statistics textbooks and the reality of dendroclimatology (or any other stat-based science), which is a point that I found while reading one of the RC links above: “both fields [stats and dendro] benefit from examining the different kinds of problems that arise in climate data than in standard statistical problems and coming up with novel solutions.”

       I went looking through the long comment thread above and couldn’t find a link to where you or someone else proves that Mann lied. I’d very much like to see that if you can put it up so I can research that claim for myself.

212. As a scientist (PhD, Chemical Engineering), it would be very helpful if someone could point me to the following:

     1. References where Mann, et al describe, in detail, what they have done so that others can replicate it. I have seen many blogs on many sites and no one to my knowledge has been able to recreate the 2 or 3 graphs that are being debated.

     2. Reference where someone has duplicated the results precisely.

     Without these, the words “cold fusion” come to mind.

213. KD asks:

     As a scientist (PhD, Chemical Engineering), it would be very helpful if someone could point me to the following:

     1. References where Mann, et al describe, in detail, what they have done so that others can replicate it. I have seen many blogs on many sites and no one to my knowledge has been able to recreate the 2 or 3 graphs that are being debated.

     2. Reference where someone has duplicated the results precisely.

     Without these, the words “cold fusion” come to mind.

     Click to access Wahl_ClimChange2007.pdf

     In this paper, we examine the MM results and criticisms of proxy data and methods
     employed by MBH, based on our own independent emulation and use of the MBH method to reconstruct Northern Hemisphere mean surface temperature over 1400–1980.

     Our implementation of the MBH method was based on published descriptions (MBH; Mann et al., 2000) and on additional material (clarifications and data) available via a “ftp” site maintained by M. E. Mann (http://www.meteo.psu.edu/∼mann/shared/research/MANNETAL98/). The same data and
     supporting material are also available through Nature online supplementary information, as updated in 2004 (Mann et al., 2004). Dr. Mann also answered clarifying questions concerning a few details of the method.

     I hope you find this helpful.

214. DeepClimate has now done the analysis for IPCC TAR Fig 2.21 which I started for the AR4 figure, based on the above discussion of what “the trick” was. As I suspected from the start here, if Mann did pad Briffa’s series with instrumental data from 1960, as Steve Mosher claimed here (and McIntyre claimed elsewhere) it makes a difference of at most 0.01 degree over a tiny portion of the graph.

     Mann clearly did pad his own (MBH) series with instrumental data in that graph. He clearly did not for Jones’ series. Whether he did or not for Briffa makes essentially no difference. Clearly Mann has made a number of mistakes – silly mistakes. But evidence of conspiracy, malfeasance, fraud? The claims from McIntyre, Mosher, and Fuller are wildly overblown.

215. Arthur Smith at 227, so Mann did wrong, but it doesn’t matter? Okay. Guess we’ll have to recall our book, pulp all remaining copies and burn the plates. Sorry to have troubled you.

216. Tom (#228) – I have not previously looked into what Mann has done, myself. Evidently he has made a number of sloppy errors of one sort or another. We’re currently having a discussion of the Tiljander case on my blog. Again it looks like more sloppiness, and stubbornness in recognizing a problem. Carelessness of that sort is never good, even if it doesn’t make any substantive difference to the results; I am interested though if you agree that the Tiljander case is the worst one so far, or do you have more damning evidence against Mann, with more substantive effect on the science he’s produced?

217. Re. 228 Tom Fuller

     Look up Hanlon’s Razor. A mistake is a far cry from malfeasance and fraud. If there were no mistakes in science then surely we’d have a fraction of the scientists who practice today.

218. I’ve loved the “Magic Schoolbus” catchphrase “Take chances! Make mistakes! Get messy!” since I first heard of it – it’s the essence of exploratory science to take wrong turns and do dumb things while trying to figure things out. But at some point, preferably prior to publication, a little more care is appropriate.

219. Brian said: “Ted has not responded to the latest round of debate, and so it’s possible that some of my arguments will be rebutted by Ted the next time he shows up.”

     Yes, I was out of town. Thank you for the detailed re-cap, Brian. I’ve since caught up on the debate and will try to clarify and/or rebut.

     Brian quoted me as saying, “But you can’t give evidence towards an ethical breech without showing the incorrectness.” Since this is characterized as “one of the most wrong things” I’ve yet said, I should address this first.

     I assumed that the context for this quote was clear. Yet this proved not to be the case, and Brian made use of an analogy to refute it. The analogy involved using proper methods of constructing explosives to commit murder by blowing someone up, which is clearly: an ethical breech in the use of a correct method.

     I would label this rhetorical device a logical fallacy of “weak analogy.” I am not claiming that the statistical methods are correctly applied; I am saying they are incorrectly applied. I further contend that this incorrect method will not be found in any statistical text. This is my “appeal to authority.” If true, then the willful and knowing use of this incorrect method would constitute an ethical breech.

     My weakest argument is that the authors did know that these methods are unsupportable in the field of statistics. I could be wrong of course. But my opinion – restated from before – is that they were smart enough to know this. In order to make this claim, I must first assert that the methods are incorrect. Thus, the appeal to authority is not a red herring (i.e., immaterial to the charge). So my quote from above – with the context inserted – should be read: “But you can’t give evidence towards an ethical breech [in the use of an incorrect statistical method] without [first] showing the incorrectness [of that method].”

     Brian further asserts that this appeal to authority is not strong enough to show the incorrectness, then summarizes the method in question, and says that both dendro-climatologists and statisticians should be consulted in regards to the correctness of this in practice. He also states that the correctness in practice will not be covered in a description of the methods found in textbooks.

     I thoroughly disagree. The “method” is that divergent data was thrown out, and this is justified by the claim that something other than temperature affected the post-1960 proxy data. The problem is not that the divergence may have been caused by something other than temperature. This is something that a dendro-climatologist could actually comment on. The problem is: you need a statistician to say that “the data was being influenced by some non-temperature effect that was new to the tree ring record.”

     In other words, the contention that the non-temperature effect did not occur elsewhere in the 600-year record relies on statistical tests. Only the expertise of statisticians are needed to assess this test. And to date, all statisticians who have examined the method have agreed that the method is flawed.

     Near the end of #225, Brian helpfully summarizes the method. I should note here that two different things are becoming somewhat conflated – perhaps due to my own writing as well. The original ‘method’ involves deleting/padding/replacing data to make the divergence go away. This new method is one that Brian seems to assume, as a way to justify – based on assumed expertise – an altering the post-1960 dendro record. I think parts 1-3 of Brian’s method are important, so I’ll restate:

     1) Some proxies correlate well with the modern thermometer record.
     2) Some tree-ring proxies correlate well with the modern thermometer record, excepting a ‘divergence’ at the end.
     3) Both sets of proxies correlate well with each other – in the pre-thermometer record – allowing for the deletion of the divergence in some tree-ring proxies.

     In order to contend that something happened post-1960 that has never happened before, you need a statistical test to perform (3) above. I am not aware if this was done in the literature. It certainly wasn’t covered in the WMO graph or the IPCC reports. Thus, these graphs present a false narrative, which is fraudulent.

     Briffa does discuss the divergence issue, and hypothesizes that something happened recently that doesn’t appear elsewhere in the proxy record. But consider what needs to be done to show, rather than suggest, that this is true. The divergent series needs to be correlated with other proxies … therefore, it is another step removed from the actual temperatures. Thus it would be a “proxy for proxies.” However, it is added to the other proxy series, and conflated with them, and treated as if it is still robust in terms of temperature.

     Part of the reason for this conflating is that the divergence proxy correlates well with *some* of the thermometer record. But again, you need a statistical test to say what the error is in the thermometer record. You need further statistical tests to add in the error *between* proxies in the pre-thermometer record. As far as I know this hasn’t been done, presumably because the range of error would make the proxy-constructed record rather meaningless, and unable to distinguish fractions of a degree.

     A suggestion that these divergent records can be deleted without these statistical tests would not be found in a statistical textbook. Anyway, this post is probably too long already, so I’ll stop here.

     • Ted (#232):

       Let me go through the points you made above.

       First, on you claim of a weak analogy, I concede your point. I was specifically intending to discuss property damage rather than murder, but I admit that even that difference wouldn’t strengthen the analogy. Here’s a better analogy. I have my Calc101 book sitting on my desk at work, and contained within it are examples of how to convert sine and cosine into both series notation and into e^ix notation. However, the calculus book doesn’t describe when one notation is most convenient for my particular application as an electrical engineer. All three notations are correct, and while theoretically all three will give the same answer, one notation will usually be preferred over another because it’s the simplest and is thus least prone to error. Therefore, I could use any of the three and be both correct and right in doing so. However, the e^ix notation is the most common in my work as an EE, and so it would be odd if I chose to use either series or sin notation. If I concluded that the series notation was the simplest for my application, I would almost certainly be challenged over my choice during a review and I would rightly face more scrutiny.

       While all three notations are correct, the choice of which is best is defined by the class of problems I face as an engineer and involves a judgment call based on my education and experience. Only if I intentionally chose a worse method in order to fake my way through something would that qualify as an ethical breach.

       In the case of statistics, statistics textbooks will describe how to correctly pad a data series for windowing/filtering, but the textbook will not describe how to best pad a particular data series for use in the field of dendroclimatology just as a calculus textbook won’t describe which sin notation is best used in the field of electrical engineering. And just as the choice of which method is best for my field involves a judgment call, so too will the best choice of data padding method involve a judgment call for the field of dendroclimatology.

       The mathematical definition of how pad data is defined in the statistics textbook, but the rightness/wrongness of any particular padding method is a matter for expert opinion and ethics rather than an issue of mathematics.

       You said

       I am not claiming that the statistical methods are correctly applied; I am saying they are incorrectly applied. I further contend that this incorrect method will not be found in any statistical text.

       My analogy directly above addresses this point as well. I have demonstrated why you won’t find discussion of the statistical method in a textbook (it’s too particular to a field to be addressed in a general statistics textbook). I have also provided an example of a situation where the choice between correct methods is a judgment call based on expertise rather than mathematics. As such, you have not proven that your appeal to textbooks as an authority is correct, nor have you demonstrated that the methods described in the textbooks were incorrectly applied.

       You said

       The problem is: you need a statistician to say that “the data was being influenced by some non-temperature effect that was new to the tree ring record.”

       I agree in part, but I’d replace the word “statistician” with “statistical test.” You ultimately do need a statistical test to verify this, but you don’t need a statistician to do the test. You need to know statistics well enough to run the test yourself and to not be led astray by the results of that test. However, many technical experts in one field or another don’t necessarily run statistical tests on every datum they take, instead they trust to their own experience. Granted this means that their experience provides an unconscious bias their decisions, but a demonstration of error due to unconscious bias is not an ethical breach. Mann has himself admitted that he now uses an “optimal” technique for padding instead of the original, questionable technique from MBH98/99. This shows that Mann has acknowledged that his initial technique was less-than-optimal and has placed his recent work on a firmer statistical grounding. This shows evidence of being willing to accept critical review and change his mind when presented with new evidence. Thus his original work may have been flawed or erroroneous, but it was not unethical. And it has been shown to be largely correct by multiple proxy reconstructions since those early papers by people unrelated to Mann, so the errors are not so large as to justify claims of unethical behavior.

       I’m not aware of whether or not a statistical test was run and published in the literature either, and I agree that this would be the preferred way to handle this as opposed to an intuitive judgment of “Huh, that doesn’t look right….” But it’s not supportable to claim that the lack of publication of a statistical test means that the graphs in the IPCC or WMO were fraudulent. Erroneous or unconsciously biased, perhaps, but not necessarily fraudulent. Even the “hide the decline” email doesn’t provide proof of fraud. Proof requires inquiry, and thus far four of five inquiries have found no fraud. We’ll know the outcome of the fifth inquiry tomorrow morning, depending on the release schedule and what time zone you’re in.

220. Re. 231 Arthur Smith

     I like your catchphrase. I’m not a scientist, but I have “first timed” in my field. As far as I’m concerned, it’s sometimes as important and valuable to know how <i.not to do it. The way to getting it right is often to get it wrong, again, and again, and again, and to get dirty and take risks, just as you say. But even then, some of the stuff you pick up on while getting it wrong helps you get something else right at a later date.

     I was once told, “Talent is that done easily which most find extremely difficult to do. Genius is that which talent finds impossible to do.” I see a lot of talent on climate oriented blogs, but I feel the genius in the debate is tied up in research facilities here and there which the talented tend to heap all sorts of misunderstandings upon.

221. Arthur, I have no problem with Mann making mistakes. I’ll bet money I’ve made more than he. It’s what has happened to science since then as a result of his refusal to admit them and trying to persist with what appears to be bad science. Have we wasted a decade to protect the Team’s reputation and signature theories?

222. Re. 234 Tom Fuller: “It’s what has happened to science since then…”

     It’s moved on, Tom, and we know more about the climate system. You guys are either looking to throw baby out with the bath water, or are enabling plenty of others who have dubious or misguided intentions to do so.

223. Mr. Bowers, I don’t believe it’s moved on. If Mann is getting interviewed by BBC on Panorama about the Hockey Stick, it seems pretty clear to me that it hasn’t moved on.

     And I don’t honestly think it’s time yet to move on. Climate change is real, and needs to be addressed. But we need to have a serious discussion about whose voices we can trust regarding which policies we should consider adopting. The Team has been the back-up band for people like Gore and Hansen, and the Hockey Stick has been ‘the’ iconic emblem.

     So far, that’s not a problem. But the people who are using the Team as instant validation have been trying actively to shut out and shut off other voices and opinions. So the validity of the Hockey Stick is still a current question. Who can we trust when we are talking about the extent and timing of future climate change? Who can we trust when discussing how much we should spend on solutions and preparation?

     I say (with some sadness–I voted for the guy three times) that we cannot trust Al Gore. Or James Hansen. Or Joe Romm. and a lot of others besides. And the Hockey Stick–and their fanatical and Puritanical defense of it–is one reason why.

224. Mr Fuller, I trust Michael Mann a damned sight more than many other characters in this tragicomedy, in which the last laugh will be on our near descendants.

     I can’t, quite frankly, even figure out your intentions with the book. Could you please clarify what the aim of the book is? Was it a scoop that you felt, as a journalist, you need to run with? Is it that simple?

     “The Team has been the back-up band for people like Gore and Hansen, and the Hockey Stick has been ‘the’ iconic emblem.”

     Just wait until sea ice disappears; there’ll likely be a whole new few emblems then. But, undoubtedly, there will be the same talking heads, plus new ones, who’ll see conspiracy beneath every climate scientist’s published full stop, and every time they attempt to warn us of anything, those who have (shall we say) “other interests than the common good” will simply pull out one book after another which make pseudo-authoratitive claims about a set of redacted data/emails, forged by second guessing, absent of firsthand experience and expertise. How often are yourself and Steven Mosher cited and referred to by policy makers, meaning that the pair of you are now a part of that policy making process?

     Indeed, who to trust? The experts, or the hacks and the hackers?

225. Ted writes:

     Part of the reason for this conflating is that the divergence proxy correlates well with *some* of the thermometer record. But again, you need a statistical test to say what the error is in the thermometer record. You need further statistical tests to add in the error *between* proxies in the pre-thermometer record. As far as I know this hasn’t been done, presumably because the range of error would make the proxy-constructed record rather meaningless, and unable to distinguish fractions of a degree.

     “As far as you know” is the epitome of an argument from ignorance. All these things have been done. Repeatedly. Locally and globally.

     I repeat, for example:

     Click to access Wahl_ClimChange2007.pdf

     Tom, have you read Wahl & Amman? Do you understand it? Or do you entirely rely on the inexpert and specious claims of McIntyre and Montford to form your opinions?

226. Luminous beauty, after what you have said about me on other weblogs, could you provide any reason why I should respond to any question you might have of me?

227. Tom, have you read Wahl & Amman? Do you understand it? Or do you entirely rely on the inexpert and specious claims of McIntyre and Montford to form your opinions?

228. J Bowers, I don’t consider McIntyre to be inexpert nor Montford’s writing to be specious.

     That aside, I have read Wahl & Amman. I am not a climate scientist, but I believe I understand it (with what I hope are the appropriate caveats due from any non-scientist).

229. Tom Fuller, thanks for the response. Now could you maybe answer my question at the head of 237. Thanks.

230. Mr. Bowers, as we wrote, “We are writing this in December 2009, long before the story is over, and even before the implications can be truly evaluated. But given intense interest in the subject, we thought it would be useful to tell this story in order to deepen understanding of what happened, who were the principal actors, and why this took place. We note that discussion of Climategate, as the scandal has been dubbed, is full of defenders of these scientists who characterize the emails as a ‘tempest in a teapot,’ saying that ‘boys will be boys’ and that the science isn’t affected. We don’t think any of those statements are true.
     As we write, the COP15 summit in Copenhagen has just finished wrangling over the commitment of hundreds of billions of dollars per year to developing countries to fight global warming. The U.S. Environmental Protection Agency has issued an endangerment finding declaring CO2 a danger to public health.
     The issue is worth $1 trillion a year, the amount that many environmentalists consider the appropriate sum to throw into the fight against global warming. With such astronomical sums at stake, getting the science right would seem to be at the heart of the discussion.
     What this scandal (and hopefully this book) shows, however, is that for the scientists involved, even more important than getting the science right was getting the message perfect. And as the scandal plays out in the future, we think we can show that for many of the participants in this story, getting the message right meant ignoring holes in the science, shutting up those who disagreed and hiding the data from those who distrusted them.”

     We also were very interested in countering the claims of those who said that the emails were taken out of context–that indeed, the behaviour of The Team looked considerably worse when context was provided.

231. Luminous beauty (Roxane?) seems to be under the impression that we are discussing MBH rather than the Briffa divergence and treatment of same. Otherwise, I can’t imagine why she cites a paper dealing with Briffa only in passing. Perhaps I’m missing something?

     She also says of me:

     “As far as you know” is the epitome of an argument from ignorance. All these things have been done. Repeatedly. Locally and globally.

     Well, no. Saying “as far as I know” simply acknowledges the possibility of ignorance, which I readily accept. And it is an invitation for additional information on the question at hand. The point, luminous, is that I do not know if anyone has ever statistically validated the idea that “something” happened in the late 20th century to create the divergence that did NOT happen elsewhere in the last ~600 years.

     I suspect the answer is “No. No one has done that.” And if they did, they decided for whatever reason not to publish. But if more information is needed to judge, perhaps Briffa’s own words will help: “In the absence of a substantiated explanation for the decline, we make the assumption that it is likely to be a response to some kind of recent anthropogenic forcing. On the basis of this assumption, the pre-twentieth century part of the reconstructions can be considered to be free from similar events and thus accurately represent past temperature variability.” [Briffa, et al, 2002 – emphasis added]

     So … Briffa simply assumed that the post-1960 decline was an anomaly and thus ignored it. Further, as far as I can tell (another acknowledgement of possible ignorance on my part), Briffa did not use the post-1960 temperature for calibration of the proxy data, in order to prevent the resulting higher temperatures in the 600-year reconstruction of past climate.

     For a more thorough examination of this issue, you may want to read this post.

     Anyway, I’m pretty confident in asserting that this missing statistical test is one dog that should have barked by now.

232. Re. 243 Tom Fuller

     “The issue is worth $1 trillion a year, the amount that many environmentalists consider the appropriate sum to throw into the fight against global warming. With such astronomical sums at stake,…”

     Worldwide taxpayer funded subsidies to the fossil fuel industry: $0.5 trillion a year. Perhaps not the daunting sum you think it is. I, for one, can see a pretty simple solution to funding half of that $1 trillion a year…

233. Re. 244 Te Carmichael: “So … Briffa simply assumed… “

     Oh, come on, Ted, do you really think “simple assumptions” are made about such things. He wasn’t making a cheese toasty, and your comment is a bit overly simplistic, don’t you think? For instance, “Briffa et al probably worked on the problem for six months…” could put it more into perspective, even though neither of us could ever know one way or another.

234. J. Bowers said, “Oh, come on, Ted, do you really think “simple assumptions” are made about such things.”

     Hahaha … yes! I think you are starting to get it. They just assumed. Probably, they would say – like Brian did – that it is a judgment call, that based on X or Y it is a reasonable assumption. But according to the published record, they did nothing more than assume.

     But this is a key point. It is absolutely critical to the argument that this assumption be made. If the divergence does not represent something new and anthropogenic, then whatever caused the divergence could have happened in the MWP, or the Roman warm period, or any other time in the past.

     Saying it is a “judgment call” doesn’t remove the possibility of bias … in fact, this is one of the few places where bias can creep in. Bias due to preconceived notions, or expectations, or just bias due to wanting to get a paper published. It is in fact essential that this sort of issue be examined thoroughly and with rigor.

     This issue also exposes one of the drawbacks of the peer-review, publication process. Now anyone can cite the paper, and summarize the conclusions, and they don’t have to mention the assumption that makes those conclusions possible. Thus, more papers can build on this work, and justify their own assumptions with statements such as “consistent with Briffa, 2002.” If a couple things like this slip through peer-review, then an entire component of the AGW hypothesis is now “proven.”

     And it is a lot of hard work to undo, to find and challenge these things. That’s why scientific paradigm shifts happen so slowly, and are hard to accept.

     But at the end of the day … yes, they simply assumed. Amazing, isn’t it?

235. J Bowers, while you and I both might advocate for redirecting fossil fuel subsidies, I think the likelihood is that climate mitigation monies would be added on to our total energy bill, and that fossil fuel subsidies would remain firmly in place.

236. Ted,

     The value of Wahl & Amman is that they show that reconstructions with and without all tree ring series are not significantly different.

     You quote Briffa as to the following:

     “In the absence of a substantiated explanation for the decline, we make the assumption that it is likely to be a response to some kind of recent anthropogenic forcing. On the basis of this assumption, the pre-twentieth century part of the reconstructions can be considered to be free from similar events and thus accurately represent past temperature variability.”

     In one’s ignorance one might believe this assumption is nothing more than wishful thinking. However, there is a substantial amount of empirical evidence in support of this assumption. Divergent boreal series are not ubiquitous. That is, not all tree ring series show divergence. Comparison of those that do with those that don’t, do not show any divergence going back thousands of years, implying that whatever the explanations for the twentieth century declines are, they were not in effect during the medieval climate anomaly. Otherwise past divergence between the two would be obvious. Furthermore, tree ring series, compared with other proxies including lake bed sediments and ice cores all tell much the same story. More corroborating evidence that twentieth century decline is uniquely associated with anthropogenic causes such as acid rain, global dimming, or fingerprint characteristics of AGW like extended spring and autumns effecting summer and late growth moisture availability in some cases or early ice break-up in marsh environments creating early season inoxia stresses on root systems in others, etc.; whether they can be fully substantiated or not. Not to mention the possibility that much divergence could very well be an artifact of end-point biases as a result of standardization methodologies.

     Now scientists, being conservative by nature (Keith Briffa in particular), are prone to be more skeptical of data for which they cannot provide substantial physical explanations. This has led to a tendency to use boreal tree ring series in recent reconstructions that do not show the divergence phenomenon and eschew those that do. Guess what? No statistically significant difference in pre-twentieth century reconstructions from earlier reconstructions that do use divergent series.

     http://www.ucar.edu/news/releases/2009/arctic2k.jsp

237. Looks like I was right about Michael Mann as said in 237.

     Penn State Completely Exonerates Climate Scientist Michael Mann On Bogus Climategate Accusations
     http://www.desmogblog.com/penn-state-completely-exonerates-climate-scientist-michael-mann-bogus-climategate-accusations

     Go on someone, we know the “W” word’s coming. Get it over with if you must, but do try to provide proof and not remote viewing.

238. I actually agree with the Penn State findings, but I note that he was cleared of charges that nobody brought forth against him. That’s a minor caveat, and I feel bad about saying it. And I don’t want to intrude on what I’m sure is a good day for him and those who like and respect him.

     However, you’re right, J Bowers in the sense that this almost certainly is not an end to this particular story.

239. Tom, I beg to differ on there being no charges brought against him. The charges were of research misconduct, as described in Penn Stste’s report starting on page 1:

     “On and about November 22, 2009, The Pennsylvania State University began to receive numerous communications (emails, phone calls and letters) accusing Dr. Michael E, Mann of having engaged in acts, beginning in approximately 1998, that included manipulating data, destroying records and colluding to hamper the progress of scientific discourse around the issue of anthropogenic global warming, These accusations were based on perceptions of the content of the emails stolen from a server at the Climatic Research Unit of the University of East Anglia in Great Britain as widely reported, Given the sheer volume of the communications to Penn State, the similarity of their content and the variety of sources, which included University alumni, federal and state politicians, and others, many of whom had had no relationship with Pel1l1 State, Dr. Eva J. Pell, then Senior Vice President for Research and Dean of the Graduate School, was asked to examine the matter. The reason for having Dr. Pell examine the matter was that the accusations, when placed in an academic context, could be construed as allegations of research misconduct, which would constitute a violation of Penn State policy,Under The Pennsylvania State University’s policy, Research Administration Policy No, 10, (hereafter referred to as RA-I 0), Research Misconduct is defined as:
     (1) fabrication, falsification, plagiarism or other practices that seriously deviate from accepted practices within the academic community for proposing, conducting, or reporting research or other scholarly activities;
     (2) callous disregard for requirements that ensure the protection of researchers, human participants, or the public; or for ensuring the welfare oflaboratory animals;
     (3) failure to disclose significant financial and business interest as defined by Penn State Policy RA20, Individual Conflict of Interest;
     (4) failure to comply with other applicable legal requirements governing research or other scholarly activities.

     RA-IO further provides that “research misconduct does not include disputes regarding honest error or honest differences in interpretations or judgments of data, and is not intended to resolve bona fide scientific disagreement or debate.”

     On November 24, 2009, two days after receipt of the allegations, Dr. Pell initiated the process articulated in RA-1O by scheduling a meeting with the Dean of the College of Earth and Mineral Sciences (Dr. William Easterling), the Associate Dean for Graduate Education and Research of the College of Em1h and Mineral Sciences (Dr. Alan Scaroni), the Director of the Office for Research Protections (Ms. Candice Yekel), and the Head of the Department of Meteorology (Dr. William Brune).

     At this meeting, all were informed of the situation and of the decision to initiate an inquiry under RA-I O. Dr. Pell then discussed the responsibilities that each individua
     would have according to the policy. Dean Easterling recused himself from the inquiry due to a conflict of interest. As the next administrator in the line of management for the college, Dr. Scm’oni was asked to take on Dean Easterling’s function in the ensuing inquiry.”

     Click to access Final_Investigation_Report.pdf

     Of course it won’t stop the accusations of whitewash. But it has to be admitted that such accusations, within context of the other inquiries so far on both Mann and CRU, are starting to come across as something from a conspiracy mentality, don’t you think?

240. To clarify further, here’s the definition of allegation given in Penn State’s “Policy RA10 HANDLING INQUIRIES/INVESTIGATIONS INTO QUESTIONS OF ETHICS IN RESEARCH AND IN OTHER SCHOLARLY ACTIVITIES”

     “Allegation is defined as any oral or written statement of possible research misconduct made to an institutional official. “

     http://guru.psu.edu/POLICIES/Ra10.html

     I think the numerous communications received by the university certainly fit the critetia.

241. J. Bowers, as was reported at the beginning and is included in the final report released yesterday at the time the Committee was convened there were no specific allegations against Mann brought from outside. These were areas of interest decided on by the Committee itself, and in my opinion lean heavily on the (perfectly understandable) side of making sure the University’s interests and reputation were protected.

     Their investigation of the first three allegations I consider rushed and incomplete. I have no professional opinion on the 4th, as I am not a scientist. I don’t want to minimize Mann’s victory, and I do hope it is a relief to him.

     But the allegations the Committee investigated were not the charges that Climate Audit would have made. In essence, the Committee found that Mann was not a threat to the University and would not be a toxic asset. And as you will no doubt see, this will not put the issue to bed, as the questions raised by McIntyre (and to a lesser extent myself) were not addressed.

     But that’s for another day. Let’s give the guy a break.

242. Re. 254 Tom Fuller:

     “there were no specific allegations against Mann brought from outside. “

     Tom, there were plenty of allegations. The list PSU came up with was based on the many and were most relevant to their policies.

     “But the allegations the Committee investigated were not the charges that Climate Audit would have made.”

     Would have? If McIntyre has a charge to make he should have made it, don’t you think?

     “Mann was not a threat to the University and would not be a toxic asset.”

     Sorry, Tom, but that’s a veiled “whitewash”. The panel investigated some incredibly serious charges. In fact, you couldn’t really get more serious in academia, as Mann’s entire career would have been flushed down the toilet and all of his prior work thrown into legitimate doubt if not the bin. The funding for Mann’s work comprised only 0.06% of PSU’s income. They could have easily thrown him to the wolves and it wouldn’t have even made a dent. This was a formal inquiry into the conduct and research of a leading academic of his field, and he came out squeaky clean. If you have no faith in the academic disciplinary process, then I suggest you look further into that subject, but it’s not taken lightly from what I read.

243. Like I said, let’s defer the battle for one day at least and give the guy a chance to enjoy it. Not that I think he cares what I write, but both he and his supporters should get a break from all this. If you really want to get after it today, there’s a thread on Climate Audit that already has quite a few comments.

244. Re. 256 Tom Fuller: “there’s a thread on Climate Audit that already has quite a few comments.”

     No. Read up on cascading conspiracy theories, which, when combined with the illogical mutterings over at CA, I have absolutely no desire to step into, thanks. I find it annoying that you suggest we don’t discuss anything here, out of respect for Michael Mann, yet suggest a different biased venue “if we must”.

     Mann has been cleared twice this year, by disinterested academics who lose everything if found to have been prejudicial. Can you match those consequences?

245. J Bowers, here is my published response: http://www.examiner.com/examiner/x-9111-Environmental-Policy-Examiner~y2010m7d3-Global-warmings-Michael-Mann-Cleared-of-the-charges-that-nobody-made

246. Pingback: The real scandal about the scandal that never was - Online Political Blog
247. Pingback: Drasties - Dutch on the World - World on the Dutch
248. Pingback: Muir Russell – what I’ll be looking for « Climate Audit

249. I find it humorous that you go from saying what Phil Jones did was fraud if it’s true, then realizing it is true, you go to saying I’m sure if that’s OK or not OK.

     What looks like bad behavior isn’t so bad if you’re the one doing it.

250. Five out of five.

251. Pingback: Mashey Potatoes, Part 1 | Watts Up With That?
252. Pingback: Manic Flail: Epic Fail | Watts Up With That?

253. There’s a new pingback or two from a couple of essays that Tom Fuller has written at WUWT where he claims that I was taking on Montford’s book. This is incorrect, as the following comment that I wrote at one of those essays can attest.
     —-
     Tom, the only mention of Montford in the post you link to is a quote from Gavin Schmidt, and there are no (zero) mentions of “The Hockey Stick Illusion.” The only mentions of Montford outside the Schmidt quote are in the comments, one of which references the Schmidt quote and two of which take you to task for relying too much on Montford’s book. There aren’t any mentions of HSI in the comments anywhere. And all of this is verifiable with a simple “find text” search in any browser.

     Furthermore, the post that contains the statistical analysis you want is not the one you link to (which is a criticism of you, Mosher, McIntyre, et al), but rather this one (http://www.scholarsandrogues.com/2010/06/03/context-climategate-emails/), a point I’ve made to you at least twice in the last few days. However, it also fails to mention Montford in either the post or the comments (except for a pingback) and there are no mentions of HSI either, again verifiable with a “find text” in your browser of choice.
     —

     I decided to put a copy of my comment here in case it doesn’t show up at WUWT for some reason.