Josef Oehmen and Fukushima – Would I have believed myself?

On the 13th of March, I posted an article called “Fukushima Nuclear Accident – A simple and accurate explanation“. This was early on in the Fukushima crisis when people were desperately hungry for understandable information, and yet there were scarce few good explanations available. The post had been written by Dr Josef Oehmen, a research scientist at MIT, in Boston. I’d stumbled across it when it had just been published on Jason Morgan’s new blog, and thought it was worth re-broadcasting, so I contacted Jason and got his and Josef’s permission to reprint.

The rest is history… Via my and Jason’s contacts and through Twitter and the blogs, it soon ‘went viral‘ , and later the Energy Collective reposted my version (with permission) and this amplified its audience even further. A group from MIT then took over management of the information, and did a few further updates, which I also mirrored. To me, it was an example of the internet at its best — exponential networking of key information.

However, the story doesn’t end there. It also created a huge amount of indignation, including a flood of vitriolic ad hominem comments on this blog that, if I’d let through the moderation queue, would have made your gentle eyes water! As the situation at Fukushima worsened, the MIT NSE group provided updates that improved upon the original information a little, and also toned down some of the stronger conclusions that had proven overly optimistic (I was also guilty of not fully appreciating the seriousness of the situation caused by the 14 m tsunami at Daiichi Plant). This updating of the information was, apparently, was the most heinous of crimes, and Josef himself was cast as the evil (and grossly unqualified) mastermind at the heart of an international conspiracy! (I was, alas, but a mere pawn in artful machinations…). The story was even taken up by New Scientist, although they got some of the detail (e.g., sequence of events) wrong.

So, what does the fiendish genius — with whom I’m since become firm internet buddies — have to say on this matter? Should people have listened to him, or should his article have been rightly consigned to ghastly the abyss of HTTP 404 errors? You decide, when you read this guest post…

Oh, and if you’d like to participate in a little 5 minute survey as part of the follow-up research that Josef is doing on this little drama, click here…

Would I have believed myself? On evaluating the quality of reports on topics that one does not know a whole lot about

Guest Post by Josef Oehmen. Josef is a research scientist in mechanical engineering and engineering systems at the Massachusetts Institute of Technology.

On Sunday, March 13, my cousin in Japan posted an email I had written to him on his blog in the early morning at 3am EST. The email explained the context of nuclear physics and engineering, as well as discussed the events at the Daiichi-1 reactor until that point. It also featured my very strong opinion that they are safe. By lunchtime, it was the second most twittered site on the internet (you can read the whole story at http://bit.ly/e1It0T). At the end of the day, it had been translated into more than 9 languages (often multiple times), and after 48 hours had been read by several million people. Two weeks into my unwanted and luckily rapidly cooling off Web 2.0 stardom, I have begun working through the trauma and reflecting. Thanks for sharing, you might think. But one question in particular came up that also has some general relevance:

Would I have believed myself if I came across that blog and had no prior knowledge of nuclear physics and engineering? Or asked another way: How do you judge the quality of TV, radio, print and internet news reporting on topics that you are only superficially familiar with?

Read the answer below. And like everything I write, it is rather lengthy!

Working in an interdisciplinary field as an academic, it is often necessary for me to judge the quality of information from areas outside my core expertise and decide whether they are reliable sources worth studying. Also, when you work with students, you start to develop little antennas when you read to judge if the student really got what she or he is writing about, and ultimately the quality of the students work (although you as the supervisor of course know everything better, well, you might not always be familiar with all the details).

So let’s take the example of my email-turned-blog, imagine I was living in Japan, had no idea about nuclear science and engineering (not too big a stretch someone just said), was looking for some info on Fukushima and came across Jason’s blog. Do I read it? All of it? What do I do then?

My approach to evaluating any sort of reports on the internet (and elsewhere) consists of 5 elements.: 2 regarding trustworthiness, 2 regarding the style (as a measure of effort put into a piece, but also a good indicator of the level of understanding of the author of the subject that he/she write about) and 1 element for content (arguably the most difficult to judge if you are not already familiar with the field). I will have to give myself credit on some of the dimensions, so I am asking you ahead of time for your forgiveness of some literary narcissism in the following.

1. Judging obvious fishiness (Trust)

When you surf the web, you come across a lot of stuff that you can safely disregard immediately. So I have two criteria for an immediate go/no-go decision at the onset:

a. Context: What is the context of the information? Blogs can be places where people put great stuff, but also incredibly stupid things (as I said, just Google my name these days). In the case of Jason’s blog, no points for great existing content, but also no minus points for tons of conspiracy theories and UFO posts. 0 points

b. Hoax potential: Would I have believed the whole story, cousin at MIT writing an email, setting up a blog to share it? Probably yes. Story looks interesting enough at first glance and setting up a blog is little enough work. Testing the opposite hypothesis: Why would anyone go through that much trouble of writing such a long text; invent such a boring cover story; and then assign the authorship to a total nobody in nuclear engineering, and not some expert in the field? So again, nothing major in favor, but also not a deal killer, 0 points.

2. Trustworthiness of the author (Trust)

Again, we have two criteria:

a. Past experience in the field. Is the author an authority in the field? Google clears that one up pretty quickly, certainly not. -1 point.

b. Bias, agenda, background: Checks out, engineering guy, MIT, probably has done his homework. 1 point.

3. Style and presentation (Style)

Is the narrative and style appealing? Again, I usually use this as an indicator of effort and level of understanding on the side of the author. Before I send the original email of to Jason, I scanned it one more time and thought to myself “Hm, this has actually turned into a nice piece of writing.” I probably would have had the same reaction scanning the text – well structured, flowing narrative, clear reasoning. 1 point.

4. Quality of the structure of the work (Style)

Does the article follow a logical structure? The article does seem well structured. It introduces the fundamentals, then progresses to describe what happened in Fukushima so far and drawing on these fundamentals. Seems to make sense. However it is not an academic treatise and strongly opinionated. Still, 1 point.

5. Content quality of the work (Content)

Here, since this is the most important category for me, I use a number of criteria:

a. Are the general fundamentals right? Are general engineering and physics fundamentals right that are used in the writing? Are the terms correctly used? Yes, 1 point.

b. Are specifics right? Are specific fundamental facts (e.g. half-life, types of elements etc.) and specific facts (sizes, amounts, temperatures, events) correct to the extent that I can verify them? Yes, 1 point.

c. Is there an uninterrupted logical flow from context and facts to interpretation? For the most part, yes. There are no logical breaks between the context, the facts being discussed in that context and the conclusions that are drawn. In its own little universe, it makes sense, no conclusions falling out of nowhere, no contradictions. However, again, the writing is not objective and strongly opinionated. But still, 1 point.

d. Are the sources given? Does the article contain sources so I could verify the claims and facts presented by the author? No, not in the narrative, not as footnotes. -1 point.

6. Possible next actions:

So, what should I do with what I just learned from reading the document? If we tally up the points for a first impression, we get 4 out of 10 points. And looking at the critical points, one of them is a biggy: No sources so I could easily verify if what the author claims is true or not. So what to do with it?

a. Disregard. This would mean thinking “oh my god, what a load of junk and a waste of time”. No, that is not what I would have done.

b. Use it to build mental model of the problem and investigate further. This means I use my newly acquired knowledge to build a mental model of the problem. What is the relevant context? What are the critical facts I need to know or monitor? That mental model is then tested (can I confirm what was said about the context, can I confirm what facts were presented?), and once that is done, run with it to grow the context (i.e. integrating understanding of spent fuel ponds) and interpret incoming facts (i.e. how dangerous is the latest venting of steam)?

c. Believe and be done with it. The information I just acquired solves my problem. I believe everything and am done with it (in this case, worrying about Fukushima).

As you can probably tell by the length of discussion of the different points above, I would have gone with b. That concludes my therapeutic reflections. And maybe you find the assessment process useful to make a more conscious choice of the news programs in TV, radio, press and internet you decide to support (I did, and that is why I love Barry and his site bravenewclimate).

Where does that leave us?

1. Help people understand the context. If you help people to understand the context, you help them to help themselves in the future. My hope is that the email made a small contribution to helping the general public, as well as some journalists, in building the context to make a better informed assessment of new facts as they come in. Do your part with your family and friends (as I had originally intended…)

2. Take a stand against mass hysteria. The email I wrote contains both an introduction to some relevant physics and engineering, as well as strong opinions about the safety of the plant you may or may not share. One part lives on on the MIT website that was created to provide some more of the same, fact-based and understandable context information; the other part has hopefully inspired a couple of people to also speak their mind in a general atmosphere of panic.

3. Demand balanced and quality reporting. Demand discussions of “possible” and “most likely” scenarios in the news. Call the newspaper editor, TV station and radio station and complain about the garbage that is still put out there. Make a conscious choice regarding your news viewing, reading and listening habits. News shows are out there to produce viewers, listeners and readers that they can sell to advertisers, not quality news. If you don’t demand it, it won’t happen.