[an error occurred while processing this directive]

PowerPoint Does Rocket Science --
and Better Techniques for Technical Reports

-- Edward Tufte, September 6, 2005



Response to PowerPoint Does Rocket Science
How to make engineers write concisely with sentences? By combining journalism with the technical report format. In a newspaper article, the paragraphs are ordered by importance, so that the reader can stop reading the article at whatever point they lose interest, knowing that the part they have read was more important than the part left unread.

State your message in one sentence. That is your title. Write one paragraph justifying the message. That is your abstract. Circle each phrase in the abstract that needs clarification or more context. Write a paragraph or two for each such phrase. That is the body of your report. Identify each sentence in the body that needs clarification and write a paragraph or two in the appendix. Include your contact information for readers who require further detail.

-- William A. Wood (email), September 8, 2005



Response to PowerPoint Does Rocket Science
This doesn't exactly fit `rocket science' but I was not sure where else to put it. A better title might be "Power Point does lifesaving--NOT."

This was an entry at the National Review blog called The Corner at 1:22 p.m. September 8 [http://corner.nationalreview.com/].

If you want to see low levels of useful data per slide, let alone irrelevance to the task at hand [presumably saving lives threatened by Katrina], you could hardly beat those referred to below.

"On lunch at work, but still would prefer no identification if referenced. Thought you might like to experience what our elite fire and hazmat volunteers are going through. This is insane.

"It appears that all people under FEMA for over two weeks must take "awareness and prevention of sexual harassment," "equal rights officer orientation," and "valuing diversity" training programs. The programs total 3-4 hours.

"The mandatory training matrix is here: http://training.fema.gov/EMIWeb/downloads/Mandatory05Matrix.doc

"The letter adopting the matrix is here: http://training.fema.gov/EMIWeb/downloads/Mandatory%20TrainingII.doc

"Powerpoints for the trainings:

http://www.training.fema.gov/emiweb/downloads/DF506/DF%20506%20Sexual%20Harassment%20Visuals.PPT

http://www.training.fema.gov/emiweb/downloads/Df434/DF%20434%20Intro%20to%20Equal%20Rights%20Powerpoints.ppt

http://www.training.fema.gov/emiweb/downloads/DF416/DF%20416%20Diversity.ppt

-- John Liljegren (email), September 8, 2005



Response to PowerPoint Does Rocket Science

I wasn't surprised to see my remark about Nature and Science contested but I was surprised to see who did the contesting, because what I see as the faults of these journals are exactly the sort of faults that you often criticize in your books. They both publish a lot of work that is fashionable and apparently exciting, but they don't insist on including the supporting information that allows readers to know exactly how the experiments were done -- half the time they wouldn't even allow authors to include this information because they would say it made the article too long. What happens in practice, therefore, is that high-profile authors will publish a claim-staking exercise in Nature or Science and then, if you are lucky, follow it up later in a journal of lower prestige with a "full paper" that includes the essential details omitted the first time round.

Let me quote (in suitably anonymized form) an e-mail that I received last week from a distinguished colleague in the US:

Thank you for bringing the (journal) paper by (authors) to my attention. I have not been keeping up on the literature (relevant comments below) and was not aware of it but absolutely agree with it. The paper by (other authors) (Nature, 2003) is pure bullshit, and the editors and reviewers responsible for letting it be published in Nature should hang their heads. (Name) and coworkers have hit the major problems (a totally incorrect assay procedure, highly suspect immunoblotting results) on the head in the third paragraph of his discussion....

This is just one isolated example, of course, but one out of many that one could cite.

-- Athel Cornish-Bowden (email), September 11, 2005



Response to PowerPoint Does Rocket Science
I'm trying to improve technical presentations in an organization where several high-level decision-makers didn't know about 1/2 mvv. The engineers there will be much better served, and decisions improved, by reporting at the level of Nature and Science (ideally articles, but even the commentaries on articles would be fine). My other models for NASA are Feynman's lectures on physics, and the A3 page (or 11 by 17 in) folded in half. You can see where we're at. If they would just write sentences, with subjects and predicates, rather than those damn bullet points.

At some of these organizations, a technical report is called a "pitch" and is presented in 10-15 minutes, or presented simply as a PP deck to look over, or shown as a one-slide executive summary, or circulated by email-attached PP slides for the cognoscenti. Some of that reporting is done in a crisis; the Boeing PP slides were prepared in 2 or 3 days when the Columbia was in trouble but still flying.

Moving from the PP slide-format to the Nature-style concise report would be an enormous improvement for any applied technical organization. Fretting about the differences between Nature and the Proceedings of the National Academy of Sciences is not relevant to improving NASA workaday technical reporting.

Nature and Science publish about 15,000 authors and co-authors a year, which means, given their high rejection rate, they disappoint perhaps 50,000 aspiring contributors each year. That is a lot of enemies to make. Nature and Science rejections probably annoy and bruise more scientists than all other scientific journals combined. Very few scientific publications have high rejection rates, in large because publication is financed by page charges (not unlike a vanity press), paid for by the author's research grants or institution.

Just about everyone who has attempted to publish has their own personal collection of injustices to retail. These horror stories describe biased, incompetent, envious referees and idiotic editorial decisions--at least at every journal with a rejection rate greater than 0%. The anger and the whiny sense of entitlement occasionally exhibited by rejected authors can become rather intense (even experienced on this little board with a contribution acceptance rate of about 60%). Publication horror stories and associated gossip are rampant in the social sciences and humanities, where rejection rates for the top journals routinely exceed 90%.

Talking to journal editors, not just the multitude of rejected authors, will fill out this picture. Over the years I've served on a dozen editorial boards of research journals and have gotten a pretty good idea of selection processes. This wisest thing I heard was from the editor of journal with a 1 in 20 acceptance rate: it is easy to identify and reject the 90% of the submissions below the line; but for the top 10% (only half will be published), it's a lottery (depending on the quirks of the referees). For NSF proposal reviewing (at least in the social sciences), it was usually very few star proposals which then led reviewers to ask how deep in the pool of routine dustbowl empiricism do we wish to dip? For journals and for grants, overall I was impressed with the care and integrity of the selection processes; I think most of us involved, other than the true believers, were seeking to find something, anything, that was good, novel, and true. I admire excellence in nearly whatever form it may take. For marginal submissions, those on the edge of accept/reject, non-meritorious factors may tend play a more important role in the decision, as it also does in faculty hiring in my experience.

The performance of a journal must be measured in aggregate and not merely by the embittered anecdotes of the rejected; that is why citations per published article and circulation numbers are relevant. Measured by citations per article, Nature and Science are close to the top, sometimes at the top. And they are by far the most widely circulated scientific journals. A measure of overall system performance is whether every minimally competent article gets published somewhere, if not in the most-cited and widely circulated journals. That is surely the case, since the median number of citations resulting from a published scientific article is zero.

-- Edward Tufte, September 11, 2005



Response to PowerPoint Does Rocket Science
The record for incremental reform in the cognitive style of PowerPoint is not promising. In the many release versions of PP, the intellectual level has not been raised. New releases have drifted toward ingrown self-parody, featuring ever more elaborated PP Phluff and presenter therapy. These changes have made the new version different from the previous version, but not smarter. There are no incentives for meaningful change in a monopoly product with an 86% gross profit margin, only incentives to make it different, somehow, from the previous release. PP competes only with itself.

-- Edward Tufte, September 20, 2005



Response to PowerPoint Does Rocket Science
Unfortunately, NASA is not re-evaluating the use of PowerPoint, instead, the "MINIMUM INTEROPERABILITY SOFTWARE SUITE" requires all federal employees and contractors employed at NASA centers to have a current version of MS Office installed on their desktop computer. Back of the envelope cost*: $13,200,000 every time a new version comes out. Appropriately, the justification for these standards is contained in a PP presentation.

For masochists: http://desktop-standards.nasa.gov/

* 60,000 on-center employees X $220/upgrade = $13,200,000

-- Robert Simmon (email), September 21, 2005



Response to PowerPoint Does Rocket Science

Here's a counterpoint to PowerPoint: the current NOAA National Hurricane Center forecast discussion for Hurricane Rita. The forecast is a succinct technical communication that effectively conveys reasoning and uncertainty.

http://www.nhc.noaa.gov/text/refresh/MIATCDAT3+shtml/ 211447.shtml

HURRICANE RITA DISCUSSION NUMBER  16
NWS TPC/NATIONAL HURRICANE CENTER MIAMI FL
11 AM EDT WED SEP 21 2005
 
THE RECONNAISSANCE PLANE WILL NOT BE IN THE AREA OF RITA UNTIL LATER
THIS MORNING. HOWEVER...SATELLITE IMAGES INDICATE THAT THE CLOUD
PATTERN IS TYPICAL OF AN INTENSE HURRICANE WITH A CLEAR EYE
SURROUNDED BY VERY DEEP CONVECTION. INITIAL INTENSITY IS ADJUSTED
UPWARD TO 120 KNOTS AT THIS TIME. HOWEVER...OBJECTIVE T-NUMBERS FROM
BOTH TAFB AND THE UNIVERSITY OF WISCONSIN CIMSS ARE PEAKING NEAR
7.0 ON THE DVORAK SCALE...SUGGESTING WINDS OF NEAR 140 KNOTS. I
WILL WAIT FOR THE PLANE TO REACH RITA TO INCREASE THE WINDS
FURTHER...IF NECESSARY. THE ENVIRONMENT IS CONDUCIVE FOR
STRENGTHENING AND RITA...AS KATRINA DID...WILL BE CROSSING THE LOOP
CURRENT OR AN AREA OF HIGH HEAT CONTENT WITHIN THE NEXT 12 HOURS OR
SO. THIS WOULD AID THE INTENSIFICATION PROCESS. THEREAFTER...THE
INTENSITY WILL BE CONTROLLED BY CHANGES IN THE EYEWALL WHICH ARE
DIFFICULT TO PREDICT. THE HEAT CONTENT IN THE WESTERN GULF OF
MEXICO IS NOT AS FAVORABLE AS IN THE AREA OF THE LOOP CURRENT SO
SLIGHT WEAKENING IS ANTICIPATED....BUT RITA IS EXPECTED TO MAKE
LANDFALL AS A MAJOR HURRICANE...AT LEAST CATEGORY THREE.   
 
THERE HAS BEEN NO CHANGE IN THE STEERING PATTERN. RITA IS MOVING
WESTWARD AT 11 KNOTS SOUTH OF A STRONG HIGH.  AS THE HIGH MOVES
EASTWARD...RITA WILL GRADUALLY BEGIN TO MOVE TOWARD THE WEST-
NORTHWEST AND NORTHWEST BASICALLY TOWARD THE TEXAS COAST. THE
OFFICIAL FORECAST IS VERY CLOSE TO THE MODEL CONSENSUS AND HAS NOT
CHANGED FROM THE PREVIOUS FORECAST.

BOTH THE GFS AND THE GFDL SUGGEST THAT THE WIND FIELD WILL EXPAND.
THEREFORE THE FORECAST WIND RADII HAVE BEEN ADJUSTED ACCORDINLY. ON
THIS TRACK AND DUE TO THE LARGE WIND FIELD ASSOCIATED WITH RITA...A
HURRICANE WATCH WILL LIKELY BE ISSUED LATER THIS AFTERNOON OR
TONIGHT.  
 
FORECASTER AVILA
 
 
FORECAST POSITIONS AND MAX WINDS
 
INITIAL      21/1500Z 24.3N  85.9W   120 KT
 12HR VT     22/0000Z 24.5N  87.9W   135 KT
 24HR VT     22/1200Z 25.0N  90.0W   130 KT
 36HR VT     23/0000Z 25.7N  92.0W   125 KT
 48HR VT     23/1200Z 26.6N  94.0W   120 KT
 72HR VT     24/1200Z 29.0N  96.5W   100 KT...INLAND
 96HR VT     25/1200Z 32.5N  97.5W    40 KT...INLAND
120HR VT     26/1200Z 35.5N  97.0W    25 KT...INLAND

-- Robert Simmon (email), September 21, 2005



Response to PowerPoint Does Rocket Science

The Rita forecast is certainly a big improvement on a typical PowerPoint presentation, but it's not beyond criticism.

  1. Why put it all in capital letters? Mixed upper/lower-case text is much easier to read and understand than all upper-case, as has been realized by the people who design traffic signs in many countries (though curiously not France, where I live) for at least thirty years. Computers have been able to cope with mixed upper/lower-case text for at least the same amount of time: surely the National Hurricane Center isn't still struggling with 1960s-vintage mainframes?
  2. I'm not sure if this is intended for the general public, but assuming that it is, can people be expected to know what TAFB, CIMSS, GFS and GFDL are?

However, I agree that it conveys reasoning and uncertainty in an honest manner, and that it is clearly intended for readers who think about what they are reading.

-- Athel Cornish-Bowden (email), September 21, 2005



Response to PowerPoint Does Rocket Science
My guess is the NWS all-caps style 1) is left over from teletype days and 2) remains because there's quite a bit of overlap between NOAA and the Navy's oceanographic community and the Navy still uses a lot of all-caps Courier. At this point I think they keep all-caps out of nostalgia.

As a New Orleans evacuee I have come to exclusively rely on NOAA reports. Interestingly I'm now following Rita closely because Tulane's medical school was supposed to reform in Houston this Saturday. That's now postponed until 1 October. I'm also staying in College Station, which is expected to get some of the wind and rain.

-- Niels Olson (email), September 21, 2005



Response to PowerPoint Does Rocket Science
The NWS caps style is also re-inforced by the fact almost all military orders (pick any service) are created in this teletype format. There is a argument that is eliminates a degree of ambiguity, at the expense of readibility. In other words, a capital letter contains the same meaning as a lower case letter. There is no implied difference of meaning based on capitalization.

But historically, it is based on telegraph and teletype styles. The format and ordering of the paragraphs can contain significance. In military communications there are generally headers and footers to tell you where you are in the message, what information to expect next, and that you have reached the end of the section or message. Assuming that you have the rosetta stone to decode often obtuse headers and footers.

By the way, this subject is very topical to me because we are frequently driven to produce reports in powerpoint rather that word. I usually object to call a powerpoint presentation a report; prefering to call it a brief or presentation. It is a very common to produce a powerpoint brief with notes pages and print the combination as a report. Even that combination leaves a lot to be desired in presenting complex issues, because all you have really done is add a text area about the same size as the powerpoint slide to amplify the slide. This, of course, means that the slide is probably not understandable to someone who is not already intimate with the subject material. But even in this case the notes pages very frequently suffer from the same choppiness.

I work in an organization where we are frequently more motivated to produce emerging findings quickly rather than conducting thorough analysis to produce quality findings. This means not only is the powerpoint brief terse and choppy, but often the results are misleading or wrong. Not to mention contentious, because it is nearly impossible to vet the findings and achieve concensus in a short period (such as less than a week in the Columbia case). I am almost sure that this situation existed in the Columbia case. And, as in my organization, you can almost assure that none of the results presented are accredited at the time they were first presented.

-- Clyde Smithson (email), September 30, 2005



Response to PowerPoint Does Rocket Science
After thinking about this over the weekend the thought struck me that the NWS style messages are more a function of data packing than anything to do with readability. Since these messages are transmitted across an electronic network that has a fixed data rate it is more important to reduce the character set so as to provide greater message bandwidth. It is probably the case that the engineers have employed a bit packing technique to maximize the amount of data carried in a word. Typically 8 or 4 bytes, 64 or 32 bits, but 2 or 1 bytes are encountered as word length of data transmitted. In a bit packing system if a value is binary (say on or off) only 1 bit is needed to represent the data, and so on based on the number of states. So why waste more data than needed.

To go back to the telegraph example, Morse code has 39 defined characters (26 letters, 10 numerals, and 3 special characters) so would require 6 bits (2 to the 6th bits = 64) to represent in a digital message. This means that in a 4 byte word (32 bits) that you could represent 5 Morse code characters and you waste 2 bits of information, a 6.25% waste. Standard ASCII, which does contain lower case and more special characters, has 128 different characters (2 to the 7th bits). A 4 byte word could contain 4 Standard ASCII characters but wastes 4 bits of information, 12.5% waste. Extended ASCII has 256 characters (2 to the 8th bits) so a 4 byte word contains exactly 4 characters. Engineers sometimes deal with the lost bits by using them to contain other information such as headers, or sometimes split characters across multiple words. This, of course, requires more software at either end of the message to encode and decode the information.

But from an economy of scale ASCII takes twice the bandwidth of Morse code and Extended ASCII takes four times the bandwidth. This does matter even today with high speed internet because much of the government and military infrastructure runs through older systems with lower data rates. Additionally, as both institutions become more network-centric, our desire to put more data through the network approaches or exceeds the capacity of the faster networks to carry all this data. This extra data flowing across the networks is not the equivalent of extra knowledge. The NWS example would provide no more information to the reader if mixed upper and lower case were used, but would consume 2 or 4 times the "data" space if ASCII or Extended ASCII were used.

For some history (http://en.wikipedia.org/wiki/ASCII#History)

-- Clyde Smithson (email), October 3, 2005



Response to PowerPoint Does Rocket Science
Here are the biographies of those on the Return to Flight Task Group who saw the NASA engineering by PowerPoint and denounced it in their final report (quoted extensively in the last 2 pages of my essay).

As I wrote in the essay above, "Both the Columbia Accident Accident Investigation Board (2003) and the Return to Flight Task Group (2005) were filled with smart experienced people with spectacular credentials. These review boards examined what is probably the best evidence available on PP for technical work: hundreds of PP decks from a high-IQ government agency thorough practiced in PP. Both review boards concluded that (1) PP is an inappropriate tool for engineering reports, presentations, documentation; and (2) the technical report is superior to PP. Matched up against alternative tools, PowerPoint loses."

The biographies of the PowerPoint reviewers are here:

-- Edward Tufte, October 18, 2005



Response to PowerPoint Does Rocket Science
For an excellent technical report on a complex engineering matter, see the NASA report on the foam loss from the external tank during the recent launch of the Discover (STS-114). The report is written in sentences and paragraphs, not bullet-point grunts and slides.

The report presents many extraordinary images of the tank post-launch. The difficult analytical issue is the lack of comparisons of the tank conditions from the previous 112 launches.

Here is the link to the report http://www.nasa.gov/mission_pages/shuttle/main/

Then go to the link "full report, 2.1 MbPDF".

-- Edward Tufte, October 23, 2005



Response to PowerPoint Does Rocket Science
What a stunningly beautiful piece of work, thank you.

This note needn't be considered for the thread. Just a couple of clean-up ideas, in addition to "are be...," it may be good to look at:

-- 5 grafs from the very bottom, "In the final report, 7 of Task Group Members... " -- First graf below the detailed slide analysis, last line: "This is a lot of insecure format... ." -- The "Marketing Strategy - Good4U" PP slide seems orphaned.

Thank you so much for the close work of your team. DJ

-- David Johnson (email), November 17, 2005



Response to PowerPoint Does Rocket Science
Many of the contributions to this thread are helpful -- especially those that point out flawed PowerPoint design. Has anyone found an example of skilled information design in a PowerPoint presentation?

I feel that we can only learn so much from examples of what NOT to do...

-- Scott L. Mitchell (email), December 9, 2005



Response to PowerPoint Does Rocket Science
PP is a competent Projector Operating System for full screen images and videos, replacing the little forward-back button in old-fashioned projector systems. PowerPoint is neither the best nor the worst Projector Operating System. It faces strong competition from the projector itself with its own forward-back controls. A Projector Operating System, however, should not impose Microsoft's cognitive style on our presentations.

PP has some low-end design tools helpful in constructing PowerPoint parodies.

PP might also help show a few talking points an informal meetings, but why not instead print out an agenda on a piece of paper?

PowerPoint may now and then benefit the bottom 10% of all presenters. PP forces the really inept to have points, some points, any points.

-- Edward Tufte, December 9, 2005



Response to PowerPoint Does Rocket Science
One of my professors, Dr Sandor Vigh, embeds what he calls zoomimage graphics in his PP presentations. I can't say I've seen anyone else do this and I haven't used Quicktime myself enough to know the buttonology how-to, but he basically puts a very high resolution image into a Quicktime frame that occupies the entire PowerPoint slide. So it's a slide projector. But he can zoom and pan! So he scans in Nettergrams, histology slides, whatever, at, say, 3000x2000 and then explores the details of the image while the projected image is always above the resolution threshold of the projector. This is vastly superior to 'mere' projection, despite the LCD projector. And look, it displays on the web!

Use the mouse to pan and shift/ctrl to zoom. Zoom in on the bones of the intermediate phalanges of the four fingers. Compare the fine lattice work of trabecular bone in the ends to the dense hollow tubes of cortical bone in the shafts.

-- Niels Olson (email), December 9, 2005



Response to PowerPoint Does Rocket Science

To follow up on Niels Olson's comment, the precise technology being used is Quicktime VR. Quicktime VR is typically used to stitch together 3-D panoramas or a 3-D view of a solid object - like Niels, I haven't used to to display two-dimensional images in this manner before. Dr. Vingh's use of it in his slides is an interesting end-run around the resolution limits of a normal computer presentation.

-- Zach Heaton (email), December 15, 2005



Response to PowerPoint Does Rocket Science
Elizabeth Lane Lawley, a professor visiting Microsoft, comments on "the culture of the deck":

http://mamamusings.net/archives/2005/11/19/the_culture_of_the_deck.php

Her experience at Microsoft is comparable to that of the NASA Return to Flight Task Group with regard to the persistent disutility of using PP decks to replace technical reports.

-- Edward Tufte, December 29, 2005



PowerPoint Does New Orleans
All but one of the committee 'final reports' for Mayor Nagin's Bring New Orleans Back Commission are out. Most are in PowerPoint. The report on levee recommendations is the lone exception.

NOTE ADDED, JULY 1, 2006 BY NIELS OLSON:

Does plain text or PowerPoint tell the story better? Compare Howard Reich's piece in today's Chicago Tribune, Crisis of culture in New Orleans, to the official Culture Committee report of Mayor Nagin's Bring New Orleans Back Commission.

-- Niels Olson (email), January 20, 2006



Response to PowerPoint Does Rocket Science
On the subject of writing, a favorite professor of mine said: "What's in the head goes on the page." The "final reports" to which Niels Olson links are astonishingly awful, and ought to be required reading for the those who would defend the general utility of Power Point. That these are "final reports" -- not merely tools to supplement the oral presentation of conclusions from actual written reports -- suggests that an antiliterate (I was going to write subliterate, but the word is insufficiently strong) approach now dominates public policy discussion in the United States. What a shame.

-- Alexey Merz (email), January 20, 2006



Response to PowerPoint Does Rocket Science
Just published is the 2nd edition of The Cognitive Style of PowerPoint. It is now 32 pages long; the original essay was 24 pages. The new edition contains the material on PowerPoint and rocket science that opens this thread, a long discussion of the causes of presentations (sorting out variation among users, content, and presentation methods), and an essay on lists. This new edition is also a chapter in Beautiful Evidence.

For more information and to order click here.

-- Edward Tufte, April 25, 2006



Response to PowerPoint Does Rocket Science
Thank you thank you thank you! I am an environmental scientist, and my company does environmental impact assessment 1-10 day training workshops -- WITHOUT Powerpoint! We have a programmer who prepares an occasional animated presentation in Flash (which is also helpful because it is only images, so we can offer it in any country, in any language) -- so I will have a projector in the room, and you should see everyone's face fall when they first enter and see the projector, and watch them light up when I tell them it is only there to scare them! Our trainings use case studies, flip charts, an occasional video or a cartoon on an overhead during the break, and music...NO Powerpoint or even Apple's lovely and way easier and more beautiful implementation of a similar system, called Keynote. I try and try to explain to prospective clients why we don't use it -- but sometimes we are actually required to use it by, gasp, their TRAINING departments....anyway, this is a long- winded way of saying that the PP disease is very hard to cure but there are many of us who simply refuse to be infected...and thanks for providing some good ammunition to use with our clients.

-- Leslie Wildesen, Ph.D. (email), June 23, 2006



Response to PowerPoint Does Rocket Science
It's not just reports that suffer.

I've recently attended a "technical" training course in which we were presented with an enormous number of PowerPoint slides, most of which were merely read aloud. Not only was this course mind-numbing to teach — I can only pity the instructor — it's also an active waste of time.

At the end of the course we were given a book containing one slide per page, along with a small amount of notes below. I've struggled to find the worst of these slides, and I've recreated one of the candidates here (90k PNG). I'm assuming direct scans would be forbidden, and I've neglected to include the "content" provider's logo.

I am still unable to find any shred of meaning in this slide, or many of the others.

-- Dan Avis (email), June 29, 2006



DISCOVERY LAUNCH
Here is a link to William Harwood's excellent account of shuttle risks in the upcoming flight, scheduled for this Saturday. This link provides context for my comments that follow. http://www.spaceflightnow.com/shuttle/sts121/060629preview/part1.html

____________________________

About 18 months ago in Houston I reviewed the shuttle Probability Risk Assessment (PRA) material for NASA. PRA works with a list of possible threats, estimates their probablilities and expected losses, and then seeks to assist decision-making for shuttle risk-reduction.

After the PRA group presented their results, I had two major suggestions:

(1) They should prepare a detailed summary matrix (on, of course, 11" by 17" paper), ordering the risks and providing, in a comments column, relevant background for each estimate. Let that intense matrix, backed up by similar more-detailed 11" by 17" arrays of risk estimates, be the main presentation device and analytical tool for making decisions. This was designed to replace their chippy and twiddly PP slides, which made a hash of their good technical work and made it difficult to assess the overall risk context.

(2) The PRA assessments did not take into account a major risk factor in both the Challenger and Columbia accidents: on-ground intellectual failures in engineering analysis. In the case of the Challenger, the analytic process on the day before the accident was seriously deficient, in the sense that--in hindsight to be sure--the Challenger would not have been launched on that very cold day (which compromised the O-rings and caused the accident) if smarter engineering analysis and better decision-making had taken place. In the case of the Columbia, better analysis and decision-making during the flight might have yielded rescue efforts to try to save the crew, which was endangered by damage to the Columbia suffered at launch. I suggested to the PRA group that on-ground analytic problems contributed to something like 1.3 of the 2.0 accidents in the 113 flights. But there was no risk assessment of such in the PRA; that is, about 65% of the directly observed empirical risk in the 113 flights was not accounted for by the PRA model. The shuttle itself was considerably less risky than what was happening on the ground in decision-making about the shuttle.

At the meeting, I also handed out Richard Feynman's famous discussion of shuttle risks, which Feynman prepared as a part of the Challenger investigation in 1987.

____________________________

The analysis for the upcoming launch of the Discovery in July 2006, as the link above indicates, was an intense evaluation of risks and trade-offs.

On the basis of reading some of the public documentation (and no direct knowledge) for the upcoming flight in the last few weeks, I think that NASA has made a reasonable and well-informed decision for the upcoming flight. It was also a contested decision. I would vote for the launch. The on-ground factors that contributed to 1.3 shuttle losses appear to be mitigated by the thorough analysis for this flight. The current risk number is a cloudy 1 in 100, which is risky but has been acceptable in the past. The cloudy contributions to risk are the recent changes in the foam, which turns Discovery into something of an experiment.

In the Discovery discussions, a telling distinction was made between "programmatic risks" and "crew risks." The programmatic risk is very high right now no matter what happens. Having flown once in 3+ years, the shuttle program might well collapse if unable to fly soon (within a year or so), or if there is another accident even if the crew escaped unharmed. This rescue scenario is itself troublesome, since the rescue launch must quickly take into account what caused the need for the Discovery crew-rescue in the first place.

-- Edward Tufte, June 29, 2006



DISCOVERY LAUNCH
Another detailed and excellent account by William Harwood on the eve of the flight:

http://www.spaceflightnow.com/shuttle/sts121/060630griffin/

There is a Feynman-like clarity to the Discovery analysis done by Michael Griffin, NASA's director. Now it just has to be confirmed empirically!

ET, Saturday, 6.53am

-- Edward Tufte, June 30, 2006



POWERPOINT FOR DISCOVERY FLIGHT READINESS REVIEW: THE FOAM SLIDES, OR "MAKE THAT CALL NOW, THAT'S 1-800-. . . . "


Here's the PP deck for "STS-121, Flight Readiness Review, External Tank Project (ET-119):"

http://images.spaceref.com/news/2006/2006.06.16.ET_FRR.pdf

This pdf file should be up in a separate window to read in parallel with the comments below:

These slides summarize the results of the enormous amount of resources (probably >$1 billion, some estimates are much higher) devoted to the external tank foam problem.

The slides do not display a sense of engineering intelligence or discipline. In the main report, there is a persistent habit of dequantification and a general absence of units of measurement. The back-up slides are more quantified and at a higher intellectual level. Several of the slides look like they were produced by a designer lacking in scientific training.

The key overview slide (page 3) is a very good idea but a presentation mess. The good idea is to have an intense and fairly detailed summary early in the presentation. But PP's lightweight resolution and lousy design tools compromise the summary slide. Students of PP design might, however, appreciate the 5 sets of orange drop-shadows, 4 wavy-purple color fields, 3 unintentionally 3D blue time-lines, 2 overactive grids, and floating-off-in-space bullets in the highlight box (with an arbitrary change from dots to dashes midstream in the box). All this stuff on one over-produced but importantslide.

In real science, every photograph has a scale of measurement built right in to the photograph. This low-resolution display method makes it impossible to do so. (Even the shuttle close-out photos, just about the most documentary type of photographs one can imagine, have no scales of measurement and no rulers in the pictures.)

The bullet lists tend somewhat to be base-touching grunts, which show effects without causes, actions without actors, verbs without subjects, and nouns without predicates. The branding with 3 logos on every slide (the title slide has 4 logos) is unprofessional, pitchy, turfy. Are we doing engineering analysis or marketing here? Some 20% of the space of every slide (already a a very low resolution display method) is devoted to branding and to the boxed-in awkward and repetitive slide titles. It is as if each and every slide has to remind the viewer what the presentation is about. So the top 20% of every slide is something to skip, perhaps putting some viewers in the mode of skipping and sliding through the rest of the slide. It is as if the top of every slide announces "nothing important here, you've seen it all many times before."

In several slides, the visually most active materials are the cross-hatched exploding 3D arrows linking the external tank to the magnified areas. Why are the arrows pointing anyway? It's just a simple linking line. The idea here of close contextualized imaging of the problem areas is a very good one, but the badly-drawn giant blue arrows are silly, and result in making the dequantified images of the foam problem areas too small.

The typography is poor, with odd hierarchies (underlined bold italic in parentheses at one point). Is "O2" the proper way for NASA contractors and NASA to write the oxygen molecule (even wikipedia uses a subscript)? Does the slide designer know how to write a subscript in PP?

The overlapping statistical graphics on page BU-2 are presented as decoration, not evidence.

The report is 33 slides long; yet about 10 slide-equivalents are essentially content-free (compulsive repetitive branding, twiddly hierarchical organization, empty space, assorted title pages, and so on). This PP fluffed-up material here and quite a bit more could easily be placed in a technical report on 4 pages of an 11" by 17" piece of paper (folded in half), an exercise left to the student.

The tone and style of the presentation seem alienated from professional engineering. It looks like the slides were prepared by a PP designer, assisted now and then by an engineer. Or maybe it is just the PP pitch style diluting the content. At an FRR?!

I hope the actual engineering for the shuttle is a lot better than the evidence for the engineering shown in this presentation.

How much does a problematic presentational style signal poor engineering? Is it just PP or a PP designer weakening the quality of evidence? Or are there deeper intellectual failures? The dequantification, the failure to follow professional engineering conventions, the infomercial tone are worrisome. There is no sign of engineering discipline here, except in the back-up slides. Thus the effect of the presentation is to suggest that there just might be some problems with foam engineering and analytical quality. A danger of problematic presentational styles, such as NASA PP, is not only that they enable sloppiness but also that they can place the truth in disrepute.

It is also a shame that all that expensive engineering work winds up being represented in this manner at a Flight Readiness Review.

-- Edward Tufte, July 1, 2006



Discovery foam problems occur on launch pad

This morning, Monday July 3, news of a small foam crack came out. On the evening of July 2, after 2 tankings and drainings for launch attempts scrubbed because of the weather, inspectors found a 5" crack in the foam on the Discovery external tank on the launch pad. Here's an excerpt from the KSC Ice/ Debris Team:

"The inboard strut for the L02 Feedline Bracket assembly at XT-1129 was found to be cracked. The damage is approx 5-6 inches long and appears to originate near the where the strut connects to the feedline and extends toward the ET. The TPS crack is approx 1/4 inch wide with an offset of approx 1/4 inch. An IPR was initiated for this item. Inboard views of the remaining visible brackets did not reveal any similar damage. Outboard views of the feedline brackets revealed areas of TPS debris in the gap between the feedline and the bracket - this condition was noticed at XT-1129, 1377, and 1623. No obvious indications of crushed foam or debris was detected at the XT-1871 and 1978 brackets."

There's a picture of the foam crack and the full inspection report here.

Note that the inspection report is written in sentences and not in the cryptic grunts of PowerPoint.

There is a research design problem or a control group problem here: are we seeing cracked foam or inspections of cracked foam? Perhaps every launch of the 114 has had some foam debris shedding, and we're only seeing small pieces and cracks now because the intensity of inspections has increased since the Columbia. Or maybe not.

-- Edward Tufte, July 3, 2006




From 9.00-9.30 pm Monday, William Gerstenmaier, NASA's head of Space Operations, gave an informative and smart news conference on the foam issue. He provided a summary of the evidence and answered questions from the space press. There's no problem as a result of the foam liberation incident. Weather permitting, the Discovery will launch tomorrow.

-- Edward Tufte, July 3, 2006



Response to PowerPoint Does Rocket Science (and the upcoming Discovery flight)
Did the news conference present the PP slides, or did they a different medium to convey the details?

-- Allan T. Grohe Jr. (email), July 5, 2006




They usually give a brief talk and then answer questions in a straightforward and intelligent manner, accompanied by occasional physical props, such as the broken-off piece of foam or model of the external tank. They did not use PP in the 8 to 10 press conferences I've viewed. You can see the press conferences and the launch by going to nasa tv at

http://www.nasa.gov/multimedia/nasatv/

Apparently the PP Flight Readiness Review for the foam (reviewed above) was something of a leak; the other FRRs at the meeting are not going to be made available. Keith Cowing, who runs NASA Watch, sent me an email saying that I might be interested in the foam FRR that he had posted at his website. You can see more on this at

http://www.nasawatch.com/

I think the press conferences are excellent, assisted by a well-informed space press. After the flight, the head of NASA Michael Griffin was asked at the press conference if he felt "vindicated" by his decision to launch. He said not at all, if anything, it was vindication for the scientific method--that is, looking at the evidence and the numbers at hand. What a wonderful thing for the Director of NASA to say. This contrasts to the PP cognitive style, which often seems to encourage presenters to pitch rather than present evidence.

-- Edward Tufte, July 5, 2006




The FRR-foam summary slide is now shown about 5 contributions up (in my review of that presentation).

And, also added, immediately above, a press conference photo of William Gerstenmaier showing the foam chip.

-- ET, July 6, 2006




Below, a link to a good account of the Discovery inspections by John Schwartz of the New York Times on the problem of distinguishing useful evidence from additional evidence, a problem that also occurs with newly developed exquisitely sensitive measurements (for example, PSA tests and the monitoring of contaminants of drinking water).

John Schwartz, New Scrutiny for Every Speck on the Shuttle, New York Times, 11 July 2006.

These issues can lead to quite subtle consequences, as my Yale colleague Alvan R. Feinstein suggested in many studies, including this one in the Archives of Internal Medicine: ". . . many breast cancers found by mammography screening have excellent prognosis not just because of early detection, but also because many of the cancers are relatively benign, requiring minimal therapy."

Sandra Y. Moody-Ayers, MD; Carolyn K. Wells, MPH; Alvan R. Feinstein, MD, MS, "Benign" Tumors and "Early Detection" in Mammography-Screened Patients of a Natural Cohort With Breast Cancer, Arch Intern Med. 2000;160:1109-1115.

(Thanks to Niels Olson for the NYT permalink above.)

-- Edward Tufte, July 10, 2006




Here's a clear technical report and press release, using a 4-page format (similar to A3 or 11" by 17", folded in half). If the report were printed as a 4-pager folded-in-half, then the June 2005/April 2006 images would fall somewhat closer together, which would facilitate comparison (although both images can be seen vertically adjacent simultaneously on, for example, a 30" monitor). From the Institute for Science and International Security (ISIS):

http://www.isis-online.org/publications/southasia/newkhushab.pdf

http://www.isis-online.org/

Here are links to the Guardian and the Washington Post accounts of the ISIS report:

http://www.guardian.co.uk/pakistan/Story/0,,1828058,00.html

http://www.washingtonpost.com/wp-dyn/content/article/2006/07/24/AR2006072400995.html

Several other ISIS reports are distinguished by their sourcing, detail, use of satellite photographs, and estimates of uncertainty. See, for example, Chinese Military Plutonium and Highly Enriched Uranium Inventories, by David Albright and Corey Hinderstein:

http://www.isis-online.org/global_stocks/end2003/chinese_military_inventories.pdf

-- Edward Tufte, July 24, 2006



Interesting presentations with full-screen dynamic graphics by Hans Rosling:

http://www.gapminder.org/

http://www.ted.com/tedtalks/tedtalksplayer.cfm?key=hans_rosling&flashEnabled=1

(These links were provided by Kindly Contributor Cesar Martin.)

-- Edward Tufte, July 25, 2006




From Nature, 13 July 2006, still more on PP, this from Martin Kemp, an Oxford art historian:

-- Edward Tufte, July 26, 2006




Here is a well-designed technical report:

http://evo.bio.psu.edu/printclock

It is about 7 pages long. Note the excellent illustrations, integration of text and images, documentation, careful citations, and different types of evidence. Note also the use of sentences and paragraphs and flowing text, not the grunts of hierarchical bullet points on slides.

This is a very high standard for a technical report, but why not start at that level?

-- Edward Tufte, August 1, 2006



In your workshops, you describe how to replace PowerPoint presentations with 11 x 17 sized reports, and provide many good arguments for why "engineering by PowerPoint" doesn't work very well.

This is a website that might interest you and your audience. It describes a process for creating A3 sized technical reports, and using them to make better decisions.

http://www.coe.montana.edu/IE/faculty/sobek/A3/index.htm

Dr. Durward Sobek of Montana State University spent six months in Japan as a grad student, interviewing and observing Toyota engineers to uncover the reasons why Toyota was able to develop cars much more quickly than other auto makers and also maintain high standards for reliability.

Toyota uses these A3 reports extensively in their engineering processes. They believe that the discipline required to accurately capture a problem on a single sheet forces the author to express the issue with both clarity and conciseness. They emphasize using visual models to express ideas rather than a lot of text, and value the ability to have all of the pertinent information within a single field of vision. The engineers are also required to bring their supporting documentation, so that the team can dive into the details when necessary.

Since then, Dr. Sobek has taught many engineers how to use A3 reports to make better technical decisions. I can tell you from my personal experience with this technique that it is amazingly powerful. By using one of these reports, we solved a technical problem within a single meeting that we had literally wrestled with for years through engineering by PowerPoint. By forcing us to make our knowledge about the problem visible in a systematic way, the tool helped us come to a deeper understanding that led to the solution.

Best regards,

Katherine Radeka

-- Katherine Radeka (email), August 2, 2006




The "A3 Process" described above begins with a good idea and then dilutes it into a Business Methodology Fad. BMFs are characterized by a germ of a good idea, but also by over-reaching, over-simplifying, excessive focus on a single idea, pitchy and enthusiastic over-simplified examples, and pretentious names ("The Toyota Method," "The Long Tail,", "The Genghis Khan Guide to Mastering the Universe," "The Takahari Guide to Infinite Profits," and so on).

In the Beautiful Evidence chapter on corrupt techniques in evidence presentations, the section on over-reaching concludes with this: "When a precise, narrowly focused technical idea becomes metaphor and sprawls globally, its credibility must be earned afresh locally by means of specific evidence demonstrating the relevance and explanatory power of the idea in its new application." (p. 151)

The A3 method, which at its heart is a good idea, requires some down-in-the-trenches detailed and complex examples. And it should avoid bullet lists in describing the method.

-- Edward Tufte, August 11, 2006



.
A remarkable account of "Death by PowerPoint," as the phrase takes on new meaning:

http://armsandinfluence.typepad.com/armsandinfluence/2006/08/death_by_powerp.html

-- Edward Tufte, August 11, 2006




From Thomas Ricks' book Fiasco:

"[Army Lt. General David] McKiernan had another, smaller but nagging issue: He couldn't get Franks to issue clear orders that stated explicitly what he wanted done, how he wanted to do it, and why. Rather, Franks passed along PowerPoint briefing slides that he had shown to Rumsfeld: "It's quite frustrating the way this works, but the way we do things nowadays is combatant commanders brief their products in PowerPoint up in Washington to OSD and Secretary of Defense...In lieu of an order, or a frag [fragmentary order], or plan, you get a bunch of PowerPoint slides...[T]hat is frustrating, because nobody wants to plan against PowerPoint slides." That reliance on slides rather than formal written orders seemed to some military professionals to capture the essence of Rumsfeld's amateurish approach to war planning. "Here may be the clearest manifestation of OSD's contempt for the accumulated wisdom of the military profession and of the assumption among forward thinkers that technology-- above all information technology--has rendered obsolete the conventions traditionall governing the preparation and conduct of war," commented retired Army Col. Andrew Bacevich, a former commander of an armored cavalry regiment. "To imagine that PowerPoint slides can substitute for such means is really the height of recklessness." It was like telling an automobile mechanic to use a manufacturer's glossy sales brochure to figure out how to repair an engine."

This raises some of the same issues discussed in the report by members of the NASA Return to Flight Task Force in the contribution at the top of this thread.

-- Edward Tufte, August 11, 2006



I have just found and bookmarked this brilliant set of material - good work and great dialog by all.

I've had my share of problems with PP and it's predicessors. Managing marketing groups in Silicon Valley, I always tried to get product managers to keep it simple. My mantra was "no more than three bullets" and "if it takes more than three, you need another slide". This approach was a recognition that presentation software was going to be used no matter what. I would say that my success rate was less than 10%. The prevailing culture often obsessed with creating slide templates that allowed as many bullets on a page as possible.

One earlier post illustrates a similar "can't see the wood for the trees" communications problem. Clyde Smithson does a creditable job of describing the efficiency of coding systems (Oct 2005) but misses that larger point. He takes a one-dimensional approach to value in communications - bandwidth is expensive - and advocates everything possible to reduce the useage of this expensive resource. This kind of thinking comes from the early days of communciations and microwave engineering. In a society dominated by Moore's Law, how valuable is that saved bandwidth compared to using a more verbose coding system like UTF that alows us to communicate in most languages on the planet?

Clearly, the message is about communication, less about cost. This kind of thinking has us guiding 100 million dollar aircraft around the sky using restricted-voice bandwidth technology that dates back to early phone systems and carbon radio microphones. Anyone who has used a full-spectrum voice system like Skype knows that fewer mistakes and mis-hears occur when all of the audio information is present. Should the lives of hundreds of passengers be placed at higher risk because the radio designer and regulatory agency saved 20 dollars on a radio costing a few hundred?

More power to your biting analysis and critical assessment of communication.

-- Brian MacLeod (email), August 20, 2006


[an error occurred while processing this directive]