This is a reading of an official United States National Security Agency document that I consider required reading for laying down groundwork to build a deeper understanding on. I intend to upload several more readings of similarly important documents and welcome any feedback.
A .pdf copy of this document can be downloaded from the excellent online leaks & archive site - cryptome.org. They also have a substack.
Please note that the italics in the below text is my own emphasis on how I pronounced particular words or phrases. This is the extent of my alteration or creative input apart from any mistakes or grammatical errors.
The Delphi Technique – Unclassified document approved for release on December 2011 by the United States’ National Security Agency as part of Transparency Case #63852
The Delphi Technique
Managers today must make decisions in the face of considerable uncertainly. Not only are data missing or incomplete, in many circumstances there are no well-defined procedures for combining the existing data objectively to produce a unique solution. To help managers in such circumstances, attempts have been made to develop methods that, while not entirely objective, do minimize the subjective element in the decision-making process. The Delphi technique is one such method.
In situations such as planning of forecasting where there is a particularly significant amount of uncertainty, informed judgment often seems more important or more easily obtainable than quantitative data. A common pracitce, therefore, is to seek such judgments from experts who are presumed to have knowledge of whatever data is available, thereby, theoretically, combining both objective and subjective bases for decisions. A group of experts rather a single source is usually consulted whenever possible – the principle that “two heads are better than one.” The Delphi technique was developed in an attempt to make the most effective use of informed intutitve judgment by creating conditions under which a group of experts can perform most ably and their answers can be combined into a single group opinion.
Technique
The term “Delphi Technique” implies more than simple “consensus of opinion.” Three features distinguish the Delphi technique from a simple round-table or panel technique:
Anonymous response.
Iteration and controlled feedback, and
Statistical derivation of a group response.
During a Delphi procedure, participants normally do not meet face to face but are questioned individually and never told which other participants gave which responses. The anonymous response, which is basic to Delphi, is intended to avoid a number of the problems inherent in face-to-face discussion:
A group may be persuaded by individuals who dominate by virtue of rank, reputation, or (sheer) personality.
A member of a group may fear “losing face” by abandoning a publicly expressed opinion.
The “bandwagon” effect often leads to adoption of solutions that are less than optimal.
Group discussions generate large amounts of “noise,” irrelevant discussion and sidetracks that contribute nothing to the central issue.
It is difficult to arrange for a number of experts to get together at one place and time.
Experiments at RAND, mostly on forecasting future trends in specific technical areas and on answering factual technical questions, indicate that anonymous responses do produce better results than face-to-face group discussion.
There is no doubt, however, that intellectual interaction can be valuable in forming judgment, and it is to provide this interaction in the absence of personal contact that the second feature of the Delphi technique is included. Responses to the first set of questions are tabulated and sent to the participants, who are aksed to respond again, using the information provided by the other responses. This feedback-and-response cycle may continue through several iterations. Sometimes respondents are asked to give reasons or are allowed to suggest further questions for data. Thus, new ideas can be disseminated for consideration by the whole group, and also respondents can get a feel for where the majority opinion lies. There is a danger, however, that the persons tabulating responses and co-ordinating the Delphi exercise can consciously or accidentally distort the feedback and bias the results.
The third feature of Delphi is production of a single group response rather than simply a tabulation of individual responses. The individual responses are aggregated in a manner that assures that every opinion contributes to the final group opinion. Exactly how to combine the opinions in itself may be a difficult problem, especially when they cannot be expressed numerically.
One modification to the basic Delphi technique is a ranking of experts and weighing of their opinions. Unfortunately, ranking the experts may be as difficult as making the original decision, and attempts to use self-ratings by each expert may measure ego as much as expertise.
Critique
In the absence of facts, certainly it is normally better to trust the judgment of experts than non-experts. Both statistical theory and experiment have shown that one is more likely to be close to the correct or best answer by sampling many responses than by depending on just one. However, the choice of experts is crucial, in both the area of choosing general classes and the area of naming specific names. The most famous uses of Delphi have been in the area of technical forecasting. For example, financial experts might be asked to predict market conditisions five years in the future, or technological experts asked by what year they would expect a specific technological advance in their field to have been made. In such cases, the choice of experts is fairly obvious.
Not all fields, however, have the right type of experts to answer certain kinds of questions. The intelligence community, for example, presents two different types of stumbling blocks to the selection of experts capable of giving judgment on important issues. First, a basic hypothesis of the Delphi method is that the experts carry a technical data base in their heads to back up their judgments, while those at the appropriate levels often must depend on others for data. Second, the intelligence community is composed of specialists and frequently does not possess people capable of judging all facets of a complex problem. For example, in a survery designed to assess the value of Elint sources, a Navy expert might be able to reply to the value of a specific source of the Navy, but he cannot assess its value to the Air Force, and therefore he cannot assess its value to the community.
A second weakness of Delphi is its tendency to ignore facts. Delphi is intended for situations where the available information by itself is insufficient for decision-making, but it is still desirable to include whatever data are available in the decision process. Delphi does not guarantee that the judges employ any factual at all in forming their opinions, nor does it encourage them to seek new data or use any existing data base. An experiment at RAND showed that supplying relevant facts to the experts along with questions did, indeed, improve answers. The selection of the facts in itself, however, may introduce bias, and presenting facts with the questions may predispose busy experts against doing any research themselves. Use of data should be maximized in a good decision making technique.
Another dange presented by Delphi’s comparative neglect of facts is the likelihood that widely held misconceptions will be continued. Scientific progress is based on the fact that a hard look at data can often reveal long-cherised ideas to be wrong, but if decisions depend solely on judgment, such recognition of error is unlikely. Many managerial decisions involve value judgments, which are particularly likely to be affected by prejudices, even among professionals, or by organizational considerations that make unbiased judgment extremely difficult.
Finally, major intelligence community problems involve a very complex system. Delphi tends to approach such problems in an oversimplified manner by isolating specifics from the system and look at them as individual and unrelated bits. The net result is that the opinions on these isolated pieces omit the serious interrelationships of these pieces. The technical way of stating this is that the systems approach is lost or overlooked.
In sum, it it erroneous to expect the Delphi technique to be applicable to all situations or to believe that the results are objectively derived. Developments of truly scientific aids to managerial decision making must still be pursued. If expert advice is the only available source of information, the Delphi technique does represent a way to elicit and combine the opinions. It must be remembered, however, that the individual responses are still subjective judgments and that statistically aggregating subjective judgments does not make the results objective.
In a field as important and complex as the production of intelligence, it is important to be aware of the validity of sources and to take care to use the best sources possible. The manager to whom the Delphi technique is recommended as a scientific management tool should realize its basically subjective nature.
Share this post