Analysis from the Edge: Information Paralysis and Decision Making in Complexity

Updated: Feb 9, 2020


Analysis From the Edge

Throughout the last 13 years of military conflict involving US forces, commanders, planners, and civilian decision makers at all levels have demonstrated an unquenchable thirst for information. Individuals and units have had to respond to multi-layered requirements for the collection of information, and dozens of new information processing tools and systems have been deployed to capture the resulting data. The small Civil Affairs community alone uses at least seven different, often competing and non-interoperable, systems for civil reconnaissance and information management. The much larger intelligence community, tasked with integrating and analyzing vast amounts of data, has hundreds of such systems. Most of these systems utilize an event-based, database-enabled ontology in which the human world operates like an engineered machine in accordance with Newtonian physics and modernist philosophy. As a result, such tools, and the people who depend on them, are ill-equipped to grapple with the complex conflicts of the twenty-first century.

Despite an ever-increasing volume of information, we are arguably less capable than ever of dealing with a world composed of complex adaptive systems characterized by their unbounded nature, diversity, and non-linearity. Indeed, small SOF elements, like their conventional counterparts, are currently bound by processes and models that, in their attempts to provide clarity to the environment, lead only to further confusion. Thirteen years of counterinsurgency operations in Afghanistan have demonstrated the peril inherent in this information overload, as well as the potential of small special operations elements to enable the military to move beyond our failed approaches and reach understanding amid complexity.[1] In doing so, commanders and staff, as consumers of information and intelligence, can be freed from reductionist ways of thinking and focus instead on developing the situational awareness that that will help them adapt strategies and achieve operational objectives.

Network-centric Warfare

In recent decades, the explosion of information technology’s capabilities, and its increasing availability and application at the tactical level, have promised a revolution in command and control for every echelon of the military. Recent attempts at doctrinal innovation have largely centered on the core problems of the complexity of modern operational environments, the potential and peril of an “informatized” battlefield, and the ill-structured nature of the problems our armed forces are being asked to address on a global scale. The concept of effects-based operations (EBO), in its various forms, is among the most famous of these attempts. EBO, however, became entangled with reductionist thinking and promised—through system-of-systems analysis and operational net assessment—much more than it could possibly deliver. A revolutionary capability in theory, EBO was supposed to enable information dominance and swift decision making to overwhelm and paralyze an adversary, resulting in his rapid, decisive defeat.[2] Instead, its users are the ones who have found themselves paralyzed by an onslaught of mostly disordered information.

Though the concept of “fog and friction” in warfare has been widely known and understood among military professionals since time immemorial, its implications are best expressed by strategist Carl von Clausewitz in his nineteenth century treatise On War.[3] As he puts it, “Everything in war is simple, but the simplest thing is difficult.”[4] Chance events and the sheer difficulty of an endeavor so intrinsic to human nature coalesce into a fog of unpredictability that has covered every battlefield in human history. For Clausewitz, it was the commander and his experience that served as the “oil capable of diminishing this friction.”[5] In the twenty-first century though, military theorists have sought to burn through the fog of war and eliminate the friction by means of information superiority and networked military organizations.

As early as 1970, before the internet and mobile technology became mainstream, futurist and sociologist Alvin Toffler described what he saw as the coming age of information overload and its dangerous implications.[6] The increasing pace of change, including rapid advancements in communication technology and faster means of transportation, were, according to Toffler’s book Future Shock, producing “decision stress” and “sensory overstimulation.”[7] This perceived loss of control was resulting in cognitive withdrawal, decision paralysis, and the inability to properly use information.[8] Future Shock and several other works in which Toffler and his wife Heidi Adelaide Toffler outlined their ideas about the future of society proved highly influential in the development of new thinking as military strategists attempted to address the effects of overabundant information on the battlefield. Other thinkers, both inside and outside the military, have looked for ways to channel the rapid pace of change and the increasing flow of information into military success.

Network-centric warfare, as this new form of information-enabled warfare came to be known, has many parents and nearly as many interpretations. The ideas and writings of the late Vice Admiral Arthur K. Cebrowski are particularly influential. Cebrowski, inspired by Air Force Colonel John Boyd’s theories of decision making in aerial combat, postulated that it was not information superiority alone that translated into victory, but the ability to transform that information into “action and behavior options” faster and more accurately than an opponent.[9] Unlike others in the network-centric warfare community who believed that technology alone would guarantee information superiority, Cebrowski understood that effectiveness was less about technology and more about information management: shared perspectives and perceptions would enable the military to better utilize the information at hand.[10] In other words, “Information superiority required more than simply knowing a lot. It required knowing more of the right things, accurately and in time to act.”[11]

In the decades since Cebrowski identified the problem, modern militaries nevertheless have only moved further away from a solution. We are bombarded with more and more data but develop less and less understanding. Something is missing—something that ties the information together and renders it meaningful and legible for those who need to use it.

Information Paralysis or Analytical Malfeasance?

Few fields within the military manage a greater volume of raw data and information than signals intelligence. The supply of metadata, databases, tools, analytical methods, and systems available to analysts seeking every “piece of the puzzle” is virtually endless. According to a 2011 New York Times article, 1,600% more data has become available to all levels of the military since 11 September 2001.[12] With the increases in signals intelligence technology and adaptation to modern information communication technologies, the data available to intelligence analysts increased by orders of magnitude compared with the general military forces. The outcome of this explosion in the volume and accessibility of information, however, is a flawed sense of understanding as nearly every echelon conducts its own analysis of the same databases through the filter of its own particular slant or interest.

Chechen Ghosts

On my first tour in Afghanistan from May 2007 to August 2008, an issue of particular interest at the division (Regional Command-East [RC-East]) and theater (International Security Assistance Force [ISAF]) levels was “foreign fighter” support to the Taliban and al-Qaeda. The priority intelligence requirements of the RC-East commander, along with nearly every RC-East Joint Intelligence Support Element (JISE) analytical product, focused to some extent on the foreign fighter issue. On one occasion, the JISE used information from a particular database to produce a map of supposed “Chechen” presence along the Afghan-Pakistan border in Kunar, Nuristan, and Nangarhar provinces. While there is no public evidence of post-Taliban era Chechen fighters in Afghanistan, rumors of their existence consistently find their way into both intelligence and open source reporting.[13] The reputation of Chechens as fearless fighters and capable leaders, combined with both the presence of lighter skinned, Caucasian-looking individuals on the battlefield and circular reporting by inexperienced human intelligence collectors, have contributed to such rumors, provoking an unwarranted special interest in Chechens among US military commanders and staffs.

This particular map, which indicated extensive “Chechen” presence across eastern Afghanistan, was briefed at a nightly commander’s update. The commanding general immediately requested additional information from the brigade commander responsible for this area, and the brigade commander, caught off guard, turned to his intelligence officer demanding to know how such an extensive Chechen presence could have been missed by his own intelligence staff. Upon hearing of this incident from the Brigade S-2, I attempted to corroborate the map and analysis produced by the JISE. Using the same tools and database, I queried all references to “Chechens” in the same geographical region and produced an identical map. A deeper look at the underlying data and geographic information system software the analyst used to produce the original map, however, revealed that any connection to Chechens was completely specious. Another source was an internal Afghan National Army report which said that American advisors had discussed the possible presence of Chechens. In most instances, the record behind the data point explicitly excluded the possibility of a Chechen presence.

In the end, not a single report associated with the map could be interpreted as evidence that Chechens were in eastern Afghanistan. But even if some of the data points had proved relevant, there remains the absurdity of accepting, without question, the existence of an extensive network of Chechens operating as a highly insular insurgent group on the Afghan-Pakistani frontier and then sharing it as intelligence in a routine commander’s update.

On another occasion another analytical section nearly persuaded a commander to conduct a direct action operation on a public call office, shared telephone held in a storefront common in Afghanistan and other developing countries, interpreting the activity of its hundreds of users to indicate that it was a key actor in multiple insurgent networks. On the surface, these appear to be simply stories of lazy analysts who could not be bothered to follow even the most basic procedures of intelligence analysis, or of commanders seeking to micromanage their units. These are, however, just a few examples among many similar ones that arose throughout my many months of service in Afghanistan as an intelligence officer and my additional tours as a civil affairs officer.

The same problems manifested in the at the division, ISAF Joint Command and ISAF headquarters levels as well as in the various special operations intelligence headquarters. Clearly, the problem was more than just an ineffective and poorly trained and led intelligence team. In fact, as a intelligence officer, I estimate that I spent well over half of my time and analytical energy preventing others from making operational mistakes based on flawed understanding gained in an environment of information paralysis. Intelligence sections at the division, CJSOTF, and higher headquarters levels simply did what they knew best: pulling records from databases, placing dots on maps, and claiming insight. More information, paradoxically, meant shallower analysis: they had all the information in the world, but no context in which to apply it.[14]

Edge Organizations and Constrained Organizations

These intelligence analysis sections failed largely because they were situated in highly modernist organizations that are poorly suited to the task of effectively using all that information to which they have access. Modernism, in this sense, is a philosophy that maintains that all problems can be solved (progressivism); that all causal relationships are knowable (logical positivism); that variables can be separated (reductionism); and, finally, that data can provide evidence of truth (empiricism).[15] Linearity, which assumes the proportionality, ad