Understanding complexity

As I have suggested, it was the most-regulated in the financial system that were in fact the most disaster-prone: big banks on both sides of the Atlantic, not hedge funds. It is more than a little convenient for America’s political class to have the crisis blamed on deregulation and the resulting excesses of bankers. Not only does that neatly pass the buck it also creates a justification for more regulation. But the old Latin question is apposite here: quis custodiet ipsos custodes? Who regulates the regulators? – Niall Ferguson: The Great Degeneration

Thinking of complex adaptive systems as merely complicated entities that can be regulated like machines can lead to disaster, as Niall Ferguson shows in his recent book. He cites the USA’s Dodd-Frank Act which is aimed at promoting stability in the financial sector but “requires that regulators create 243 rules, conduct 67 studies and issue 22 periodic reports“. Simple principles, such as transparency, would work much better in the complex, and emotion-driven, world of finance. After all, money is a common human fiction that requires us to believe in it. Human systems are complex.

As organizations get larger, their original simplicity gets harder to maintain. Organizations reach their maximum cohesiveness above 150 people, based on anthropologist Robin Dunbar’s research. Beyond this size, knowing everybody in person becomes impossible. Intermediate layers of power and delegation begin to develop with more than 150 people and companies then enter the realm of complication.

Most of today’s larger organizations have a complicated structure. To enable growth and efficiencies, more processes are put in place, just like the financial regulators have done. This is what management schools have been teaching for over half a century. New layers of control and supervision appear, silos are created, and knowledge acquisition is formalized in an attempt to gain efficiency through specialization. To compensate for all of these rules, organizations put significant effort into compliance training. But this too is a myth, as some of the best trained people have been involved in disasters like the BP oil spill and the Enron collapse.

Today’s large, complicated organizations are now facing complex business environments that require agility in simultaneously learning and working. Typical strategies of optimizing existing business processes or cost reductions only marginally improve the organization’s effectiveness. Faster evolving markets challenge the organization’s ability to react to customer demand. Decision-making becomes paralyzed by process-based operations and chains of command and control; thereby decreasing agility. Training, as “the” solution to workplace learning needs, fails to deliver and then gets marginalized, often being the first department to have its budget cut.

Organizations, public and private, need to understand complexity, instead of simply increasing complication through rules, regulations, and control processes. This lack of understanding is the major barrier to success in the network era. As the image below by Yaneer Bar-Yam shows, a networked civilization requires 1) more laterally connected organizations, 2) fewer hierarchies, and 3) more diversity.

A schematic history of human civilization reflects a growing complexity of the collective behavior of human organizations. The internal structure of organizations changed from the large branching ratio hierarchies of ancient civilizations, through decreasing branching ratios of massive hierarchical bureaucracies, to hybrid systems where lateral connections appear to be more important than the hierarchy. As the importance of lateral interactions increases, the boundaries between subsystems become porous. The increasing collective complexity also is manifest in the increasing specialization and diversity of professions. Among the possible future organizational structures are fully networked systems where hierarchical structures are unimportant. – Y. Bar-Yam, Complexity rising: From human beings to human civilization, a complexity profile, EOLSS UNESCO 2002

Dave Pollard has a very clear post on how to address complexity from an organizational perspective. He also elaborates on 16 attributes of effective ways to address complex problems. It’s a list worth keeping handy.

As we come to understand complex predicaments better, we’re learning that the best approaches to them are very different from what works best for simple or complicated problems. Because all the variables cannot be known, and because cause-and-effect relationships cannot be established in complex situations, analytical approaches (like systems flowcharts) used in complicated problem-solving simply won’t work.

The best approaches in complex situations are, well, complex. They entail the use of many different techniques, some of which we are not very good at, and some of which are quite sophisticated, novel, or nuanced. – Dave Pollard, Complexity: It’s not that simple

Once we understand that we are dealing with complexity, and that many of our analytical approaches and control processes are not optimal, then we will be able to build structures for the network era. Over-engineering for complex social work environments is counterproductive. Here is an example from our past, that could work in our future.

In the Six Nations culture, power was distributed but the roles were clear. There were specific roles for each of the member tribes, namely Wolves (Pathfinders); Turtles (Problem Formulators); Bears (Problem Solvers). According to the book Systems Thinking: Managing Complexity and Chaos, solving problems and making governance decisions went like this:

Wolves – Set direction, and identified relevant issues
Turtles – Defined the problems
Bears – Generated alternatives and recommended solutions
Turtles – Checked on the potency of the recommended solutions
Wolves – Integrated the solutions, kept the records, communicated the decisions

What is interesting is that there were clear checks and balances to the dominant wolves, as only the turtles could define the problem, and it was up to the bears to recommend solutions. The wolves could only take action on those problems, with a finite set of solutions. It was simple, but it ensured 1) increased lateral connections, 2) limited hierarchies, and 3) increased diversity of ideas.