Are All Complex Systems Always Unpredictable?

Recently, I have been reading many popular, social science books including books by economist Daniel Kahneman, philosopher Nassim Taleb, sociologist Malcom Gladwell and media writer Derek Thomson.

In their books all these authors discuss complex systems, chaos theory, graph theory and network theory. They all usually arrive at the broad conclusion that all complex systems are unpredictable.

We humans tend to overgeneralize, which is one common way of arriving at wrong conclusions. These writers use some specific observations and reasoning to make a general statement, that all complex systems are unpredictable.

Many complex systems are unpredictable most of the time, but some complex systems can be predictable, given the right expertise and information.

First I want to state that all these writers have interesting observations and thoughts, and present valuable ideas that one can learn from.

My Dad once told me, that if you really consider someone intelligent and worth learning from, you have to find something in that person’s views, that you disagree with, otherwise you are just blindly following the person’s opinion and not analyzing enough.

In the context of complex systems the above mentioned writers all overgeneralize to arrive at a wrong conclusion. But they arrive at their wrong conclusions in different ways of overgeneralization.

• Kahneman, a statistically minded economist, in his book “Thinking, Fast and Slow” discusses that intuition can work in simple settings with limited outcomes, but not in complex systems such as the financial markets. Most of his reasoning is very statistics focused with some shades of psychology. As evidence, he presents the lack of statistical significance in the performance of professional money managers as a whole for outperforming the index.

However, Kahneman assumes that since there is no statistical significance in the performance of a group of investment managers in public markets, that no specific investment manager is able to do this. This is an over-generalization that transfers a conclusion about the total sample to each member of the sample.

• Taleb, a former options trader, in his book “Antifragile” states that complex systems cannot be predicted. He considers complex systems driven by volatility and guided by responses to stressors. And he discusses concepts like Black Swans. Taleb dislikes the simplified mathematical and model based approach to finance and economics in the currently accepted theory. I also dislike and disagree with many parts of the current financial and economic theories.

However, Taleb assumes that because the entire socio-economic system is really complex and highly unpredictable, that other complex systems/subsystems including the economic and financial system have to be unpredictable as well. This is an overgeneralization that transfers the behavior of an entire system to its subsystems.

• Gladwell is a sociologist and in his book “Outliers” discusses the role of chance in drastic outcomes of highly successful people. He argues that people, who we consider outliers, got a lucky break at some point and their exceptional abilities were random and not predetermined. Luck is an important component in life and clearly a shaper of everyone’s future.

However, Gladwell assumes that since luck played such a big role in the destinies of some outliers he analyzed, therefore most people with exceptional impact have succeeded primarily because of luck or unpredictable external circumstances (this is bad logic). He concludes that the complexity of the system does not allow for a predictability of an outlier outcome. This is an overgeneralization, that transfers the conclusion from individual instances of outcomes to all instances.

• Thomson, a media writer, in his book “Hit Makers” uses his knowledge and background to illustrate the evolution of popular phenomena in film, music, graphic art and other forms of media. He traces the random instances that made an obscure media phenomenon become a global media phenomenon. He later discusses some ideas about predictability of hits and chaos theory and extends it to the spread of all new phenomena.

However, Thomson assumes that because popular media phenomena are driven by large degrees of randomness, that all disruptive social phenomena are unpredictable. This is an overgeneralization that transfers a certain dynamic in a specific field to phenomena in other fields.

An interesting side-observation to many social science books is that most of them quote the same limited number of social science experiments. They seemingly base their conclusions on the same limited knowledge base.

In a sense most knowledge itself is created by generalization. Many social scientists create knowledge by observing a limited number of semi-experimental outcomes and making a conclusion about a larger set. Therefore this knowledge can often lead to overgeneralization.

One contributor to overgeneralization is that most people only have experience in limited number of settings and fields. Their perspective is heavily shaped by this exposure. Information from a limited number of perspectives makes a person more likely to overgeneralize patterns from one field to fields where one does not have deep experience or expertise.

We can call this a perspective bias.

The future is really difficult to predict but not all parts of it are unpredictable. Granted this predictability is very difficult at an early stage of a phenomenon, but for example in a disruptive phenomenon there is usually a point where the phenomenon becomes much more predictable.

However, this tipping point does not occur when the phenomenon’s external expression tips. In reality it occurs much earlier. It occurs at a point when enough related phenomena have created a critical density of proportional, positive feedback to the phenomenon and therefore constitute a strong, self-reenforcing, positive feedback system.

At this point the system is fairly predictable. I specifically said fairly because unpredictable single occurrences, like a car accident that kills the leader of a movement, a natural catastrophe that devastates an entire city, or a virus outbreak, can derail the path of any phenomenon. Therefore diversification in predictions is always useful.

But in absence of such drastic, random, single influences many systems can become predictable much earlier than visible by their outward behavior.

The key to predicting the future of a complex system is identifying the existence and creation of these proportional, positive feedback loops early enough. One needs to look at a phenomenon as a much bigger system that incorporates everything else that interacts with this phenomenon. And if one is able to identify these proportional, positive feedback loops early, one can predict the future with significant certainty.

The skill of capturing these feedback loops in a specific field is usually honed by prolonged, in-depth exposure and analysis of similar phenomena in that field. After a while one recognizes common modes of positive feedback in a field and hence becomes better at capturing them early.

A somewhat related thought is the power law distribution that we often hear about in case of disruption. The reason disruption tends to have a power law distribution is because of this significant positive feedback loop aspect.

A proportional, positive feedback reinforces and strengthens a phenomenon. In the next step this feedback grows proportionally, since the phenomenon itself grew. Therefore the feedback reinforces the phenomenon even more and creates a self-reinforcing cycle. One can also look at positive feedback in terms of derivative calculus.

Mathematically the power law distribution is simple to explain, as it is the mathematical nature of the exponential function. The exponential function’s derivative is proportional to the function itself. This means that the function’s change is proportional to the current state. Hence as the current state increases by the change, it increases the change of the next state and this cycle starts to reinforce itself.

This type of behavior is very common in nature, as the exponential function is a solution to many partial differential equations that govern the physical world. But in real life there is also a damping function that increases non-linearly with the phenomenon and therefore leads to a finite distribution and not an unstable blowout.

In social systems this damping can have many shapes. Here are some simplified examples:

• For compound interest one mode of damping is a progressive tax.
• For Silicon Valley’s network effect some modes of damping are the high rent prices, traffic and talent shortage.
• For a high growth disruptive startup the damping is the inefficiency of scale and bureaucracy of structures.

In the real world there is no pure exponential behavior, because a pure exponential function by its nature would grow infinitely, consume everything, become unstable and not exist anymore. Every existing natural phenomenon that is not in current disruption, must be in a negative feedback mode within a range around its current equilibrium.

But there are many phenomena that exhibit phases of exponential behavior because of positive feedback loops. For disruption to occur the feedback has to become larger and grow faster than the damping of the system. If one is able to identify this early enough, one can predict some aspects of the future. In a social setting this is difficult and requires expertise and experience, but it is not impossible as many of the above mentioned social science authors claim.

When it comes to the predictability of complex systems, disruption is probably one of the easiest components to capture. Non-disruptive phenomena are usually much harder to capture and track early on.

Disclaimer:

Since I am a dyslexic, I am prone to spelling and grammar mistakes. Hopefully it does not distract from the substance of the article.

Thank you for reading this article :)