Data Driven Decision Making Is A Farce

In recent years, the explosion of data has been unprecedented; however, our decision-making, along with the managerial and social problems we face, has not improved even though tool and technology has improved—in fact, for many, it has worsened. The root cause? A significant educated and technocratic world clings to the misguided belief that data alone leads to better decisions. In reality, data is USELESS and even DETRIMENTAL without a deeper understanding of the systems to which the data belongs. Since all observations are theory-laden, the data you collect is a function of your understanding of those systems. If your understanding is flawed, one may collect wrong set of data or even misinterpret the right data. So in lack of understanding of system, even the most accurate data becomes worthless or harmful.

Understanding and Data works together and make up system in itself.
With a solid grasp of a system, you often need no data at all. When you truly understand a system, you already know what's possible and what's not. Just as the laws of science inform us about the what's possible in physical world, the laws of systems can clarify what is feasible within a any systems including techno-social systems, of which modern corporations are perfect example of. This understanding allows the human mind to eliminate certain possibilities and conceive new system designs, transforming the previously impossible into the possible. Data is useful in two situations:

1. It can help refute misunderstandings about systems, or
2. It can gauge the intensity of the forces driving a system, allowing us to predict how the system will behave if we do nothing and what might change if we intervene in specific ways.

Thus, understanding and data form an interdependent system, but understanding is supreme.

1. Two individuals with different understanding will seek different data. This should be obvious. If you agree , then data you seek is function of understanding.
2. Also two individual with different understanding will arrive at different conclusions, even when examining the same data. Even this should be obvious, but I just watched series on Alexander to this example comes to mind. Consider the historical war between Alexander and Darius: Alexander concluded that a red moon during a lunar eclipse was a signal to march forward, while Darius's team saw it as a sign of clear defeat and decided to retreat. Both leaders believed they knew the truth, yet both were wrong; neither had any understanding of how lunar eclipses occur or why the moon appeared red. Now Alexander's belief favored him by boosting the morale of his team, and Darius's belief brought the morale of team down, which was critical factor in the outcome of the battle. The point here is not to discuss historical fact or how Alexander was right or Darious was wrong.The point is interpretation of data, in this case redness of moon is function of your understanding of system, in this case the understanding of How moon becomes red and Why Moon becomes red. The outcome can't determine who was more knowledgeable. Instead of lunar eclipse, had this been a solar eclipse, the interpretation of data based on similar understanding could have been detrimental of Alexander and favorable to enemy. So to repeat, the point is that interpretation of data is function of your understanding of system,

If data you seek is function of you understanding and your interpretation of data is function of your understanding [two different concept], then what's supreme is understanding and not data. No way saying, data is not important, but understanding is waaaaay important that data. That seems like common sense, but our action does not align with that common sense understanding. There is clearly

1. little focus on methods of how do we know if our understanding is correct
2. way too much focus on collecting data driven by our understanding and interpretation of that data using our understanding.

So data gets better but understanding does not, where as we are supposed to in reverse route of improving understanding before improving data. If all our decisions are understanding driven and not data driven then people who claim that we shall make data driven decision are saying, there is no issue with our understanding, get more data. Now the statement that you have great understanding but less data THAN the loop that your understanding could be potentially flawed, lets discuss that before we seek what data to collect and improving quality of that data.

We seek data accuracy, when we should be seeking more understanding accuracy
Data collection can be erroneous, and it's relatively easy to identify faulty data. Improvements in data collection, data scrubbing, and statistical techniques—rooted in math and technology—allow us to eliminate poor-quality data or, at the very least, recognize it for correction. However, how do we correct our understanding of systems? This question is rarely asked because many assume their understanding must be correct simply because it resonates with them. This is absurd—everyone believes they are knowledgeable simply because their understanding seems logical to them.

Understanding saves us when data misguides us
Understanding involves explaining how a system works and why it functions in a particular way, while many management decisions most of the time merely seeks justifications. We will discuss different between Explanation vs Justification in a different posts. The challenge with data-driven decision-making is that as systems become more complex, data can justify virtually anything. Many people have read books about spurious correlations but often fail to grasp their core message, including the authors themselves. The true message should be that, in complex systems, data can prove any relationship; yet the correlations discussed in these books are called spurious not because data says so, but because have a fundamental understanding systems that those variables can't have any causal relationship. For instance, while data might suggest a correlation between increased ice cream sales and higher homicide rates, we know there is no causal relationship between the two, as the explanations do not fit. If our explanations were incorrect, we would be in serious trouble. In the case of ice cream and homicide relationship, we got lucky, but how do we ascertain the validity of our understanding when we are dealing in complex situation? We can't say, an explanation that makes sense to us is valid explanation. I am sure you can sense the graveness of wrongness in that statement. 'How do we Know what we know is knowledge' is the question that we have no publicly engaged and we need to engage in that. We cannot rely on the simple justification that "it makes sense to me," as this is a perfect way for data to mislead us. So how do we verify that our understanding is correct? This inquiry requires delving into philosophy and epistemology. That's why at SystemsWay often begins with philosophy—the love of knowledge—and epistemology plays a key role in determining how we know what we know is knowledge. Our discussions in SystemsWay are deeply rooted in Systems Philosophy. Unfortunately, this approach is often overlooked in business schools, and philosophy is generally regarded as a despised subject in society. Will discuss some other time.

Analysts don't better their understanding of systems with data
Analysts have 'a' understanding of a system. Analysis if expanded will be termed analytical reductionist thinking. Seeing systems as linear cause of effect mechanistic system taught to us in physics and engineering . No wonder a person can possess extensive financial data, including P&L and balance sheets, yet still struggle to grasp the business as a system, because thought data is correct, their understanding is not because corporation is not a mechanistic system. For the same reason, a person can possess extensive data about economy and yet struggle to grasp Economy As A System because economy is not a mechanistic system. Even for system as simple as Bug Management As A System, a person can possess extensive bug data yet, still have no understanding of Bug Management As A System. In many companies, I have observed individuals with detailed data on bugs—categorized by SLA, SLO, OOSLA, P0/1/2/3, and organized by team etc—still constantly struggle to keep management of bugs under control. Entire organization from CTO till individual developer at ground, get consumed with constant escalation. Employees become frustrated with the constant pressure from above to meet SLA requirements before operational meetings. In the context of bug management, nearly everyone ends up misrepresenting the situation. Developers might assign bugs to others at the last moment, mark them as "not reproducible" to remove them from the queue, or deprioritize them altogether or countless other techniques. Management then says, people are behaving bad, where as any human would behave that way in that system. If you criminalize creation and lack of capacity of individual to fix bugs when system de-capacitate individual from fixing bogs, you will get same social mis-behavior described above. People were not bad. The understanding with with people have designed bug management systems is bad. Because most people are mostly good most of the time provided systems in which they are embedded allows them to be, people mis-behaving in the system should point us to question the systems design and our understanding, but we go on a reverse route, blaming people for mis-behaving and hold them accountable with more and more data. WHY? Because our understanding is that systems are perfect, so it's must be people who are not doing what we pay them todo. That's a false understanding not all the time, but most of the time. The systems we are dealing with social or techno-social system, but leaders, analysts even developer who suffer deeply in these system see them as mechanical systems. More data is not going to help you better system if thinking which you are developing the understanding of the system is not correct one.

It's time that before you seek data, question the understanding with which you seek data. That's hard. How can one know that understanding that make sense to them is correct or not ? That has to be learnt. People say 21st century is century of complexity. I say 21st century will be century of understanding, rather re-understanding. The understanding that brought us so far, will alone not take us further far. It's time to question the thinking with which we develop our understanding. Interested, join the Systems Thinking workshop waitlist.