Enterprise Data Warehouse Implementation: A Cybernetic-Boydian Approach

Technical projects are often treated as linear or “waterfall” processes, with clean sequential steps that are easily tracked and managed. While this linear, waterfall approach has its place in establishing high-level milestones for a project, it is inevitably frustrated by the non-linear, more complex nature of real-world projects.

From the perspective of a linear project plan, any deviation from the expected sequence due to unforeseen changes or customer feedback is seen as an unwelcome interruption or failure. Does this have to be the case? Is there any project management paradigm that can more naturally handle the inevitable turbulence that arises in all projects?

Cybernetics is a collection of ideas developed around “regulatory and purposive systems” that are driven by feedback mechanisms. The term is derived from the Greek word kubernētēs or “steersman”, comparing the “steering” or “regulatory and purposive” aspect of these systems to the actions of a ship’s steersman, who monitors the ship’s movements and makes corrections to maintain the proper course. The word kubernētēs is also the root of the more familiar English word “governor” which captures the same sense of purposive control. Accordingly, cybernetic control mechanisms are often referred to as “governing” mechanisms as well.

The canonical example of a simple cybernetic system is the thermostat. A thermostat combines a sensor (which detects the current temperature) and a governing or regulatory mechanism (which turns the air conditioning or heat on or off) in order to drive the system as a whole (the thermostat and room) toward a desired or purposive state — namely the temperature that has been programmed into the thermostat. More complex cybernetic systems are found everywhere both in nature and the human-made world, from ants to stock markets. All these systems exhibit behaviors which are fed back into the system as inputs, producing further behaviors according to often-complex sets of governing rules.

Around the same time that cybernetics was emerging as a field of study and a way of thinking about complex systems, Colonel John R. Boyd of the United States Air Force was developing his own set of theories, which would eventually become among the most prominent military ideas of the 20th century.

Boyd was a fighter pilot, and the most important concept of his strategic theory is an abstraction of his experience in reacting to the fast-paced events of air-to-air combat. Boyd’s key concept is called the “decision cycle”, or the “OODA loop”, with OODA standing for Observe, Orient, Decide, Act. Observation is the direct collection of data via the senses. Orientation is the analysis and synthesis of this data to form a coherent mental perspective. Decision is the process of choosing an optimal course of action, given the current perspective. And finally, Action is the physical playing-out of the decisions that were made. Feedback also occurs at each step of this process, since observation is a continuous process that goes on throughout the entire decision cycle.

The key insight of Boyd’s theory is that an entity (be it a single person, a business, a military organization, etc) will be successful in attaining its goals to the extent that it is able to “tighten the loop” or execute the repeating decision cycle as quickly and efficiently as possible.

Executing this loop quickly for a single fighter pilot is largely a matter of reflexes and experience, but for larger teams and organizations it is a much more complex task. Boyd developed a comprehensive framework for assessing nested OODA loops at various levels, from the individual pilot or soldier all the way up to the strategic and operational level of generals and nations.

While Boyd’s theory was developed in the context of adversarial or competitive dynamics like battle or business competition, it can be extended simply enough to simple productive activities like the creation of an enterprise data warehouse (EDW), with all of its essential insights intact.

If one combines a cybernetic analysis and Boyd’s ideas for optimizing the decision cycle, the resulting synthesis is a cybernetic-Boydian approach. This approach provides a flexible framework to drive fast, responsive and efficient planning and decision-making. In this paper, we will develop the ideas of this cybernetic-Boydian approach further, as applied in the context of an EDW implementation project.

Before an EDW project even begins, the stage must be set by establishing clear lines of communication, areas of responsibility, and expectations for all team members involved. One central principle of Boydian strategy is pushing “power to the edge”, in the sense of decentralizing decision-making and empowering all team members to take on a proactive stance. Micro-management leads to stagnation, and the goal of an engineering team is to operate efficiently and react quickly to problems as they arise.

Furthermore, the most valuable knowledge about the nuts and bolts of the system is not concentrated at the top of the org chart— it is held disproportionately by the engineers who are engaging most closely with the data. Engineers should be empowered to provide suggestions and feedback to project architects and managers based on what they see on the ground, and to make decisions where appropriate. This process of decentralized two-way communication exemplifies the cybernetic principle of governance via feedback.

At the highest level of abstraction, an EDW implementation project is a single iteration of the decision cycle (or OODA loop) of a business, and accordingly it begins with observation. This initial project-level observation requires gathering detailed requirements on the current state of all infrastructure, data sources, data structures, reports, KPIs and business rules. The purpose of a data warehouse is to combine data from many sources and present a unified and clear picture of the operational status of the business. This can only be accomplished if the EDW team invests the necessary effort to understand all the complexities of the business logic and data. From a cybernetic perspective, the team’s output will only be as good as the quality of the feedback provided, and the only way to ensure high-resolution feedback is to develop fine-grained understanding of the customer’s business processes and data structures.

Another equally important part of the observation step is developing a detailed understanding of the required end state and deliverables for the EDW. The team must establish the content and form of desired reports, the desired state of the EDW infrastructure, frequency of data refreshes, and all other relevant details of the desired end product. Imagine a thermostat that is unsure what temperature it was programmed to target — an EDW project without a clear definition of deliverables will be just as broken.

Once the initial high-level observation has taken place, the next step is orientation. Orientation is the analysis and synthesis of the data collected during the observation stage. The team must dive as deep as time allows into the business logic, the relationships within the data and the steps required to reach the desired end state. As the requirements and prerequisites are set out in more detail, this will tease out potential issues and required changes that serve as further feedback to refine the proposed solution.

The real world rarely allows for a clean linear development from initial requirements to deliverables, and the most effective way to minimize the impact of unexpected changes is by discovering them as early as possible, which can only happen through in-depth exploration and discussion of the system in both its current and future states.     

The next step in the process (“decide”) involves taking in the data and analysis of the first two steps of the OODA loop and finding a best course of action. These decisions include which EDW architecture to use, what kinds of dimension or fact tables to use, how to configure data pipelines, what reporting solution to use, and many more critically important choices. Each decision must incorporate all the available information from all team members, and contrary to what the colloquial sense of the term “governor” might imply, many of these decisions can be made cooperatively by the whole team — not necessarily by a single person.

The “decide” step is traditionally seen as the most important in project management and leadership, but from a purely cybernetic perspective it ought to be a natural consequence of the determinative steps that preceded it. If the desired state is well-defined and the “sensors” of the cybernetic system are providing high-resolution information to those making the decisions, the best choice will often be clear.

The final step of the OODA loop — action — is important because it represents most of the actual day-to-day work of building an EDW. Planning and decision-making is important to a project, but the daily work of turning plans into reality is where those plans either succeed or fail. Effective action requires a team that is not only competent but also motivated and clear in purpose. Each member of the team ought to have a clear understanding of the end goals of the project and know how their day-to-day work is bringing those goals closer to realization.

Concurrently with this high-level decision cycle, there are decision cycles occurring at every level of abstraction within the project, from cycles spanning weeks or days centered around designing and implementing each report, server, database, or table; all the way down to the level of each individual as they observe, orient, decide and act countless times each hour. The cybernetic-Boydian approach seeks to streamline and accelerate all of these nested, concurrent cycles by encouraging clear and frequent communication and high-resolution understanding at all levels of the project.

As a cybernetic system, the EDW project does not terminate cleanly after completing the sequence of steps. Each action produces a new set of circumstances to which the team must observe, orient, decide and react once again. The success or failure of each minute task in the project feeds back into the system as a whole, affecting the team’s morale, the necessity for other tasks, and sometimes even the desired end state of the project.

Even the process of observing, orienting and deciding can generate new feedback that shifts priorities or reveals previously-unknown challenges. Ultimately, the team must be resilient and responsive in reacting to these emergent surprises. The point of the cybernetic-Boydian approach is not to foresee every possibility, or to attempt to exercise perfect control over every step of the project, but to create an environment of adaptive and communicative engagement that is capable of efficiently handling the unforeseen.