Introduction
Innovation implies change, and this change can often be disruptive to stable organizations, teams, partner networks and other business/social ecosystems. The rise of Generative AI with a Data Fabric built according to Lean and Agile principals is just such a disruption.
Just like in natural ecosystems, a culture of adaptability becomes more advantageous in volatile business/social ecosystems. Conversely, being highly adaptable can hinder the ability to take advantage of stability. As a result, those who have been benefiting from a culture that reinforces stability will ultimately lose out when confronted by more adaptable and agile counterparts better able to take advantage in advancements in data integration and AI unless leadership is able to make their culture readily embrace adaptability and a tolerance for risk.
Considering that teams, organizations, and entire economies in business can be described as business/social ecosystems, there are parallels between workings of biological ecosystems. Exploring these connections can deepen our understanding of how changes unfold in ecosystems in general. This understanding can be crucial in discerning how cultural shifts and advancements in technology impact the management and utilization of data for Artificial Intelligence (AI) applications. It can also offer valuable insights into how and when businesses integrate new data driven AI capabilities within their operational frameworks.
The inflection point for biological and business ecosystems
When stable ecosystems face disruptions either from external factors or the internal evolution of populations, they attempt to find a new balance. This transition, however, is not a gradual one. As the disruptive process unfolds, the ecosystem reaches a tipping point, leading to an exponential acceleration of change. Once this tipping point is reached, equilibrium is lost, and populations within the ecosystem rapidly decline due to the disruption as they are replaced by more nimble or resilient competitors.
Consider the example of coral reef ecosystems. As ocean temperatures continue to rise, expansive reef systems like the Great Barrier Reef are currently in the process of and have already undergone significant decline. I personally witnessed this change a couple of years ago when my family visited the Alligator Reef lighthouse in the Florida Keys. Having snorkeled there frequently in the mid-1990s, I could clearly see how the coral reef had transformed over the past twenty-five years.
The contrast was stark: the once abundant sea fans, bustling schools of fish, and thriving live coral had given way to a few remaining sponges, a sparse population of fish, and a seafloor scattered with bleached coral remains. Despite this disheartening reality, there is a glimmer of hope, as coral reefs are gradually starting to regenerate. Nevertheless, this regeneration process is expected to take millions of years without intervention.
Marine ecologists, such as Nichole Price of the Bigelow Laboratory for Ocean Sciences in Maine have documented the migration of coral species towards latitudes between 20 and 35 degrees north and south of the Equator, driven by warming ocean temperatures. Concurrently, other researchers, like those at the Lirman lab at the University of Miami, are cultivating corals with genetic traits that enhance their adaptability to higher water temperatures and pollutants.
In the coral reef ecosystem, it is the specific coral species and their populations that demonstrate higher adaptability—through migration and resistance to elevated water temperatures—that are surviving compared to the previously larger but less adaptable populations. This is because there is an energy cost associated with adaptability in stable biological ecosystems, which limits the success of more adaptable organisms during times of equilibrium and provides them an advantage only when the ecosystem encounters disruptive inflection points. The same is true of social ecosystems as well.
66 million years ago – After benefiting from 165 million years of stability, dinosaurs became extinct while adaptable mammals thrived.
This dynamic of adaptability and its cost exists across various levels, from teams and business units to entire industries and the human population as a whole. For instance, the value placed on traits and behaviors demonstrating adaptability becomes more apparent when faced with an existential threat to an entire species.
In social ecosystems, inflection points are driven by fear and greed
Inflection points within all ecosystems follow an exponential pattern. In biological ecosystems, these points typically indicate an exponential surge in the rate of change among one or more species' populations. However, in social ecosystems, inflection points manifest differently, marking shifts in behavior. Above all, behavioral shifts stem from either a significant event, such as the Attack on Pearl Harbor, or a culture of innovation, as seen during the early stages of companies like Apple, Facebook, and Netflix, which were born out of such events, like the inception of a new company or market entry.
Beneath these transformations lies fear —fear of external threats, the fear of missing out (FOMO), the fear of defying cultural norms and losing organizational support, the fear of lagging behind competitors due to insufficient innovation, the fear of being perceived by others as less than capable. And greed – the desire to dominate new markets as they emerge and existing markets as they are disrupted. Exponential growth in awareness in social ecosystems creates both fear and greed in the minds of participants and drives the behavioral changes indicative of an inflection point..
The inflection point in Generative AI is not just about the exponential rise in awareness, it’s about how that awareness creates the emotions. It is fear and greed that ultimately drive greater adoption and technological advancement. It encompasses the exponential growth of FOMO, the fear of job displacement due to AI, and the apprehension about losing personal and/or organizational competitiveness. Consequently, an increasing number of individuals are actively incorporating AI into their daily lives, initiating an exponential transformation in the human social ecosystem. This surge in AI usage attracts substantial funding for the development of new AI capabilities, leveraging AI as a competitive advantage that fuels additional revenue and profits.
While some may argue that excitement serves as an intrinsic catalyst for behavioral change, this notion primarily applies to a small percentage of individuals within most social ecosystems. Consider the Technology Diffusion curve, as popularized by innovation pioneer Geoffrey Moore, which emphasizes that only 2.5% of the total population typically leads such changes.
The situation is even more challenging in the realm of Information Technology (IT). IT leaders are typically selected for their ability to maintain the stability of systems, such as overseeing upgrade projects and IT business systems, rather than fostering disruptive innovation. Consequently, excitement alone cannot drive significant innovation and change within organizations, unless the majority of organizational members are carefully chosen from a pool of innovators, as often seen in startup environments. This is a key reason why large corporations, government entities, and other major organizations rarely spearhead disruptive changes. Instead, they tend to evolve gradually and incrementally over time unless catalyzed by a significant triggering event.
The rise of AI, Lean Data, and the Data Fabric
Just as external forces can disrupt ecosystems, evolutionary changes to behaviors of one or more participants can disrupt other participants as the ecosystem attempts to restore equilibrium. The rise of Artificial Intelligence is just such a change. While one might argue that AI is an evolution in humanity itself, that is not what I mean and is a question for another day. What I mean is that the use of AI by individuals represents an adaptive behavior that is in the process of disrupting all levels of social ecosystems as AI technologies evolve and adoption increases.
In biological systems, a brain, functioning as a natural form of intelligence, requires a nervous system to connect and process sensory information and carry commands to the body’s muscles and organs. Similarly, an AI instance requires a “digital” nervous system that pre-processes and delivers clean data to AI instances in a way that is secure and compliant with data privacy requirements. AI also needs a way to issue commands to digital and physical systems for automation and other use cases. And all of this must occur at an exponentially higher speed and scale than traditional approaches to data integration. The Data fabric (at least how I define it) is exactly that, a digital nervous system for AI.
But for Generative AI to reach its potential with sensitive enterprise data, data pipelines need to be exponentially quicker and cheaper to build and maintain. To make that possible, building a data fabric using the principles of Lean Data is critical.
Just as Lean Manufacturing principles help streamline manufacturing assembly lines, Lean Data (See my blog post on Lean Data here) is a set of principles and processes that accelerates the rate of change in building pipelines that deliver data to and from AI instances.
Generative AI’s inflection point
Since the beginning of 2023, I've had numerous discussions with technologists, CTOs, researchers, and other professionals concerning the potential disruption brought about by AI. Despite the customary excitement surrounding emerging technologies (anyone remember the Blockchain hype?), many remain doubtful about the recent buzz around AI, often pointing out that AI has been in existence for decades. What sets the current state of AI apart, however, is the increased awareness among the general public (not just data scientists) about the practical applications of AI in their daily lives. Consequently, the use of generative AI tools such as Chat GPT are experiencing exponential growth in usage - an inflection point to be sure. This is the AI inflection point where the disruption of roles, companies, industries and societies begins in earnest.
In biological ecosystems, an inflection point occurs when an exponential function, such as birth or death rates, reaches a critical threshold, causing the growth rate to dwindle and eventually turn negative, leading to an accelerated decline in the population. In the case of coral reef ecosystems, reef structures begin to deteriorate due to a higher rate of coral organism deaths (bleaching) compared to the rate of new coral growth. This triggers an exponential process where the dwindling number of organisms are unable to reproduce, while a growing percentage of coral organisms perish due to ecosystem stress.
Similarly, in social ecosystems, a parallel exponential process unfolds, but in the opposite direction. Just as elevated water temperatures have triggered an inflection point (collapse) in coral reef ecosystems, the perception of AI as a disruptor to the existing order has prompted a shift in people's behaviors, leading to an inflection point in the disruption of various organizations and industries that make up social ecosystems.
However, depending on how one defines the ecosystem and its constituent populations, several organizational and industry ecosystems are either nearing, have already reached, or have surpassed the disruptive inflection point of AI. Moreover, the use of AI is not a singular point of disruption, as various types of AI exist at different stages of maturity and adoption. On the other hand, data fabrics have not undergone a similar inflection point as AI. This is not because exponential growth in the availability of clean data, serving as critical building block for AI-driven disruption, does not pose a threat to the stability of industries and organizations. Rather it is because the need for data fabrics built with the principles of Lean Data has not been broadly recognized by those charged with managing data because of the lack of a precipitating event, a lack of understanding of lean/agile principles by most data scientists and engineers, and because most data folks tend to be risk averse and linear in their thinking.
The Lean Data Fabric’s inflection point
In social ecosystems, the inflection point is when the realization of a disruptive change in a specific group reaches a stage where an increasing number of participants begin to take action based on that awareness, leading to exponential growth. Geoffrey Moore's concept of the "Chasm" in "Crossing the Chasm" precisely represents this inflection point, where the adoption of new technologies expands rapidly from "early adopters" to "mainstream adopters." Notably, this expansion pertains to actual users embracing the technology.
Regarding Generative AI, the term "user" now encompasses anyone with computer access and an internet connection. While it remains vital to apply Lean Data principles to construct Data Fabrics that can efficiently deliver clean data at an exponential scale as the Generative AI landscape evolves, the composition of the team is critical. Teams and organizations responsible for data integration primarily originate from the data science community, boasting expertise in probability and statistics, mathematics, data modeling, analysis, and artificial intelligence. However, they often lack exposure to operational disciplines such as Lean/Agile, IT operations management, and process automation. And most importantly, there is no precipitating event for most data science organizations to overcome the status quo and embrace a disruptive change in the way they manage data, at least not yet. Many data organizations might be talking about Data Fabrics, but they are looking at it as a sustaining innovation (as a technology) not as a disruptive innovation (as a new way of doing things). This will not work for most. Clayton Christensen, a thought leader on innovation discusses extensively in his writing that no technology is disruptive by itself. Rather, it is how the technology is employed, the business model, that makes it disruptive or sustaining. The value of the Lean/Agile Data Fabric is that it allows organizations to change the way data integrations are built – if organizations don’t change the way they operate, these efforts will fail to create the value business leaders are expecting from their data integration and AI initiatives. This is exactly why many IT organizations 20 years after the introduction of the principles of DevOps, the application of Lean/Agile to software development, still struggle with implementation. Similarly, the Data Fabric (implemented with Lean Data) represents the application of Lean/Agile to data integration. I expect the transformation in data integration to play out in a similar way, albeit in a compressed timeline given the fact that the nature of the need for data fabrics to support generative AI is a far more powerful precipitating event than the one for DevOps (user driven, more efficient software delivery).
Life in the food chain – Strategies for surviving and thriving
Many technology people fancy themselves as “disruptors”, but the reality is that technology people are just like everyone else. They’re raising families and building careers to create safety for themselves and their families and trying to gain recognition in their communities. Most people get jobs at seemingly stable companies where disruption experiences significant pushback. But the winds of change are blowing, driven by advances in AI and other precipitating change events. It’s important to recognize that unless you are on the cusp of retirement, AI is going to disrupt your life and you will experience much greater success if you are the disruptor (not the disruptee). But when and how to go about this?
Situational Awareness
To begin, it’s critical to develop situational awareness of the stage of disruption for each AI and operational data component for each ecosystem you call home including your team, your greater organization, and your industry. Finding answers to the questions below is a great start, but you’ll also need to do your homework to understand the current state of AI, including what the possible use cases are, both current and emerging.
Which versions of AI, Data Fabrics, and Lean Data are reaching an inflection point in your ecosystems? You’ll need to understand this for each AI type/use case and supporting systems like Lean Data and technologies like the Data Fabric as well as how all those pieces fit together to enable business outcomes.
Are there disconnects where one technology component (e.g. Generative AI with enterprise data) has reached an inflection point, but supporting technologies (e.g. Lean/Agile Data Fabric) have not?
Team/Individual
How is your personal productivity being improved with the emerging technology?
Are others in similar roles outside the team using the new technology to greater or lesser effect?
Are other team members using the new technology to greater or lesser effect?
How resistant to change are other team members? Are they emotionally invested in the status quo?
Business unit/Company
Is the company reaching an inflection point? To what extent has fear/greed kicked in with respect to the broader leadership team in the adoption of AI for the company?
With AI. For the most part, business executives recognize that AI has to be part of their core business strategy, but that doesn’t necessarily mean right now.
With Lean/Agile Data Fabric. It’s very unlikely that senior leadership recognizes the fact that AI needs a better method of ingesting clean data if an AI strategy is to succeed, but there will be individuals (remember the 2.5%) who will that you will need to partner with.
Industry
Is the industry reaching (or has reached) a precipitating event where the entire industry is about to be (or is being) disrupted?
How would various combinations of Generative AI with Lean/Agile Data Fabrics negate the coming disruption or position your company (or team) to dominate through that disruption?
Does your team, unit and/or other company leadership understand the implications of the coming disruption and are they motivated to challenge the status quo as a result?
Develop your strategy – don’t push too far too fast
Now that you’ve established situational awareness, it’s time to get to work. This is where you put together your strategy to navigate around the defenders of the status quo (or convert them) and partner with the innovators. It’s critical to remember though that you can’t incentivize individuals or teams to act if they don’t possess both awareness of the benefits of the new technologies and methodologies AND are motivated to act even in the presence of the risk of change. Instead, the approach is to avoid allowing new technology adoptions being perceived as threatening to established individuals and teams or to convince those participants to join the initiative if they exhibit openness to being educated on the subject and are likely to experience an emotional reaction once they realize the truth. A couple of years ago, I had a conversation with a senior VP (now retired) at UPS that claimed Amazon was not a disruptive threat, even given the fact Amazon represented more than 10% of UPS package deliveries and was competing directly with UPS with little blue vans scouring suburban neighborhoods. And then there’s the fact of Amazon’s history of using their vendor partners/customers own data to compete with them directly. In such cases, it’s important to recognize that such an obviously wrong opinion is rooted in emotion and is very difficult to change. It’s better to work around these leaders than spend a lot of time trying to change their opinion, until you can point to specific business outcomes you can give them credit for. Also, if an organization has too many of these folks, it’s probably better to think about moving on – that ecosystem is too far from an inflection point to be able to embrace change. It may also be the right choice for that organization at that point to NOT change until a precipitating event occurs, meaning you either embrace the status quo yourself or move on.
To build momentum, it’s also important to use a Lean/Agile approach for delivering business value. Few leaders at this point are willing to wait years for a waterfall project to yield fruit.
Partner with fellow disruptors
Even if you’re a CEO, you can’t build an organizational culture of innovation that rewards adaptability over stability by simply hiring. You have other executives whose replacement would be too disruptive to current operations, and then of course, there’s the board. Consequently, it’s critical to identify and partner with individuals in the current ecosystem who are open to challenging the status quo because they feel (or can be made to feel) fear/greed/excitement about the existential threat and opportunity of using Generative AI with Enterprise Data. It’s also critical to make identifying these attributes part of the hiring process as well.
Walt Carter, in his book “We Can't Stay Here: Becoming A Great Change Captain” discusses the need to bring “misfit toys” (from the 1964 TV special “Rudolph the Red Nosed Reindeer”) into your organization to support change initiatives. By this, he is referring to individuals that exhibit adaptability, a preference for collaboration vs. competition, and a tendency to prioritize “us” vs. “me” - traits often penalized in organizations because most organizations prioritize stability over adaptability, yet critical for any ecosystem seeking to get in front of being disrupted.
Sell the car not the assembly line
The 1st Industrial revolution was one of the most disruptive, yet overall positive events in human history. And it all started with the rise of enabling systems and technologies like the rise of capitalism and the corporation, the rise of machines that made use of non biological forms of energy for manufacturing goods and mining, and the rise of nationalism to provide both funding and regulation. But one of the often overlooked drivers of the 1st revolution is the invention of the printing press in 1436. This invention unlocked an exponential advance in the ability of humans to communicate ideas broadly which made it possible for social ecosystems to reach inflection points in years rather than centuries.
The disruption of the Roman Catholic ecosystem via the Protestant Reformation is one such example. When Martin Luther nailed his “Ninety-five Theses or Disputation on the Power and Efficacy of Indulgences” on the door of the Wittenburg Castle Church in 1517, that was an inflection point in a social if there ever was one. And it was made possible because Luther was able to use the printing press to quickly and widely share his ideas in a way that made millions fearful that the Catholic Church had become too greedy and corrupt. This same type of mass communication in a more modern context is exactly why Generative AI has reached an inflection point, but Lean/Agile Data Fabrics have not. Martin Luther was able to reach outside of the clergy (a specialized group of experts unwilling to self-disrupt) to the broader population and create an emotional response in that much larger group to drive disruption in the specialist group.
Nearly 400 years later, Henry Ford achieved something similar when he disrupted the emerging automotive industry. While Ransom Olds received the 1st assembly line patent in 1901, the assembly line was nothing new. A similar process was being utilized by meat packers as early as 1870’s Chicago. In fact, it was a visit to a Swift and Company slaughterhouse by one of Ford’s employees, William "Pa" Klann, sometime around 1906 that is widely credited with introducing Ford to the concept. Once armed with the efficiencies of the assembly line, Ford was able to build awareness in a mass market, the emerging middle class and use that awareness to create FOMO for new middle class consumers wanting something that previously was only available as a luxury good because of cost. Unfortunately for most of Ford’s competition, their engineers were not adaptable enough to navigate the shift to mass production and their car companies ultimately failed.
“It is not the strongest of the species that survive, nor the most intelligent, but the one most responsive to change.”
It was the geometric growth in awareness and corresponding motivation to act in the larger ecosystem of middle class users that created the disruption inflection point in the smaller ecosystem of the automotive industry. The inflection point was therefore not the advent of assembly line in automotive manufacturing. The inflection point was awareness that middle class people could now afford a car grew exponentially to the point where the emergence of the new middle class market disrupted the automotive industry. To be successful innovating with Generative AI with data delivered by a Lean/Agile Data Fabric, you must do the same – Sell the business outcome, not the assembly line.
Your results must link directly to something your team, division, and company cares about; getting vertical alignment like this takes a lot of time and effort, but is a requirement for success. And the choice of business outcomes to tackle must also be ones that map most easily to each level in the hierarchy.
Run the Skunk works strategy for ecosystems that are behind
You’ve already done your homework and achieved situational awareness so you know the business outcomes possible by adopting AI. Now to navigate the landmines. While the ideal situation is one where there is a culture of innovation and everyone is motivated to collaborate and invest personally in transformation, that is an unlikely scenario. More likely, you will have some number of leaders in the organization who will try to block you because of their stake in the status quo, even if your company and/or industry is experiencing significant disruption. The best strategy here is to find and partner with one or more business champions who do want disruptive change and are willing to support your initiative from a budgetary perspective as well as work around the organizational obstacles. This will only work if your partners in data driven AI disruption have political cloud significantly greater than the naysayers and you’ve carefully selected projects that solve business cases for data driven AI that:
Avoid interference with existing systems protected by political interests in the organization (even if they work for you)
Initial install and 1st sprint completion are less than 3 months in duration
1st sprint creates a clear and significant business impact you don’t need to be technical to understand
Creates integration work that is easily repurposed for other items on the business’ AI and/or Data wish list
Know when to move on
In some cases, you’ll finish your investigation on where the people in your ecosystems are with respect to being disrupted by AI and Lean Data via Data Fabrics and realize you cannot foster the cultural changes needed to prepare for the coming disruption. A successful strategy for avoiding ecosystem disruption with data driven generative AI may not be possible because of the current state of culture in that ecosystem; you may have to decide to play the long game by waiting for the political landscape to shift and/or a company/industry inflection point, content yourself with working in an ecosystem on the wrong end of disruption (good for folks nearing retirement) or start the process of moving on now. Forcing the issue by fighting unwinnable political battles is not a good strategy. You’ll likely lose and suffer a loss in reputation as you’re pushed out anyways.
Steve Sasson, inventor of the Digital camera in 1975
This is actually a common situation. While Steve Sasson invented the digital camera in 1975, the leadership of his employer, Kodak, refused to market a digital camera until it was too late. Sasson continued working on digital cameras for Kodak until his retirement in 2009, but although Kodak made royalties for the patent, the leadership’s unwillingness to self cannibalize their traditional photography business ultimately led them to bankruptcy. Did Steve Sasson make a mistake staying at Kodak until the end? It depends on his priorities. Just like Steve Sasson, some of us will have to make some hard choices depending on our values, goals, priorities and where we are with respect to our careers.
Conclusion
It’s important to recognize that unless you are on the cusp of retirement, #AI is going to disrupt your life and you will experience much greater success if you are the disruptor (not the disruptee). It all comes down to the culture of the social and business ecosystems you are a part of and when those ecosystems reach inflection point were exponential growth in awareness turns to change in behavior en masse.
The pace of adoption of Generative AI and Lean Data at an exponential scale requires innovators to recognize and take advantage of disruptive inflection points. This necessitates fostering a culture of innovation and aligning with individuals open to change. Understanding the role of fear and greed in driving transformation is crucial, as seen in the adoption of AI. Recognizing the need for agile methodologies in data integration underscores the importance of putting the principals of Lean Data to work building a Data Fabric that can source the clean data needed at exponential scale to take advantage of Generative AI with Enterprise Data.
Please share!
Tyler Johnson
Cofounder, CTO PrivOps