Thanks to its proven power when implemented properly, in this article Will Cappelli CTO and VP Product, EMEA at Moogsoft talks about how AIOps has become a buzzword that many vendors are laying claims to. Don’t believe everything you hear.
AIOps has become a necessity for taming the dizzying operational complexity that IT Ops and DevOps teams face as they support their organizations’ digital transformation efforts.
Using artificial intelligence and machine learning, AIOps helps DevOps and IT Ops teams automate and streamline IT incident management, accelerate problem detection and fixes, and prevent outages of critical services.
But vendor approaches to AIOps differ. One key question is: Should AIOps be delivered as a function of existing enterprise and systems management products, or instead as an independent platform that organizes those other technologies, and orchestrates their interaction?
Learn More: The Major Roadblocks To a Successful Big Data Analytics ImplementationOpens a new window
The superior option is unequivocally the latter. Here’s why.
An abridged history of IT Operations
In 2000, the modern IT world began to take shape, as the Internet first linked together computing devices across the globe—albeit in an imperfect and rudimentary way. At this point, these devices generated an exabyteOpens a new window of self-descriptive data about their operations, half of which was either irrelevant or duplicated.
Four big vendors — IBM, CA, HP and BMC — provided large clunky software to manage this operational data via systems management, network management, asset management, and service desk tools.
Five years later, virtualization emerged as a layer on top of the Internet, while self-describing data had increased by an order of magnitude to a zettabyteOpens a new window — 10 with 19 zeros after it. The quality of this data had worsened, with 60% being noise or duplicated. Struggling to innovate, the big four IT management vendors nurtured a venture capital community to promote the creation of startups with modern technology … that they could then acquire.
By 2010, cloud computing emerged, and self-descriptive data ballooned to a yottabyteOpens a new window , with 70% being noise or duplicated. At this point, the big four vendors were displaced by specialized vendors organized into eight submarkets:
- Event Correlation
- Smart Alerting
- Service Desk & CMDB
- Process Automation
- Topology Analytics
- Time Series Databases
- Log Management
- Monitoring (including end user, application, infrastructure and network monitoring)
This specialization was made necessary by the complexity that the cloud, virtualization, and the Internet created in the global computing infrastructure.
Fast forward to 2015. Complexity had worsened with the atomization and hyper-modularization of the global compute infrastructure as a result of the adoption of technologies like microservices. Self-describing data reached a brontobyte, 80% of which was noise and duplicated.
To make matters worse, the number of products in the eight submarkets had ballooned, and they didn’t interoperate with each other. At this point it was virtually impossible for enterprises to coordinate the operations of these eight different types of technology. It became obvious that we needed a capability for automated observation and analytical response to coordinate what those eight subtypes of technology were doing. As a result, the idea for AIOps became well-articulated at that point.
By 2020, we’ll be up to a gegobyte of self-descriptive data, with a whopping 90% of it being noise or duplicated. Complexity will worsen due to the adoption of continuous integration / continuous delivery (CI/CD) software pipelines. For CI/CD environments to succeed, we need a third element: CA, or continuous assurance of the process that generates digital services, and AIOps plays a central role in that.
Learn More: How Does Outsourcing Help Reduce Software Development Costs?Opens a new window
What we need from AIOps
To deal with this gegobyte of highly suspect data and manage increasingly complex IT environments, organizations need 5 different types of AIOps algorithms, or 5 dimensions of functionality:
- data selection
- pattern discovery
- inference
- communication
- robotics (automation)
In the real-world, here’s how it works.
An AIOps workflow ingests heterogeneous data from many different sources. Using entropy algorithms, it removes noise and duplication, which can amount to more than 90% of the data, and selects only the truly relevant data. It then correlates this relevant information using various criteria, like text, time and topology.
Next, it discovers patterns in the data, and infers which data items signify causes, and which signify events. It then communicates the result of that analysis to a collaborative environment, which will support automated responses to what has been discovered.
At almost every step along the way in that workflow — from the initial ingestion of the data to the ultimate response to what has been discovered — it’s necessary to have automated observation, analysis, and automated response to what has been found.
Learn More: Taking on the Patching and Configuration Crisis: 3 Critical Steps for 2020Opens a new window
In defense of independent AIOps platforms
Vendors in the eight submarkets only provide partial AI capabilities, often relying mostly on antiquated and inefficient rules-based correlation, and semi-automated statistical analysis. The reason for their weak or non-existent AI capabilities is that they haven’t devoted the necessary research effort and investment. They’ve belatedly realized that they must provide AIOps in order to remain competitively relevant, and are trying to come up with AIOps-like capabilities in a hurry.
On top of that, their AI offerings are domain specific and don’t interconnect each other’s tools, which is a central requirement of fully-functioning and true AIOps, and one of a platform solution’s core values.
Indeed, a key reason why AIOps became an important factor in the market was precisely to link these different types of technologies together. Although these vendors claim to offer AIOps, their respective solutions are somewhat myopic.
All of them stay away from that role of organizing and integrating what they do with all of the other pieces of the puzzle. It’s precisely in that integration that AIOps platforms deliver their core value. In other words, it’s only through the intelligent integration of all these different functionalities that we can manage that gegobyte of self-descriptive but highly corrupt data.
Let us know if you liked this article on LinkedInOpens a new window , TwitterOpens a new window , or FacebookOpens a new window . We would love to hear from you!