Why did VMware acquire Wavefront? The start of the answer to this question comes with an understanding of what Wavefront is (or was). Wavefront was started by former Google engineers who set out to build a monitoring system for the commercial market that had the same features and benefits as the monitoring system that Google had built for itself.
Due to the massive scale of Google, such a system would have to have two key attributes:
- The ability to consume and process massive amounts of data very quickly. In fact, the Wavefront website make the claim, “Enterprise-grade cloud monitoring and analytics at over 1 million data points per second.”
- The ability to quickly find what you want in this massive ocean of data
So, it is clear that the folks at Wavefront viewed modern monitoring to be a big data problem, and it is clear that some people at VMware were willing to pay a fair amount of money for a monitoring system that took a real-time and highly scalable approach to monitoring.
Why is modern monitoring a big data problem?
Rather than just assume that VMware and Wavefront are right about the idea that modern monitoring is a big data problem, let’s look at the underlying trends in the IT industry to determine what is changing monitoring in this manner. It is not enough that Google (and by extension other public clouds like Amazon Web Services and Microsoft Azure) have a big data monitoring problem. The question at hand is whether or not monitoring for the typical enterprise has become a big data problem.
The key insight is that the IT environment today could not be more different that it was as recently as five or 10 years ago. Ten years ago, there was one language (Java), and an application ran on one operating system, which ran on one physical server. Applications were updated as infrequently as possible, and changes in general were made as infrequently as possible.
The image below depicts the typical “IT stack” at an enterprise today.
What is so different about the modern IT environment is the following:
- In response to an unlimited demand for business functionality implemented in software, agile development and DevOps were invented to speed code to market.
- In response to the same pressures, new languages and run times were created.
- The above two changes lead to a very diverse application stack, with frequently arriving and changing applications.
- At the infrastructure layer, everything (compute, networking and storage) is virtualized and is often subject to automated management.
In summary, the modern IT stack now consists of very diverse application stacks, with a rapid rate of change (many changes per hour) running on an abstracted and dynamic infrastructure.
How does monitoring have to respond?
If you look at how monitoring is done today, it really has not changed in response to the changes in the IT stack listed above. Monitoring today consists of many different vendors, each collecting a slice of the total data, analyzing it, alerting on it, displaying it in dashboards and providing in reports.
Now, it might be tempting to try to find one vendor that can monitor that entire stack for you. Before you go down that road, remember what happened the last time that was tried. It was called Business Service Management with offerings from BMC, CA, HP and IBM, and it failed miserably (20 years ago) because even then the pace of innovation was so high that each vendor could not keep up. So they acquired companies to fill the gaps in their product lines, and they were never able to integrate them, which resulted in the mess, which in turn resulted in the failure of the BSM suites.
So, the first very important realization is that due to the accelerating pace of innovation in the industry, monitoring must remain a multi-vendor problem. This is so because various parts of the monitoring problem are “whole company” problems that require a significant investment of capital in intellectual property to solve.
Monitoring must also generally change to embrace the following principles:
- If the stack is diverse (especially at the application layer), then each component and layer of the stack needs to be monitored.
- Transactions need to be monitored from their inception to their end in the application system (browser to database and back again).
- The entire stack needs to be monitored from the top of the the stack to the infrastructure (browser to hard disk or storage device and back again).
- So, the number of things that need to be monitored increases dramatically.
- If the environment is dynamic due to frequent changes at the application layer and automation at the infrastructure layer, then monitoring needs to be much more frequent. Every five minutes is no longer frequent enough. Every minute is no longer frequent enough.
The big data approach to monitoring
If we accept that monitoring is a multi-vendor problem due to the diversity of the problem, and we accept that the granularity of monitoring and the frequency of monitoring must increase due to the dynamic nature of the stack, then monitoring is a real-time multi-vendor problem.
There are then two approaches to implementing real-time big data monitoring:
- Have every vendor integrate with every other vendor and try to maintain a nightmare of a compatibility matrix.
- Have every vendor integrate with a common high-performance, big data back end especially built for the real-time multi-vendor monitoring problem.
The diversity of the modern application stack, and the rate of change at both the application and infrastructure layers, requires that monitoring become more granular and more frequent across the multiple vendors required to cover the IT estate. This turns monitoring into a multi-vendor big data problem.
- November 2020(46)
- October 2020(79)
- September 2020(72)
- August 2020(64)
- July 2020(71)
- June 2020(74)
- May 2020(50)
- April 2020(71)
- March 2020(71)
- February 2020(58)
- January 2020(62)
- December 2019(57)
- November 2019(64)
- October 2019(26)
- September 2019(24)
- August 2019(15)
- July 2019(24)
- June 2019(55)
- May 2019(82)
- April 2019(77)
- March 2019(71)
- February 2019(67)
- January 2019(77)
- December 2018(46)
- November 2018(48)
- October 2018(76)
- September 2018(55)
- August 2018(63)
- July 2018(74)
- June 2018(64)
- May 2018(65)
- April 2018(76)
- March 2018(82)
- February 2018(65)
- January 2018(80)
- December 2017(71)
- November 2017(72)
- October 2017(75)
- September 2017(65)
- August 2017(97)
- July 2017(111)
- June 2017(87)
- May 2017(105)
- April 2017(113)
- March 2017(108)
- February 2017(112)
- January 2017(109)
- December 2016(110)
- November 2016(121)
- October 2016(111)
- September 2016(123)
- August 2016(169)
- July 2016(142)
- June 2016(152)
- May 2016(118)
- April 2016(60)
- March 2016(86)
- February 2016(154)
- January 2016(3)
- December 2015(150)