The numbers show that hybrid infrastructures are the new normal:
- 69% of businesses of all sizes have embraced hybrid clouds (Flexera).
- 88% of cloud-based apps share data and services with on-premises apps (MicroFocus).
- 45% of enterprises see hybrid solutions as their top priority for 2019 (Flexera).
As discussed in our blog post the Rise of the Hybrid Cloud the dramatic growth of hybrid infrastructures is being driven by benefits such as enhanced flexibility, cost optimization opportunities, and support for the agile DevOps culture. But hybrid clouds also come with their challenges, such as determining how to consistently apply security and compliance processes and how to avoid performance issues resulting from the differences between private and public cloud SLAs. Perhaps the biggest challenge of them all, however, is implementing seamless management and monitoring across complex hybrid architectures.
This blog post explores the challenges of monitoring hybrid cloud environments and suggests some best practices that can mitigate their risks.
Why It’s Hard to Monitor Hybrid Clouds
Recent surveys show that one of the main concerns today in moving workloads to the public cloud is achieving effective end-to-end monitoring of software and hardware stacks deployed across on-premises infrastructures and private and public clouds. According to Keysight Technologies, 61% of the IT professionals they surveyed are concerned about their ability to securely deliver cloud traffic to on-premises monitoring systems. Similarly, 93% believed that packet-level visibility is a critical component of security monitoring, yet only 19% thought that they have comprehensive, realtime access to network packets in a hybrid cloud infrastructure.
In addition to the security vulnerabilities they present, hybrid cloud monitoring challenges can also impact business-critical metrics related to performance and availability. With poor visibility and fragmented monitoring stacks, it takes longer to troubleshoot and resolve issues, leading to unacceptable levels of latency (too low) and downtime (too high).
In short, there is a lot at stake in overcoming the obstacles to effective hybrid cloud monitoring. One of the major stumbling blocks is that the different components of a hybrid infrastructure require different monitoring approaches and tooling, but it is unlikely that the organization’s existing monitoring frameworks can effectively track both on-premises and cloud environments. In fact, it is estimated that only 15% of legacy network security tools or appliances have been fully “cloudified,” i.e., all of their on-premises capabilities have been modified to work on the cloud. To make things even more complicated, most of these enterprise-grade monitoring systems underwent significant customization when they were deployed in customer environments. Getting so many disparate and complex systems to work together to provide meaningful, real-time insight into the health of a hybrid cloud infrastructure is close to impossible.
Another major hybrid cloud monitoring obstacle is IT’s limited visibility into the public cloud components of the infrastructure. Public cloud providers do offer monitoring and logging services, such as AWS CloudTrail, Amazon CloudWatch, Azure Monitor, and GCP’s Stackdriver Monitoring. However, the Cavirin-sponsored 2019 AWS Cloud Security Report (registration required) reveals that only 21% of respondents believe that their organizations effectively use the monitoring, logging, and alerting capabilities of AWS CloudTrail or Amazon CloudWatch.
Other issues that make it difficult to monitor hybrid clouds include:
- The difficulty of discovering, creating and maintaining an up-to-date topology for the hybrid environment.
- The sheer scale of metrics that are generated across a complex hybrid environment.
- The siloed nature of cloud provider tools. They are good, but even if they are being used effectively, they cannot provide a full picture across hybrid/multicloud environments.
- The challenge of implementing agent-based legacy monitoring solutions in the highly dynamic and often ephemeral cloud environment. In general, it is exceedingly difficult to instrument modern apps and services for logging and monitoring.
- The dearth of IT personnel skilled in configuring and managing hybrid cloud environments.
