Saturday, 20 June 2026

ETCD in Red Hat OpenShift: Why Heartbeat Monitoring Matters and How VMFS Storage Can Make or Break Your Cluster

Introduction

When discussing Red Hat OpenShift performance and reliability, most administrators focus on worker nodes, application pods, networking, and ingress traffic. However, the true foundation of every OpenShift cluster is ETCD.

ETCD is the distributed key-value store that acts as the source of truth for Kubernetes and OpenShift. Every object in the cluster-including deployments, pods, services, routes, secrets, config maps, and node information-is stored in ETCD.

Because ETCD sits at the heart of the OpenShift control plane, even minor performance degradation can impact the entire cluster. One of the earliest indicators of ETCD health is heartbeat latency. In many VMware-based OpenShift deployments, ETCD heartbeat issues are not caused by Kubernetes itself but by underlying storage latency, particularly within VMFS-backed datastores.

This article explains:

The role of ETCD in OpenShift
Why heartbeat monitoring is critical
How storage latency affects ETCD
The relationship between ETCD and VMware VMFS storage
A practical script to monitor ETCD latency and heartbeat health

Understanding ETCD in OpenShift

ETCD is a distributed, strongly consistent key-value database used by Kubernetes and OpenShift.

Every operation performed through the Kubernetes API eventually results in ETCD activity.

Examples include:

Creating a pod
Scaling a deployment
Updating a secret
Modifying a route
Registering a node
Updating cluster configuration

In a standard OpenShift deployment, ETCD runs on the control plane nodes and uses the Raft consensus algorithm to maintain consistency across multiple members.

The architecture ensures that:

Data is replicated across ETCD members
Changes are committed only after consensus
Cluster state remains consistent even during failures

Without a healthy ETCD cluster, OpenShift cannot function reliably.

The Importance of ETCD Heartbeats

ETCD relies on the Raft consensus protocol.

Within Raft, a leader node periodically sends heartbeat messages to follower nodes.

These heartbeats serve several purposes:

Confirm leader availability
Maintain cluster synchronization
Prevent unnecessary elections
Ensure consistency among ETCD members

A healthy ETCD cluster continuously exchanges heartbeats with minimal latency.

If followers stop receiving heartbeats within the expected timeframe, they assume the leader has failed and initiate a leader election.

While occasional elections are normal, frequent elections indicate underlying infrastructure problems.

Why ETCD Heartbeat Monitoring Is Essential

1. Detects Infrastructure Problems Early

Heartbeat delays are often the first warning sign of:

Storage latency
Network congestion
CPU starvation
Memory pressure

Administrators who monitor ETCD heartbeat behavior can identify issues before applications begin failing.

2. Prevents Control Plane Instability

Delayed heartbeats can trigger unnecessary leader elections.

This can result in:

Kubernetes API delays
OpenShift console slowness
Failed deployments
Operator degradation
Cluster upgrade interruptions

3. Improves Troubleshooting

Many OpenShift issues initially appear as:

API server slowness
Cluster operator degradation
Node instability

ETCD heartbeat metrics help quickly identify whether the root cause lies in the control plane infrastructure.

ETCD and Storage: A Critical Relationship

Unlike traditional databases that often depend on throughput, ETCD depends heavily on low latency.

Every ETCD transaction involves:

Writing data to disk
Performing fsync operations
Replicating changes to followers
Receiving acknowledgements

Because these operations occur constantly, storage latency directly affects ETCD responsiveness.

A few milliseconds of additional latency may seem insignificant, but in ETCD environments those delays accumulate rapidly and impact heartbeat processing.

Understanding VMware VMFS Storage

Many enterprise OpenShift clusters run on VMware vSphere.

In these environments, virtual machines often reside on VMFS (Virtual Machine File System) datastores.

VMFS provides:

Shared storage access
High availability
vMotion support
Centralized datastore management

While VMFS is highly reliable, ETCD performance ultimately depends on the performance characteristics of the storage array beneath VMFS.

How VMFS Storage Affects ETCD Heartbeats

Storage Latency

The most common cause of ETCD performance degradation is storage latency.

When datastore latency increases:

WAL writes become slower
Backend commits take longer
Raft operations are delayed
Heartbeats arrive late

Administrators may begin seeing warnings such as:

"leader failed to send out heartbeat on time"

Datastore Contention

Many organizations place numerous virtual machines on the same datastore.

Examples include:

Database servers
Backup workloads
Application servers
OpenShift control plane nodes

When storage-intensive workloads compete for resources, ETCD often becomes one of the first services affected.

Queue Depth and Storage Congestion

Storage arrays have finite resources.

Issues such as:

Excessive queue depth
Controller bottlenecks
SAN congestion
Storage processor saturation

can increase datastore response times and negatively impact ETCD heartbeat processing.

Common Symptoms of Storage-Induced ETCD Problems

Administrators may observe:

Frequent ETCD leader elections
Slow API server responses
Delayed pod scheduling
Cluster operator degradation
OpenShift console timeouts
Upgrade failures
Node status update delays

In many cases, these symptoms originate from storage latency rather than ETCD software defects.

Key ETCD Metrics to Monitor

The following metrics are particularly important:

Metric	Purpose
etcd_disk_wal_fsync_duration_seconds	WAL write latency
etcd_disk_backend_commit_duration_seconds	Backend database commit latency
etcd_server_leader_changes_seen_total	Leader election frequency
etcd_network_peer_round_trip_time_seconds	Peer communication latency
etcd_server_has_leader	Leader availability

A sudden increase in these metrics should trigger investigation.

ETCD Health Monitoring Script

The following script can be executed from a system that has access to the OpenShift cluster and is authenticated using the oc command-line utility. (Please note : I will be sharing if requested in the comment only ) .

Interpreting the Results

Healthy Cluster

Typical characteristics:

WAL fsync latency below 10 ms
Backend commit latency below 25 ms
Minimal leader changes
No heartbeat warnings

Warning State

Typical indicators:

WAL fsync latency above 50 ms
Commit latency above 100 ms
Occasional leader changes
Sporadic heartbeat warnings

At this stage administrators should review datastore performance.

Critical State

Typical indicators:

Persistent latency above 100 ms
Frequent leader elections
API responsiveness issues
Control plane instability

Immediate investigation of storage infrastructure is recommended.

VMware Metrics to Correlate

Whenever ETCD latency rises, VMware administrators should review:

Datastore latency
Device latency
Kernel latency
Queue depth
Storage throughput
Storage controller utilization

Correlating OpenShift ETCD metrics with VMware performance statistics often leads directly to the root cause.

Best Practices

To maintain healthy ETCD performance:

Use Fast Storage

Prefer:

SSD-backed datastores
NVMe-backed storage
Low-latency SAN configurations

Isolate Control Plane Workloads

Avoid placing:

Backup servers
Large databases
High-I/O applications

on the same datastore as OpenShift control plane nodes.

Monitor Continuously

Track:

ETCD latency
Heartbeat behavior
Leader changes
VMware datastore performance

Investigate Heartbeat Warnings Immediately

Heartbeat warnings are often early indicators of larger infrastructure issues.

Ignoring them can eventually lead to cluster instability.

Conclusion

ETCD is the most critical component of the OpenShift control plane. While administrators often focus on CPU, memory, and networking, storage latency remains one of the most common causes of ETCD performance degradation.

Heartbeat monitoring provides an early-warning system that can reveal underlying infrastructure problems before they affect applications. In VMware environments, VMFS datastore performance directly influences ETCD write latency, commit operations, and Raft heartbeat processing.

By continuously monitoring ETCD heartbeat health, storage latency, and VMware datastore metrics together, organizations can significantly improve OpenShift stability, reduce unexpected outages, and ensure a resilient Kubernetes platform.

Note : This ETCD latency we experienced and solved in our on prem clusters, we welcome your way of experience and admin challenges related to ETCD.

Sunday, 14 June 2026

DevSecOps in 2026: The Trends Reshaping the Pipeline and the Tools Behind Them

A few years ago, "DevSecOps" mostly meant adding a security scanning step somewhere near the end of a CI/CD pipeline and calling it a day. That era is over. In 2026, security is being pulled earlier into development, automated more aggressively, and — increasingly — built to handle a development process where a meaningful share of code is written by AI rather than humans.

This post walks through where DevSecOps stands right now: the trends driving the biggest changes, the tools teams are actually using, and a few recent events that show exactly why this space needs constant attention.

The State of the Market

DevSecOps adoption has moved well past "early trend" status. A growing share of organizations now build software using DevSecOps practices, and the shift isn't just about checkbox compliance — security teams report real confidence gains, with automation increasingly seen as core to the approach rather than a bolt-on. Most teams now run some combination of static analysis (SAST), software composition analysis (SCA), container scanning, and infrastructure-as-code (IaC) checks as standard practice.

But adoption of tools isn't the same as maturity. The honest state of things in 2026 is that most teams have plenty of scanners running — the harder problem is what happens after a scanner finds something. If high-risk findings sit untriaged, more scanning just means more noise.

Trend 1: Consolidation and Alert Fatigue

The single biggest theme this year is consolidation. Teams that adopted a different point solution for each concern — one tool for SCA, another for SAST, another for container scanning, another for secrets — are now drowning in disconnected alerts from a dozen dashboards.

The response has taken two forms:

Unified platforms. Tools like Wiz and GitLab's built-in security suite aim to cover multiple domains (cloud posture, container security, SAST, DAST) from a single product, reducing the number of places a team has to look.
Aggregation layers. Open-source tools like DefectDojo sit on top of an existing mix of scanners and consolidate their output into one place — useful for teams that don't want to rip out tools that already work well individually, but need a single view of what actually matters.

A related shift is policy as code (PaC) — instead of manual compliance checklists, security and compliance rules are written as code (often using Open Policy Agent) and enforced automatically across pipelines and infrastructure. This turns "did we follow the policy?" from an audit question into something the pipeline can answer for itself.

Trend 2: Cloud-Native Security Goes Mainstream (CNAPP)

Cloud-Native Application Protection Platforms — CNAPPs — bundle together Cloud Security Posture Management (CSPM), Cloud Workload Protection (CWP), and IaC security into one platform that automatically discovers and protects new cloud resources as they're deployed.

This category got a lot more attention in 2026 after Google's roughly $32 billion acquisition of Wiz closed — one of the largest deals in the cloud security space, and a clear signal of how central cloud-native posture management has become to the broader security stack. For teams running multi-cloud or heavily containerized environments, CNAPP-style tooling is increasingly treated as table stakes rather than a "nice to have."

Trend 3: Supply Chain Security Gets Serious — and Gets Tested

Software supply chain security — SBOMs (Software Bills of Materials), artifact signing, and dependency provenance — has moved from "good idea" to "expected practice," with open-source tools like Syft and CycloneDX for SBOM generation and Sigstore for artifact signing becoming close to standard.

But 2026 also delivered a blunt reminder of why this matters: Trivy's own GitHub Action was compromised twice in the same month through tag poisoning — meaning a tool widely used to scan for supply chain risks briefly became a supply chain risk itself. It's a useful (if uncomfortable) case study: pinning action versions to commit hashes rather than mutable tags, and treating your security tooling's own dependencies with the same scrutiny as your application code, isn't paranoia — it's now a documented attack pattern.

Trend 4: AI Is Changing Both Sides of the Equation

AI shows up in DevSecOps in two very different ways right now, and it's worth keeping them separate:

AI as a target. AI coding assistants are generating code — and APIs — faster than security review can keep up, and a large share of security professionals now use or plan to use AI/ML for code review specifically because of this volume increase. At the same time, AI agents are increasingly consumers of internal APIs and systems, which creates a new category of identity and access management problem: how do you apply least-privilege and audit trails to a non-human actor making autonomous decisions?

AI as a tool. On the defensive side, AI-driven triage is one of the most-cited 2026 trends — using models to cut through alert noise, prioritize findings by actual exploitability rather than raw CVE severity, and in some cases draft remediation pull requests automatically. At a recent major security conference, "AI agent security" was reportedly the subject of products from a striking number of vendors in a single week — a sign of how quickly this sub-category is forming, even if much of it is still maturing.

The Tools Teams Are Actually Using

Rather than a generic "top 10" list, here's a more practical breakdown by what each tool is genuinely good at — because in 2026, "what's the best DevSecOps tool" is less useful than "what's the best tool for this specific job."

For container and dependency scanning (SCA): - Trivy (open source, by Aqua Security) remains the default choice for fast, free container image scanning, IaC misconfiguration checks, secrets scanning, and SBOM generation — all from a single CLI binary, with no feature gates. - Grype is a close open-source alternative focused specifically on container image CVE scanning, built on well-maintained vulnerability databases. - Snyk offers a more polished, dashboard-driven experience with automated fix pull requests and exploitability-based prioritization — useful when alert noise is the bigger problem than scan coverage.

For static analysis (SAST): - Semgrep has become a popular open-source option for fast, customizable static analysis across many languages. - Snyk Code offers a proprietary AI-assisted SAST engine with real-time IDE feedback, which Trivy notably does not provide at all.

For infrastructure as code: - Checkov and tfsec are widely used for Terraform and broader IaC policy scanning, often used together for breadth and speed respectively.

For runtime and cloud posture: - Wiz focuses on deep runtime visibility and IAM risk detection across cloud environments — particularly strong for cloud-heavy teams, and now backed by Google's resources following the acquisition. - Falco (a CNCF open-source project) handles runtime threat detection specifically for Kubernetes. - Sysdig is notable for forensics — capturing deep system call data so security teams can reconstruct what happened inside a container even after it's gone, which matters because containers are ephemeral by nature.

For secrets detection: - TruffleHog and GitGuardian both scan git history (not just current code) for leaked credentials and high-entropy strings — important because a secret committed and later removed is still exposed in history.

For pulling it all together: - DefectDojo remains the leading open-source option for aggregating findings across all of the above into a single, manageable view — arguably more important in 2026 than any individual scanner, given how fragmented the tool landscape has become.

A common pattern for teams building a stack from scratch: Trivy and Semgrep cover the widest surface area at zero licensing cost (containers, IaC, OSS CVEs, and SAST), with a commercial tool like Snyk layered in specifically for its developer workflow integration and automated remediation — rather than trying to replace the open-source layer entirely.

What This Means in Practice

If there's one thread connecting all of this, it's that tool count was never the goal — closing the loop between detection and remediation is. A team running ten scanners that never triages findings is, in a meaningful sense, less secure than a team running two scanners that consistently fixes what it finds.

For 2026 specifically, three things are worth prioritizing regardless of company size:

Pin your CI/CD dependencies — including your security tools' own GitHub Actions — to commit hashes, not tags. The Trivy incident this year is exactly why.
Don't treat AI-generated code differently in your pipeline — it should go through the same SAST/SCA/IaC gates as anything else, ideally with no exceptions for "the AI wrote it quickly."
Invest in triage and aggregation before adding more scanners. If your team already has signal it isn't acting on, a new tool will only add more signal you won't act on.

What's your stack looking like in 2026 — consolidated platform, mixed open-source, or somewhere in between? I'd be curious to hear what's working (and what isn't) in the comments.

API Discovery for enterpise

If you ask most engineering leaders "how many APIs does your organization actually have?", the honest answer is usually some version of "we're not entirely sure." That uncertainty used to be a minor governance headache. In 2026, it's a serious security and compliance problem — and it's the reason API discovery has become one of the fastest-growing categories in enterprise tooling.

This post breaks down what API discovery actually means, why it has become urgent now, and which tools are worth knowing about if you're evaluating this space.

What Is API Discovery?

At its simplest, API discovery is the process of automatically finding, cataloging, and documenting every API running across an organization's environment — internal services, partner integrations, third-party connections, and anything exposed publicly. This includes the APIs your teams know about, but more importantly, the ones they don't.

Three categories tend to come up constantly in this space:

Shadow APIs — endpoints that exist and are live, but were never registered in any official inventory
Zombie APIs — old or deprecated endpoints that were supposed to be retired but are still reachable
Orphaned APIs — services left behind after a team moved on, a project was shelved, or ownership changed without documentation following

Modern discovery tools find these by analyzing network traffic, inspecting API gateways, scanning code repositories and CI/CD pipelines, or some combination of all three.

Why This Has Become Urgent

API sprawl isn't new, but a few trends have made it dramatically worse recently.

AI coding assistants are generating APIs faster than teams can track them. What used to take a developer a day to scaffold can now be generated in minutes with tools like GitHub Copilot or other AI code generators. The speed is genuinely useful, but it widens the gap between "API exists" and "API is documented, secured, and governed."

AI agents are now API consumers too. It's no longer just human developers calling APIs — autonomous agents are increasingly calling them as part of automated workflows. Every one of those calls is a dependency that needs to be tracked, authenticated, and rate-limited appropriately, which multiplies the surface area that needs visibility.

Most enterprises run multiple API gateways, not one. It's common for one team to standardize on Kong, another on AWS API Gateway, and a third on Apigee or MuleSoft — each with its own catalog. The result isn't usually a discovery problem so much as a fragmentation problem: APIs get documented in one place, monitored in another, and governed nowhere consistently.

Put together, this means the old approach — a spreadsheet someone updates occasionally — simply can't keep up anymore.

The Current Tooling Landscape

The tools in this space generally fall into a few overlapping groups: security-focused discovery platforms, gateway-aggregation tools, and developer-collaboration networks. Here's a look at some of the names that come up most often.

Security-First Discovery Platforms

These tools approach API discovery primarily through the lens of attack surface management — finding what's exposed, flagging risk, and feeding that into broader security workflows.

Levo.ai takes a runtime-first approach, using lightweight sensors to observe API traffic at the OS layer before encryption. This lets it build a live inventory across REST, GraphQL, gRPC, and SOAP without requiring code changes, and it enriches each discovered endpoint with details like authentication method and sensitive data exposure.
Salt Security focuses on identifying APIs through traffic analysis and behavioral patterns, which is particularly useful for catching APIs that were never intentionally published anywhere.
StackHawk pairs discovery with API-focused dynamic application security testing (DAST), so once an API is found, it can be tested immediately rather than just logged for later review.
Traceable.ai, Akto, Qualys, Rapid7, Invicti, Akamai, and Orca Security all show up regularly in this category too, generally differentiated by how deeply they integrate with existing security stacks and cloud environments.

Gateway Aggregation and Inventory Tools

For organizations dealing with the "multiple gateways, multiple catalogs" problem specifically, some tools focus on pulling everything into a single unified view.

DigitalAPI.ai is built around this exact use case — aggregating APIs across Kong, Apigee, AWS API Gateway, Azure, MuleSoft, and others into one inventory, which helps when different teams have standardized on different platforms over time.

Developer Collaboration and Internal Discovery

Not every discovery problem is a security problem — sometimes it's simply "our own developers don't know this API already exists, so they build a duplicate."

Postman's Private API Network addresses this directly. Available on Postman's enterprise tier, it gives internal developers a searchable network of workspaces, collections, and documented APIs built by other teams — reducing duplicate work and onboarding time. Postman frames this as a "git-driven workflow," where the catalog stays in sync with what's actually in source control rather than relying on someone remembering to update a wiki.
The same company's Public API Network is worth knowing about too, even if it's a different use case — it's reportedly one of the largest public directories of APIs, with well over 100,000 listed, complete with ready-to-fork collections and documentation.

How to Think About Choosing One

If you're at the stage of evaluating these tools, a few questions tend to matter more than feature checklists:

Internal or external focus (or both)? Security platforms tend to emphasize finding what's exposed to the outside world; collaboration tools like Postman's network focus more on internal reuse and developer productivity.
How does it discover APIs? Traffic analysis, gateway integration, and code/CI scanning all surface different things. Traffic-based discovery tends to be best at finding shadow APIs that nobody registered anywhere; code-based discovery tends to be better at catching things before they ever go live.
Does it just find APIs, or also act on what it finds? Some tools stop at inventory and risk scoring. Others, like StackHawk, chain discovery directly into testing — which matters if your goal is closing gaps quickly rather than just measuring them.
Multi-gateway reality check. If your organization genuinely runs more than one API gateway (most do), make sure whatever you pick can actually aggregate across all of them, not just the one your team happens to use.

The Bottom Line

API discovery has moved from a "nice to have" governance exercise to something closer to table stakes — driven less by any single trend and more by the combined effect of AI-accelerated development, AI agents as API consumers, and the simple reality that most large organizations have never run a single, unified API stack.

Whichever category of tool fits your organization best, the underlying goal is the same one teams have always had: being able to answer "what APIs do we actually have running right now?" with confidence — not a guess.

Have you evaluated any of these tools, or run into shadow APIs in your own environment? Let us know in the comments — we'll be covering individual platforms in more depth in upcoming posts.