When security teams respond to a breach, they often move fast — sometimes too fast to catch every detail. That was the case for Grafana in May 2024, when a single GitHub workflow token slipped through the company’s otherwise thorough incident response. The missed token gave attackers access to private repositories, stolen source code, and sensitive business contacts. This incident, triggered by the TanStack npm supply-chain attack, offers five critical lessons for any organization using CI/CD pipelines. The story of this breach is also a masterclass in why grafana token rotation demands more than speed: it demands completeness, verification, and a clear-eyed view of your entire workflow inventory.
Lesson 1: One Missed Token Can Undo an Entire Incident Response Plan
Grafana detected malicious activity from compromised TanStack packages on May 1. The company immediately activated its incident response plan, which included rotating GitHub workflow tokens across its environment. A significant number of tokens were changed in short order. Yet one token remained untouched. That single omission allowed the attacker to enter Grafana’s private repositories.
This outcome is not unusual. In complex CI/CD environments, tokens exist in workflows that span multiple repositories, actions, and service integrations. A DevOps engineer may rotate a token in one repository but forget about a fork, a stale branch, or an action that references the same token indirectly. When you rotate 95 percent of your tokens, the remaining 5 percent can still cause a full breach.
Why completeness matters more than speed
Speed during incident response is valuable, but it can create blind spots. Pressure to contain a threat often leads teams to focus on the most obvious or active tokens first. Less conspicuous tokens — those used by workflows that run only weekly or that reside in archived repositories — are easier to overlook. Grafana’s own post-incident review found that a specific workflow originally deemed not impacted was in fact compromised. The assumption that a workflow was safe led directly to the missed rotation.
“We performed analysis and quickly rotated a significant number of GitHub workflow tokens, but a missed token led to the attackers gaining access to our GitHub repositories.” — Grafana incident update
Actionable steps to avoid this gap
First, create a central token inventory before any incident occurs. This inventory should list every workflow token, its purpose, its expiration date, and which workflows or repositories reference it. Second, during a breach response, rotate every token in that inventory, not just the ones you suspect are compromised. Third, use a script or automation tool to verify that no token remains unchanged after rotation. Fourth, run a post-rotation audit that compares the list of all tokens before and after the incident. If any token shows the same value, investigate immediately.
Lesson 2: A Complete Token Inventory Is the Foundation of Secure Grafana Token Rotation
Grafana’s incident reveals that you cannot rotate what you do not track. The company’s initial assessment missed a workflow that had been compromised, which meant its token inventory was incomplete at the time of rotation. Incomplete inventories are shockingly common. A 2023 survey by a cloud security firm found that 37 percent of organizations could not list all their CI/CD secrets, including tokens, across their environments.
For organizations using GitHub Actions, the challenge is especially acute. Workflow tokens can be embedded in YAML files, referenced across repository secrets, or inherited through reusable actions. When a supply-chain attack hits, the affected workflows may not be the ones you expect. The TanStack attack infected dozens of npm packages, and Grafana’s CI/CD pipeline consumed a malicious package that appeared legitimate. The info-stealer module executed inside Grafana’s GitHub environment, exfiltrating workflow tokens to the attackers.
How to build a comprehensive token inventory
Start by using GitHub’s built-in API to list all repository secrets, environment secrets, and organization secrets. This gives you a baseline. Next, audit all workflow YAML files to identify which tokens are referenced and in which contexts. Many tokens are used through the secrets context, but some appear directly in environment variables or as inputs to composite actions. Pay special attention to workflows that trigger on events like workflow_dispatch or repository_dispatch, as these can bypass standard guardrails.
Additionally, scan for tokens in third-party actions. If your workflow calls an action from the marketplace, that action may request permission to access certain secrets. The action you trust today could be compromised tomorrow, as the TanStack case demonstrates. Maintain a list of every external action your workflows depend on and rotate any token that action can access.
Lesson 3: Supply-Chain Attacks Demand Broader Scope Than You Think
The Grafana breach did not originate in Grafana’s own code. It started with the TanStack npm supply-chain attack, part of the Shai-Hulud malware campaign attributed to TeamPCP hackers. Dozens of TanStack packages infected with credential-stealing code were published on the npm registry. Grafana’s CI/CD workflow consumed one of those malicious packages, and the info-stealer module executed in its GitHub environment.
This scenario is increasingly common. In 2024, supply-chain attacks on the npm ecosystem rose by an estimated 42 percent compared to the previous year, according to data from a software supply-chain monitoring platform. Attackers no longer need to find a vulnerability in your application — they can compromise a dependency you trust and let that dependency do the work for them.
Why standard incident response falls short
Most incident response playbooks focus on internal threats: compromised credentials inside your own systems, misconfigured cloud resources, or vulnerable application code. Few playbooks account for a malicious dependency that executes in your CI/CD pipeline and exports your tokens to an external server. The Grafana team responded admirably by detecting the activity on May 1 and deploying a plan. But the plan itself was built around the assumption that the affected workflows were well-known. In reality, the compromised package had infected a workflow that Grafana initially believed was safe.
Widening the scope of your response
After a supply-chain incident, do not limit your investigation to the package itself. Map every workflow that depends on the compromised package, directly or transitively. npm packages can have deep dependency trees, and a package you use indirectly through another dependency may also be infected. Use a software bill of materials (SBOM) to identify all dependencies in your CI/CD pipeline. Then audit every workflow that uses any of those dependencies, not just the one that triggered the alert. This approach would have revealed the missed workflow in Grafana’s environment before the attacker could act on the leftover token.
Lesson 4: Automated Pentesting Tools Cannot Verify Rotation Completeness
Automated penetration testing tools deliver real value. They can answer one question reliably: can an attacker move through the network once they gain initial access? But they were not designed to test whether your controls block threats, whether your detection rules fire, or whether your cloud configurations hold. Most importantly, they cannot check whether every token in your environment has been rotated after an incident.
Grafana’s own guidance on automated pentesting makes this distinction clear. The company built tools to test network movement but acknowledged that those tools do not validate controls, detection rules, or cloud configs. The same limitation applies to token rotation verification. An automated pentest might reveal that an attacker can reach a private repository, but it will not tell you which token is still valid and missing from your rotation list.
You may also enjoy reading: Foxconn Confirms Cyberattack: 5 Facts on Nitrogen Ransomware.
What you actually need to validate
After the Grafana incident, Grafana Labs published a guide covering six surfaces that organizations should validate instead of relying solely on automated pentesting. These surfaces include token inventory completeness, workflow dependency maps, secret rotation logs, detection rule triggers, cloud configuration drift, and access control enforcement. Each surface requires manual review or specialized automation beyond standard pentesting tools.
For token rotation specifically, build a simple script that compares the current values of all GitHub secrets against their values before the incident. If any value remains unchanged, flag it. Then verify that the unchanged token is not used by any workflow that could have been exposed during the supply-chain window. This manual verification step is what Grafana’s original response missed.
Lesson 5: Data Exposure Is Not Limited to Customer Information
When Grafana first disclosed the breach, the company confirmed that intruders stole source code but assured there was no customer impact. The hackers would not receive a ransom payment. Later, the investigation revealed that the intruder also downloaded operational information, including business contact names and email addresses used in professional relationships. This data included details Grafana uses for its business — not customer production data, but still sensitive enough to cause concern.
The company stressed that this was not customer production data and that no customer production systems or operations had been compromised. The codebase was not modified, so downloaded code is safe. Yet the exposure of business contacts and operational details carries its own risks. Attackers can use this information for targeted phishing campaigns, social engineering against partners, or further reconnaissance.
Why non-production data still matters
Many organizations focus their breach response on customer data and production systems. These are legitimate priorities. But operational data — emails, project plans, vendor lists, internal documentation — can be equally valuable to attackers. In Grafana’s case, the exposed business contacts could be used to impersonate Grafana employees or partners. The operational information could reveal internal processes, upcoming product features, or business relationships.
Grafana Labs promised to notify impacted customers directly if the ongoing investigation reveals additional risk. This promise is standard practice, but organizations should extend the same diligence to business partners and employees whose contact data may have been exposed.
Steps to protect non-production data during rotation
First, classify all data that your CI/CD workflows can access, not just production data. Second, ensure that tokens with access to operational data are rotated with the same urgency as tokens that touch customer information. Third, after an incident, review logs for any access patterns that suggest data exfiltration beyond source code. If business contacts were accessed, notify the affected individuals so they can watch for targeted phishing attempts. Fourth, consider segmenting tokens by data classification so that a single token compromise does not expose both source code and business operations.
The Grafana breach shows that a single missed token can cascade into multiple types of data exposure. Protecting customer data is essential, but operational data also deserves vigilant protection.
The five lessons from Grafana’s incident center on one core truth: grafana token rotation cannot succeed without a complete inventory, a wide investigative scope, verification beyond automated tools, and protection for all data types. Speed in incident response matters, but completeness matters more. A single oversight — one workflow deemed safe when it was not, one token left unchanged — can undo hours of diligent work. Organizations that build token inventories before an incident, audit workflows after a supply-chain attack, validate rotation completeness manually, and protect both customer and operational data will be better prepared to avoid the same fate.
