Aikido

Google API keys keep working after you delete them

Written by
Joe Leon

tl;dr When you delete a Google API key, it says it’s immediately deleted. Our testing says ~23 minutes. During that window, an attacker with a leaked key keeps access to your data and enabled APIs (including Gemini). You have no way to revoke it faster or confirm when it stops working. Google closed our report as “won’t fix”.

When you delete an API key, you expect access to end immediately. 

Google API keys don’t work this way. Revocation propagates gradually across Google's infrastructure. Some servers reject the key within seconds, others keep accepting it for 23 minutes.

An attacker holding your deleted key can keep sending requests until one reaches a server that has not caught up. If Gemini is enabled on the project, they can dump files you have uploaded and exfiltrate cached conversations.

The GCP console will not show the key, and it will not tell you the key is still working. You are trusting Google's infrastructure to eventually catch up.

Authentication should not be eventually consistent

Many of Google Cloud's services are eventually consistent by design. In this model, updates propagate gradually across its servers rather than all at once. This tradeoff lets Google scale globally and stay fast, and for most services, the lag is invisible. But for authentication, that tradeoff is harder to justify.

Credential revocation delays are exploitable. A few months ago, Eduard Agavriloae disclosed a 4-second delay that enabled deleted AWS access keys to create new credentials. Four seconds was enough to matter on AWS. 

Given recent attention to Google API keys used to access Gemini, we set out to measure how long Google's API key revocation window remains open.

What is a revocation window?

The revocation window is the time between when you delete a key and the last successful authentication. 

The revocation window is the time between the key’s deletion and the last accepted request

If the window is a few microseconds, the behavior matches what users expect. If it’s longer, every extra second gives attackers more time to misuse a stolen key.

Measuring the revocation window

To measure Google’s API key revocation window, we ran 10 trials over two days.

In each trial, we created an API key, deleted it, and sent 3-5 authenticated requests per second until no valid response came back for several minutes.

We picked that rate because we don’t control how Google routes our requests to different authentication servers across the world. The volume was meant to rotate through as many of them as possible per trial, while still respecting Google's infrastructure. We can't say whether a higher request rate would extend the observed window. An attacker has no reason to throttle the way we did, so it's worth noting our numbers may not represent the worst case.

For completeness, we also spot checked our work a few weeks later to ensure the behavior we saw was not due to transient network issues.

23 minutes

Across ten trials, the longest window was nearly 23 minutes! That is an incredibly long time for a deleted key to still authenticate successfully.

The shortest window was nearly 8 minutes, and the median was around 16 minutes. Across trials, the success rate was highly unpredictable: one minute after deletion, one trial saw 79% of requests succeed while another saw only 5%.

This chart plots the percentage of authenticated requests succeeding by minute for each trial.

An attacker holding a stolen key does not see a clean cutoff or a predictable decay. Access keeps working, unevenly, until it eventually just stops. 

Tracing a single trial in GCP’s console

To illustrate the inconsistency in Google's infrastructure, we monitored one of our trials using the "Traffic by Credential" graph in GCP. The lower (blue) line shows successful authentications and reflects the length of the revocation window.

This chart plots requests per second during our testing window. The upper line shows invalid API requests, and the lower line shows valid requests.

We didn't expect to see any other data, but a second line appeared. The upper (green) line plots rejected requests and is labeled apikey:UNKNOWN. We had (wrongly) assumed that invalid requests would simply be dropped without any project attribution. Instead, GCP includes them in this graph.

To better understand the mysterious apikey:UNKNOWN value, we attempted to authenticate with a days-old deleted API key. Surprisingly, those requests showed up in the same graph, bundled into the same apikey:UNKNOWN group.

Our best guess is that Google preserves project attribution across key deletion in case users decide to restore the credential. 

Screenshot of the “Restore deleted credentials” button in GCP

For IR teams investigating a leaked credential, the apikey:UNKNOWN value can be confusing. Any request made with any deleted Google API key gets bundled into the same UNKNOWN category, making it hard to tell which requests relate to a particular credential. A request in that bucket could mean a threat actor is still trying to authenticate with the deleted key, or it could be a legitimate service running on an unrelated, outdated one.

If you're inside the 23-minute revocation window, look for valid authentications using the leaked key. If you're outside that window, the risk appears extremely low.

Regional differences in consistency

We ran the first experiment from a residential IP address on the East Coast of the US. We hypothesized that running a similar experiment across VMs in different GCP regions might expose additional inconsistencies.

We spun up a VM in three regions: us-east1, europe-west1, and asia-southeast1. We then ran 5 trials. For each trial, we deleted a single API key and sent requests from each of the three VMs in parallel.

Two things stuck out. First, immediately after a key is deleted, VMs in different regions saw very different authentication success rates. The table below maps the percentage of valid requests in the first minute by region.

This table shows the percentage of valid authentication requests in the first minute of testing across all 5 trials broken out regions.

Look at trial 1: us-east1 82%, europe-west1 60%, asia-southeast1 32%. VMs further from the US picked up the deletion faster, which is the opposite of what you'd expect. We can't say exactly why from the outside. Google's request routing is more complex than "VM region equals server region," and a VM in Singapore isn't necessarily talking to servers in Singapore. But the pattern was consistent across trials, which points to something about regional infrastructure, caching, or routing affinity driving the difference.

Second, the regional gap was not just a first-minute thing. Across the full window, asia-southeast1 had a median request authentication success rate of 22%, while us-east1 and europe-west1 both came in around 49%. Asia stayed lower minute by minute, not just at the start.

Whatever's driving these differences, the server’s location clearly shapes how a deleted key behaves after deletion.

Other Google credentials

Our trials all used keys with access to Gemini, but we observed the same behavior with keys scoped to other GCP APIs, such as BigQuery and Maps. The delay is a property of the credential type, not of which APIs are enabled on the project.

For completeness, we tested two other Google credential types:

  • New Gemini API keys (AQ. prefix). Deletion propagated in ~1 minute. 
  • Google Service Account keys. Deletion propagated in ~5 seconds.  

Both run at Google scale, and both revoke much faster than the 23 minutes we measured for Google API keys. 

Disclosing to Google

We reported this to Google. Google closed the report as “won’t fix”. The team's position, as we understand it, is that propagation delay is a known property of the system and not a security issue.

While Google publicly documents that their IAM API is eventually consistent, they don’t specifically document this behavior for Google API keys.

[Caption: Screenshot from Google’s IAM API documentation page.]

Broken User Expectations

Distributed systems at Google's scale are hard, and this is not a critique of the GCP IAM team. But a 23-minute revocation window is fundamentally at odds with what users expect from a delete button. The gap between that expectation and Google's behavior highlights four problems:

1. The UI is extremely misleading. 

When you delete the key, Google removes it from your view and tells you "Once deleted, it can no longer be used to make API requests." That statement is demonstrably false. The user has no way to know whether the key is still live, no way to speed up revocation, and no way to confirm when it has fully stopped working. 

Screenshot from Google's current API key deletion dialog. It say that once deleted, keys can no longer be used to make API request. But you can restore your credential from the deleted credentials page.

2. Google built faster revocation for other credential types. 

Service account credential revocations propagate in about 5 seconds. Gemini's newer API key format propagates in about a minute. Both run at Google scale. Both suggest this is technically solvable for Google API keys, too. 

3. Long consistency windows are not compatible with authentication. 

The expectation when you delete a credential is that the credential is dead. Even a few seconds of delay matters, as Eduard's AWS research showed last year.

4. Long revocation windows break Just-in-Time credential minting. 

A service provider that wants to dynamically mint Google API keys has to build in a 23-minute buffer after revocation before the key is guaranteed dead. That is incompatible with how JIT is supposed to work.

Working around the window

Until Google ships faster revocation, the responsibility for closing this gap falls on users. Two things help.

1. Treat key deletion as a 30-minute operation, not an instant one. 

If you are responding to a leaked Google API key, assume the key is live for 30 minutes after clicking delete. That gives a bit of a buffer beyond the maximum 23 minutes we observed. Plan the rest of your incident response around that window.

2. Watch usage during the window. 

Under "Enabled APIs and services" in the GCP console, review API requests by credential. If you see unexpected usage from that credential after deletion, someone could be actively exploiting it.

Large eventual consistency windows are the wrong design choice for credential revocation. Until Google changes that, treat every key deletion as a 30-minute operation and watch the window for misuse.

Share:

https://www.aikido.dev/blog/google-api-keys-deletion

Subscribe for news

4.7/5
Tired of false positives?

Try Aikido like 100k others.
Start Now
Get a personalized walkthrough

Trusted by 100k+ teams

Book Now
Scan your app for IDORs and real attack paths

Trusted by 100k+ teams

Start Scanning
See how AI pentests your app

Trusted by 100k+ teams

Start Testing

Get secure now

Secure your code, cloud, and runtime in one central system.
Find and fix vulnerabilities fast automatically.

No credit card required | Scan results in 32secs.