37 Million Customers Lost, 41 Days of Invisible Attack: Rethinking API Security from the T-Mobile Case
January 2023. T-Mobile disclosed an API data breach affecting 37 million customer accounts. The attacker had begun “unauthorized” use of an API on November 25, 2022. For 41 days, no one noticed. Names, addresses, phones, emails, birth dates—everything leaked. How did this happen? The answer is simple but painful: They could not protect an API whose existence they did not know. Salt Security’s analysis is clear: This was a “Shadow API” case. Security teams were unaware of this API’s existence, the data it processed, and its security posture. It was outside the scope of the API management and security program. And this was T-Mobile’s 8th breach. Since 2018. They had agreed to pay $350 million for a 2021 breach. But the real question is: Didn’t T-Mobile have an API Security product? They probably did. So why didn’t it work?Field Reality: “Should We Buy an API Security Product?”
In meetings over the past few months with many banks, public agencies, and defense industry customers—with security teams, infrastructure, system architects, service managers, and enterprise architects—I saw a common pattern. The same question: “Should we buy an API Security product?” From what they said: they had attended many presentations, run PoCs, but still had no clear answers. Every API Security vendor tells a different story; they couldn’t tell which was right. Some talk about behavior analysis, some API Discovery, some promise ML-based threat detection, some say “full visibility” but don’t (or can’t) explain in detail with real scenarios. Some even claimed they “catch problems in mid-air” (they literally told this to one of our customers). I asked the same question in every meeting: “When you say API Security, what do you mean and what do you expect?” I asked this not only to active customers but also to experts when the topic turned to security. When we went into details, their real needs were:- To know all APIs exposed and consumed inside and outside the organization (lately they also want to master internal API inventory).
- To master API inventory.
- To know who owns and is responsible for each API (or at least who to hold accountable when something goes wrong).
So What Exactly Happened at T-Mobile?
For 41 days the attack went undetected because:- The API was outside central management (shadow API).
- It did not go through a gateway.
- There was no visibility.
- No behavioral baseline had been built.
- Anomaly detection was not in place.
“No API Security product can see traffic that does not pass through an API Gateway. It cannot protect what it cannot see. It is useless for Shadow APIs.”So the order is critical:
- Gateway first → All traffic through a single point.
- Then Management → Build inventory, assign ownership.
- Security last → Only then does protection make sense.
The Hard Truth: How API Security Products Work
A crucial point must be understood here. API Security products are not blind, but they need eyes to see. They need:- To operate by observing API Gateway or WAF traffic.
- To analyze HTTP request–response pairs.
- To learn normal behavior patterns.
- To detect anomalies.
- API Discovery (which APIs exist?)
- Behavioral Analysis (what is normal?)
- Anomaly Detection (where is the deviation?)
- Shadow API Detection (where are unregistered APIs?)
- Threat Intelligence (what are the attack patterns?)
Is the SOA Story Repeating?
When I saw this situation, my years of experience on the SOA side came to mind. Back then everyone was talking about SOA. Global vendors sold products under the SOA name. Enterprise Service Bus (ESB) products, SOA governance platforms, service registries… Millions of dollars were spent. In the end, most projects built with those products failed. The ones that continued did so because no one dared touch them. Why did they fail? Because they trusted the product and said “the product will solve everything.” Global vendors marketed their products very well as SOA. Sales presentations were great. Demos were impressive. But the forgotten truth was:SOA (Service-Oriented Architecture) is not a product; it is an architectural approach or a philosophical paradigm.Like Lego pieces, you have to build it. You must first understand the architectural principles, then define your governance model, then build your infrastructure step by step. Buying a product did not mean having an SOA architectural approach. Is the same movie now repeating in API Security? Many organizations act with this logic:
SOA is not a product; it is an architectural approach for designing systems.
- “Let’s buy an API Security product first; security is the priority.”
- “We’ll think about the gateway later.”
- “Inventory will emerge gradually.” In other words: we’ll figure it out on the way. (That may work for non-technical topics, but in technical architecture it often ends in disaster.)
What About the Data?
It’s not only field observations; the data points to the same reality: Gartner 2024 Market Guide for API Protection:“Shadow and dormant APIs cause, on average, larger data leaks than other breaches.”Salt Security & LevelBlue Research (2024):
“94% of organizations experienced a security issue in production APIs in the past year. Many businesses have limited visibility into API inventory.”Dark Reading (November 2024):
“The most urgent problem for organizations is the lack of inventory of all external APIs. Attackers are exploiting this gap.”OWASP API Security Top 10:
“#9 Improper Inventory Management: Poor management of API assets. Lack of awareness of all APIs leads to security vulnerabilities.”The pattern is clear: Without Gateway and inventory, API Security products stay blind.
PART 2: The T-Mobile Case – Deep Dive
41 Days: Anatomy of an API Breach
The T-Mobile case is an excellent case study of how “theoretical risks” in API security turn into real disaster. Here’s what happened chronologically: November 25, 2022 – Day Zero: The attacker gained unauthorized access to one of T-Mobile’s APIs. But this was no ordinary API. It was a “Shadow API”—unregistered, undocumented, and unknown to security teams. November 26 – January 4 (40 days): Silence. Complete silence. The attacker accessed 37 million customers’ data every day, systematically: names, billing addresses, emails, phone numbers, birth dates, T-Mobile account numbers. Security teams? Unaware. SIEMs did not alarm. API Security products were silent. Because we’re talking about an API they couldn’t see. January 5, 2023 – Detection: After 41 days, an anomaly was detected. How? Details were not fully disclosed; likely through another security layer or manual check. January 6, 2023 – Stop: T-Mobile stopped the attack only 1 day after detection. Notable point: If it could be stopped in 1 day once detected, why was it invisible for 41 days?Root Cause: Shadow API and API Sprawl
Salt Security’s in-depth analysis made the problem clear:“An API whose existence is outside the scope of the API management and security program, undocumented. Security teams are unaware of these APIs’ existence, the data they process, and their security posture.”This is not just “we forgot an API.” It’s the result of a systematic problem called “API Sprawl”:
1. Shadow APIs
- Definition: Unregistered APIs that no one knows about.
- How they appear: Developers create them for testing and forget; legacy, undocumented endpoints; uncontrolled growth in microservices; unintegrated APIs from acquired companies.
- At T-Mobile: This category caused the breach.
2. Zombie APIs
- Definition: APIs thought to be unused but still active.
- Why dangerous: No security updates, no monitoring, known vulnerabilities, thought “disabled” but still running.
3. Ghost APIs
- Definition: APIs that exist in documentation but behave differently in reality (teams integrating often suffer from “Internal Server Error” and similar mismatches).
- Problem: Gap between documentation and real behavior; security tests based on docs miss real vulnerabilities; version mismatches. Widespread use of an API Developer Portal helps with these issues but not with inventory.
Scale of API Sprawl
The T-Mobile case is not isolated. Salt Security and LevelBlue’s 2024 research shows:- 94% of organizations had a security issue in production APIs in the past year.
- Most businesses have limited visibility into API inventory.
- The average enterprise has 200–500 unregistered APIs.
8 Breaches, $350 Million
T-Mobile’s breach history: 2018 (2M customers), 2019 (prepaid, another breach), 2020 (employee emails, customer data), 2021 August (54 million—largest), 2022 April (SIM swap), 2022 Nov 25 (our case starts), 2023 January (37 million—Shadow API breach). Cost of the 2021 breach: T-Mobile agreed to pay **150 million to cybersecurity. Question: Despite that payment and investment, how did the same type of breach happen again in 2023?Why Did It Repeat?
Salt Security’s analysis is clear: Lack of visibility (incomplete API inventory), no gateway (traffic not centralized), governance weakness (no API lifecycle management), Shadow APIs (continuously created, not detected). T-Mobile likely did buy API Security products, strengthened the security team, ran penetration tests, and did compliance work—but did not build a central API Gateway, inventory all APIs, create a mechanism to detect Shadow APIs, or institutionalize API lifecycle management. Result: They bought security products but did not give them “eyes to see.” Shadow APIs stayed invisible and the attack continued for 41 days.3 Lessons from the T-Mobile Case
1. Spending money ≠ Being secure. You can spend $500 million, but if you don’t follow the right order, you repeat the same mistake. 2. Visibility comes first. An attack invisible for 41 days = missing Gateway and inventory. You must see first, then protect. 3. API Sprawl is real. Shadow, Zombie, and Ghost APIs keep appearing. You need systematic processes to curb them, not just tools.PART 3: 3 Critical Questions Before Buying an API Security Product
T-Mobile paid 150 million in cybersecurity after the 2021 breach. They probably bought top API Security products. They still had the same type of breach in 2023. The problem was not the products. The problem was the foundations. If you’re also debating “should we buy an API Security product?” in your organization, ask yourself these 3 questions first.QUESTION 1: Do You Have an API Inventory?
”How many APIs do you have?”
Answers I got in the field:- “We don’t know exactly but around 200” (Turned out there were 847.)
- “Each department has its own APIs; no one knows the total.”
- “We don’t even know if we have APIs in our legacy systems.”
- “Since we moved to microservices, we’ve lost control.”
No Inventory, No Security
An API inventory is not just an Excel list. A complete inventory should include: name & version, endpoints, HTTP methods, hosting; owner (Product Owner), technical lead, business unit, lifecycle state; authentication, authorization, data types (PII, PCI), compliance (GDPR, KVKK, PCI-DSS); tech stack, dependencies, SLA, rate limiting.What Happens When You Buy API Security Without Inventory?
Real scenario: A financial institution bought a 2M spent, most APIs still invisible.“Buying API Security without inventory is like running a police patrol in a city with no map. You don’t know which streets to protect.”
QUESTION 2: Do You Have a Central API Gateway?
”Does all your API traffic pass through a single central point?”
This question is even more critical. API Security products work by observing Gateway traffic. Distributed architecture (highest risk): Each API exposed differently; no central visibility. Central Gateway architecture: All traffic through one point; full visibility.Why API Security Stays Blind Without a Gateway
Principle: (1) Traffic monitoring, (2) Baseline learning, (3) Anomaly detection, (4) Threat intelligence. This loop requires SEEING the traffic. In T-Mobile: the Shadow API didn’t go through the gateway; the API Security product couldn’t see the traffic; it stayed blind for 41 days; no behavioral baseline; no anomaly detected.”We Use a WAF; Isn’t That Enough?”
No. WAF and API Gateway are different. WAF: Layer 7 attacks (SQL injection, XSS), pattern-based, not API-aware, cannot build API inventory, cannot detect API-specific attacks (BOLA, Mass Assignment). API Gateway: manages API traffic, rate limiting/throttling, builds API inventory, request/response transformation, gives API Security products full visibility.“Assuming you’re secure because you have a WAF is like assuming you know every room in the building because there’s a security guard at the door.”
The Numbers
Gartner 2024: Shadow API rate 68% without central Gateway, 23% with. That’s a 3x difference.QUESTION 3: Can You Distinguish Internal and External APIs?
”Which of your APIs are exposed to the internet and which are internal only?”
This seems simple but for most organizations it isn’t. Real case: An e‑commerce company said “We only have 12 external APIs. The rest are internal.” After a network scan: 47 APIs were exposed to the internet; 35 were exposed “by mistake”; developers had opened them for testing and forgot to close; some had been open for years. External APIs: Intentionally exposed, usually better protected, monitored, documented. Internal APIs (accidentally external): Minimal security, designed “for internal use,” weak or no auth, not monitored—a gold mine for attackers.OWASP API Security Top 10 #9
Improper Inventory Management: API hosts and deployed versions not properly managed; old versions still running, deprecated APIs not removed, no or outdated inventory, unclear internal/external split.How Shadow API Detection Works
(1) Analyze Gateway traffic, (2) Compare with inventory, (3) Flag endpoints not in inventory, (4) Behavior analysis. This requires: a Gateway (traffic visibility), an inventory (reference), and internal/external classification (risk level). Otherwise the product only detects what it sees; the 360 unseen APIs remain Shadow.BOTTOM LINE: Saying NO to All 3 = Money Lost
| Question | If NO, result |
|---|---|
| Do you have an API inventory? | API Security product cannot know what to protect. No reference. Shadow APIs cannot be detected. |
| Do you have a central Gateway? | API Security product cannot see traffic. Behavioral analysis doesn’t work. Anomaly detection impossible. |
| Can you separate internal vs external? | Risk level cannot be set. Accidentally exposed APIs cannot be found. No prioritization. |
PART 4: What the Data Says – Industry Research and Statistics
Reports from the last 12 months lay bare an uncomfortable truth: We are in an API security crisis, and the problem is foundational, not technical—it won’t be fixed by a product. And it’s a clear sign that the time for a culture change in your organization may have already passed. Summary figures: 94% had production API security issues; 71% don’t have 100% accurate inventory; 68% have Shadow APIs; 62% increase in API attacks; 48% have incomplete or outdated API inventory; 40% of web attacks via APIs (2024); 31% leaked data via APIs; 3x difference in Shadow API rate between those with and without a Gateway. Conclusion: This is not a few “bad examples.” It’s a systematic, widespread, growing crisis. Root cause is the same everywhere: No API inventory, no central Gateway, no visibility, Shadow APIs out of control. Result: Data breach, customer loss, reputation damage, legal penalties, millions in losses.PART 5: “You Can’t Protect What You Can’t See” – Why Visibility Is Critical
A golden rule in cybersecurity:“You can’t protect what you can’ see.”T-Mobile couldn’t see the attack for 41 days. Why? Because they couldn’t see that API. Three levels of visibility:
- Asset Visibility — Which APIs exist? Count, hosting, endpoints, internal/external, versions.
- Traffic Visibility — How are your APIs used? Who calls, how often, parameters, response times, error rates. Baseline for behavioral analysis and anomaly detection requires seeing traffic.
- Data Visibility — What data do your APIs process? PII, financial, health, credentials, GDPR/KVKK/PCI-DSS. OWASP API3:2023 Excessive Data Exposure: APIs may return more data than needed; frontend uses 3 fields but backend sends 20. Without Data Visibility you can’t detect this.
PART 6: Gateway → Management → Security – Step-by-Step Roadmap
Wrong order: Buy API Security → Deploy → Only 30–40% of APIs visible → Shadow APIs not detected → Low ROI, disappointment → Project shelved. Right order: STEP 1: GATEWAY (Visibility) → STEP 2: MANAGEMENT (Inventory & Governance) → STEP 3: SECURITY (Protection & Anomaly). Each step builds on the previous. Skipping = failure.STEP 1: API Gateway – Foundation (0–6 months)
Goal: 100% visibility. Gateway selection and setup (months 1–2): HA, scalability, rate limiting/throttling/caching, monitoring/logging, security tool integration, Kubernetes/OpenShift, APIOPS/CI-CD. Quick wins: central auth, rate limiting for DDoS, basic logging. API migration (2–4): Priority: External (highest risk), PII (GDPR/KVKK), high-traffic, legacy (shadow risk), internal. Discovery & baseline (4–6): All APIs through Gateway; learn traffic patterns; build baseline. Output: 100% Gateway coverage, central visibility, baseline ready.STEP 2: API Management – Inventory & Governance (6–12 months)
Goal: Control and governance. Don’t skip this even if you already use a Gateway. API inventory (6–8): name, version, endpoints; owner, tech lead, business unit; data classification; SLA, rate limits; auth; compliance. Lifecycle (8–10): Design → Develop → Test → Deploy → Monitor → Deprecate → Retire. Governance: new APIs must be in inventory; deprecated closed in a controlled way; version standard; documentation mandatory. Shadow API elimination (10–12): Gateway traffic vs inventory; find Shadows; assign ownership; integrate or shut down. Output: Up-to-date inventory, clear ownership, Shadows eliminated, governance in place.STEP 3: API Security – Protection & Anomaly (12–18 months)
Goal: Proactive security. Now buy an API Security product. You have 100% visibility, inventory, baseline, and governance. Platform selection (12–13): Gateway integration, behavioral analysis, OWASP API Top 10, threat intelligence. PoC on production traffic, Shadow detection rate, false positives, latency impact. Deployment & tuning (13–15): Integrate, baseline learning (2–4 weeks), policies, alert tuning. Operationalization (15–18): SOC training, playbooks, incident response, KPIs. Output: Attack detection in minutes, not 41 days. Final advice: Don’t say “the product will fix everything,” “let’s skip a step,” or “security first, foundations later.” Move step by step; don’t break the order Gateway → Management → Security. “18 months may seem long. But T-Mobile had 8 breaches in 8 years and paid $350M+. Which path do you choose?”PART 7: Tomorrow May Be Too Late – Check Today
“Nice article but it won’t happen to us. We’re different.” T-Mobile thought so too. Even after paying $350 million. Are you really different? Ten simple questions: Can you say how many APIs you have in 5 minutes? Have you said “Where did this API come from?” in the last 6 months? For a random API, who’s the business owner? Do all APIs go through one Gateway? Do you have APIs written 3+ years ago still running? New vendors/integrations? Can every team deploy their own API freely? Does your Security product see all APIs? Any API security incident in the last 2 years? Do you sleep well at night? Your answers show the real picture.The One Thing I Learned from SOA
Twenty years ago with SOA we said “the product will fix it,” tried to skip foundations, were impatient—and failed. We’re making the same mistake in API Security. But one thing is different: The API economy is far more critical than SOA. We don’t have the luxury of getting it wrong.Final Word
“The most dangerous API is the one you don’t know."You can buy the best API Security product. You can spend millions. But if your APIs don’t go through a Gateway, that product only protects what it sees—not what it doesn’t. Can you see 100% of your APIs? Can you detect your Shadow APIs? Does all API traffic pass through a single point? If you answer NO to these, you’re very close to T-Mobile’s 41 days.
"The most expensive vulnerability is the one you don’t see."
"The biggest mistake is saying ‘it won’t happen to us.’”

