Bytes down - Azure Cloud horror story

Writing is a catharsis

A good provider, a faithful client

I'm a tech CEO, a tech founder, a flamboyant engineer turned sales turned marketing specialist turned entrepreneur superhero. I build and I ship, just like Naval Ravikant. It would make me unstoppable, he said.

But let's be real. I'm not unstoppable. I'm just a guy selling software to broke SMBs, especially restaurants and retailers. And honestly, I can barely sell, but that’s a story for another day.

I'm so non-unstoppable that I’m doing all the DevOps at my company myself because I can't afford more engineers. But that's fine. I manage the Azure services, and we've been running on their servers for years. Azure offers tons of features, really modern ones. I would describe it as such: horrendous UI, good back-ends, and convenient integrations. My company isn't a new client; we've been running fairly large databases and applications on their servers for four years now. We've paid tens of thousands of dollars, got a full year of credit through the Microsoft graduate program, had pairing sessions with their engineers, and no incidents... until two weeks ago.

When you're a long-term client, with a substantial amount of money paid on time for four years, it's common practice to tell your provider you'll pay on a specific date. When you've complied with those promises for years, it becomes a routine process: "Hello Azure, I have a pending invoice that I can't pay today, but I will next week." They usually give you a date, and you can sleep well. All is fine in the best of worlds... well, usually.

July 10th meant July 6th

Some support engineer based in India told me I had until July 10th to settle the invoice. In any case, large cloud providers, at least serious ones, never shut down critical services without notice. I was assured repeatedly that we had until July 10th to make the payment, which has been our standard routine for years. These assurances came after multiple confirmations from the support team. Despite these assurances, somebody dropped the ball, and as a result, the operations of all out applications were completely ruined over the weekend… right in the Middle of Copa-America, and me going full airplane mode “me time”.

We sell software to restaurants, and the disruption was catastrophic. We lost clients' money and new clients it took months to sign. This wasn't just any application; it was a critical payment and accounting system, and the disruption was severe for my business and theirs. We lost money, they lost money, and we lost clients because they lost clients. I had to call developers DB re-initiation and jobs on a Sunday, and we were still not finished.

To give you a clearer picture: some restaurants use my solutions to pay at the table. When they tried to pay, their whole payment system was showing a blue screen. Small retailers use my software to invoice and sell at the counter; they couldn't do that either. Massive disruption. Big mess.

Spiraling into absurdity

So, picture this: It's the weekend, and my company's critical applications are down. We sell software to restaurants and retailers, and everything is showing a blue screen of doom. Naturally, I try to call Microsoft's support, but guess what? Microsoft blocks support from shut-down subscriptions. Brilliant. So, I have to shell out for premium support from another subscription.

Now, I'm bouncing from one support engineer to another—first in India, then Costa Rica. I'm switching time zones like a jet-setter, but nobody is competent enough to help. It's like a tournament of cluelessness.

In the middle of this chaos, something truly bizarre happens: I start getting spammed by an OpenAI bot named Niro. He's sending me emails, apologizing profusely for shutting down my OpenAI subscription and promising to look into it. Except, I wasn't even contacting OpenAI. Thanks, Niro, but next time.

After four hours on calls and what feels like the entire world checking into my emails, we finally get things reactivated that night. Crisis somewhat averted.

In the end, bureaucracy and incompetence combined to give me a weekend from hell. Microsoft, you've outdone yourself this time… but that’s not finished.

Spiraling into oblivion

A week later, I'm back at it, trying to ensure things stay active. I'm gunning for reparations or at least a delay. But nope, Microsoft says no dice. The only solution is to pay every single invoice right now and ask for a refund. Unfair? Absolutely. But fine, I do it. And guess what? The whole payment platform bugs out. Money leaves my bank, but Microsoft can't confirm they received it. The support ticket vanishes into the ether, nobody calls me back, and I'm left pulling my hair out (I don’t have much left though)

I raise a new support ticket to ask about my payment and subscription status. And oh boy, here comes the kicker: "You should make a payment right now, or it can get shut down." It's a weekday, hundreds of users are relying on the software, and I'm officially an insomniac. So I ask for a delay and try to confirm the payment since they don't accept any other proof. Apparently VMs got shut down randomly, as nobody can tell me when it will get deactivated.

Then things get even crazier. Azure support starts calling me, thrilled to announce my subscription is active! Oh, joy! But can you tell me until when? Turns out they can't really speak English: "Until you pay." The whole ticket said I PAID BUT IT DOES NOT REFLECT PAYMENT. Communication breakdown at its finest.

Then, the absurdity meter goes off the charts. I get a call from their team in the Philippines. "Hello sir, I'm glad to announce you have no deactivated subscription on Microsoft because … as I can see, you have no subscription at all." Wow, my software must be running on magic clouds.

But wait, there's more! I start getting calls from support wanting to close my case because I have no active subscription. I ask them to actually do some work instead of work about work: "No, the mission is important, sir. We need to avoid duplicate tickets and close them." Because, as the other dude said, I have no subscription.

Finally, a glimmer of hope. I get a call from a wonderful agent in Costa Rica. Best support I've had by far. They help close the case with grace and competence. But the saga has left me scarred.

So, Microsoft, thanks for the memories. I'm off to AWS.