Don’t get burned by cloud providers: 3 keys to prep for an outage

Yes, the Cloud is great for companies looking to save money, implement business solutions quickly and efficiently, and respond rapidly to changing business conditions – but it’s not without risks. One recent incident shows that IT’s job doesn’t end when a cloud provider takes over.

Recurly provides subscription billing services, credit card storage and related services to companies that do business over the Internet. On Monday, its primary encryption device failed. The problem then cascaded to the backup slave device. In the process, the encryption keys protecting the credit cards used to process subscriptions were corrupted.

The crux of the problem: Recurly didn’t back up the encryption keys needed to access the billing info.

After the hardware failure, the vendor was unable to access the billing info to process payments because the encryption keys were either wiped out entirely or corrupted.

And since there were no back-ups available, the process of restoring the service that processes the recurring payments has been slow and painful. As of September 12, some customers’ data has been fully restored, some will be restored within the next few days — and the rest are out of luck. They will have to get in touch with their customers and ask them to re-enter their billing info.

Judging by the comments scattered across various IT news sites, Recurly’s customers are pretty upset. So how do you avoid getting burned by your cloud provider?

  • For one thing, it’s vitally important that you back up the data you put in the Cloud. Check to make sure you have incremental, functioning backup copies. Don’t assume anything.
  • Also, encrypt backup copies of critical business data, such as your customer’s credit cards, and don’t forget to back up the encryption keys.
  • Finally, even if you carefully vet each one of your cloud providers, hardware failures are a fact of life. You’ll be better off if you dump the “it’ll never happen to me” mindset and come up with a plan of action before a problem occurs.