Yesterday, January 2nd, between 5:00 and 8:00 PM PST, if you were on EEBA’s website you may have noticed that every time you attempted to create a new transaction or edit one, you were getting a “Whoops” message saying an error occurred. If you were attempting to register an EEBA account (either on the website or via the device), you would have also been unable to. Thanks to all of those that let us know about the issue.

We wanted to take a moment to explain what happened and what we’re doing to make sure it doesn’t happen again.

The Problem

We discovered that a disk on a database server had filled up and was causing the errors when attempting to update the database. Our automated monitoring did not detect the issue until it became a problem. A few notes:

1. You were able to browse the website and see your transaction history, envelopes, etc.
2. But, once you tried to create or edit a transaction (or an account or update an envelope), you would have gotten an error.
3. There was no data lost. On the website you would have known that you encountered the problem because you would have gotten a message saying that the edit you made was not saved.
4. If you were entering or editing transactions on the device, these were sync’ed once everything was back up and running properly. That’s one of the benefits of EEBA’s full offline capability in the app. The app acted the same way as it would if your data connection bad. You might not have noticed anything at all. Or, you may have noticed that the app reported trouble syncing to the server (and that it would retry).

Here’s what we’re doing

We’re changing our monitoring systems to provide earlier detection of issues like this and reviewing other steps that we can take to ensure that this doesn’t happen again.

We know that you rely on EEBA to help keep you on budget and we’re committed to keeping the service highly available. Sorry for the inconvenience and feel free to let us know if you have any questions!