SporeStack Blog

Server Topup Bug

On 2024-09-02 we were alerted about a serious (but rare) bug in our server topup logic. It was fixed shortly after (basically the same day in UTC time) and should not be a problem moving forward.

Quick Summary

Impact

Conditions

Servers being topped up to have more than one year before expiration.

Problem

Token funds were lost during the topup operation, despite the expiration not being extended.

Explanation

Server topup requests between 2023-10-02 and 2024-09-02 that received a HTTP 422 response with a message saying that server expirations can't exceed one year would remove the funds that would have been spent.

If you've never had a server with expiration close to a year out, you should not have been impacted by this. In the previous 90 days, only the customer who reported this was impacted by the bug. We are unable to determine if other customers were impacted.

If you may have been affected by this or are curious, please read on!

Details

We have a long time customer who noticed some pecuilar behavior around server topups. With the customer's report, we were able to track it down to a pretty serious bug, although one that few people would run into.

On 2023-10-02 we changed server topups to only allow up to one year out and no longer. This can be seen in our API changelog.

Unfortunately, the bug is due to logic ordering that we would attempt to draw funds from the token first, then throw an error if the expiration was more than a year out. If you had funds to go more than a year out but that topup would have put your server over the one year mark, we "ate" the funds and threw an error.

This is certainly not what we meant to do. We try very hard to make sure token funds are accounted for and don't "vanish" mysteriously.

Unfortunately due to our 90 day data/log retention, we could not be sure that no one else ran into the same issue. In the previous 90 days, no other customers were affected. It's possible that between 2023-10-02 and 2024-06-04 that other customers were impacted, but we have no way to determine this for certain. It's not likely to be many, but we'd still like to make it right with customers who reasonably believe they were impacted by this bug.

If you attempted to topup a server between 2023-10-02 and 2024-06-04 past one year, while having the funds to do so, please send us a token message (preferred) or an email (support at this domain).

Please note that the default max for a topup is 90 days. Thus to approach a one year expiration, you would have to issue several topup requests to approach the one year mark.

We recommend anyone wanting their servers to last until intentional deletion (if ever) to use autorenew. When you use autorenew on your servers that you intend on keeping around, your token stats will show the daily cost of servers and how long until funds run out. This makes it very easy to project with multiple servers how many funds you'll need. It's also safer because if autorenew fails to maintain a week's worth of expiration, the server is powered off. This gives you a signal for a whole week that funds have run out before deleting the server.

Root Cause Analysis

In larger companies a Root Cause Analysis or post-mortem is often given after serious downtime or bugs.

In this case, the bug was a simple change in logic that appeared to be fine, but was flawed.

The most likely way to catch the bug would have been to make sure funds were not drawn after such a topup request. We have a test case like that in now, and normally have a wide range of test coverage ranging from obvious to paranoid.

We take our code quality very seriously. We currently have 83% test coverage. Not that percentage of test coverage is the only indication of code quality, but it's an indicator as to how much time is spent writing tests and trying to insure the stability and trustworthiness of SporeStack.

The nature of the bug and forgotten test case is that the most likely way we could have caught the bug is with peer code reviews. Unfortunately, SporeStack is not currently large enough to be able to have peer code reviews as we don't have enough staff for it. This might lead the astute reader to assume that "we" is "I" and that would be correct. We have had help in the past and it was my mistake to not encourage more code reviews as part of the process. But largely it's been a one man show, reviewed by the same man. It's not ideal for software quality. To our knowledge this is the first time funds have been lost by a bug since we launched in 2017.

I/we am very sorry for this mistake, however. Customer trust is very important to us. We can't be certain there's no other possible fund-eating bugs in SporeStack, but I will say that it's pretty thoroughly used, tested, and even this bug was pretty surprising. A shame though, nonetheless, and hopefully one that won't happen again.

In this instance while there is a test case to cover this particular bug moving forward, the nature of the bug wasn't one where one exact change could reliably prevent future occurences. I hope that we can have more staff when business is high enough to warrant such, and at that time we can improve peer review.

In the mean time we have been continually improving code quality and stability. This may sound like a cop-out, but we really do take this seriously and it appears that unfortunately this has come down to human error of a nature that we can't entirely prevent moving forward, other than making blanket statements like that we will be more careful, as if we haven't been trying to be already.

We are very sorry, though. This is certainly never what we intended to happen and hopefully very few people/tokens have been impacted.

Possible Questions

This would not have occured under normal autorenew operation. Autorenew will maintain an expiration of one week out.