After writing an article about my first experience with serverless architecture, I felt pretty good about what I've learned and achieved. However, what happened to me the next morning left me with sour taste in my mouth.
After 2 weeks' holiday I got up this morning and got ready for work. I checked my mails and messages, took a shower and ate breakfast - you know, the daily routine. Then, at 7:43 I received a budget notification from Amazon. At this point, you can already guess what happened.
At first glance, it doesn't seem like much. With $83.28 forecasted, I expected an actual total of about 5$ or less. I quickly logged into the Amazon console, went straight to billing and was greeted with a whopping $800 forecasted cost and a $180 actual cost. I was left with a light headed feeling, it's a lot of money for me - you could probably tell by the budgeted value in the mail.
Not wasting any time, I quickly dug into the logs and found the mistake I made. It's a bug I found while writing my blog post yesterday, and I forgot to change the code and deploy it...
My S3 bucket is set up so that when a new file is added, it triggers a Lambda function that determines the cache duration based on the file type and stores it in its meta data. Because the object is overwritten, it triggers the event again because it's seen as a new object - which in turn triggers the Lambda function again. The check to handle that behavior was present, I checked if the meta data was already there and if so would just terminate the script with a simple return statement. That's exactly the part where it all went wrong, because of a refactor, I forgot the return statement and it just continued overwriting the file again. Thus, creating an infinite loop.
I suspect it had been running since a little before I published my blog post, the logs were full of it.
I thought my day couldn't get any worse, a few hours later, it did. I guess the costs update periodically, because it updated and the cost went up again.
The actual cost is now $206 and over $1000 forecasted, it makes me think twice about using pay-per-use services in the future. One little mistake can cost a lot of money, the budget notifications were very late so there was very little I could do against it.
Right now I'm continuously refreshing the billing page on Amazon, hoping it wont go any higher - I probably wont be sleeping much tonight.
I'm not saying serverless is bad, but you should be very careful with it. Keep an eye on your logs, test everything again and again. Set up a budget alarm, even though I still ended up with $206 costs, it could have been much much worse without one.
Some stats for the people interested, you can clearly tell S3 and Lambda have been very active.
This is probably the most stupid thing I ever did. One missing
return; ended up costing me $206.
Update: I wrote a follow up, be sure to check it out too!