Huge Security Hole in Sendgrid

We had a serious security incident over the weekend that took us by surprise. Although no customer accounts were compromised, it was a really close call.

Like lots of companies that want to be sure our emails successfully arrive in your inbox, we use a third-party email service, Sendgrid, to ensure deliverability.

A few weeks ago, we had received a transcript of a chat with Sendgrid tech support that was clearly someone trying to social engineer access to our account. Though Sendgrid didn’t fall for that attempt, we alerted them to the probing and asked them to please make sure that future social engineering attempts wouldn't work. They replied and set our minds at ease:

As a policy, we will never change an account's credentials or email address for a user, especially over a chat or email ticket.

We will provide the links or instructions for the user to do so, but those pages can only be accessed with the proper credentials.

However, it turns out that the policy was ignored this weekend, and someone managed to convince Sendgrid over the phone to change the email address on the account. We got an email from them, but by that point it was already too late. The hacker had logged into Sendgrid and taken control.

He had registered a domain, chunkhost.info, and “Would you please change our email from support@chunkhost.com to support@chunkhost.info?” sounded convincing enough that Sendgrid went through with it without bothering to verify anything.

So why would someone want to take over our Sendgrid account?

Sendgrid has a feature that allows you to BCC every outgoing message to a separate email address. Once they activated that feature, they initiated password resets on the two accounts they were after, both of which are Bitcoin-related.

The password reset email was indeed delivered to our customer, but also BCC'd to the attacker. With the password reset link, they could change the password and access our customers' accounts.

Luckily, the affected customers were both using our Two-Factor Authentication feature. This means you not only need a password, but a token generated by your phone to log in.

Our customers' accounts were protected and the attackers were stymied. But it was really close.

Within about 20 minutes, we’d noticed what was happening and blocked their access. We disabled password resets, reset all sessions, and switched to local mail relaying. Once we were confident that things were locked down, we had a back & forth with Sendgrid. Yesterday, they told us this:

It appears that the email address on file was changed...to support@chunkhost.info by our system, which pretty much confirms your suspicion that these people convinced one of our representatives to change the email address on file. After the email address was changed, they were able to simply request a new password and gain account access. This should have never happened and we take things like this very seriously. I apologize that you've had to deal with this and I will make sure that we re-iterate with out [sic] staff that we have policies like that in place for a reason.

Be careful with third-party mail senders!

We are continuing to send our own email while we explore other options, but other companies should take notice and not make the mistake we did. If your accounts are ever a target for break-ins (especially if you do anything related to bitcoin!), protect yourself and your customers by sending your own mail.