Heroku

Heroku has now revealed that the stolen GitHub integration OAuth tokens from last month further led to the compromise of an internal customer database.

The Salesforce-owned cloud platform acknowledged the same compromised token was used by attackers to exfiltrate customers' hashed and salted passwords from "a database."

Heroku's update comes after BleepingComputer reached out to Salesforce yesterday.

Like many users, we unexpectedly received a password reset email from Heroku, even though BleepingComputer does not have any OAuth integrations that use Heroku apps or GitHub. This indicated that these password resets were related to another matter.

Heroku explains forced password resets

This week, Heroku started performing forced password resets for a subset of its user accounts after last month's security incident, without fully explaining why.

On Tuesday night, some Heroku users received emails titled "Heroku security notification - resetting user account passwords on May 4, 2022," advising users that their account passwords were being reset in response to the security incident. The reset would also invalidate all API access tokens and require users to generate new ones, explained the email.

But, the original security incident being referred to involved threat actors stealing OAuth tokens issued to Heroku and Travis-CI and abusing these to download data from private GitHub repositories belonging to dozens of organizations, including npm.

"On April 12, GitHub Security began an investigation that uncovered evidence that an attacker abused stolen OAuth user tokens issued to two third-party OAuth integrators, Heroku and Travis-CI, to download data from dozens of organizations, including npm," GitHub had previously disclosed.

These tokens had earlier been used by Travis-CI and Heroku OAuth applications to integrate with GitHub to deploy applications.

By stealing these OAuth tokens, threat actors could access and download data from GitHub repositories belonging to those who authorized the compromised Heroku or Travis CI OAuth apps with their accounts. Note, GitHub's infrastructure, systems, or private repositories themselves were not impacted by the incident.

But, that still did not explain why would Heroku need to reset some user account passwords—until now.

It turns out the compromised token for a Heroku machine account obtained by threat actors also allowed unauthorized access into Heroku's internal database of customer accounts:

"Our investigation also revealed that the same compromised token was leveraged to gain access to a database and exfiltrate the hashed and salted passwords for customers' user accounts," explains Heroku in an updated security notification.

"For this reason, Salesforce is ensuring all Heroku user passwords are reset and potentially affected credentials are refreshed. We have rotated internal Heroku credentials and put additional detections in place. We are continuing to investigate the source of the token compromise."

A YCombinator Hacker News reader alleged that the "database" being referred to might be what was once called "core-db."

The reader in question is Craig Kerstiens of PostgreSQL platform CrunchyData, who has previously been affiliated with Heroku.

"The latest report states about 'a database' which is presumably the internal database," says Kerstiens.

"I don't want to speculate too much, but it seems [the attacker] had access to internal systems. GitHub were the ones that detected and noticed it and reported to Heroku. Do not disagree that there should be more clarity, but best to follow up with Salesforce on that."

BleepingComputer reached out to Kerstiens who confirmed writing these comments.

Customers call vague disclosure a 'train wreck'

Heroku's original disclosure of the security incident stated that unauthorized access had been related to GitHub repositories belonging to accounts that used Heroku's compromised OAuth tokens.

"The compromised tokens could provide the threat actor access to customer GitHub repos, but not customer Heroku accounts," the company had previously stated.

But the password reset emails rightfully prompted concerns among customers that Heroku's investigation may have uncovered further malicious activity by the threat actors that was not being disclosed.

Some YCombinator Hacker News readers dubbed the disclosure "a complete train wreck and a case study on how not to communicate with your customers."

In its quest to be more transparent with the community, Heroku has shed some light on the incident, starting a few hours ago.

"We value transparency and understand our customers are seeking a deeper understanding of the impact of this incident and our response to date," says Heroku.

The cloud platform further stated that after working with GitHub, threat intel vendors, industry partners and law enforcement during the investigation it had reached a point where more information could be shared without compromising the ongoing investigation:

"On April 7, 2022, a threat actor obtained access to a Heroku database and downloaded stored customer GitHub integration OAuth tokens. Access to the environment was gained by leveraging a compromised token for a Heroku machine account. According to GitHub, the threat actor began enumerating metadata about customer repositories with the downloaded OAuth tokens on April 8, 2022. On April 9, 2022, the attacker downloaded a subset of the Heroku private GitHub repositories from GitHub, containing some Heroku source code.

GitHub identified the activity on April 12, 2022, and notified Salesforce on April 13, 2022, at which time we began our investigation. As a result, on April 16, 2022, we revoked all GitHub integration OAuth tokens, preventing customers from deploying apps from GitHub through the Heroku Dashboard or via automation. We remain committed to ensuring the integration is secure before we re-enable this functionality."

By contrast, another third-party integrator, Travis-CI, disclosed on the business day following GitHub's original notification that no customer data had been impacted by the incident.

Heroku users are advised to continue monitoring the security notification page for updates related to the incident.

Update, May 5th, 2022 09:30 AM ET: We confirmed the quoted reader in the piece is indeed Kerstiens.

Related Articles:

GitHub enables push protection by default to stop secrets leak

GitHub now can auto-block token and API key leaks for all repos

GitLab affected by GitHub-style CDN flaw allowing malware hosting

GitHub comments abused to push malware via Microsoft repo URLs

Fake cheat lures gamers into spreading infostealer malware