Gentloo Linux GitHub Hack

The Gentoo Linux team recently revealed the reasons for the hack that took place on it’s GitHub organization account roughly a week ago. The simple reason: bad security practices.

What ultimately ended up with the hacker taking control of the Gentoo Linux GitHub account and inserting malicious code designed to remove user data, was the result of faulty password security practices. According to the Gentoo Linux team in a post-hack summary report: “Evidence collected suggests a password scheme where disclosure on one site made it easy to guess passwords for unrelated webpages.”

That’s it. The hacker’s guessed the password.

According to the summary, hackers were able to not only gain access but take over control by simply gaining access to a password of an organization administrator. The wiki summarized the hack as follows: “An unknown entity gained control of an admin account for the Gentoo GitHub Organization and removed all access to the organization (and its repositories) from Gentoo developers. They then proceeded to make various changes to content. Gentoo Developers & Infrastructure escalated to GitHub support and the Gentoo Organization was frozen by GitHub staff. Gentoo has regained control of the Gentoo GitHub Organization and has reverted the bad commits and defaced content.” Luckily for the team, the damage was somewhat limited.

The team shared lessons learned from the experience:

What went well

  • Gentoo responded quickly to reports of problems.
  • GitHub responded quickly and had a function to hide the impacted organization.
  • Gentoo quickly removed account access once the entry point was located.
  • GitHub provided audit logs that helped map out the incident.

What went badly

  • Initial communications were unclear and lacking detail in two areas.
    • How can users verify their tree to be sure they had a clean copy?
    • Clearer guidelines that even if users got bad copies of data with malicious commits, that the malicious commits would not execute.
  • Communications had three avenues (www.gentoo.org, infra-status.gentoo.org, and email lists.) Later we added a wiki page (this page) and were inconsistent on where to get updates.
  • GitHub failed to block access to the repositories via git, resulting in the malicious commits being externally accessible. Gentoo had to force-push over them as soon as this was discovered that.
  • Credential revocation procedures were incomplete.
  • We did not have a backup copy of the Gentoo GitHub Organization detail.
  • The system report is not mirrored from Gentoo, but is stored directly on GitHub.

Lucky items

  •  Numerous Gentoo Developers have personal contacts at GitHub, and in the security industry and these contacts proved valuable throughout the incident response.
  • The attack was loud; removing all developers caused everyone to get emailed. Given the credential taken, its likely a quieter attack would have provided a longer opportunity window.
  • The method by which the attackers pushed commits (force pushing their commits) made downstream consumption more conspicuous; this would have blocked git from silently pulling in new content to existing checkouts on ‘git pull’.

Ensure that your organization has real-time visibility of your security health with BAP.