PyPi vs Idiotic Developers

Alright. I'm astounded that this even needs to be talked about in the first place, but, here we go.

A few days ago, PyPi (the de-facto language-specific package manager for Python) began requiring the maintainers of packages marked as "critical" to use two-factor authentication (or 2FA) before pushing updates to said packages. Pretty sensible, right? Given the many, many, many issues that other language-specific package managers (such as NPM) have had with malicious code being pushed to critical packages, compromising thousands of production systems, requiring 2FA before pushing updates seems like a reasonable precaution.

Unfortunately, certain software developers don't agree. The person which has been making headlines recently is "Markus Unterwaditzer", the maintainer of "atomicwrites", a Python package used by over 300 other packages and countless devices and servers. Apparently, our dear Markus isn't much a fan of two-factor authentication, writing on Twitter (in reference to a screenshot of an email sent to him by PyPi informing him of the new requirement for 2FA):

"hi, we've solved supply chain security by enforcing security policies on your free labor" -- wtf??

He then followed up this tweet with another rather disturbing update:

nice, I just deleted the atomicwrites package, then uploaded a new version. now it's no longer a critical project.

Markus Unterwaditzer's legendary tweet

Oh boy. So, not only did dear Markus choose to bypass security mechanisms which many would consider to be bare minimum good practice, but by deleting and reuploading the atomicwrites package, he broke the dependencies for all of the over 300 packages depending on atomicwrites, in addition to the update mechanism for every single machine out there in the wild which depended on his package. PyPi's admins were quickly alerted to the situation, and had to repair the damage done by Markus' little maneuver.

Markus has since abandoned atomicwrites, writing a note on the project's GitHub repository stating that it is now unmaintained. Fortunately not many followers of Markus' Twitter account approved of this rather creative decision, but a few commenters on the Hackaday article "This Week In Security: Retbleed, Post-Quantum, Python-atomicwrites, And The Mysterious Cuteboi" were more sympathetic of Markus' side of the story. One commenter, under the name "Mike Stone", wrote:

[Markus Unterwaditzer] is correct that they're imposing an additional cost on something they get for free. It appears the PyPi team failed to use any of the well-known best practices for dealing with creators you aren't paying, like communicating ideas and getting buy-in _before_ declaring them mandatory.
It also looks like the PyPi team is dumping the workload for their problem (security and consistency of the overall collection) onto unpaid contributors. At minimum, they should have a well-reasoned, persuasive explanation for why they aren't willing to do the work themselves.

This Week In Security: Retbleed, Post-Quantum, Python-atomicwrites, And The Mysterious Cuteboi

So, apparently Markus isn't alone: there are a variety of other undereducated and, quite frankly, idiotic individuals out there who are apparently unaware of basic security practices or the nature of PyPi's decision! Let me explain.

Package Repositories and Security

The package repositories of most software distributions (Debian, Red Hat, OpenBSD, and so on) are curated by the repository maintainers. These repository maintainers are responsible for deciding what packages to include in the repositories, what versions of those packages are included, and ensuring that those packages are both functional and secure. This allows end-users to be confident in the performance and security of their system.

Repositories such as PyPi, NPM, Cargo, and others, operate very differently. Rather than being curated by package maintainers, these repositories allow any developer to publish their software for free, with no curation or verification. This makes these packages far more similar to the Google Play Store or the Apple App Store, where published software is only subjected to basic automated anti-malware screening, and is otherwise left uncurated (so long as it doesn't do anything to gain the attention of the publisher). This results in tons and tons of poorly written and insecure software being published to repositories such as PyPi's.

On its own though, this isn't necessarily a bad thing. People are of course free to simply not install any of the malware or cruft present in PyPi's repositories; the end-user is expected to be aware of the quality and security of the software they use. Unfortunately, end-users are dumb. Most of the time end-users either don't have the time or the interest in manually verifying changes to every single update to every single piece of software they use. The ability to not have to worry about things like this are one of the major draws of the repositories provided by software distributions.

It is generally understood by the public that software published to services like PyPi aren't necessarily secure, and as such people are (usually) careful about choosing what software from PyPi they use. However, once a person has decided to use (and often begin depending on) a piece of software published through PyPi, they generally don't keep monitoring it for continued quality and security; it's assumed that the maintainers of that software will continue doing the great work they've been doing. But, what if the person publishing an update to PyPi isn't the package's maintainer?

Oops, All Bitcoin Miners!

Services like PyPi (and in particular NPM, which has hilariously bad security practices and has had a large number of high-profile incidents as a result in just the past year alone) are frequently exploited to distribute malware onto the machines of unsuspecting victims. If the PyPi account of a package maintainer is compromised, the malicious actor is able to push malicious code onto machines which operate under the assumption that PyPi's distribution of that package is secure. Obviously, this is a big, big problem. Since, by the very nature of the service they provide, PyPi does not and cannot manually curate every single update for every single one of the thousands of packages they host, end-users are entirely reliant on the accounts of package maintainers to remain secure (see platforms like NPM which have had a number of vulnerabilities allowing developer accounts to be compromised), and for those package maintainers themselves to continue operating in good faith (see the NPM packages 'colors' and 'faker', which were intentionally sabotaged by their maintainer).

Fortunately, PyPi doesn't need to curate everything manually: Users of software distributed through PyPi are generally trusting the DEVELOPER of the software they use, not the PUBLISHER. PyPi is seen as nothing more than a way of automatically fetching the latest updates and managing dependencies of required software, it is the package maintainers which are held responsible for the quality and security of that software. So, the only responsibility PyPi has to their users (in terms of assuring quality and security of the software they distribute) is ensuring that they are, in fact, distributing software provided by the correct package maintainer, and not a malicious actor. In order to do this though, PyPi obviously needs a way of identifying who is the correct maintainer, and who is a malicious actor.

You can lead a horse to water...

This is where two-factor authentication comes in. PyPi accounts are the mechanism through which package maintainers are authenticated, so it is obviously of the utmost importance that these accounts remain secured. Since there have been numerous incidents of account being broken into via compromised passwords (and even at least one incident of NPM packages being compromised by malicious actors simply contacting support claiming to be the official maintainer and to have lost their password), it makes sense to impose additional mechanisms to ensure accounts remain secure. One mechanism which has been used by all sorts of Internet services for decades is two-factor authentication. Since two-factor authentication mechanisms like TOTP are standard and incredibly common, it makes total sense to require package maintainers, especially the maintainers of high-profile packages which could cause massive damage if compromised, to use two-factor authentication as an additional layer of security.

In fact, PyPi has gone the extra mile in terms of their 2FA deployment. Instead of just supporting the ubiquitous TOTP mechanism, they're also supporting the FIDO U2F mechanism, which allows the use of hardware security keys and are generally much more secure. But wait, there's more! PyPi has also gone above and beyond in providing FREE hardware security keys to the maintainers of packages they've deemed critical, providing not one but two keys, just in case the first one fails. For free. They're giving away 4,000 Titan Secure Keys, in both USB-A and USB-C variants, to the maintainers and owners of the over 3,500 packages they've marked "critical" (out of the over 350,000 packages total, at the time of this writing).

Of course since FIDO U2F is a standard protocol, there's no need to use a Titan brand key; any security key supporting the standard, such as those manufactured by the very common Yubikey brand, can also be used. And if you don't have (and don't want to buy) a hardware security key, you can always still just use TOTP, which is supported by any device, with hundreds of free programs to choose from.

...but you can't make it drink.

Despite the complaints of Mike Stone, PyPi is not, in fact, "dumping the workload for their problem ... onto unpaid contributors". Certainly, it appears that they've invested a considerable amount of their time and money into ensuring the quality and security of the service they provide. They have gone above and beyond in ensuring that they are not "imposing an additional cost on something they get for free", by allowing maintainers to choose from multiple standard authentication mechanisms, and providing free hardware security keys to users who want to take extra steps to secure their accounts. Additionally, history and common sense provide a very clear "well-reasoned, persuasive explanation" for requiring the use of 2FA.

So, it would seem, it is in fact the package maintainers who "aren't willing to do the work themselves". PyPi has done everything within their power to make the deployment of two-factor authentication as easy and painless as possible, providing clear documentation, public announcements, free security keys, and support for standard mechanisms. Additionally, PyPi is far from the first service to implement these precautions, with 2FA already being supported by a very large number of Internet services, and quickly becoming REQUIRED by various others. It is not uncommon nowadays for web services to provide large, attention-grabbing banners requesting you enable 2FA if you have not done so already. 2FA has for a long time now been seen as a basic, bare-minimum mechanism for ensuring account security.

The actions of Markus Unterwaditzer come across and childish, idiotic, and dangerous. This sort of approach to basic security mechanisms like 2FA are one of the largest reasons for the exploitation of services like PyPi for distributing malware. As a person who, like most people, rely on a large number of software packages maintained by a large number of developers, Markus' attitude to basic preventative measures is a massive red flag that screams "Do not use this guy's software if you don't want a Bitcoin miner running on your production system".

Yes, the administrators of services like PyPi have some responsibility to ensure the quality and security of their distribution systems, but it is absolutely the responsibility of the package maintainer to make use of the tools given to them. It is not PyPi's fault if an idiotic software developer posts their password to Facebook accidentally. It is not PyPi's fault if an idiotic software developer refuses to use two-factor authentication.

Closing Thoughts

Personally, I would like to see services like PyPi make two-factor authentication mandatory for ALL accounts, not just those designated as critical. In general I'm not a fan of these sorts of services, as they have a large number of issues not faced by the repositories of software distributions. Nevertheless, in this case, I am impressed at the steps PyPi has taken to prevent the distribution of malware through their services, and although they could improve things further by requiring this mechanisms for all users, I feel that they have done their part.

If you are a software developer looking to better secure the distribution of your software, there are a wide variety of tools available to you. A very common method which has been in use for decades is signing your source releases with an OpenPGP key, providing both redistributors and end-users with a way to verify the integrity of your releases. Documenting your releases in multiple places (such as on a personal website, in a Git repository, on a mailing list, in an IRC channel, and so on) can also help people to sniff out compromised releases, hopefully before they cause too much damage.

Additionally, you can recommend that your users install software through the repositories of software distributions rather than downloads pre-built binaries or source releases from your site directly, adding an additional layer of review that malicious releases need to pass through before being shipped to end-users. Generally, though, users will install software through their distribution's package repository anyways, simply out of convenience; to further enforce this, you can opt to not ship pre-built binaries of your software at all. This also makes your job easier, as you no longer have to deal with the myriad other issues that come with shipping pre-built binaries ("Does this work on aarch64? Does this work on i386? When will you release a Windows version? Why do I get libxyz.so.1.2.3: cannot open shared object file: No such file or directory? Hello???").

As for package maintainers who are combative or disapproving of basic security mechanisms like two-factor authentication, I can only hope that they better educate themselves. All I can say regarding Markus Unterwaditzer abandoning atomicwrites is: good riddance. That sort of neglectful software development is already too common, and we do not need more of it.