If they had write access, then leaked personal data is the least of anyone's worries. The real concern is how close the hackers came to infiltrating the image source for virtually every modern microservices system. If you could put a malicious image in say alpine:latest for even a minute, there's no telling how many compromised images would have been built using the base in that time.

And even if you hosted your own distribution and notary (like we now do for openSUSE and SUSE), you can't force Docker to check the signatures of all images from that server!

Only docker.io/library/* has enforced image signing and the only other option is to globally enforce image signing which means "docker build" will result in unusable images out-of-the-box.

If you look at something like OBS (the Open Build Service that openSUSE and SUSE use to provide packages as well as host user repos), the signing story is far better (and OBS was written a very long time ago). All packages are signed without exception, and each project gets it's own key that is managed and rotated by OBS. zypper will scream very loudly if you try to install a package that is unsigned or the key for a repo changes without the corresponding rollover setup. And keys are associated with projects so a valid rpm from a different project will also produce a warning and resolution prompt. That's how the Docker Hub should've been designed from the start.

(Disclaimer: I work for SUSE, on containers funnily enough.)

Why not?

Regardless, I think it's certainly an excellent hardening step.

They'd really need to be signatures attached to the images, not just hashes.

Use proper cryptography and don't roll your own!

For example, when Linux Mint's ISOs were briefly backdoored, the attackers also changed the checksum shown on the website: https://www.zdnet.com/article/hacker-hundreds-were-tricked-i...

Nowadays, it's arguably a best-practice when designing a new protocol or storage format to simply make all checksums cryptographically strong unless there's a reason not to. I think that might be where the confusion is coming from.

I guess if you were really sure you had obtained a checksum prior to the service compromise, then that would give reasonable assurance the image was not tampered with.

Assuming you have fetched a given image and captured its sha in a config file in your version control (e.g. a kubetnetes manifest), then whenever you deploy a container you are sure that you're not affected by exploits happening _after_ you saved the fingerprint.

Using SHA-256 as you describe works well and is widely used by package systems to ensure that source code being built does not change. Signatures can be better for determining if the initial image is valid if the signing is done away from the distribution infrastructure since development machines might be more secure than distribution infrastructure (and if not you will have problems either way). You still need to get the correct public key to start with. However, if you do have a good image by luck or actually secure enough distribution infrastructure then SHA-256 will certainly let you know that you get the same one later. Signatures almost always sign a cryptographic hash of the message and not the message itself.

See this issue: https://github.com/docker/hub-feedback/issues/873

Definitely wouldn't have helped prevent the compromise.

Even then just read access to code often allows enough info for leveraging/escalating privilege.

I'm not 100% sure if Docker hub uses deploy keys for repos it has access to thru the integration, but at least previously there was an option to manually add one to repository if it couldn't access it otherwise.

Their newer GH apps permission model allows fine-grained access to only specific repos (and also only read access e.g.). However their older Oauth flow only allows full access to everything. And 99% of GH integrations still seem to use the older authentication method.

This is also something that many CI providers suffer from. There are only few that already support GH apps.

With how much of the internet blindly pulls images from it, the potential gain from hijacking just one high-profile one would be monumental.

Shortcuts all around -- kind of reminds me of MongoDB. Sad it's the primary player...

How is this issue specific to Docker? Anyone can download a random library off github, use a shady linux distribution, or install utility tools loaded with spyware.

I don't think Docker aims to solve issues relating to trusting upstream software. It's a tool to help package applications, just like how tar allows you to package files. What you put in it is up to you.

How is this github any different?

I wonder if docker allows this and on the other hand if that's even feasible for say application images, given that applications must be updated a lot for security reasons. Of course if the Dockerfile's parent reference is not pinned, that does only help to some degree...

docker pull ubuntu@sha256:45b23dee08af5e43a7fea6c4cf9c25ccf269ee113168c19722f87876677c5cb2

To be fair, the docker.io/library/* images are signed but no other images are and there are a bunch of issues with how the signing policies work for users that want to enforce that some images must be signed.

Installing known-vulnerable old versions of legitimate software can be just as bad as installing custom malware.

And as I said, only official-library Docker images are signed. All other images are unsigned and even for third-party repos you can't force Docker to verify all images from a given repo (you have to enable it globally, which breaks the utility of a local "docker build").

[+] Arch is the only counterexample I can think of and I'm not even sure if my memory is correct.

Strawman. Anyone can use Debian Stable or at least Testing.

Nobody read the source code for this exact reason: “the community is here to read it so I won’t".

In Debian we review and vet packages.

If Github was compromised, it would be easy and obvious to insert malicious code in a repository, but hide those changes from anyone on the github website.

Images on Docker Hub don't even need to share their Dockerfile, to talk of all the source/etc that went into their build.

> Which you can avoid by forking the mainline repo and depending on your fork.

If github was compromised, the it would be pretty easy to generate forks with the same compromised code.

Actually only short names go to docker hub, one can setup their own registry and use it via dns names.

Example: docker pull quay.io/letsencrypt/letsencrypt

One day, there's going to be a colossal compromise, and that might finally change where we place security in the priority chain.

I do wish developers would be a little wiser around these things, especially when they see companies taking such huge amounts of capital. I found it quite depressing to watch the unquestioning way development communities assimilated the docker worldview.

But no, not Docker.

You're totally right; with as important as their registry is to well funded attackers, and as startup-y and "agile" as they are, and as godawful as the security practices are that underlie their tools and standards... they hadn't a chance. They still don't.

There is no reason to expect them to get better.

[0] https://twitter.com/WHHackersBR/status/1118393568656334850

(that said, google main page vulnerable to xss is kind of like... what, we're afraid someone will take over google and put some cryptominers on the google.com main page?)

6% of the people received a specific email saying the body of their email was accessed and they had to backtrack.

> This unauthorized access could have allowed unauthorized parties to access and/or view information related to your email account (such as your e-mail address, folder names, the subject lines of e-mails, and the names of other e-mail addresses you communicate with), but not the content of any e-mails or attachments, between January 1st 2019 and March 28th 2019.

Notice it says your email account. The whole email is about the account of the recipient, not those of other recipients. Given that they explicitly worded it this way and people clearly misinterpreted it to mean something else, I hope you can forgive me for being a little skeptical of third-party anecdotes that suggest Microsoft claimed nobody's email contents were accessed...

Why wasn’t that the case before?!

There's always a way to enhance your processes, monitor more indicators, etc. or otherwise improve your security.

It wasn't authentication credentials, but still.

> The bigger problem for Google isn’t the crime, but the cover-up. The vulnerability was fixed in March, but Google didn’t come clean until seven months later when The Wall Street Journal got hold of some of the memos discussing the bug. The company seems to know it messed up — why else nuke an entire social network off the map? — but there’s real confusion about exactly what went wrong and when, a confusion that plays into deeper issues in how tech deals with this kind of privacy slip.

(https://www.theverge.com/2018/10/9/17957312/google-plus-vuln...)

as long as you're using software somewhere in the stack that isn't like maturity level 5, AND you don't have constant audits looking for novel attacks on working-as-intended systems, you're pretty much guaranteed to inherit (or create) a vulnerability at some point, and if you're important enough it will get exploited. the reason that doesn't mean we should start modeling computer systems as "living organisms that eventually get old and die" and should keep modeling security like war is that when you get hit, you can respond. all the layers matter, and insofar as Microsoft or Google do it right, they primarily do it right by having a mature process for monitoring, patching, isolating, etc.

as for docker hub though, yeah i'm totally with you. i'm just saying we shouldn't overestimate the preventive capacity of anyone, honestly. if you're doing anything important over the internet at all, you're making some compromises somewhere.

here are 2 links to things i handwaved at above, for example's sake:

https://www.wired.com/story/microsoft-email-hack-outlook-hot...

https://www.forbes.com/sites/kateoflahertyuk/2018/10/09/goog...

As I understand it, there's no element of signing from the actual devs of an image, just from the central trust service of Docker Hub.

There already have been questionable images hosted there ... just by users uploading compromised images. No hacking needed.

I thought, maybe I need to log out and try if the new password works. I clicked on Log Out link, the website has refreshed and I was still logged in.

They did just post the notice in a banner at the top of https://hub.docker.com

https://success.docker.com/article/docker-hub-user-notificat...

I’m just hoping that I was using a password manager by then.

Any word as to the cause of this? Was something important stored in plaintext, etc.?

Eh? Doesn’t let you use what without an account?

Anyone can pull images anonymously. An account is only for publishing.

https://github.com/docker/docker.github.io/issues/6910

Mac: https://download.docker.com/mac/stable/Docker.dmg

Windows: https://download.docker.com/win/stable/Docker%20for%20Window...

Notice the big "Please Login to Download" button.

The problem is more that they make it harder to find if you don’t log in, which is really not great. But if you don’t want to create an account there is certainly no need to do so.

This going by the way it's worded "single hub database with a subset of non financial data"

Luckily I use both 2fa and random password for github, would suck to loose that account ;)

(Side note: this obviously wouldn't have prevented the current attack)

> Data includes usernames and hashed passwords

How are they hashed? And specifically, can we expect them to be already cracked?

It may also explain some suspicious behaviour / source of compromise in the past (we know when the issue was uncovered, not when the first dump was taken)

Though, as they say that passwords need changing, we can safely assume that their salting, hashing and work factor were insufficient and not following best practice. Just like the lack of 2FA.

Or did the get access to a partition of the user data? How is this even possible?

Some very old backup that had only 5% of earliest users?

Some log file which had plain-text creds of approx 5% users?

Or did they discover the attack as it was happening and kicked-out the attacker in the middle of a data download (only 5% complete)?

Also, why rely on users to change their passwords? Is there a security log I can check?

NPM is bad because the Javascript ecosystem is fast-moving with loose builds that have thousands of dependencies that are all bundled and run insider consumer's browsers.

It'd be a legit criticism of ruby gems or CPAN, but linux distros are an entirely different kettle of fish, and most of the mainstream distros take security pretty seriously

Not impossible, nothing ever is, but fairly difficult.

[1] https://whydoesaptnotusehttps.com

Now, should apt use TLS by default? Ideally, yes. A secure transport is better than an insecure one regardless of what you're sending through it. But unfortunately it's not as simple as that. Most CDNs charge extra for TLS, and many existing free mirrors of packages don't provide TLS at all. Also, using HTTP allows for proxies to cache packages.

Unfortunately, as we discovered recently, apt had not been distrustful enough of HTTP metadata (which was a pretty big mistake since the entire design of package managers is that they must distrust the transport, especially if it's completely insecure like HTTP).

https://www.theregister.co.uk/2019/01/22/debian_package_mana...

https://whydoesaptnotusehttps.com

I hope this doesn't hurt docker too badly. I really like the hub / auto build service.

By replicating the images (or packages) that you need into your own account, you can minimise the possibility of a bad actor replacing a well-known image with something untrusted.

An alternative is to side-cart a service like Notary (https://docs.docker.com/notary/getting_started/) in order to establish a chain of trust for images. If an image gets changed, Docker will refuse to use it and you will be warned that it is untrusted.

Biased opinion on an alternative registry:

- Cloudsmith: https://cloudsmith.io/l/docker-registry/

But you've got other options, such as:

- Self-hosted: https://github.com/docker/distribution)

- Cloud-specific (e.g. ECR, GCR, ACR, etc.)

- Sonatype Nexus: https://www.sonatype.com

- ProGet: https://inedo.com/proget

- Gitlab: https://gitlab.com

- Artifactory: https://jfrog.com/artifactory/

If you're missing the auto-build functionality, this can be achieved reasonably easily with any of the mainstream and awesome CI/CD services out there, such as:

- SemaphoreCI: https://semaphoreci.com/

- CircleCI: https://circleci.com/

- DroneCI: https://drone.io/

Disclaimer: I work for Cloudsmith, and still think Docker Hub is great. :-)

Back to the images bit first:

Base images are only referenced/pulled at build time. So if you've already built your own image and stored it, it'll contain all of the layers necessary to run it without explicitly pulling from Docker Hub.

In the case that you're building new images (likely), it'll need to pull the base images from Docker Hub. However, if you pull the base image(s) from Docker Hub first, you can tag them and store them in your local (or hosted) registry, then refer to those explicitly instead.

For example (using a Cloudsmith hosted registry):

  docker pull alpine:3.8
  docker tag alpine:3.8 docker.cloudsmith.io/your-account/your-repo/alpine:3.8
  docker push docker.cloudsmith.io/your-account/your-repo/alpine:3.8

  FROM alpine:3.8

  FROM docker.cloudsmith.io/your-account/your-repo/alpine:3.8

Caveat: The downside is that you have to trust those dependencies at the exact point you pull them down, so I concede it is still not perfect without rebuilding the lot. :-)

Nevertheless: The base images will be pulled from dockerhub by default and I am not sure we should trust them. Do we have any better alternatives for this?

https://blog.madhuakula.com/some-tips-to-review-docker-hub-h...

Say I use Mongodb and enabled authorization. Will I be fine then? How to discover unauthorized access?

In the worst case, if someone notifies you of a configuration problem or some software bug that allows anonymous access to your database or the ability to remove logs, you may have to assume the entire database was compromised since the existence of that configuration issue or software bug.

> I imagine it is a one way hash

All hashes are one way. If it's lossless and can be reverted, it's a compression algorithm or isomorphism or encryption or cipher or any of a number of other things, but not a hash.

> I don’t even know how it can get decrypted.

It is not decrypted, but brute forced. For example, even if you can't algorithmically figure out what the input to md5sum is that gives you '1f3870be274f6c49b3e31a0c6728957f', you could apply md5sum to every word in the dictionary in a matter of seconds and find out that 'apple', when md5summed, has that output. You would then have one possible password for that hash (though technically there are infinitely many inputs that have that output).

The only way we can know how computationally difficult it is to brute force the password hashes is if we know the following: 1) the hash algorithm used (and other inputs like cost factor) and 2) the entropy of the salt used. Those two together lets us calculate the amount of computation needed to try one brute force "guess". Individual password's difficulty to be brute forced can then be calculated from their entropy (e.g. 'apple' has less entroy than '2SEZb'), to determine the average number of inputs needed to be tried, multipled by the cost of each attempt. Given that difficulty, you can then estimate how long an attacker will take to find your password by estimating how much computational power they have at their disposal.

In general, if you randomly generate 10+ character passwords and docker used best practices, the answer is that any attacker will not get your password in under a thousand years, and if you use a password which has been leaked before or is a dictionary word (or simple variation), it can be found on the order of minutes to days.

The most famous application of rainbow tables is one of Microsoft's family of terrible password hashes LM. But the reason it breaks that wide open is not just the lack of salt, it's also that LM hash only works for 7 character passwords, and up to 14 chars are supported by doing two entirely separate hashes - so you can craft rainbow tables for all possible 7 character inputs and then reverse the hash.

http://archive.fo/bKGKq

"Sorry, it's not you. It's us, but we are working on it!"

And the main usecase is k8s. So docker is just an implementation detail its relevancy is waning imo

Some organizations took the risk of running docker taking images directly from docker hub. They were relaying the security of the images to them.

Some organizations are going to panic now and host their own registry. Which they need to protect as well. But in general it will create a better decentralized ecosystem.

I think this is good for the docker community in general.

- Change your password on https://hub.docker.com

- Check https://github.com/settings/security

- Reconnect oauth for Automated Builds

- Roll over effected passwords and API keys stored in private repos / containers

Quick take:

- Password hashes

- Github tokens

- Bitbucket tokens

- Your Automated Builds might need new tokens

Checking my github logs - It looks like they've known about this for at least a full 24 hours. Most people aren't going to have this looked at until Monday which kind of sucks. Hopefully there is more of a postmortem coming.

Is anyone from github able to comment on this as well?

There doesn't seem to be a way for us to tell if a repo was read by these keys over that time period.

  The following SSH key was added to the foo/bar repository 
  by myorg-dockerhub-user:

  Docker Cloud Build
  39:31:51:be:d6:00:c4:ef:c7:74:c2:16:66:33:93:06

  If you believe this key was added in error, you can 
  remove the key and disable access...

It seems like Docker Hub is implemented as an OAuth app [2], where these granular options are not available and you have to grant access to all your repositories.

[1] https://developer.github.com/apps/differences-between-apps/

[2] https://docs.docker.com/docker-hub/builds/link-source/

honest question, what's the point of using OAuth when the Authz is so coarse? Why not augment to have scopes per repo? Is it considered bad practice to have have a variable (repo name) as a scope?

> Service user (or machine/bot account) suggested

> Attaching your personal GitHub or Bitbucket account to this Docker Hub organization will allow other organization owners to create builds from your private repositories. We suggest using a service user (also referred to as a machine user or bot account).

c.f.: https://docs.docker.com/docker-cloud/builds/automated-build/...

Seems worthwhile to do this, if you're an enterprise or otherwise have sensitive private repos. But I agree that it would be better to have an easier per-repo authorization system, since many users won't bother going through the hassle of setting up a service account.

> c.f.: https://docs.docker.com/docker-cloud/builds/automated-build/....

Did they remove this language from your link? I don't see it anymore.

https://developer.github.com/apps/differences-between-apps/

https://www.netlify.com/docs/github-permissions/

Also not sure what access permissions you need but deploy keys are repo level.

https://developer.github.com/v3/guides/managing-deploy-keys/...

Machine users are another option.

https://developer.github.com/v3/guides/managing-deploy-keys/...

- retrieve a list of all repos to display in the autobuild setup page

- setup webhooks for the gh repo that should be built via dockerhub autobuild

- setup a deploy key for said repo, so that it can be cloned

I removed the dockerhub oauth on github side, after setting up autobuild. My builds on push to master and tag are still working. So it seems possible to remove dockerhubs write access to your github repos after the autobuild setup, which really seems to be a good idea.

EDIT: it finally worked, 4th attempt, and very slowly. Looks like something isn't working 100% as it should

EDIT 2: aaaand I can't login now with the new password. A password reset did work, but it looks like their password database is under some stress at the moment.

Specifically to make the password database more secure, the generation of password hashes is very computationally intensive by design (e.g. that's the whole point of something like bcrypt vs. sha1)

Password systems really shouldn't be designed to handle a 10x or 100x load without some slowdown. If they could handle that, it means their password DB probably isn't as hardened as it should be.