In the past, I’ve seen many people use git repositories to store sensitive information related to their projects. Lately, I’ve also been seeing some people even announce that they’re storing API keys on their private GitHub repositories. I’m writing this article because I believe that people should understand the risks of storing API keys with your code.
This article is not intended to be read as a permanent solution to the problems you might have with storing API keys. Instead, it’s my own analysis of the problem and my suggestions on how to fix it. So, to begin with, what exactly is the problem with storing sensitive information near your code on a git repository?
Why you shouldn’t store API keys on git repositories
Storing API Keys, or any other sensitive information, on a git repository is something to be avoided at all costs. Even if the repository is private, you should not see it as a safe place to store sensitive information. Let’s start by looking at why it’s a bad idea to store API keys on public git repositories.
By nature, a public git repository can be accessed by anyone. In other words, anyone with an Internet connection can access the contents of a public git repository. Not only that, but they can also browse all the code inside the repository and possibly even run it. If you store an API key on a public repository, you are publishing in the open so that anyone can see it.
A recent search for client_secret on GitHub revealed that there are more than one 30,000 commits that potentially expose an API key and secret. In some cases, you can just copy and paste the code and immediately access the API. This problem is becoming so important that some companies invest in resources to make sure that there aren’t any leaked API keys and secrets. Last year Slack started to search for exposed API tokens and invalidate them proactively. This action prevents malicious access to Slack’s accounts but can’t possibly find all the leaked tokens.
So, this is happening on public git repositories. What about the private ones? Why is that an issue? Private git repositories, especially those hosted on services such as GitHub, GitLab, and Bitbucket, are exposed to a different type of risk. Whenever you integrate a third-party application with one of the services mentioned before, you might be opening your private repositories to those third parties. Those applications will be able to access your private repositories and read the information contained therein. While that alone doesn’t create any risk, imagine if one of those applications becomes vulnerable to attackers. By getting unauthorized access to one of those third-party applications, attackers might gain access to your sensitive data, including API keys and secrets.
So, where should API keys be stored?
Fortunately, there are many alternatives for securely storing API keys and secrets. Some of them let you use your git repository and simply encrypt the sensitive data. Other tools are more sophisticated and automatically decrypt sensitive information as part of a deploy workflow. Let’s look at some of the available solutions.
git-remote-gcrypt
The first solution lets you encrypt a whole git repository. git-remote-gcrypt does that by adding functionality to git remote helpers so that a new encrypted transport layer becomes available. Users simply have to set up a new encrypted remote and push code into it. Read on if you’re looking for a more fine-grained solution that lets you encrypt individual files.
git-secret
git-secret is a tool that works on your local machine and encrypts specific files before you push them to your repository. Behind the scenes, git-secret is a shell script that uses gpg to encrypt and decrypt files that might have sensitive information.
git-crypt
Another solution is git-crypt. It is very similar to git-secret in the way it operates, but it has some interesting differences. The first thing to notice about git-crypt is that it is a binary executable and not a shell script, as git-secret is. Being a binary executable means that to use it you first have to compile it, or you need to find a binary distribution for your machine. If you’re using a Mac you’re lucky because HomeBrew offers a git-crypt ready-to-install package—the only thing you have to do is run brew install git-crypt on a terminal.
BlackBox
BlackBox is a tool created by Stack Overflow, the company behind popular Q&A communities such as Stack Overflow itself, Server Fault, and Super User. BlackBox is a quite robust tool as it works not only with git but also with other version control systems like Mercurial, and Subversion. It also supports the encryption of small strings and not just entire files. It does that when working with puppet and uses puppet’s hiera, a key-value lookup tool for configuration data. Having the ability to encrypt and decrypt individual strings makes BlackBox a great solution for securing API keys and secrets.
Heroku Configuration and Config Vars
If you’re working with Heroku you should not store any sensitive information such as API keys and secrets on your git repositories. Heroku itself offers a solution that lets you set configuration variables. Your application can then access the contents of those configuration variables during runtime by accessing the corresponding environment variables. Even though the values are not encrypted, this solution lets you avoid using your git repository for storing API keys. Dokku, which is an Open Source solution similar to Heroku, offers the same capabilities.
Docker secrets
At the end of the spectrum of possible solutions is Docker secrets. This solution has been introduced by Docker in February 2017 and is gaining popularity ever since. Docker secrets let you define variables that are encrypted and made available to specific services during runtime. Secrets are encrypted both during transit and at rest. This approach makes Docker secrets the perfect solution for storing and using API keys and secrets in a secure and encrypted way.
Summary
By now you should be aware of the dangers of storing sensitive information such as API keys and secrets on public and also private git repositories. Understanding the potential ways in which your repositories might be exposed is key to assessing and mitigating the risks associated with information leaks. This article also proposes a few different solutions that let you encrypt API keys and secrets so that you can securely use your code repositories. I’m sure there are more solutions out there that can help you achieve the same results.
Top comments (28)
If you're running in AWS, you can also make use of Parameter Store, which seems to fly under the radar for a lot of people (being buried three layers deep inside another product probably contributes to that). It allows you to set general configuration parameters, as well as secrets which are encrypted at rest using KMS. At my job, we're starting to use Parameter Store a lot, even when we're testing our applications locally. Our keys never have to live on our machines - as long as we're logged into the AWS CLI, our apps can access them in the same way they would in production.
Hi Joseph. Thanks for sharing information about the AWS Parameter Store! I didn't know about it, and it certainly looks like a good solution.
What about ignoring the .env file or any file you use to save your keys and secrets with .gitignore?
This is how I* do it: .env is ignored, and there's an .env.example included in the repository so that a developer knows which keys are expected/needed/available.
It's what we currently do at work. The secret files are in a KeePass database. It's cumbersome to deal with, especially when you want to checkout a fresh version of a project. Even with a script that copies the files to the projects that should contain them.
Ofcourse this has to do with KeePass as our solution for storing the
.gitignored files.I'm glad the author made the effort of listing some alternatives ways to protect files, and others responded with even more ways to do it. Because at some point I'll be fed up with that KeepAss database and this article is where I'll refer back to :)
Environment variables are by far the simplest way to go. There’s a bit of initial coordination when multiple developers need the same keys - anyone have a favorite way to do initial setup?
Salt Stack - a better alternative to Puppet, Chef, etc. - also comes with built-in support for encrypting secrets while you store them in your version control. Basic idea is simple:
If not using something like Docker, then Salt is one of the best options out there for managing your server configuration (though in my opinion THE best), and supports this out of the box.
docs.saltstack.com/en/latest/ref/r...
Hi Duke. Salt Stack looks very interesting. Thanks for sharing it—I'll add it to my list of alternatives.
Yea it does a lot of things very well, and is built with components that you can take into use separately.
Some examples of things you can do with Salt Stack:
Basically Salt Stack can take care of most of your server management needs.
Where do you store the encryption keys that you use to encrypt the keys you've stored on Github? Chicken? Egg?
I suffer from the same dilemma all the time. I considered Hashicorp Vault as a credentials store, but you still need an access token to access the stored credentials, which means I need to store this token somewhere in my deployment tool (in my case, Ansilbe). Ansible has the ability to encrypt variables, but then I need to store that encryption key somewhere too. If you are automating all deployments (like I do), then you always need to store a secret somewhere. it's a never-ending cycle...
I suppose it's a matter of figuring out the most secure egg (or chicken, if you prefer), a lesser of all the evils. Meh.
I've been working on Torus CLI for sharing secrets between humans and machines from development to production.
It integrates directly into your workflow, so you can model the way you store your secrets to the way you organize and deploy your code. All of the secrets are encrypted on the client using an elliptical curve keypair derived from your password.
Torus makes it easy to centralize all of your secrets and configuration, making it easy to share secrets in development, manage the configuration used directly in your CI/CD flow to ensure build secrets never touch disk, or in production by injecting secrets directly into a process using
torus runvia environment variables.To give someone access, simply invite them to your org and add them to the appropriate teams. No decrypting files using gpg, dealing with binary merge conflicts, or educating users on how to keep secret keys or files safe.
When it comes time to rotate a secret, with one command it's out of rotation, you just need to deploy to bring everything up to date. Most importantly though, when someone leaves your company or changes teams, it's really easy to track down which secrets need to be rotated using the torus worklog command.
Since the post and comments already touch on configuration mgmt solutions too, here is Ansible's answer to this problem - ansible-vault docs.ansible.com/ansible/2.4/vault...
Thanks for sharing information about ansible-vault, Pavlo!
Some questions:
Where to store the secret key for the secrect storage?
How do you version and deploy the secrets?
How do you share the secrets in a team; per version and environment (dev, test, staging, prod)?
Environment variables are some kind of a "system wide global variable"
and not threat safe (see Laravel) in PHP. I would therefore not recommend it.
I came to the conclusion that a simple
env.phpis safe enough and works everywhere. This file is excluded via.gitignore. A developer can copy a versioned template of this file (example.env.php) and adapt it to his needs.For all PHP projects I've worked on, I've used github.com/vlucas/phpdotenv for all keys. I learned of it after working with Laravel projects.
For anyone on Rails, the new encrypted credentials in version 5.2 are pretty nice. Before, I had to store sensitive information in my .env file locally as well as Heroku config. Now, I can just store it in an encrypted YAML file within the repo and keep only the encryption key in my environment.
I want to create chrome extension, which allows for everyone Create and Read files from MY repository. It's possible? And I can't hide API..
+Looks like all permissions allows not only create, but delete also..
Hi Bruno, excellent post! I have used your article as an inspiration to write a brief tutorial on how to use AWS KMS and OpenSSL for protecting security-sensitive data in public GitHub/ Bitbucket repositories. Hope that is of any use for people who are looking to implement an encryption mechanism for their project.
Any opinions on untracked, manually maintained env vars? Just for personal/small team projects, of course.
This is what I usually do, with various packages (the vlucas package above for PHP, dotenv for node/JS) for managing the variables. I find working off a .env file is more applicable and cross platform, as various deploy tools can keep them secret and inject them for you and keeping them outside of a git repo.
At that point it's an issue of communicating those keys to the rest of the team/documenting them securely. I don't have a great solution for that. Password managers like LastPass or Dashlane could work, but might get awkward as you scale up team size?
I've seen LastPass used for that sort of thing, and it's workable. I think a solution that fits a team that's grown past that is just a config script that is run on first setup that pulls all dev configs from a separate, secure store, which maybe pings a different company employee and then generates a one-time token for access, but that might be more or less security or configuration than some orgs need.
Hmm interesting. Thanks for sharing these different tools.
Thanks for your kind comment, Andy!
Hi thank you a lot for sharing.What about storing API key not in project but in system env variables !?
I create template files like "credentials.default.config" and put them on git. My app then read a "credentials.config" file that is not in the repository. This is the safest method I found.