Software Configuration Management (SCM) Security

David A. Wheeler

March 13, 2004; Revised Jan 30, 2011

Introduction

Software development is often supported by specialized programs called "Software Configuration Management" (SCM) tools. SCM tools often control can read and modify the source code of a program, keep history information (so that people can find out what changed between versions, and who changed them), and generally help developers work together to improve a program under development.

Problem is, the people who develop SCM tools often don't think about what kind of security requirements they need to support. This mini-paper describes briefly the kinds of security requirements an SCM tool should support. Not every project may need everything, but it's easy to not notice some important requirements if you don't think about them. There are two basic types of SCM tools, "centralized" and "distributed"; the basic security needs are the same, but how these needs can be handled are different between the two different types. I'm primarily concentrating on basic SCM tools (like CVS, Subversion, GNU Arch, Bitkeeper, Perforce, and so on). Clearly related tools include build tools, automated (regression) test tools, bug tracking tools, static analysis tools, process automation tools, software development tools (such as editors, compilers, and IDEs), and so on.

The Security Basics

Fundamentally, there are some basic (potential) security requirements that any system needs to consider. These are:

  1. confidentiality: are only those who should be able to read information able to do so?
  2. integrity: are only those who should be able to write/change information able to do so? This includes not only limiting access rights for writing, but also protecting against repository corruption (unintentional or malicious). Changesets must be made atomically; if 3 files change in a changeset, either all or none should be committed.
  3. availability: is the system available to those who need it? (I.E., is it resistant to denial-of-service attacks?)
  4. identification/authentication: does the system safely authenticate its users? If it uses tokens (like passwords), are they protected when stored and while being sent over a network, or are they exposed as cleartext?
  5. audit: Are actions recorded?
  6. non-repudiation: Can the system "prove" that a certain user/key did an action later? In particular, given an arbitrary line of code, can it prove who was the individual that made that change and when? Can it show all those who approved/accepted it, as a path?
  7. self-protection: Does the system protect itself, and can its own data (like timestamps, changesets, other data) be trusted?
  8. trusted paths: Can the system make sure that its communication with users is protected?
  9. resilience to security algorithm failures: If a given security algorithm fails (such as the hash function or encryption), can the algorithm be easily replaced to protect past and future data? (Added 2005-03-02, after the revelation of serious problems in SHA-1).
  10. privacy: Is the system designed so it's not possible to retrieve information that users want to protect? For example, spamming is a serious problem; it may be desirable to NOT record real email addresses, at least in some circumstances. If there is a "secret branch" where security patches are located, try to not store its location in the dataset. This is similar to confidentiality, but you might not even trust an administrator... the notion is to NOT store or depend on data you don't want spread.

An SCM has several assets to protect. It needs to protect "current" versions of software, but it must do much more. It needs to make sure that it can recall any previous version of software, correctly, as well as the audit trail of exactly who made which change and when. In particular, an SCM has to keep the history immutable - once a change is made, it needs to stay recorded. You can undo the change, but the undoing needs to be recorded separately. Very old history may need to be removed and archived, but that's different than simply allowing history to be deleted.

The Threats

Okay, so what are the potential threats? These vary, and not all projects will worry about all threats. Nevertheless, it's easier to provide a list of threats and the counter-measures an SCM should support.

Individual projects may choose to not employ a given counter-measure, since they may decide that's not a threat for them. For example, open source software (OSS) projects may decide that there's no "threat" of unauthorized reading of software, since the code is open to reading by all. However, that may not always be true - many OSS projects hide changes that reveal security vulnerabilities until the new version is ready for deployment. Thus, it's difficult to make simple statements like "projects of type X never need to worry about threat Y". Instead, it's simpler to list some potential threats, and then projects can decide which ones apply to them (and configure their SCM system to counter them).

Outsiders without privileges

An outsider (not a developer or administrator) may try to read or modify assets (software source code or history information) when they're not authorized to do so. SCM systems should support authorization (like login systems), and support a definition of what unauthorized users can do. An SCM system should support configurations that allow anonymous reading of a project and/or its history, since there are many cases where that's useful. However, SCMs should also support forbidding anonymous read access. That's even true for OSS projects, since as I noted above, sometimes OSS projects want to hide security fixes until they're ready for deployment.

Normally unauthorized users shouldn't be allowed to modify a source repository, so an SCM should support that (and should make that the default). In rare cases, it's possible to imagine that even this constraint isn't true, especially if the SCM tool is designed to be used for resources other than source code. Most Wiki systems such as Wikipedia allow anonymous changes; they work instead by protecting the history of changes so that everyone will know exactly what's changed, instead of preventing writing of the primary data. Such approaches are rare for software code; for example, the Wikipedia software itself (as stored in its trusted repository) can only be changed by a few privileged developers. However, it is conceivable that software documentation and code would be maintained by the same SCM software, and perhaps a few projects would allow anyone to update the documentation as long as all changes were tracked and could be easily reversed.

The underlying identification and authentication system (the login system) can use intrusion detection systems to detect likely attempts to forge privileges (e.g., by detecting password guessing attacks, or detecting improbable locations of a login). The underlying login system could also support enabling limits (e.g., delays after X login attempts, or only permitting logins from certain Internet Protocol address ranges for certain developers). However, these mechanisms need to not create a denial-of-service attack; otherwise, an attacker might try to forge logins not to actually log in, but to prevent legitimate users from doing so.

Non-malicious developers with privileges

An SCM system should support protected logins (e.g., if it uses passwords, it should protect passwords during transit and while they're stored). Once users are authenticated, an SCM system should be able to limit what users can do based on the authorization that's implied.

SCM systems could usefully limit reading to particular projects, say. Limiting reading of specific files inside a project can be useful, but it often isn't as useful inside a branch developers must access because developers often need the entire set of files to develop (e.g., to recompile something). But limiting who can read changes in certain branches could be vital for some projects. For example, it is common for security vulnerabilities to be reported to a smaller group of people than the entire development staff, and for the patch to be developed by specially trusted developers without full knowledge of all developers. This is particularly true for open source software projects, but it's also sometimes true for other projects. This kind of functionality can also be important for projects such as military projects with varying degrees of confidentiality; most of the program may be "unclassified", but with a poor or stubbed algorithm; there may be a better classified algorithm, but it will need to be maintained separately. Ideally, the SCM should be trustworthy enough to protect that data, though in practice such trust is rarely granted; an SCM should instead gracefully handle importing the "unclassified" version and automatically merging the "classified" data on equipment trusted to do so.

Limiting writing of specific files inside a project can be much more useful, since in some projects some users "own" certain files. In many situations it doesn't make sense either, but an SCM system should still support limiting which developers can make which changes.

Malicious developers with privileges (and attackers with their credentials)

An area often forgotten by SCM systems is handling malicious developers. You know, the ones who intentionally insert Trojan horses into programs. Denying they exist doesn't help; they do exist. And even if they didn't, there's no easy way for an SCM to tell the difference between an authorized malicious developer and an attacker who's acquired an authorized developer's credentials.

A malicious developer might even try to make it appear that some other developer has done a malicious deed (or at least make it untraceable). They can use their existing privileges to try to gain more privileges. A malicious developer might try to modify the data used by a CM system so that it looks like someone else made the change (e.g., provide someone else's name in a ChangeLog entry). A malicious developer might try to modify a CM "hook" to make it appear that some other developer has inserted malicious code (perhaps to avoid blame or frame the other developer). A malicious developer might modify the build process, e.g., so that when another developer builds the software, the build system attempts to steal credentials or harm the developer.

Since developers have the privileges to read and change data, malicious developers (and attackers with their credentials) are harder to counter. But there are counter-measures that can be used against them. Here are some reasonable measures:

  1. Make sure that developers can't corrupt the repository. As a counter-example, GNU Arch allows developers to share a writeable directory as a repository. That's very convenient, but if you're worried about malicious developers, that's not enough; a malicious developer could easily remove data or corrupt it in such a way that it'd be hard to tell who caused the problem (there's current effort to create an "archd" server that would probably counter this problem).
  2. Make sure that all developer actions are logged in a non-repudiable, immutable way. That way, even if someone makes a change, it's easy to see who made what changes, in any time in the future. That "someone" may be a malicious developer, or an attacker with the credentials (e.g., cryptographic keys) of a developer -- but in either case, once you find out who did a malicious act, the SCM should make it easy to identify all of their actions. In short, if you make it easy to catch someone, you increase the attackers' risk... and that means the attacker is less likely to do it. In practice, this can be done by requiring that all changes be cryptographically signed by each developer (at the developer's side). You can't just make people log into a central server and trust its results, or have a central server sign everything; if the server is subverted you won't be able to trust any of that data. Immutable backups also help; that way, if history is changed, it can be easily detected. Implied here is that there is a relatively easy way to undo changes in a later version; after all, if it's easy to identify exactly what a developer did, those changes can be undone.
  3. Make sure all developer actions can be easily reviewed later. A simple action to show exactly what's been changed recently will make it easy for new changes to be reviewed - and possibly set off alarms.
  4. Have tools to record and/or require others' review. If you really want to make sure that malicious code doesn't get through, the best method known is to make sure that some other person (who is unlikely to be colluding) reviews the code. Thus, ways to cryptographically sign that a person reviewed anothers' changes can be helpful, as long as the reviewer's signature can't be forged, and as long as the signature clearly indicates what was reviewed. A review could be at a brief "I briefly scanned for malicious code" all the way to "I deeply analyzed every line for correctness", so the SCM tool should support recording to what level the review occurred too. Note that the Linux kernel development process now has people adding a "Signed-off-by:" tag to each changeset; this is primarily for licensing reasons, but still, it's helpful in identifying all the other parties who looked it at so that how the change got there (but note that in Signed-off-by, a person may or may not change it further).
  5. Support automated checking before acceptance, including detection of suspicious/malicious changes. An SCM system should make it possible to enforce certain rules before accepting a change (at some level): such as enforcing formatting rules, requiring a clean compile, and/or requiring a clean run of a regression test suite (in a suitably protected environment). It should be possible to watch changes to find "suspicious" changes: the first time that developer has modified a given file, code that looks a like a Trojan horse, formatting/naming style that's significantly different than this developer's normal material, attempts to send email or other network traffic during a code build, and so on. This is basically intrusion detection at the code change level. It should also be possible for an automated process to quickly check for hints of "stolen" code before accepting anything (e.g., to detect copyright-encumbered code), by calling to programs such as Eric S. Raymond's comparator.
  6. Support authentication/cryptographic signature key changes and re-signing. No matter what protection is put in place, a developer's secrets (e.g., their login passwords or private keys) may be acquired by an attacker. Thus, an SCM (along with its support environment) need to support changing such secrets. In particular, it may be useful to "cycle" developer private keys, having developers switch to new private keys, ensuring that the old keys will not be accepted for newer changes, and possibly destroying all copies of the older private keys (so that they cannot be stolen by anyone). Since private keys may be compromised, once such a compromise has been detected, it should be possible to invalidate the compromised keys and re-sign data (once it's checked) with new cryptographic keys. This is yet another reason to support multiple signature keys (in addition to supporting multi-person review).
  7. On login, acquisition, and commit, report the "last time" and source location (e.g., IP address) where reading and writing (committing) were performed. Although this doesn't deal with a malicious developer, it does increase the likelihood that an attack using stolen credentials will be detected. After all, the developer is mostly likely to know the last time that they read from and wrote to some repository, so they'll be able to detect when someone else forges their identity. Ideally, this would be resistant to repository attacks.

On April 11, 2004, Dr. Carsten Bormann from the University of Bremen sent me an email about a specialized attack that he terms the "encumbrance pollution attack". In an encumberance pollution attack, the attacker inserts material that cannot be legally included. To understand it, first imagine an SCM with perfectly indestructible history. The attacker steals developer credentials, or is himself a malicious developer, and checks in a change that contains some encumbered material. "Encumbered" material is simply material which cannot be legally included. Examples include child pornography, slanderous/libelous statements, or code which has copyright or patent encumberances. This could be very advantageous, for example, a company might hire a malicious developer to insert that company's code into a competing product, and then sue the competitor for copyright infringement, knowing that their SCM system "can't" undo the problem. Or a lazy programmer might copy code that they have no right to copy (this is rare in open source software projects, because every line of code and who provided it is a matter of public record, but it proprietary projects do have this risk). Any SCM can record a change that essentially undoes a previous change, but if the history is indestructable and viewable by all, then you can't get rid of the history. This makes your SCM archive irrevocably encumbered. This can especially be a problem if the SCM is indestructably recording proposals by outsiders! An SCM system could be designed so that a special privilege allowed someone to completely deletion the history data of illegal changes, of course. However, if there are special privileges to delete history data, it might be possible to misuse those privileges to cause other problems.

One mechanism for dealing with an encumberance pollution attack is to allow specially-privileged accounts to "mask" history elements; i.e., preventing access to certain material by normal developers so that it's no longer available, so that the material isn't included in later versions (essentially it work like an "undo" against that change). However, a "mask" would still record the event in some way so that it would be possible to prove that the event occurred at a later time. Perhaps the system could record a hash of the encumbered change, allowing the encumbered material to be removed from the normal repository yet proving that, at one time, the material was included. A "masking" should include a cryptographic signature of whoever did the masking. This mechanism in particular requires careful design, because the mechanism should be design so that it doesn't permit other attacks.

Most SCM systems have multiple components, say, a client and server. Even GNU arch, which can use a simple secure ftp server as a shared repository, has a possible server (the ftp server). Clients and servers should resist attack from other potentially subverted components, including loss of SCM data.

Repository attacks

Many repositories have themselves undergone attack, including the Linux CVS mirror, Savannah, Debian, and Microsoft (attackers have acquired, at least twice, significant portions of Windows' code). In 2011, kernel.org was subverted. Thus, a good SCM should be able to resist attack, even when the repository it's running on subverted (through malicious administrators of a repository, attacker root control over a repository, and so on). This isn't just limited to centralized SCM systems; distributed SCM systems still have the problem that an attacker may take over the system used to distribute someone's changes. In 2011, attackers subverted Linux's kernel.org site, but it's believed there was little damage to the source code repositories due to the nature of git.

An SCM should be able to prevent read access, even if the repository is attacked. The obvious way to do this is by using encrypted archives. But there are many variations on this theme, primarily in where the key(s) are stored for decryption. If the real problem is just to make sure that backup media or transfer disks aren't easily read, the key could simply be stored on a separate (more protected) media. The archive keys might only be stored in RAM, and required on bootup; this is more annoying for bootup, and an attacker is likely to be able to acquire the data anyway. The repository might not normally have the keys necessary to decrypt the archive contents at all; it could require the developer to provide those keys, which it uses and then destroys. This is harder to attack, but a determined adversary could subvert the repository program (or memory) and get the key. Another alternative is to arrange for the repository to not have the keys necessary to decrypt the archive contents at any time. In this case, developers must somehow be provided with the keys necessary to do the decryption, and essentially the repository doesn't really "know" the contents of the files it's managing!

Preventing write access when an attacker controls a repository is a difficult challenge, especially since you still want to permit legitimate changes by normal developers. Since the attacker can modify arbitrary files in this case, the goal is to be able to quickly detect any such changes:

  1. Cryptographic signing of changes can help significantly here, since this makes it possible to detect changes by anyone other than the authorized developers. Clearly, the list of public keys needs to be protected; this can be protected in part by ensuring that the list is visible to all developers, and having tools automatically check that the public listed key is correct (each developer's tool checks that the key listed is really that developer's key).
  2. Changeset chaining can help detect problems (including unintentional ones). Basically, as changes are made, a chain recording those changes can be recorded and later checked. This is typically done using cryptographic hashes, possibly signed so you know who verified the chain. Note that this is also useful for detecting accidental corruption.
  3. Automated tools to detect if "my" change has been altered. Any given developer will know what changes they checked in. So, record that information locally/separately, and check it later. That way, someone can modify the repository to remove the latest security fix, but the developer of the change can quickly tell that it's been removed.
  4. Immutable backups, and tools to check them, can help as well. If a repository's history is changed, that change can be compared with backups. Be careful that a corrupted tool won't create misleading backups, and make sure that the repository can't give one view to backup tools, and another view to whoever actually takes and uses the program.
  5. Simple, transparent formats can help make it harder to hide attacks. Data that is stored in simple, well-understood formats that can be analyzed independently (e.g., a signed tarfile of patches) tend to be more resistant to attack than data structures that presume that no other process will manipulate the data contents (e.g., typical databases).

Related Work

There only seems to be a little related work available on the topic. Lynzi Ziegenhagen wrote a Master's thesis for the Naval Postgraduate School about revision control tools for "high assurance" (a.k.a. secure) software development projects: Evaluating Configuration Management Tools for High Assurance Software Development Projects ( also available at StormingMedia). A commentary on that paper is also available. The OpenCM project has published some papers, including Jonathan S. Shapiro and John Vanderburgh's Access and Integrity Control in a Public-Access, High-Assurance Configuration Management System (Proc. 11th USENIX Security Symposium, 2002, San Francisco, CA, 2002).

Another related paper is "Configuration Management Evaluation Guidance for High Robustness Systems" by Michael E. Gross (Lieutenant, United States Navy), March 2004.

The Trusted Software Methodology included a number of configuration management requirements; in particular, its upper level requirements were specifically designed to counter malicious developers. See "Trusted Software Methodology Report" (TSM), CDRL A075, July 2, 1993, and in particular its appendix A (which defines the trust principles). The Common Criteria includes a number of configuration management requirements (see in particular part 3 in the ACM section).

Security for Automated, Distributed Configuration Management by Devanbu, Gertz, and Stubblebine examine a completely different problem (one which is important, but not the one in view here).

Jeronimo Pellegrini's Apso (prototype software) is a framework for adding secrecy to version control systems (currently for Monotone). In 2006 a conference paper was published that described it. In 2011 he mentioned that he hasn't had much time to work on it any more, but the ideas may be of interest to others.

There is a vast amount of literature about SCM systems, as well as papers discussing or evaluating particular systems. That includes my own Comments on OSS/FS Software Configuration Management (SCM) Systems.

Conclusions

All of this can't prevent all attacks. But such an SCM system can make the attacks much harder to perform, more likely to be detected, and make detection much more rapid. Here are some examples:

  1. A malicious developer could insert a few lines into a build process that said "when you compile, email to me your private key data" - then, once they had the private key, remove that line, and then forge other changes as that unsuspecting developer. But an SCM system with all of the capabilities above would make it much harder to hide this. The change with these malicious instructions would be clearly labelled as from that developer, and later changes would be labelled as being from that developer or one of the compromised systems - and removing the change later would record yet another change that might be detected.
  2. A malicious attacker might take over the repository, and repeatedly remove a critical patch to a security vulnerability. Still, the removal could be detected by the creator of the patch, and actions such as changing to a different repository could be performed. Trying to change older copies would likely be detected by chaining and comparisons with backups.

It's my hope that SCM systems will have more of these capabilities in the future. I'm happy to note that some SCM developers have considered these issues. Aegis has a nice side-by-side comparison comparing a version of this paper with Aegis' capabilities. Bazaar-NG has considered these security ideas. Hopefully others will consider these issues too.


Feel free to see my home page at http://www.dwheeler.com. Paul Stadig's "Thou Shalt Not Lie: git rebase, ammend, squash, and other lies" is related.