Open Standards, Open Source

by David A. Wheeler, 2006-04-12 (revised 2008-10-10)

This paper discusses how open standards aid free-libre / open source software (FLOSS) projects, how FLOSS aids open standards, and then examines a mystery: Today, essentially every successful open standard is implemented by FLOSS. This paper considers why that is, and what it means if this is predictive. In particular, this suggests that an open standard without a FLOSS implementation is probably too risky for users to require, and that developers of open standards should encourage the development of at least one FLOSS implementation.

Open standards are standards that are publicly defined and anyone can implement, where both their process of definition and who can implement the standard is non-discriminatory. More importantly, open standards create an open market where users can switch between competing implementations -- allowing users to fairly select and re-select the best products for their circumstances. In the software world, examples include TCP/IP (the basis of the Internet) and HTML (the key document format of the World Wide Web). A more precise discussion and definition of what an open standard for software is -- and an example of how to use that definition -- is given in “Is OpenDocument an Open Standard? Yes!”; I discuss software open standards and their relationship to security in “Open Standards and Security” (as reported by NewsForge).

In contrast, free-libre / open source software (FLOSS) is software whose license gives users the freedom to run the program for any purpose, to study and modify the program, and to redistribute copies of either the original or modified program (without having to pay royalties to previous developers). Examples include the Linux kernel, Apache web server, and Firefox web browser. Many FLOSS programs are directly developed by for-profit companies, and in any case, FLOSS software is commercial software.

Open standards do not require the use of FLOSS programs; open standards are by definition neutral on whether proprietary or FLOSS implementations are used. Yet there is a relationship between open standards and FLOSS. This paper briefly discusses how open standards aid FLOSS projects, and how FLOSS aids open standards.

I then move on to a mystery: Today, essentially every successful open standard is implemented by FLOSS. In fact, I haven’t found a single counter-example. There’s no fundamental requirement for this, because by definition open standards do not discriminate between FLOSS and proprietary implementations. I discuss what this observation could mean, including the possibility that you may be able to use this observation to predict the future, and even make wise decisions based on those predictions.

Open standards aid FLOSS projects

Open standards aid FLOSS projects, and it’s not hard to see why:

  1. Open standards make it easy for users to adopt a FLOSS program, because users of open standards aren’t locked into a particular implementation. Instead, users who are careful to use open standards can easily switch to a different implementation -- including a FLOSS implementation. It’s been often noted that organizations that wisely avoid proprietary extensions to HTML can easily use various web browsers (such as Firefox).
  2. Open standards also make it easier for FLOSS developers to create their projects, because the standard itself helps developers know what to do. Creating any interface is an effort, and having a pre-defined standard helps reduce that effort greatly.

However, note that open standards aid proprietary projects in exactly the same way, so open standards are very neutral in their help. Open standards make it easy to switch to a proprietary program that implements the standard, and aid implementors of proprietary programs by giving them the same information. If a user uses an open standard, they can switch from a FLOSS implementation to a proprietary one, as well as from proprietary to FLOSS, if there are both proprietary and FLOSS implementations.

FLOSS aids open standards

FLOSS aids open standards, too:

  1. FLOSS implementations can help create and keep open standards open. A FLOSS implementation can be read and modified by anyone; such implementations can quickly become a working reference model that demonstrates what the specification means (clarifying the specification) and demonstrating how to actually implement it. Perhaps more importantly, by forcing there to be an implementation that others can examine in detail, specifications that are unnecessarily hard to implement or use simply won't get any traction - any implementation can reveal spec flaws, and though proprietary implementations can hide some of the issues, a FLOSS implementation reveals all the problems in a proposal. "The Rise and Fall of CORBA" by Michi Henning (ZeroC) states that "the open source community has done a much better job of adhering to [best practices for developing specifications] than have [non-open-source] industry consortia."
  2. FLOSS implementations can help rapidly increase use of the open standard. FLOSS programs can typically be simply downloaded and tried out, making it much easier for people to try it out and encouraging widespread use. FLOSS is not “free” in terms of total cost of ownership (TCO), because there’s usually a cost of training, support, and so on; nevertheless, FLOSS tends to cost much less than proprietary implementations, pressuring proprietary implementations to limit their prices as well. This lower cost of the proprietary implementations also tends to encourage widespread use of the standard, since users are more likely to use a standard if it is not too expensive to use.

Indeed, FLOSS products can by themselves essentially define a (de facto) open standard, because their very existence makes it easy to spread them. Paul Vixie said "compared to the power and relevance of open source software, [the internet standards group IETF] may as well be declared dead for all the relevance it has at this stage". Paul Vixie notes that "zero configuration networking" succeeded by creating a FLOSS implementation first, and that FLOSS programs (like BIND and Sendmail) have essentially created or enforced many interfaces.

One interesting property of FLOSS deserves mentioning: Self-competition. From a user's point of view, open standards are really interesting if there is a marketplace of competing component suppliers that implement the open standard, and allows them to switch to a competitor. So if there is only one implementation of an open standard, and it's proprietary, there's still lock-in (with its problems) due to lack of competition. The same thing doesn't happen with FLOSS, even if there's only one FLOSS implementation. It turns out that a FLOSS program embeds the seeds of competition in it, for several reasons:

  1. Multiple organizations can support the same FLOSS component, so if you get bad support, you can switch.
  2. Anyone or any group that dislikes how the FLOSS project is going can copy the project source code and try to establish a "forked" (competing) project. The ability to create fork means that a FLOSS project always contains the seeds of a competing project.
  3. People who want the FLOSS project to support certain capabilities can often hire someone to add those capabilities to the product, and they can choose who hire.
  4. Most larger FLOSS projects operate in a "Darwinian" process... different contributions are sent to the project, and then someone (typically the project lead) decides which ones are good enough to include. This "bake-off" is internal competition inside a project, but has the same result: the best approaches end up getting accepted.

Of course, there can still be multiple different FLOSS implementations of an open standard, just as there can be multiple proprietary implementations. The HTTP standard has multiple FLOSS implementations, including Apache and lighttpd. The TCP/IP suite has been separately implemented by the *BSDs and by the Linux kernel. Text editors (which depend on the simple standards for text files) have been implemented for a long time with great interoperability; the running competition between the various emacs and vi/vim camps has been going on for a long time.

Mystery: Successful open standards have FLOSS implementations

At the April 2006 LinuxWorld (Boston) someone made an extraordinary claim: “Today, all successful open standards have FLOSS implementations.” I’ve learned that “all” is rarely true for any statement; this statement should be easy to disprove.

Except I can’t disprove it. Typical examples of open standards (such as TCP/IP, HTTP, HTML, DNS, SMTP, POP3, IMAP, IPSec, SSH, SSL, C, C++, and OpenDocument) all have at least one FLOSS implementation. When I started looking at the specialty open standards, I kept finding FLOSS implementations there too! I looked at niche standards such as Z39.50 for search queries... and found that yes, they have them too. Even open standards developed by non-traditional organizations have FLOSS implementations (e.g., GEDCOM for exchanging genealogical data). There are some open standards without a FLOSS implementation, such as some of the OSI protocol stack components, but these are also examples of failures (TCP/IP, which had strong FLOSS support, demolished the OSI stack long ago). Historically this wasn’t a true statement, but at some point in the past this became true, and it shows all the signs of staying true. There may be a successful open standard without a FLOSS implementation, but the difficulty in finding an example suggests that having a FLOSS implementation is very much the rule, not the exception.

In some sense this is very surprising; there is no fundamental requirement that an open standard must have a FLOSS implementation. For example, almost no organization that creates open standards requires this, and many open standards are based on proprietary implementations (not FLOSS ones). The discussions above on how FLOSS helps open standards, and how open standards help FLOSS projects, could explain why they often occur together... but by themselves they fail to predict this 100% (or nearly so) coverage.

I find unanswered questions much more interesting -- they are a mystery, and often we learn about something more fundamental when we examine mysteries.

So why do essentially all open standards have a FLOSS implementation? One possible explanation is that the aid given by FLOSS to open standards is much more important than previously credited. For example, the availability of a reference model has become so important that perhaps open standards creators essentially always ensure there is a FLOSS implementation, through their development process. There is something to this claim, in the sense that I can point to specific standards’ development process which explicitly used FLOSS as a reference model. The development of OpenDocument often depended on FLOSS implementations -- often when a change in the specification was proposed, its impact on a particular reference implementation that could be viewed by all was examined. This is a good thing, because checking out implementation issues before saying the standard is complete is more likely to produce a good standard. The W3C and IETF often use FLOSS implementations as reference implementations, to help ensure that they’re creating a good standard. But this does not happen all the time, which means this would only explain why this circumstance is common -- it still fails to explain the essentially universal coverage of open standards by FLOSS projects.

Another explanation comes to mind, which I think is more likely. If it’s true, it gives us insight into FLOSS’ current role in information technology (IT). Here’s the possible explanation: FLOSS has become extremely widespread in IT, so much so that if an open standard is successful, there will naturally be a FLOSS implementation. This is more insightful than it appears at first. There are many studies showing FLOSS’ strong presence in many specific markets, such as web servers, mail servers, and so on. But it’s much harder to answer the question “how widespread is FLOSS in the entire IT industry, in all segments?” This is especially hard to answer when you’re interested identifying “key” components, because it’s hard to define “key” in an unambiguous way. Saying “key” components have an open standard is imperfect, but it isn’t a bad estimate; people only bother to work to create an open standard if it is a key interface for them. And it’s reasonable to believe that the larger FLOSS community would concentrate on implementing open standards first -- again, efforts to define open standards tend to focus on key interfaces, and since they’re defined they’d tend to be the interfaces that would be the easiest for FLOSS projects to implement as well.

If that’s true, then essentially all successful open standards are implemented by FLOSS because FLOSS is itself so widespread in IT. At some point in the past FLOSS grew so rapidly that we reached a major milestone: total FLOSS implementation of the key interfaces in the IT industry (as identified by open standards). That suggests that FLOSS is not just implemented in a few key places, but in fact has universally taken root across the IT industry... not just in a few widely-trumpeted situations, but the “unsexy” ones as well.

Can we predict the future with this?

Theories are only interesting if they can be used to predict future events. And this theory makes a prediction! If it’s true that all successful open standards have a FLOSS implementation, it follows that an open standard will not be successful unless and until a FLOSS implementation becomes available. You can’t predict the other way -- having a FLOSS implementation is no guarantee that an open stanard will succeed -- but the lack of a FLOSS implementation appears to be a serious problem. So:

  1. If you are a software acquirer, and trying to decide if you want to require a particular open standard in your requirements, it appears that it would be reasonable to ask “is there a FLOSS implementation, or will there be one soon?” If there is no FLOSS implementation of an open standard, you should consider avoiding the standard, because it’s riskier to depend on.
  2. If you are developing an open standard, you should try to ensure that there will be at least one FLOSS implementation. Frankly, I’d suggest having it developed while the standard is being developed -- that way, you can use the FLOSS implementation to vet the standard (to make sure it’s a good one) and as a concrete way to explain the meaning of the specification and aid other implementors.

When you think about it, this makes sense. FLOSS is so pervasive in our industry today that if there are no FLOSS implementations, it suggests that it’s not really an open standard (perhaps there are some legal traps), that it’s viewed as likely to fail, or perhaps there’s some other critical problem. Also, if there’s a competing open standard that has a FLOSS implementation, those who prefer FLOSS products may use that instead... creating a move of users away from the FLOSS-less standard. Having competing standards is a bad idea anyway, but if this observation is predictive -- and it’s reasonable to think it might be -- then it gives us at least one tool to help figure out the eventual winner.

Conclusions

Open standards and FLOSS each support the other. But more interestingly, successful open standards seem to always have at least one FLOSS implementation, and above I speculate why this is so.

This suggests a simple rule: An open standard without a FLOSS implementation is probably too risky for users to require. It also suggests that developers of open standards should encourage the development of at least one FLOSS implementation, since they appear to be essential for acceptance of the standard. That doesn’t mean that all users will use the FLOSS implementation, but they will at least have the option.

It’s been noted that creating open standards is not easy -- and that’s true. But there’s also broad agreement that closed standards only benefit the market leaders, not the users -- such situations quickly lead to dependency, stagnation, a lack of innovation, and extremely high unnecessary costs. Nothing worthwhile is easy, but many organizations have realized that open standards are vital to them and work hard to encourage them (examples include the European Commission, as demonstrated by the “European Interoperability Framework”, and the U.S. DoD). We need to continue to encourage open standards, and work to understand their relationship with other IT trends, so we can deploy open standards in an even smarter way in the future.