David A. Wheeler's Blog

Open government: Default release as OSS and Open Access

Sun, 28 Feb 2010 22:49 GMT

U.S. government agencies are soliciting ideas on how to make them more transparent, participatory, collaborative and innovative.

Please support proposals to release government-funded works by default as open access (for research papers) or as open source software (for software). An example is the proposal to the National Science Foundation (NSF) called Public funding = Public viewing. This proposal recommends that publicly funded projects must be published as open access and all data and code shared as open source software. Please vote for this, make helpful comments, and so on. Similarly, please vote for and/or add similar proposals for other agencies where they apply.

If "we the people" pay for research and development, then "we the people" should normally get the results. I can see the need for exceptions — particularly for classified works — but those should be exceptions. In short, I think this kind of proposal makes sense.

As I've commented before, Government-developed Unclassified Software should by default be released as Open Source Software, and research papers produced from U.S. government funding should be open access. So please make sure that U.S. agencies know this. Thanks.

Free/Libre/Open Source Software's big win: Jacobsen/JMRI v. Katzer

Mon, 22 Feb 2010 15:56 GMT

There’s been a major legal victory for Free/Libre/Open Source Software (FLOSS): Jacobsen v. Katzer. Articles like Bruce Perens’ “Inside Open Source’s Historic Victory” and A Big Victory for F/OSS: Jacobsen v. Katzer is Settled give many of the specifics; here is a quick summary.

Bob Jacobsen is a high-energy physicist who developed (as a hobby) the Java Model Railroad Interface (JMRI) Project. JMRI is a set of FLOSS Java tools for configuring and controlling model railroad trains. Matthew Katzer used loopholes in the law to patent ideas that Jacobsen and others had created and publicly discussed first, domain-squatted, tried to embarass Jacobsen to Jacbonsen’s employer, and used part of Jacobsen’s JMRI software in Katzer’s own product without complying with the JMRI license (by not providing the required credit). The JMRI has a short summary of this unpleasant fight, as well as lots of details and court papers.

What’s impressive was that Bob Jacobsen stuck through a very hard series of events. At first the court didn’t seem to understand FLOSS at all, and Jacobsen was handed some very unpleasant defeats. At one point, Jacobsen had to pay over $30,000 of his own money.

But Jacobsen persevered, and won critical rulings and a final settlement that is really a complete victory for him. In 2008 the United States Court of Appeals for the Federal Circuit vacated the district court’s ruling and held that the terms of the Artistic License (a FLOSS license) are enforceable. The court said, “Open source licensing has become a widely used method of creative collaboration that serves to advance the arts and sciences in a manner and at a pace that few could have imagined just a few decades ago”. On February 18, 2010, the parties finally settled. Among other terms, Jacobsen has won $100,000, Katzer is forbidden to use Jacobsen’s software, and the two patents at issue have been disclaimed. What’s more, the rulings stemming from this case have created a precedent that FLOSS licenses are legally enforceable, eliminating a lot of uncertainty, and because there is a final settlement it is not possible to appeal the case. Strictly speaking, the precedents do not automatically apply everywhere in the U.S., but even where they do not strictly apply, they will still have a strong weight.

This result is critically important to FLOSS. If FLOSS developers could not enforce their licenses, the probable result would be that a lot of such software would never be written. The Amici Curiae brief by Creative Commons Corporation et al. and the Software Freedom Law Center Amicus Brief in Jacobsen v. Katzer both do a nice job explaining why getting this ruling right was so important.

So, my hat’s off to Bob Jacobsen. Through his persistence, he’s made the world better for all of us.

California: Open Source Software is Okay!

Sat, 09 Jan 2010 13:07 GMT

The California state government has officially declared that it’s okay to use open source software inside the California state government. On January 7, 2010, the California Office of the State Chief Information Officer (OCIO) released Information Technology Policy Letter (ITPL) 10-01, titled “Open Source Software Policy” . A key purpose of ITPL 10-01 is to “formally establish the use of Open Source Software (OSS) in California state government as an acceptable practice”, and the first sentence of its policy statement is that “The OCIO permits the use of OSS”. It even includes the ten-point open source definition (OSD) as promulgated by the Open Source Intiative, to make sure that there’s no misunderstanding.

I think this is a big deal. Officially saying “it’s okay to use free/libre/open source software (FLOSS)” is really important before FLOSS can get widespread use in governments. Most technologists already understand the potential advantages of FLOSS, but they encounter a lot of resistance when they try to use or develop FLOSS in large organizations like governments. Far too many middle managers are instinctively afraid of change from “the way we’ve always done it”. For example, they may be afraid of unseen problems, or afraid their bosses will rake them over the coals later. Far too often the middle managers have misunderstandings about FLOSS, too. For example, many managers still believe the myth that “you can’t get support” and are unaware of the many companies that do provide such support. Companies that make competing proprietary products are delighted (of course) when governments don’t consider their competition... but in an era of tight budgets, it doesn’t make sense for governments to ignore competing (and often less expensive) products. When top officials give official “top cover” permission to consider FLOSS, then the technologists and middle managers are far more likely to fairly and honestly consider them.

Also, the fact that it’s California matters. The economy of the California is larger than most countries (if it were a country, it would be third through tenth in the world depending on how you measure it). Anything the state of California does can influence other states and countries; acts like this further legitimize the user of Free/Libre/Open Source Software (FLOSS).

Of course, the state of California isn’t the only government organization to release a memo officially declaring that it’s okay to use free/libre/open source software (FLOSS). Just looking inside the U.S., the U.S. DoD did this in 2003, the Office of Management and Budget (OMB) released a somewhat similar memo in 2004 that applied to the entire U.S. federal government, the U.S. Navy did this in 2007, and the the U.S. DoD released clarifying guidance in 2009 re-emphasizing this point. And that’s only a few examples from U.S. government organizations; the examples from around the world are legion. It’s really difficult to get people to change what they do... as you can tell from the number of times that various U.S. federal government organizations have had to state and re-state it. Still, they really do have an effect. Official policy statements that FLOSS is used, such as the one California just released, are a necessary first step to changing things from “the way we’ve always done things”.

Moglen on Patents and Bilski

Thu, 31 Dec 2009 10:48 GMT

Eben Moglen has a very interesting presentation on patents (including comments on Bilski) that was originally presented on Nov. 2, 2009. Software patents and business method patents have been a disaster for the U.S. and world economy, and he has some interesting things to say about how we got here (and how it could be fixed).

One interesting point he made, which I hadn’t heard before, is that there is a fundamental conflict between the patent system and the Administrative Procedure Act of 1946 (aka the APA). Nearly all of the U.S. government must obey the APA before creating new rules and regulations. According to the APA, U.S. agencies must keep the public informed, provide for public participation in the rulemaking process, establish uniform standards for rulemaking and adjudication, and provide for judicial review. In particular, agencies normally have to perform a cost-benefit analysis.

But the patent system pre-existed the APA. Patents, since they are government-created monopolies, can constrain people in the same ways that any other rule or regulation can. However, the government does not follow the APA to determine if each proposed patent should be granted. Instead, the old patent process was essentially grandfathered in instead, as a special exception to the APA. Because the APA is not considered when examining each patent, no one in government asks the normally-required question “How will each proposed patent be publicly reviewed before it is granted?”. Patents on ideas that are patently obvious are routinely granted, in part because there is no public review before they are granted and because the patent office (by policy) ignores most information available to the public. All because the patent-granting process is not required to enable public participation in the rulemaking process, in this case, the process for permitting the granting of a patent. Also, when examining a patent to determine if it should be granted, no one asks normally-obvious questions like:

“What is the result of the cost-benefit analysis for this patent? Is there good evidence that granting this government-created monopoly would have societal benefits that greatly exceed all of that monopoly’s costs to society?”
“If the costs exceed the benefits, what is the smallest term that could be granted in which the benefits to society exceed their costs?”.

Because the patent system predates the APA, all potential harms to society from a patent are completely ignored during the patent examination process. If patents were individually considered as new regulations under the APA, such questions would need to be carefully considered. That’s an interesting point Moglen makes.

It’s my hope that the Supreme Court will clearly stop software patents. We shall see.

U.S. research should be open access

Sun, 13 Dec 2009 19:24 GMT

The Office of Science and Technology Policy (OSTP) has launched a “public consultation on Public Access Policy”, to see if research funded by U.S. grants should be made available as open access results. I think this is important — I believe publicly-funded unclassified research should actually be made available to the public.

Historically, the U.S. pays a fortune for research, the results are written up as papers for journals, and then various publishers acquire total rights to these papers and charge exorbitant monopoly fees for them. The result: Most U.S. citizens cannot afford to see the research their taxes pay for.

The basic question here is really straightforward: Should publicly-funded research results be made available directly to the public instead? Or, should private companies continue to gain ownership over publicly-funded results, for either nothing or a tiny fraction of the public’s costs?

A small number of journal publishers and societies strongly want to keep things the way they are, of course. It makes sense from their point of view; everybody likes free (or nearly free) money! Historically, this arrangement was created because it can be expensive to publish and manage paper. However, that rationale has become completely obsolete. Few people want the paper any more — they want the research, on-line, without a paywall. And don’t give me nonsense about the “costs” of peer review. Many journals don’t pay their reviewers (the reviewers do it gratis), and even if they did, the total control they gain is still unjustified; the U.S. government spends far more per paper than they do for review.

The current sequestering of research is not good for science or the country. I’m currently reading the interesting book “Are We Rome?” by Cullen Murphy, and I can’t help but see some parallels. Chapter 3 is all about “when public good meets private opportunity”. Private organizations may pay for private research, and then keep their results private. But when the public pays for research, it should be shocking if it does not get released back to the public. And by “released back”, I mean released back at no fee at all.

So who will pay for the printing, complex peer review, storage, and fancy indexing of these research results? I think the very question shows a failure to understand current technology, but let’s answer it anyway. Most peer review isn’t paid-for anyway, and if it is, it’s a tiny cost compared to the research itself. Storage? Don’t make me laugh; for $100 I can buy storage for the all of the U.S. research papers for a year. Indexing? The government shouldn’t be doing serious indexing at all!! Just put it on a government site with a basic form filled out (title, authors, date, keywords, abstract, and a link to the actual paper on the government site). If it’s not behind a paywall, the many commercial search systems will index it for you.

I do think there should be a centralized government repository of such papers. If it’s distributed, then papers could be lost without anyone knowing it. I think they should be freely redistributable, so others can copy what they want, but a centralized repository would make sure that we keep all of them available forever. Also, bandwidth costs can be reduced by scale. There’s a risk that they all get lost at once, but it’s easier to copy everything if there’s one place to start from. If it’s a complicated site, then they’ve done it wrong.... for each paper there should be a simple “summary” page with title, authors, etc., and the actual paper itself.

OSTP cites the experience of NIH; NIH did wonderful work for releasing as open access, and in my mind the real problems are that they didn’t go far enough. First, NIH has a one-year embargo... if I already paid for it (and I did), why should wealthy people and organizations get the results first? Second, NIH only considers the actual papers, not the data and software programs that support the works... yet often those are more important. If they were funded by the public, then the public should get them (unless they’re classified, of course, but then they shouldn’t be released at all). I’m sure there are complications and exceptions, but a “default open access” policy would go a long way.

So please, tell the OSTP that the U.S. should release government-funded research as open access publications, available to anyone on the Internet without a paywall. In short, if “we the people” paid for it, then “we the people” should get it. For more information, see this Request for Information (RFI)

Trusting Trust, DDC, and Free-Libre/Open Source Software (FLOSS)

Fri, 13 Nov 2009 14:02 GMT

As I noted in my blog, I’ve just released my dissertation “Fully Countering Trusting Trust through Diverse Double-Compiling (DDC). But what does that mean for Free-Libre/Open Source Software (FLOSS)? In short, it’s fantastic news for FLOSS, but to explain why that’s so, I need to backtrack first.

The “trusting trust” attack is a nasty computer attack that involves creating a subverted compiler in such a way that it even subverts compilers. It was originally reported in a 1974 security evaluation of Multics, but most people heard about it from Ken Thompson’s 1984 Turing Award presentation (Ken Thompson is a creator of Unix). This attack is incredibly nasty, and what’s worse, until now there’s been no effective countermeasure to it. Indeed, some have claimed that it could not ever be countered, making the whole idea of “computer security” a non-starter.

The “trusting trust” attack appears to be especially devastating to FLOSS. The problem is that with the trusting trust attack, the source code that people review does not correspond to the executable that’s actually running, and that seems to completely torpedo the “many eyes” review that FLOSS makes possible. The whole world could carefully review a program’s source code, but it wouldn’t matter if the compiler turns it undetectably into something malicious.

Thankfully, there is an effective countermeasure, which I’ve named Diverse Double-Compiling (DDC). You can see my dissertation which explains what it is, proves that it works, and even demonstrates it with several compilers including GCC. (I will be giving a public defense of it on November 23, 2009, if you’d like to come.) This means that source code review, such as mass review of FLOSS code, can now actually work.

But there’s more, because there’s an interesting catch with DDC. DDC counters the trusting trust attack, but it’s only useful for people who have access to the compiler source code. Fundamentally, DDC is a technique for determining if a compiler executable corresponds with its source code, but only people who have the source code can apply DDC to see if that’s true. What’s more, only people who have access to the source code will find the statement “the source and executable correspond” particularly useful. (You could use trusted intermediaries, but this requires total trust in those intermediaries, making such claims far weaker than claims that anyone can check.) What’s more, DDC is actually useful beyond what we normally think of as compilers, because you can redefine “compiler” as including other parts (such as the operating system). In that case, you can even show that the system’s executables all correspond to their source code. But you can only use DDC to counter the trusting trust attack if you have access to the source code.

So we now have a radical change. Now that DDC has been shown to work, we can see that software with available source code (including FLOSS) has a fundamental security advantage over other software. That doesn’t mean that all FLOSS is more secure than all proprietary software, of course. But FLOSS already had a general security advantage because it better meets Saltzer & Schroeder’s “Open design principle” (as explained in their 1974-1975 papers). Now we have an attack — the trusting trust attack — for which FLOSS has a fundamental security advantage. The time of ignoring FLOSS options, because of misplaced notions that FLOSS cannot be as secure as proprietary software, needs to come to an end.

Notes about the DoD and OSS memo

Wed, 28 Oct 2009 17:01 GMT

Yesterday I posted about the new 2009 DoD memo about open source software. I'm delighted to see that the word is getting out. Slashdot, Linux Weekly News, and LXer.com all mentioned the new memo and even pointed to my post. Others are noting the new memo too, including CNet's Matt Asay, InformationWeek's J. Nicholas Hoover, InformationWeek's Serdar Yegulalp, NetworkWorld, and The H. Dan Risacher has posted on Slashdot some background and history for this new 2009 DoD memo. He notes, for example, that "The lawyers were by far the biggest delay" in getting this memo released.

There's some supporting information for this memo at the DoD Free Open Source Software (FOSS) Communities of Interest (COI) site, which posts the memo itself and a supporting DoD Open Source Software Frequently Asked Questions (FAQ) document.

To help potential users, I've updated my presentation Open Source Software (OSS) and the U.S. Department of Defense (DoD), which I hope will clarify some things. I should also remind people about the 2003 MITRE study "Use of Free and Open Source Software (FOSS) in the U.S. Department of Defense", which showed that in 2003 Free/libre/open source software (FLOSS, FOSS, or OSS) was already widely used in the DoD.

New DoD memo on Open Source Software

Tue, 27 Oct 2009 18:02 GMT

The U.S. Department of Defense (DoD) has just released Clarifying Guidance Regarding Open Source Software (OSS), a new official memo about open source software (OSS). This 2009 memo should soon be posted on the list of ASD(NII)/DoD CIO memorandums. This 2009 memo is important for anyone who works with the DoD (including contractors) on software and systems that include software... and I suspect it will influence many other organizations as well. Let me explain why this new memo exists, and what it says.

Back in 2003 the DoD released a formal memo titled Open Source Software (OSS) in the Department of Defense. This older memo was supposed to make it clear that it was fine to use and develop OSS in the DoD. Unfortunately, as the new 2009 memo states, "there have been misconceptions and misinterpretations of the existing laws, policies and regulations that deal with software and apply to OSS that have hampered effective DoD use and development of OSS".

This new 2009 memo simply explains "the implications and meaning of existing laws, policies and regulations", hopefully eliminating many of those misconceptions and misinterpretations. A lot of the "meat" is in the Attachment 2, section 2 (guidance), so let's walk through that:

Point (a) says that "In almost all cases, OSS meets the definition of 'commercial computer software' and shall be given appropriate statutory preference..." This confirms what I've been saying for years (see FLOSS is commercial software). This is important in the U.S. government contracting world, because government contracts are required to prefer commercial computer software. If OSS is commercial software, then it needs to be preferred in the same way that proprietary software is preferred (compared to rebuilding everything from scratch). It also makes many decisions easy; there are lots of rules for "commercial software", and since OSS is "commercial software", then contractors already know what the rules are.
Point (b) says that the DoD is "required to conduct market research... [and should] include OSS [in the research] when it may meet mission needs". That's a sea change all by itself — that means that people will typically need to justify not considering OSS options. What follows this statement is perhaps even more important... here we have an official government document acknowledging specific advantages OSS can have. The memo says that "There are positive aspects of OSS that should be considered", noting "the continuous and broad peer-review enabled by publicly available source code", "the unrestricted ability to modify software source code enables the Department to respond more rapidly to changing situations, missions, and future threats", the reduced risk of lock-in, the possibility of lower costs, and so on. Let's be clear: this is not a statement that the DoD will only use OSS, indeed, it clearly states that "the software that best meets the needs and mission of the Department should be used, regardless of whether the software is open source". But it does tell people that there may be some very pointed questions if the OSS options aren't considered, and that's good for both the military and the taxpayer.
Point (c) counters a weird problem specific to the DoD. DoD Instruction 8500.2 has a control called "DCPD-1 Public Domain Software Controls". This control says that if you use a binary program, you must either have a warranty or the source code for a program. That makes sense; if there's a defect in the program, the government wants to know that it will be fixed by someone else or have the rights to fix it themselves. Problem is, people decided to ignore the second part, and re-interpreted this control as "cannot use open source software", which is not what it says. Hopefully this terrible misconception will start going away soon.
Point (d) says, "The use of any software without appropriate maintenance and support presents an information assurance risk. Before approving the use of software (including OSS), system/program managers, and ultimately Designated Approving Authorities (DAAs), must ensure that the plan for software support (e.g., commercial or Government program office support) is adequate for mission need". This actually hits at a number of issues. Some people say nonsense like "OSS has no support", which is silly; Red Hat just entered the S&P 500, primarily through paid support. But it is true that when you use software — any software — for a critically important function, you better have a plan for maintenance and support. This isn't unique to OSS; it's simply a true statement in general. OSS does add additional options for how you do support, including self-support and fully competed support, but you still need to plan for it.
Point (e) notes that there "is a misconception that the Government is always obligated to distribute the source code of any modified OSS to the public", and then explains the real situation.
Point (f) says that "software source code and associated design documents are 'data' as defined by DoD Directive 8320.02, and therefore shall be shared across the DoD as widely as possible to support mission needs..." This strikes a big blow at contractors who try to "hoard" software that the government paid to develop. In short, DoD directives already require that such software be shared with others.
Point (g) says, "Software items, including code fixes and enhancements, developed for the Government should be released to the public (such as under an open source license) when all of the following conditions are met...". This certainly doesn't say that all DoD software will be released as OSS (nor would you expect that), but now the DoD is clarifying the issues to be considered before releasing software as OSS. It's actually a simple 3-part test:
1. "The project manager, program manager, or other comparable official determines that it is in the Government’s interest to do so, such as through the expectation of future enhancements by others."
2. "The Government has the rights to reproduce and release the item, and to authorize others to do so...."
3. "The public release of the item is not restricted by other law or regulation..."

But perhaps most important is this memo's opening statement: "To effectively achieve its missions, the Department of Defense must develop and update its software-based capabilities faster than ever, to anticipate new threats and respond to continuously changing requirements. The use of Open Source Software (OSS) can provide advantages in this regard...". As with the later part (b), here we have an official government document acknowledging that OSS can have a significant advantage. What's more, these potential advantages aren't necessarily just minor cost savings; OSS can in some cases provide a military advantage. Which is a more-than-adequate justification for considering OSS, as I have been advocating for years.

I'm really delighted that this memo has finally been released. I participated in the original brainstorming meeting to create this memo (as did John Scott), and I reviewed many versions of it, but many, many other hands have stirred this pot since it began. It took over 18 months to create it and get it out; getting this coordinated was a very long and drawn-out process. My thanks to everyone who worked to help make this happen. In particular, congrats go to Dan Risacher, who led this project to its successful completion.

By the way, if you're interested in the issue of open source software in the U.S. military/national defense, you probably should look at Mil-OSS (at least, join their mailing list, and consider going to their upcoming conference; I was a speaker at their last one). If you're interested in the connection between open source software and the U.S. government (including the military), you might also be interested in the upcoming GOSCON conference on November 5, 2009 (I'm one of the speakers there too).

CVC3 License Changed to BSD

Sat, 17 Oct 2009 15:07 GMT

CVC3 is one of the better automated theorem provers. Given certain mathematical assertions, it can in many cases prove that certain claims follow from them. Some tools that can prove properties about programs use CVC3 (and/or similar programs). For example, the Frama-C Jessie plug-in for C and Krakatoa for Java use Why, which can build on one of several programs including CVC3.

Problem is, CVC's license has historically been a problem. I understand that its authors intended for CVC3 to be Free/Libre/Open Source Software (FLOSS), but unfortunately, it was released with additional license clauses that resulted in yet another non-standard license. This was an unfortunate mistake; as I note in my essay on GPL-compatible licenses, it is absolutely critical to choose a standard FLOSS license when releasing FLOSS. In this case, the big problem was the addition of an "indemnification" clause that was really scary; to some, at least, it seemed to imply that if the CVC3 authors were sued, anyone who used or copied the program was obligated to pay their legal bills. Interpreted that way, no one wanted to touch the program... how could any user possibly know their risks? Fedora eventually ruled that this license was non-free (aka not FLOSS), and thus could not be included in Fedora. There was a less-serious problem that if you made a change to the program, you had to change the name... since the program couldn't even compile without a change (at the time), this meant that you had to change the name almost instantly. There is a reason that people have converged on standard FLOSS licenses; if your lawyer says you need to add non-standard clauses, be wary, because the result may be that few people can use your program.

I'm delighted to report that this has a happy ending. CVC3's license has just been changed to a straight BSD license - a well-known license that is universally acknowledged as being FLOSS. This means that there are no licensing problems for Linux distributions. Only about a day after he found this out, Jerry James has submitted a CVC3 package to Fedora. So, I expect that in a relatively short time we'll see CVC3 available directly in common Linux distribution repositories.

I think this is a helpful step towards open proofs, which are cases where an implementation, its proofs, and the necessary tools are all FLOSS. Having a good tool like CVC3 to build on makes it easier to develop useful tools. My hope is to mature formal methods tools so that they can be more scaleable, applicable, and effective than they are today. It's clear that a single little tool cannot possibly do the job; we need suites of tools that can work together. And this is a promising step in that direction.

Auto-DESTDIR released!

Wed, 12 Aug 2009 10:31 GMT

I’ve just released Auto-DESTDIR, a software package which helps automate program installation on POSIX/Unix/Linux systems from source code. If you have the problem it solves — automatic support for DESTDIR — you want this!

A little background: Many programs for Unix/Linux are provided as source code. Such programs must be configured, built, and installed, and that last step is normally performed by typing “make install”. The “make install” step normally writes directly to privileged directories like “/usr/bin” to perform the installation. Unfortunately, most modern packaging systems (such as those for .rpm and .deb files) require that files be written to some intermediate directory instead, even though when run they will be in a different filesystem location (because of security issues). This redirection is easy to do if the installation script supports the “DESTDIR convention”; simply set DESTDIR to the intermediate directory’s value and run “make install”. Supporting DESTDIR is a best practice when releasing software. Unfortunately, many source packages don’t support the DESTDIR convention. Auto-DESTDIR causes “make install” to support DESTDIR, even if the provided “makefile” doesn’t support the DESTDIR convention. Auto-DESTDIR is released under the “MIT” license, so it is Free-libre/open source software (FLOSS).

Auto-DESTDIR is implemented using a set of bash shell scripts that wrap typical install commands (such as install, cp, ln, and mkdir), These wrappers are placed in a special directory. The run-redir command modifies the PATH so that the directory with these scripts is listed first, and then runs the given command. The make-redir command invokes “make” using run-redir, along with some extra settings to simplify things. For more information on this approach, and why this is a good way to automate DESTDIR, see the paper Automating DESTDIR, especially its section on wrappers.

So please take a look at the Auto-DESTDIR software package, if you have the problem it solves.

SPARK released as FLOSS (Free/ Libre / Open Source Software)!

Fri, 05 Jun 2009 17:41 GMT

The SPARK toolsuite has just been released as FLOSS (Free/ Libre / Open Source Software) by Praxis (its creator). This is great news for those who want to make software safer, more reliable, and more secure. In particular, this means that Tokeneer is now an open proof. If you haven’t been following this, here’s some background.

Software is now a part of really critical systems (ones that need “high assurance”), yet often that software is not as safe, reliable, or secure as it needs to be. I believe that in the long term, we will need to start proving that our very important programs are correct. Testing by itself isn’t enough; completely testing the trivial “add three 64-bit integers” program would take far longer than the age of the universe (it would take about 2x10^39 years). The basic idea of using mathematics to prove that programs are correct — aka “formal methods” — has been around for decades. There are a number of cases where formal methods have been applied successfully, and I’m glad about that. And yet, applying formal methods is still relatively rare. There are many reasons for this, such as inadequate maturation and capabilities of many formal methods tools, and the fact that relatively few people know how to apply formal methods when developing real programs. But what, in turn, is causing those problems? It’s true that applying formal methods is a hard problem that hasn’t received the level of funding it needs, but still, it’s been decades!

I believe one problem hindering the maturation and spread of formal methods is a “culture of secrecy”. Details of formal method use are often unpublished (e.g., because the implementations are proprietary or classified). Similarly, details about formal methods tools are often unshared and lost (or have to constantly re-invented). Biere’s “The Evolution from LIMMAT to NANOSAT” (Apr 2004) gives an example: “From the publications alone, without access to the source code, various details were still unclear... Only [when CHAFF’s source code became available did] our unfortunate design decision became clear... The lesson learned is, that important details are often omitted in publications and can only be extracted from source code. It can be argued, that making source code of SAT solvers available is as important to the advancement of the field as publications”

This “culture of secrecy” means that researchers/toolmakers often don’t receive adequate feedback, researchers/toolmakers waste time and money rebuilding tools, educators have difficulty explaining formal methods (they have no examples to show!), developers don’t understand how to apply it (and it has an uncertain value to them), and evaluators/end-users don’t know what to look for.

I believe that a way to break through this “culture of secrecy” is to develop “open proofs”. But what are they? An “open proof” is software or a system where all of the following are free-libre / open source software (FLOSS):

the entire implementation
automatically-verifiable proof(s) of at least one key property, and
required tools (for use and modification)

Something is FLOSS if it gives anyone the freedom to use, study, modify, and redistribute modified and unmodified versions of it, meeting the Free software definition and the open source definition.

Imagine if we had a number of open proofs available. There could be small open proofs that could be used for learning (e.g., as examples and use in class exercises). There could be proofs of various useful functions and small applications, so developers could see how to scale up these techniques, directly reuse them as components, or use them as starting points but add additional (proven) capabilities to them. When problems come up (and they will!), toolmakers and developers could work together to find ways to mature the tools and technology so that they’d be easier to use (e.g., so more could be automated). In short, imagine there was a working ecosystem where researchers/toolmakers/educators, developers of implementations to be proved, and evaluators/end-users could work together by sharing information. I believe that would greatly speed up the maturing of formal methods, resulting in more reliable and secure software.

In this context, Praxis has just released the SPARK GPL Edition. This is their SPARK toolsuite (a formal methods tool) released under the GNU General Public License aka GPL (the most common FLOSS license). So, what’s that?

SPARK is a variant of the Ada programming language, designed to enable proofs about programs (by adding and removing some features of Ada). The additions are in special comments, so SPARK programs can be compiled by a normal Ada compiler like GNAT (which is part of gcc). The Open Proofs page on SPARK has some information on SPARK. The page What is Special About SPARK Contracts? gives a nice quick introduction to SPARK, which I will quote here. It points out that the Ada line:

        procedure Inc (X : in out Integer);

just says there is some procedure “Inc” that may read a value X, and may write it out, but that’s it. In SPARK, you can add much more precise information, and the SPARK tools can then check to see if they are true. For example, if you say this using SPARK:

        procedure Inc (X : in out Integer);
        --# global in out CallCount;
        --# pre  X < Integer'Last and
        --#      CallCount < Integer'Last;
        --# post X = X~ + 1 and 
        --#      CallCount = CallCount~ + 1;

then the SPARK tools will ensure at compile-time (not run-time) that:

X and global variable CallCount must be read by at least one path and must be updated by at least one path through the procedure
Inc can only called when both X and CallCount are less than Integer’Last. The “pre” means “precondition”.
After Inc runs, both X and CallCount will always be incremented by one (X~ refers to the initial value of X). The “post” means “postcondition”.

You can learn more about SPARK from the book High Integrity Software: The SPARK Approach to Safety and Security” by John Barnes. Sample text of Barnes’ book is available online. The open proofs page on SPARK has more information.

This means that the “Tokeneer” program is now an open proof. Remember, to be an open proof, a program’s implementation, proofs, and required tools have to be open source software. Tokeneer was a sample program written to show how to apply these kinds of techniques to actual systems (instead of trivial 5-line programs). The Tokeneer program itself, and its proofs, have already been released as open source software. Many of the tools it required are already FLOSS (e.g., fuzz and LaTeX for its formal specifications, and an Ada compiler to compile it). Now that SPARK has been released as FLOSS, people can examine this entire stack of software to make improvements in all the technologies, as well as learn from them and create improved implementations. No, this doesn’t suddenly make it trivial to make proofs about complex programs, but it’s a step forward.

If you are interested in making future software better, please help the open proofs project. You don’t need to be a math whiz. For example, if you know how to do shell scripting, please help us package some promising formal methods tools (like SPARK) so they are easy to install. It’s hard to get people to try out these tools (and give feedback) if they’re too hard to install. If you know of formal methods software that is rotting in some warehouse, try to get it released as FLOSS. I think all government-funded unclassified research software should be released as FLOSS by default, since “we the people” paid for it! If you’re interested in the latest software technology, try out a few of these formal methods tools, and release as FLOSS any small programs and proofs you develop with them. Send the toolmakers feedback, or write down their strengths and weaknesses to help others understand them. SPARK is a tool that can be used, right now, in certain circumstances. I have no illusions that today’s formal methods tools are ready for arbitrary 20 million line programs. But if we want future software to be better than today, we need to figure out how to mature formal methods technology and make it better-understood so that it can mature and scale. I think making top-to-bottom worked examples and starting points can help us get there.

Parchment: Running the Z-machine

Thu, 28 May 2009 20:03 GMT

I just learned of fun web application called Parchment. Parchment lets you play interactive fiction (I.F., aka "text adventure games") using just your web browser. It only works with I.F. in "Z-machine" format, but that's a very common format.

So go to the parchment site and try out something from their long list of interactive fiction... now you don't need to install anything! That includes my small replayable puzzle "Accuse" (my Accuse source code is already available).

If you want more information about it, here's a brief post about Parchment by its author, Atul Varma. Atul built this based on an existing program, Thomas Thurman's Gnusto. Both are open source software (using the GPLv2 license). Once again, this demonstrates the neat thing about community-developed software; one person developed a program for one circumstance, and another extended it for a different circumstance.

There are several tools available for creating interactive fiction. I've been watching Inform 7 for a while, with interest, because it takes a radically different approach to writing code. Inform 7 is a natural-language programming language that tries to actively exploit features of natural language to make developing these kinds of things easier. You can see a brief Inform 7 tutorial if you're curious, as well as the full Writing with Inform documentation. Inform 7 isn't itself OSS, though significant portions are; inform 6 (a key substrate) and many other portions including the Inform 7 standard rules are released under the Artistic License 2.0. The extensions are released under the "Attribution Creative Commons licence"; that's not normally a license used for software, but I think it'd meet the criteria for OSS, and Fedora approves of this license for content. I hope that someday the rest will be released as OSS as well. The logic behind Inform 7 is described in "Natural Language, Semantic Analysis and Interactive Fiction" by Graham Nelson. If you're interested in some of the technical stuff behind it, the text of the Standard Rules, the text of the extensions, Inform 7 for programmers, and the Chart of Rules can tell you more.

Wikipedia changes its license

Sat, 23 May 2009 12:36 GMT

The Wikimedia Foundation (WMF) will change the licensing terms on all its materials — including Wikipedia. Now, all of its existing material will be released under the Creative Commons Attribution-ShareAlike (CC-BY-SA) license in addition to the current GNU Free Documentation License (GFDL). The WMF says “This change is meant to advance the WMF’s mission by increasing the compatibility and availability of free content.” This means that Wikipedia material can now be combined with the vast amount of CC-BY-SA licensed material, and Wikipedia can now include the volumes of CC-BY-SA material (that material will just be CC-BY-SA). It also makes it easier to use Wikipedia material (and other material from the Wikimedia Foundation).

I think this is a good thing overall. Incompatible licenses are a real scourge on community-developed works. Past experience shows that license incompatibility can be a real problem for free-libre/ open source software (FLOSS or OSS), in particular. Bruce Perens warned about FLOSS license incompatibility back in 1999! As I argue in Make Your Open Source Software GPL-Compatible. Or Else, you should release free-libre/ open source software (FLOSS) using a GPL-compatible license. You don’t need to use the GPL, but using a GPL-compatible license (like the MIT, BSD-new, LGPL, or GPL) so means that people can combine your software with other software to create larger works. I show how this works in The Free-Libre / Open Source Software (FLOSS) License Slide, which has a simple graph showing how common FLOSS licenses can work together. Wikipedia articles aren’t software, but the principles still apply - licenses need to enable community-developed works, not disable them.

Now, nothing is perfect. One nice benefit of the GNU Free Documentation License (GFDL) is that it requires that readers be able to get editable versions whose format specification is available to the public (for details, see its text on “transparent” copies). This is a really nice feature of the GFDL; it counters some of the problems of proprietary formats.

The GFDL has many problems, though, when used for short works like Wikipedia articles or images. Most obviously, it requires that you include the entire text of the license with each work (see GFDL 1.3 section 2). That’s no problem for large manuals, which is what the GFDL was designed for, but it’s a big problem for short works. Nobody likes having a license longer than the article it’s attached to! This is one reason why CC-BY-SA is so widely used for short works - and since Wikipedia is primarily a large set of short works, it makes sense. Which is why I (and many others) voted to approve this change.

Now it’s certainly true that people also complain that the GFDL allows the addition of unmodifiable sections. But many GFDL items don’t have them, and Debian determined through a formal vote that “GFDL-licensed works without unmodifiable sections are free [as in freedom]”.

I should also give credit to the Wikimedia Foundation (WMF), Richard Stallman of the FSF, and Lawrence Lessig, who worked together to make this possible.

For more on the Wikimedia license modification, you can see Wikimedia license FAQ, Lawrence Lessig’s post on GFDL 1.3, GFDL 1.3: Wikipedia's exit permit, FDL 1.3 FAQ, and An open response to Chris Frey regarding GFDL 1.3.

Government-developed Unclassified Software: Default release as Open Source Software

Fri, 22 May 2009 20:44 GMT

I’d like to see this idea seriously considered and discussed: By default, unclassified software which the government paid to develop should be released to the public as open source software (unless there’s a good reason not to).

Why? Well, If “we the people” paid to develop it, then “we the people” should get it! I think this idea fits into the good government ideal of data transparency; after all, software is data. Currently, we have a lot of waste and unnecessary costs due to loss, re-development, and/or government-created monopolies. The government is not a venture capitalist (VC); people who need a VC should go to a VC.

Let me focus specifically on the United States. I think this idea easily fits into the broader ideas of transparency and open government, including the Memorandum on Transparency and Open Government. Look at all the excitement over data.gov, indeed, Apps for America having a contest to develop software to use data from data.gov.

Indeed, there’s a long history of U.S. laws specifically set up to make data available. Most obviously, Freedom on information act (FOIA) requests make it possible to extract information from the U.S. government. 17 USC 105 and 17 USC 101 prevents the U.S. government from claiming U.S. copyright on a work “prepared by an officer or employee of the United States Government as part of that person’s official duties”. So this idea would be an extension of what’s already gone on.

Let me focus on research, and how this idea could help advance technology. Think of all the advantages if software developed by U.S.-funded research could be reused by other research projects and commercial firms. For example, imagine if other researchers could simply extend previous work by modifying previously-developed software, instead of re-building yet another version from scratch. Anyone could take commercialize the research making it more likely that it would be commercialized instead of being lost in the archives shown at the end of Raiders of the Lost Ark. Some argue that giving sole rights is the only way to commercialization, but that’s just not true; open source software is commercial software, so this is simply a different and fairer path to commercialization. In contrast, the current system inhibits all kinds of technical progress; Biere’s “The Evolution from LIMMAT to NANOSAT” (Apr 2004) found that “important details are often omitted in [research] publications and can only be extracted from source code... [Making source code available] is as important to the advancement of the field as publications”. Originally I thought of this idea for research software, and it’s not hard to see why. But when I starting thinking about the reasons for doing this — especially “if ‘we the people’ paid to develop it, then ‘we the people’ should get it” — then I realized that this principle applies much more broadly.

An open government directive isn’t out yet, but they’re clearly working on it. Please submit this - and other ideas like it - to them. I think there’s a lot of promise, but they can only enact and refine ideas that they’ve heard of. If you like this idea, please vote for it.

If this happened, I envision a two-stage process: (1) release of the software as an archive (so it can be downloaded), and (2) some of it will get picked up and used to start an active OSS project. The second stage might not happen for many years after the first, and that’s okay. Some will ask “how will people find it”, but I think that’s the wrong question. There are many commercial search engines that can find code, but they can only find stuff that’s web-accessible; let’s give them something to find.

Perhaps this should be done in stages. For example, perhaps it'd be best to start with software developed by research. Researchers are supposed to share their results anyway (under most cases), and the lack of software release often inhibits research (e.g., it's harder to check or repeat results). You could then broaden this to other types of software.

I’m sure there will need to be exceptions. There would need to be some sort of guidelines to figure out when to grant those exceptions, and those guidelines should be developed though lively discussion. Most obviously, if it’s a special ingredient necessary for national security, then it should be classified and not revealed in any form. I would not expect weapon systems or intelligence software to be released (though sometimes generic functions developed in them could be released). Export controls would still apply. But the exceptions should be that: Exceptions.

Wikipedia for childrens' schools

Mon, 11 May 2009 12:49 GMT

Wikipedia is a cool project. But if you want to hand an encyclopedia to younger children or to schools, Wikipedia is not a great choice. Wikipedia is not “child-safe”, nor is intended to be; it includes a lot of “adult” content. Also, Wikipedia constantly suffers vandalism; the vandalism is often repaired quickly, but that’s little comfort to parents and teachers. There’s also the problem of Internet access; schools typically employ blocking software, and blocking software is fundamentally not smart. Since Wikipedia mixes material that’s okay for children with stuff that is not, Wikipedia often gets blocked by schools for children. Some schools for children just don’t have Internet access at all, for a variety of reasons. All of this makes it hard for such schools to directly use Wikipedia.

Wikipedia for schools is a cool project that compensates for this. It’s a free, hand-checked, non-commercial selection from Wikipedia, targeted around the UK National Curriculum and useful for much of the English speaking world. The current version has about 5500 articles (as much as can be fit on a DVD with good size images) and is “about the size of a twenty volume encyclopaedia (34,000 images and 20 million words)”. It was developed by carefuly selecting for content, then checking for vandalism and suitability by “SOS Children volunteers”. You can download it for free from the website, or as a free 3.5GB DVD.

I also see this as a future model for Wikipedia — allow people to edit, but have a separate vetting process that identifies particular versions of an article as vetted. Then, people can choose if they want to see the latest version or the most recent vetted version. To some, this is very controversial, but I don’t see it that way. A vetting process doesn’t prevent future edits, and it creates a way for people to get what they want... material that they can have increased confidence in. The trick is to develop a good-enough vetting process (or perhaps multiple vetting/rating processes for different purposes). This didn’t make sense back when Wikipedia was first starting (the problem was to get articles written at all!), but now that Wikipedia is more mature, it shouldn’t be surprising that there’s a new need to identify vetted articles. Yes, you have to worry about countries to whom “democracy” is a dirty word, but I think such problems can be resolved. This is hardly a new idea; see Wikimedia’s article on article validation, Wikipedia’s pushing to 1.0, WikiQA by Eloquence, and FlaggedRevs. I am sure that a vetting/validation process will take time to develop, and it will be imperfect... but that doesn’t make it a bad idea.

So anyway, if you know or have younger kids, check out Wikipedia for schools. This is a project that more people should know about.

FLOSS doubles every 14 months!

Thu, 07 May 2009 18:11 GMT

I just took a look at Red Hat's 2009 brief to the European Patent Office on why software patents should not be allowed. It's a nice brief, noting that software patents hinder software innovation, and that there is a sound legal basis not to expand availability of such patents in Europe. (Here's Red Hat's press release, and Glyn Moody's comments (ComputerWorld UK) on it).

Their brief points to another paper with very interesting results: "The Total Growth of Open Source" by Amit Deshpande and Dirk Riehle (Proceedings of the Fourth Conference on Open Source Systems (OSS 2008). Springer Verlag, 2008. Page 197-209). In this paper, they analyze the growth of more than 5000 open source software projects, and show that "the total amount of source code as well as the total number of open source projects is growing at an exponential rate." In their conclusion they state that the "total amount of source code and the total number of projects double about every 14 months."

That is an extraordinary rate of growth. Exponential growth can start small, but when it continues it will completely flatten anything not growing exponentially (or growing as fast). This result is consistent with my earlier work, More than a Gigabuck: Estimating GNU/Linux's Size, which also found very rapid growth in free/libre/open source software (FLOSS).

So if you're interested in software trends, take a look at "The Total Growth of Open Source" and Red Hat's brief to the EPO on software patents. I think they're both worth reading.

Why copyright damage limits don't hurt FLOSS

Wed, 22 Apr 2009 23:50 GMT

There's a move afoot to argue that copyright infringement penalties should bear a rational relationship to the value of what was infringed. You might think that this could harm Free/Libre/Open Source Software (FLOSS), but I don't think so. Here's why.

First: This is all being brought to a head by the current file-sharing lawsuit against Boston University graduate student Joel Tenenbaum, which raises a number of interesting questions. One issue that I find particularly interesting is the issue of statutory damages: Are fines from $750 to $150,000 per song (worth at most $1), non-commercially shared without permission, even legal under the US Constitution? Or, are these fines so excessive that they are unconstitutional? Ars Technical gives a brief summary of the case, if you haven't been following it. The Free Software Foundation (FSF)'s Amicus Brief in Connection with defendant's motion to dismiss on grounds of unconstitutionality of copyright act statutory damages as applied to infringement of single MP3 files argues that these penalties grossly exceed the crime; the FSF argues that the "State Farm/Gore due process test applicable to punitive damage awards is likewise applicable to statutory damages, and in particular bars the suggestion that each infringement of an MP3 file having a retail value of 99 cents or less may be punishable by statutory damages of from $750 to $150,000 -- or from 2,100 to 425,000 times the actual damages".

Frankly, I think the FSF and Tenenbaum have a reasonable argument on this point. People who shoplift a CD from a store would definitely pay penalties when caught, but those penalties would bear some relationship to the value of the property stolen, and would be far smaller than a file-sharer. This notion that the "punishment should fit the crime" is certainly not new; Proverbs 6:30-31 talks about thieves paying sevenfold if they are caught. That doesn't make such actions right - but unjust penalties aren't right either. I think a lot of the problem is that copyright laws were originally written when only rich people with printing presses could really make and distribute many copies of material. Today, 8-year-olds can distribute as much information as the New York Times, and the law hasn't caught up.

But does the FSF risk subverting Free/Libre/Open Source Software (FLOSS) by making this argument? After all, FLOSS developers also depend on copyright law to enforce certain conditions, and often charge $0 for copies of their software. If the penalties would be limited to "7 times the original cost", would that make FLOSS development impossible?

I don't think there's any problem, but for some people that may not be obvious. The difference is that in a typical music copyright infringement case, the filesharer could purchase the right to do what they're doing for a relatively low price, something typically not true for FLOSS. For example, under normal circumstances it's perfectly legal to buy a song for $1, and then transfer that song to someone else (as long as you destroy your own copies), so sharing that song with 10 people is legal after paying $10.

In contrast, violations of FLOSS licenses often can't be made legal by simply buying the rights. If you violate the revised BSD license by removing all credits to the original author, there's typically no "alternative" legal version available for sale without the author credits. (Indeed, under legal systems with strict "moral rights" it may not even be possible.) Similarly, if you violate the GPL by releasing binary software yet refusing to release its source code, there's often no way to pay additional money to the original authors for that privilege. In some cases, GPL'ed software is released via a dual-use license (e.g., "GPL or proprietary"), with the proprietary version costing additional money; in those cases you do have a value that you can compare against. In cases where there is a value you can compare against, then you should use that value to help determine the penalty. Otherwise, a much stiffer penalty is justified, because there is no method for the infringer to "buy" his or her way out, and their actions risk making functional products (not just entertainment) unsupportable. As noted in the United States Court of Appeals for the Federal Circuit case 2008-1001, JACOBSEN v. KATZER, the court essentially found that failing to obey the conditions of an open source software license led to copyright infringement. (For more on this particular case, see New Open Source Legal Decision: Jacobsen & Katzer and How Model Train Software Will Have an Important Effect on Open Source Licensing.)

So I think that it does make sense to limit copyright penalties based on the value of the original infringed item... but that doing so does not (necessarily) put FLOSS development processes at risk.

Microsoft loses to Open Source Approaches (Encarta vs. Wikipedia)

Mon, 20 Apr 2009 17:06 GMT

The competition is over. On one side, we have Microsoft, a company with a market value of about $166 billion (according to a 2009-04-20 NASDAQ quote). On the other side, we have some volunteers who work together and share their results on the web using open source approaches.

And Microsoft lost.

As pointed out by Chris Dawson (ZDNet), Mike Jennings (PC Pro), Naomi Alderman (the Guardian), Noam Cohen (NY Times), Adam Ostrow (Mashable), and many others, Microsoft Encarta (Microsoft’s encyclopedia project) has folded, having failed to compete with Wikipedia. It’s not even hard to see why:

Wikipedia is cheaper than Encarta (no-cost vs. cost)
Wikipedia is easier to start using. If you have a web browser, you have Wikipedia. In contrast, you have to specially install Encarta, and it does not work on all platforms.
Wikipedia is more up-to-date than Encarta. It often took years before Encarta entries got updated, even on trivially obvious issues such as death dates.
Wikipedia has far more material. Wikipedia has far more articles, and generally it has far more material in each article.
Wikpedia’s material has fewer legal restrictions, so users are allowed to do more with Wikipedia results. Creating mash-ups and reposting portions is part of today’s world.

One lesson to be learned here is that it sometimes doesn’t matter how large a company is; changes in technology may mean that they may abandon something in the future. Plan that the future will change, even if a company seems invincible. It’s easy to pick on Microsoft here, but the same can be said of IBM, or Oracle, or anyone else. Tying yourself completely to any one company is, in the long term, a mistake. Thus, you need to have a reasonable escape plan if a company folds or stops supporting a product that you depend on.

Another lesson to be learned here is that proprietary approaches can be beaten by open source approaches. That doesn’t mean it must happen every time, of course. But clearly open source approaches can, at least sometimes, dominate their proprietary competition.

In the long term, it simply doesn’t matter if a company has more money if an open-sourced competitor can produce a better product, make it available at a lower cost, and can sustain that process indefinitely. Given those three factors, proprietary vendors will lose to an open-sourced competitor unless there’s a key differentiator that is sufficiently valuable to users. In such cases, having more money is just an opportunity to lose more money; it gives no benefit of scale. Microsoft’s Encarta team tried to compete by adding special materials (like fancy graphics and sound). I’m sure that Encarta managers convinced themselves that because they were spending money to develop these materials, that users would pay for Encarta instead. They were wrong. In the end, users were more interested in good, timely information than in fancy graphics, and Encarta simply didn’t have a chance. Open source approaches were simply better at providing the encyclopedia people wanted than proprietary approaches were.

The obvious question to me is, are there any lessons that apply to software too? Wikipedia uses free / libre / open source software (FLOSS) principles, but Wikipedia is an encyclopedia not a FLOSS program. Indeed, software is different than encyclopedias in many ways, for example, people can easily switch encyclopedias (while the lock-in and network effects of software are well-known), and far more people can participate in encyclopedia development than in software development. But I still think there are lessons to be learned here. This Encarta vs. Wikipedia battle should make it clear that no proprietary company — no matter how well-resourced it is — is invulnerable to open source competition. Developers of products with FLOSS-like licenses give up some privileges that the law permits them to have, and in return, they can often drastically reduce their development costs and increase the breadth of the result (because the development efforts can be shared among many developers). At a certain point, FLOSS-like projects can end up like a snowball rolling down the hill; they gain so much momentum that even large sums of money — or being the first — aren’t enough to counter them. As a result, even proprietary companies with massive cash resources do not always win. In summary, it doesn’t matter if you have lots of money; if your product costs more and does less (from the user’s point of view), you must change that circumstance, eliminate all competition, or suffer failure of the product.

Releasing FLOSS Software

Mon, 13 Apr 2009 17:44 GMT

If you’ve written (or started to write) some Free/Libre/Open Source Software (FLOSS), please follow the time-tested community standards for releasing FLOSS software when you want people to be able to install it from source code. Unfortunately, a lot of people don't seem to be aware of what these conventions are. This really hit me in my recent OpenProofs work; we're trying to make it easy to install programs by pre-packaging them, and we've found that some programs are a nightmare to package or install because their developers did not follow the standard conventions.

So I've released a brief article: Releasing Free/Libre/Open Source Software (FLOSS) for Source Installation, to help people learn about them. For the details, I point to the GNU Coding Standards (especially the release process chapter) and the Software Release Practice HOWTO. I also point out some of the most important conventions that will make building and installing your software much easier for your users:

Pick a good, simple, Google-able name.
Identify the version (using simple version numbers or ISO dates), and include that in the release filename as NAME-VERSION.FORMAT.
Use a standard, widely-used, GPL-compatible FLOSS license — and say so.
Follow good distribution-making practice, in particular, make sure tarballs always unpack into a single new directory named NAME-VERSION.
Use the standard invocation to configure, build, and install it: ./configure; make; make install.
Support the standard ./configure options like --prefix, --exec-prefix, --bindir, --libdir, and so on.
Create a makefile that can rebuild everything and uses makefile variables (including applicable standard makefile variable names and targets).
Have “make install” support DESTDIR.
Document the external tools/libraries needed for building and running, and make it easy to separate/reuse them.
If you patch an external library/tool, get the patch upstream.
Use standard user interfaces. For command line tools, use “-” single-letter options, “--” long-name options, and “--” by itself to signal “no more options”. For GUI tools, provide a .desktop file.

To learn more, see the whole article: Releasing Free/Libre/Open Source Software (FLOSS) for Source Installation.

Fixing Unix/Linux/POSIX Filenames

Tue, 24 Mar 2009 18:16 GMT

Traditionally, Unix/Linux/POSIX filenames can be almost any sequence of bytes, and their meaning is unassigned. The only real rules are that "/" is always the directory separator, and that filenames can't contain byte 0 (because this is the terminator). Although this is flexible, this creates many unnecessary problems. In particular, this lack of limitations makes it unnecessarily difficult to write correct programs (enabling many security flaws), makes it impossible to consistently and accurately display filenames, and it confuses users.

So for those of you who understand Unix/Linux/POSIX, I've just released a new technical article, Fixing Unix/Linux/POSIX Filenames.

This article will try to convince you that adding some limitations on legal Unix/Linux/POSIX filenames would be an improvement. Many programs already presume these limitations, the POSIX standard already permits such limitations, and many Unix/Linux filesystems already embed such limitations - so it'd be better to make these (reasonable) assumptions true in the first place. The article discusses, in particular, the problems of control characters in filenames, leading dashes in filenames, the lack of a standard encoding scheme (vs. UTF-8), and special metacharacters in filenames. Spaces in filenames are probably hopeless in general, but resolving some of the other issues will simplify their handling too. This article will then briefly discuss some methods for solving this long-term, though that's not easy - if I've convinced you that this needs improving, I'd like your help figuring out how to do it!

So - take a peek at Fixing Unix/Linux/POSIX Filenames. If you have ideas on how to help, I'd love to know.