SPARK released as FLOSS (Free/ Libre / Open Source Software)!
The SPARK toolsuite has just been released as FLOSS (Free/ Libre / Open Source Software) by Praxis (its creator). This is great news for those who want to make software safer, more reliable, and more secure. In particular, this means that Tokeneer is now an open proof. If you haven’t been following this, here’s some background.
Software is now a part of really critical systems (ones that need “high assurance”), yet often that software is not as safe, reliable, or secure as it needs to be. I believe that in the long term, we will need to start proving that our very important programs are correct. Testing by itself isn’t enough; completely testing the trivial “add three 64-bit integers” program would take far longer than the age of the universe (it would take about 2x10^39 years). The basic idea of using mathematics to prove that programs are correct — aka “formal methods” — has been around for decades. There are a number of cases where formal methods have been applied successfully, and I’m glad about that. And yet, applying formal methods is still relatively rare. There are many reasons for this, such as inadequate maturation and capabilities of many formal methods tools, and the fact that relatively few people know how to apply formal methods when developing real programs. But what, in turn, is causing those problems? It’s true that applying formal methods is a hard problem that hasn’t received the level of funding it needs, but still, it’s been decades!
I believe one problem hindering the maturation and spread of formal methods is a “culture of secrecy”. Details of formal method use are often unpublished (e.g., because the implementations are proprietary or classified). Similarly, details about formal methods tools are often unshared and lost (or have to constantly re-invented). Biere’s “The Evolution from LIMMAT to NANOSAT” (Apr 2004) gives an example: “From the publications alone, without access to the source code, various details were still unclear... Only [when CHAFF’s source code became available did] our unfortunate design decision became clear... The lesson learned is, that important details are often omitted in publications and can only be extracted from source code. It can be argued, that making source code of SAT solvers available is as important to the advancement of the field as publications”
This “culture of secrecy” means that researchers/toolmakers often don’t receive adequate feedback, researchers/toolmakers waste time and money rebuilding tools, educators have difficulty explaining formal methods (they have no examples to show!), developers don’t understand how to apply it (and it has an uncertain value to them), and evaluators/end-users don’t know what to look for.
I believe that a way to break through this “culture of secrecy” is to develop “open proofs”. But what are they? An “open proof” is software or a system where all of the following are free-libre / open source software (FLOSS):
Imagine if we had a number of open proofs available. There could be small open proofs that could be used for learning (e.g., as examples and use in class exercises). There could be proofs of various useful functions and small applications, so developers could see how to scale up these techniques, directly reuse them as components, or use them as starting points but add additional (proven) capabilities to them. When problems come up (and they will!), toolmakers and developers could work together to find ways to mature the tools and technology so that they’d be easier to use (e.g., so more could be automated). In short, imagine there was a working ecosystem where researchers/toolmakers/educators, developers of implementations to be proved, and evaluators/end-users could work together by sharing information. I believe that would greatly speed up the maturing of formal methods, resulting in more reliable and secure software.
In this context, Praxis has just released the SPARK GPL Edition. This is their SPARK toolsuite (a formal methods tool) released under the GNU General Public License aka GPL (the most common FLOSS license). So, what’s that?
SPARK is a variant of the Ada programming language, designed to enable proofs about programs (by adding and removing some features of Ada). The additions are in special comments, so SPARK programs can be compiled by a normal Ada compiler like GNAT (which is part of gcc). The Open Proofs page on SPARK has some information on SPARK. The page What is Special About SPARK Contracts? gives a nice quick introduction to SPARK, which I will quote here. It points out that the Ada line:
procedure Inc (X : in out Integer);
just says there is some procedure “Inc” that may read a value X, and
may write it out, but that’s it.
In SPARK, you can add much more precise information, and the SPARK tools
can then check to see if they are true.
For example, if you say this using SPARK:
procedure Inc (X : in out Integer);
--# global in out CallCount;
--# pre X < Integer'Last and
--# CallCount < Integer'Last;
--# post X = X~ + 1 and
--# CallCount = CallCount~ + 1;
then the SPARK tools will ensure at compile-time (not run-time) that:
You can learn more about SPARK from the book High Integrity Software: The SPARK Approach to Safety and Security” by John Barnes. Sample text of Barnes’ book is available online. The open proofs page on SPARK has more information.
This means that the “Tokeneer” program is now an open proof. Remember, to be an open proof, a program’s implementation, proofs, and required tools have to be open source software. Tokeneer was a sample program written to show how to apply these kinds of techniques to actual systems (instead of trivial 5-line programs). The Tokeneer program itself, and its proofs, have already been released as open source software. Many of the tools it required are already FLOSS (e.g., fuzz and LaTeX for its formal specifications, and an Ada compiler to compile it). Now that SPARK has been released as FLOSS, people can examine this entire stack of software to make improvements in all the technologies, as well as learn from them and create improved implementations. No, this doesn’t suddenly make it trivial to make proofs about complex programs, but it’s a step forward.
If you are interested in making future software better, please help the open proofs project. You don’t need to be a math whiz. For example, if you know how to do shell scripting, please help us package some promising formal methods tools (like SPARK) so they are easy to install. It’s hard to get people to try out these tools (and give feedback) if they’re too hard to install. If you know of formal methods software that is rotting in some warehouse, try to get it released as FLOSS. I think all government-funded unclassified research software should be released as FLOSS by default, since “we the people” paid for it! If you’re interested in the latest software technology, try out a few of these formal methods tools, and release as FLOSS any small programs and proofs you develop with them. Send the toolmakers feedback, or write down their strengths and weaknesses to help others understand them. SPARK is a tool that can be used, right now, in certain circumstances. I have no illusions that today’s formal methods tools are ready for arbitrary 20 million line programs. But if we want future software to be better than today, we need to figure out how to mature formal methods technology and make it better-understood so that it can mature and scale. I think making top-to-bottom worked examples and starting points can help us get there.
path: /oss | Current Weblog | permanent link to this entry
Parchment: Running the Z-machine
I just learned of fun web application called Parchment. Parchment lets you play interactive fiction (I.F., aka "text adventure games") using just your web browser. It only works with I.F. in "Z-machine" format, but that's a very common format.
So go to the parchment site and try out something from their long list of interactive fiction... now you don't need to install anything! That includes my small replayable puzzle "Accuse" (my Accuse source code is already available).
If you want more information about it, here's a brief post about Parchment by its author, Atul Varma. Atul built this based on an existing program, Thomas Thurman's Gnusto. Both are open source software (using the GPLv2 license). Once again, this demonstrates the neat thing about community-developed software; one person developed a program for one circumstance, and another extended it for a different circumstance.
There are several tools available for creating interactive fiction. I've been watching Inform 7 for a while, with interest, because it takes a radically different approach to writing code. Inform 7 is a natural-language programming language that tries to actively exploit features of natural language to make developing these kinds of things easier. You can see a brief Inform 7 tutorial if you're curious, as well as the full Writing with Inform documentation. Inform 7 isn't itself OSS, though significant portions are; inform 6 (a key substrate) and many other portions including the Inform 7 standard rules are released under the Artistic License 2.0. The extensions are released under the "Attribution Creative Commons licence"; that's not normally a license used for software, but I think it'd meet the criteria for OSS, and Fedora approves of this license for content. I hope that someday the rest will be released as OSS as well. The logic behind Inform 7 is described in "Natural Language, Semantic Analysis and Interactive Fiction" by Graham Nelson. If you're interested in some of the technical stuff behind it, the text of the Standard Rules, the text of the extensions, Inform 7 for programmers, and the Chart of Rules can tell you more.
path: /oss | Current Weblog | permanent link to this entry
The Wikimedia Foundation (WMF) will change the licensing terms on all its materials — including Wikipedia. Now, all of its existing material will be released under the Creative Commons Attribution-ShareAlike (CC-BY-SA) license in addition to the current GNU Free Documentation License (GFDL). The WMF says “This change is meant to advance the WMF’s mission by increasing the compatibility and availability of free content.” This means that Wikipedia material can now be combined with the vast amount of CC-BY-SA licensed material, and Wikipedia can now include the volumes of CC-BY-SA material (that material will just be CC-BY-SA). It also makes it easier to use Wikipedia material (and other material from the Wikimedia Foundation).
I think this is a good thing overall. Incompatible licenses are a real scourge on community-developed works. Past experience shows that license incompatibility can be a real problem for free-libre/ open source software (FLOSS or OSS), in particular. Bruce Perens warned about FLOSS license incompatibility back in 1999! As I argue in Make Your Open Source Software GPL-Compatible. Or Else, you should release free-libre/ open source software (FLOSS) using a GPL-compatible license. You don’t need to use the GPL, but using a GPL-compatible license (like the MIT, BSD-new, LGPL, or GPL) so means that people can combine your software with other software to create larger works. I show how this works in The Free-Libre / Open Source Software (FLOSS) License Slide, which has a simple graph showing how common FLOSS licenses can work together. Wikipedia articles aren’t software, but the principles still apply - licenses need to enable community-developed works, not disable them.
Now, nothing is perfect. One nice benefit of the GNU Free Documentation License (GFDL) is that it requires that readers be able to get editable versions whose format specification is available to the public (for details, see its text on “transparent” copies). This is a really nice feature of the GFDL; it counters some of the problems of proprietary formats.
The GFDL has many problems, though, when used for short works like Wikipedia articles or images. Most obviously, it requires that you include the entire text of the license with each work (see GFDL 1.3 section 2). That’s no problem for large manuals, which is what the GFDL was designed for, but it’s a big problem for short works. Nobody likes having a license longer than the article it’s attached to! This is one reason why CC-BY-SA is so widely used for short works - and since Wikipedia is primarily a large set of short works, it makes sense. Which is why I (and many others) voted to approve this change.
Now it’s certainly true that people also complain that the GFDL allows the addition of unmodifiable sections. But many GFDL items don’t have them, and Debian determined through a formal vote that “GFDL-licensed works without unmodifiable sections are free [as in freedom]”.
I should also give credit to the Wikimedia Foundation (WMF), Richard Stallman of the FSF, and Lawrence Lessig, who worked together to make this possible.
For more on the Wikimedia license modification, you can see Wikimedia license FAQ, Lawrence Lessig’s post on GFDL 1.3, GFDL 1.3: Wikipedia's exit permit, FDL 1.3 FAQ, and An open response to Chris Frey regarding GFDL 1.3.
path: /oss | Current Weblog | permanent link to this entry
Government-developed Unclassified Software: Default release as Open Source Software
I’d like to see this idea seriously considered and discussed: By default, unclassified software which the government paid to develop should be released to the public as open source software (unless there’s a good reason not to).
Why? Well, If “we the people” paid to develop it, then “we the people” should get it! I think this idea fits into the good government ideal of data transparency; after all, software is data. Currently, we have a lot of waste and unnecessary costs due to loss, re-development, and/or government-created monopolies. The government is not a venture capitalist (VC); people who need a VC should go to a VC.
Let me focus specifically on the United States. I think this idea easily fits into the broader ideas of transparency and open government, including the Memorandum on Transparency and Open Government. Look at all the excitement over data.gov, indeed, Apps for America having a contest to develop software to use data from data.gov.
Indeed, there’s a long history of U.S. laws specifically set up to make data available. Most obviously, Freedom on information act (FOIA) requests make it possible to extract information from the U.S. government. 17 USC 105 and 17 USC 101 prevents the U.S. government from claiming U.S. copyright on a work “prepared by an officer or employee of the United States Government as part of that person’s official duties”. So this idea would be an extension of what’s already gone on.
Let me focus on research, and how this idea could help advance technology. Think of all the advantages if software developed by U.S.-funded research could be reused by other research projects and commercial firms. For example, imagine if other researchers could simply extend previous work by modifying previously-developed software, instead of re-building yet another version from scratch. Anyone could take commercialize the research making it more likely that it would be commercialized instead of being lost in the archives shown at the end of Raiders of the Lost Ark. Some argue that giving sole rights is the only way to commercialization, but that’s just not true; open source software is commercial software, so this is simply a different and fairer path to commercialization. In contrast, the current system inhibits all kinds of technical progress; Biere’s “The Evolution from LIMMAT to NANOSAT” (Apr 2004) found that “important details are often omitted in [research] publications and can only be extracted from source code... [Making source code available] is as important to the advancement of the field as publications”. Originally I thought of this idea for research software, and it’s not hard to see why. But when I starting thinking about the reasons for doing this — especially “if ‘we the people’ paid to develop it, then ‘we the people’ should get it” — then I realized that this principle applies much more broadly.
An open government directive isn’t out yet, but they’re clearly working on it. Please submit this - and other ideas like it - to them. I think there’s a lot of promise, but they can only enact and refine ideas that they’ve heard of. If you like this idea, please vote for it.
If this happened, I envision a two-stage process: (1) release of the software as an archive (so it can be downloaded), and (2) some of it will get picked up and used to start an active OSS project. The second stage might not happen for many years after the first, and that’s okay. Some will ask “how will people find it”, but I think that’s the wrong question. There are many commercial search engines that can find code, but they can only find stuff that’s web-accessible; let’s give them something to find.
Perhaps this should be done in stages. For example, perhaps it'd be best to start with software developed by research. Researchers are supposed to share their results anyway (under most cases), and the lack of software release often inhibits research (e.g., it's harder to check or repeat results). You could then broaden this to other types of software.
I’m sure there will need to be exceptions. There would need to be some sort of guidelines to figure out when to grant those exceptions, and those guidelines should be developed though lively discussion. Most obviously, if it’s a special ingredient necessary for national security, then it should be classified and not revealed in any form. I would not expect weapon systems or intelligence software to be released (though sometimes generic functions developed in them could be released). Export controls would still apply. But the exceptions should be that: Exceptions.
path: /oss | Current Weblog | permanent link to this entry
Wikipedia for childrens' schools
Wikipedia is a cool project. But if you want to hand an encyclopedia to younger children or to schools, Wikipedia is not a great choice. Wikipedia is not “child-safe”, nor is intended to be; it includes a lot of “adult” content. Also, Wikipedia constantly suffers vandalism; the vandalism is often repaired quickly, but that’s little comfort to parents and teachers. There’s also the problem of Internet access; schools typically employ blocking software, and blocking software is fundamentally not smart. Since Wikipedia mixes material that’s okay for children with stuff that is not, Wikipedia often gets blocked by schools for children. Some schools for children just don’t have Internet access at all, for a variety of reasons. All of this makes it hard for such schools to directly use Wikipedia.
Wikipedia for schools is a cool project that compensates for this. It’s a free, hand-checked, non-commercial selection from Wikipedia, targeted around the UK National Curriculum and useful for much of the English speaking world. The current version has about 5500 articles (as much as can be fit on a DVD with good size images) and is “about the size of a twenty volume encyclopaedia (34,000 images and 20 million words)”. It was developed by carefuly selecting for content, then checking for vandalism and suitability by “SOS Children volunteers”. You can download it for free from the website, or as a free 3.5GB DVD.
I also see this as a future model for Wikipedia — allow people to edit, but have a separate vetting process that identifies particular versions of an article as vetted. Then, people can choose if they want to see the latest version or the most recent vetted version. To some, this is very controversial, but I don’t see it that way. A vetting process doesn’t prevent future edits, and it creates a way for people to get what they want... material that they can have increased confidence in. The trick is to develop a good-enough vetting process (or perhaps multiple vetting/rating processes for different purposes). This didn’t make sense back when Wikipedia was first starting (the problem was to get articles written at all!), but now that Wikipedia is more mature, it shouldn’t be surprising that there’s a new need to identify vetted articles. Yes, you have to worry about countries to whom “democracy” is a dirty word, but I think such problems can be resolved. This is hardly a new idea; see Wikimedia’s article on article validation, Wikipedia’s pushing to 1.0, WikiQA by Eloquence, and FlaggedRevs. I am sure that a vetting/validation process will take time to develop, and it will be imperfect... but that doesn’t make it a bad idea.
So anyway, if you know or have younger kids, check out Wikipedia for schools. This is a project that more people should know about.
path: /oss | Current Weblog | permanent link to this entry
FLOSS doubles every 14 months!
I just took a look at Red Hat's 2009 brief to the European Patent Office on why software patents should not be allowed. It's a nice brief, noting that software patents hinder software innovation, and that there is a sound legal basis not to expand availability of such patents in Europe. (Here's Red Hat's press release, and Glyn Moody's comments (ComputerWorld UK) on it).
Their brief points to another paper with very interesting results: "The Total Growth of Open Source" by Amit Deshpande and Dirk Riehle (Proceedings of the Fourth Conference on Open Source Systems (OSS 2008). Springer Verlag, 2008. Page 197-209). In this paper, they analyze the growth of more than 5000 open source software projects, and show that "the total amount of source code as well as the total number of open source projects is growing at an exponential rate." In their conclusion they state that the "total amount of source code and the total number of projects double about every 14 months."
That is an extraordinary rate of growth. Exponential growth can start small, but when it continues it will completely flatten anything not growing exponentially (or growing as fast). This result is consistent with my earlier work, More than a Gigabuck: Estimating GNU/Linux's Size, which also found very rapid growth in free/libre/open source software (FLOSS).
So if you're interested in software trends, take a look at "The Total Growth of Open Source" and Red Hat's brief to the EPO on software patents. I think they're both worth reading.
path: /oss | Current Weblog | permanent link to this entry
Geocities, a web hosting site sponsored by Yahoo, is shutting down. Which means that, barring lots of work by others, all of its information will be disappearing forever. Jason Scott is trying to coordinate efforts to archive GeoCities' information, but it's not easy. He estimates they're archiving about 2 Gigabytes/hour, pulling in about 5 Geocities sites per second... and they don't know if it'll be enough. What's more, the group has yet to figure out how to serve it: "It is more important to me to grab the data than to figure out how to serve it later.... I don't see how the final collection won’t end up online, but how is elusive..."
This sort of thing happens all the time, sadly. Some company provides a free service for your site / blog / whatever... and so you take advantage of it. That's fine, but if you care about your site, make sure you own your data sufficiently so that you can move somewhere else... because you may have to. Yahoo is a big, well-known company, who paid $3.5 billion for Geocities... and now it's going away.
Please own your own site — both its domain name and its content — if it's important to you. I've seen way too many people have trouble with their sites because they didn't really own them. Too many scams are based on folks who "register" your domain for you, but actually register it in their own names... and then hold your site as a hostage. Similarly, many organizations provide wonderful software that is unique to their site for managing your data... but then you either can't get your own data, or you can't use your data because you can't separately get and re-install the software to use it. Using open standards and/or open source software can help reduce vendor lock-in — that way, if the software vendor/website disappears or stops supporting the product/service, you can still use the software or a replacement for it. And of course, continuously back up your data offsite, so if the hosting service disappears without notice, you still have your data and you can get back on.
I practice what I preach. My personal site, www.dwheeler.com, has moved several times, without problems. I needed to switch my web hosting service (again) earlier in 2009, and it was essentially no problem. I just used "rsync" to copy the files to my new hosting service, change the domain information so people would use the new hosting service instead, and I was up and running. I've switched web servers several times, but since I emphasize using ordinary standards like HTTP, HTML, and so on, I haven't had any trouble. The key is to (1) own the domain name, and (2) make sure that you have your data (via backups) in a format that lets you switch to another provider or vendor. Do that, and you'll save yourself a lot of agony later.
path: /misc | Current Weblog | permanent link to this entry
Why copyright damage limits don't hurt FLOSS
There's a move afoot to argue that copyright infringement penalties should bear a rational relationship to the value of what was infringed. You might think that this could harm Free/Libre/Open Source Software (FLOSS), but I don't think so. Here's why.
First: This is all being brought to a head by the current file-sharing lawsuit against Boston University graduate student Joel Tenenbaum, which raises a number of interesting questions. One issue that I find particularly interesting is the issue of statutory damages: Are fines from $750 to $150,000 per song (worth at most $1), non-commercially shared without permission, even legal under the US Constitution? Or, are these fines so excessive that they are unconstitutional? Ars Technical gives a brief summary of the case, if you haven't been following it. The Free Software Foundation (FSF)'s Amicus Brief in Connection with defendant's motion to dismiss on grounds of unconstitutionality of copyright act statutory damages as applied to infringement of single MP3 files argues that these penalties grossly exceed the crime; the FSF argues that the "State Farm/Gore due process test applicable to punitive damage awards is likewise applicable to statutory damages, and in particular bars the suggestion that each infringement of an MP3 file having a retail value of 99 cents or less may be punishable by statutory damages of from $750 to $150,000 -- or from 2,100 to 425,000 times the actual damages".
Frankly, I think the FSF and Tenenbaum have a reasonable argument on this point. People who shoplift a CD from a store would definitely pay penalties when caught, but those penalties would bear some relationship to the value of the property stolen, and would be far smaller than a file-sharer. This notion that the "punishment should fit the crime" is certainly not new; Proverbs 6:30-31 talks about thieves paying sevenfold if they are caught. That doesn't make such actions right - but unjust penalties aren't right either. I think a lot of the problem is that copyright laws were originally written when only rich people with printing presses could really make and distribute many copies of material. Today, 8-year-olds can distribute as much information as the New York Times, and the law hasn't caught up.
But does the FSF risk subverting Free/Libre/Open Source Software (FLOSS) by making this argument? After all, FLOSS developers also depend on copyright law to enforce certain conditions, and often charge $0 for copies of their software. If the penalties would be limited to "7 times the original cost", would that make FLOSS development impossible?
I don't think there's any problem, but for some people that may not be obvious. The difference is that in a typical music copyright infringement case, the filesharer could purchase the right to do what they're doing for a relatively low price, something typically not true for FLOSS. For example, under normal circumstances it's perfectly legal to buy a song for $1, and then transfer that song to someone else (as long as you destroy your own copies), so sharing that song with 10 people is legal after paying $10.
In contrast, violations of FLOSS licenses often can't be made legal by simply buying the rights. If you violate the revised BSD license by removing all credits to the original author, there's typically no "alternative" legal version available for sale without the author credits. (Indeed, under legal systems with strict "moral rights" it may not even be possible.) Similarly, if you violate the GPL by releasing binary software yet refusing to release its source code, there's often no way to pay additional money to the original authors for that privilege. In some cases, GPL'ed software is released via a dual-use license (e.g., "GPL or proprietary"), with the proprietary version costing additional money; in those cases you do have a value that you can compare against. In cases where there is a value you can compare against, then you should use that value to help determine the penalty. Otherwise, a much stiffer penalty is justified, because there is no method for the infringer to "buy" his or her way out, and their actions risk making functional products (not just entertainment) unsupportable. As noted in the United States Court of Appeals for the Federal Circuit case 2008-1001, JACOBSEN v. KATZER, the court essentially found that failing to obey the conditions of an open source software license led to copyright infringement. (For more on this particular case, see New Open Source Legal Decision: Jacobsen & Katzer and How Model Train Software Will Have an Important Effect on Open Source Licensing.)
So I think that it does make sense to limit copyright penalties based on the value of the original infringed item... but that doing so does not (necessarily) put FLOSS development processes at risk.
path: /oss | Current Weblog | permanent link to this entry
Microsoft loses to Open Source Approaches (Encarta vs. Wikipedia)
The competition is over. On one side, we have Microsoft, a company with a market value of about $166 billion (according to a 2009-04-20 NASDAQ quote). On the other side, we have some volunteers who work together and share their results on the web using open source approaches.
And Microsoft lost.
As pointed out by Chris Dawson (ZDNet), Mike Jennings (PC Pro), Naomi Alderman (the Guardian), Noam Cohen (NY Times), Adam Ostrow (Mashable), and many others, Microsoft Encarta (Microsoft’s encyclopedia project) has folded, having failed to compete with Wikipedia. It’s not even hard to see why:
One lesson to be learned here is that it sometimes doesn’t matter how large a company is; changes in technology may mean that they may abandon something in the future. Plan that the future will change, even if a company seems invincible. It’s easy to pick on Microsoft here, but the same can be said of IBM, or Oracle, or anyone else. Tying yourself completely to any one company is, in the long term, a mistake. Thus, you need to have a reasonable escape plan if a company folds or stops supporting a product that you depend on.
Another lesson to be learned here is that proprietary approaches can be beaten by open source approaches. That doesn’t mean it must happen every time, of course. But clearly open source approaches can, at least sometimes, dominate their proprietary competition.
In the long term, it simply doesn’t matter if a company has more money if an open-sourced competitor can produce a better product, make it available at a lower cost, and can sustain that process indefinitely. Given those three factors, proprietary vendors will lose to an open-sourced competitor unless there’s a key differentiator that is sufficiently valuable to users. In such cases, having more money is just an opportunity to lose more money; it gives no benefit of scale. Microsoft’s Encarta team tried to compete by adding special materials (like fancy graphics and sound). I’m sure that Encarta managers convinced themselves that because they were spending money to develop these materials, that users would pay for Encarta instead. They were wrong. In the end, users were more interested in good, timely information than in fancy graphics, and Encarta simply didn’t have a chance. Open source approaches were simply better at providing the encyclopedia people wanted than proprietary approaches were.
The obvious question to me is, are there any lessons that apply to software too? Wikipedia uses free / libre / open source software (FLOSS) principles, but Wikipedia is an encyclopedia not a FLOSS program. Indeed, software is different than encyclopedias in many ways, for example, people can easily switch encyclopedias (while the lock-in and network effects of software are well-known), and far more people can participate in encyclopedia development than in software development. But I still think there are lessons to be learned here. This Encarta vs. Wikipedia battle should make it clear that no proprietary company — no matter how well-resourced it is — is invulnerable to open source competition. Developers of products with FLOSS-like licenses give up some privileges that the law permits them to have, and in return, they can often drastically reduce their development costs and increase the breadth of the result (because the development efforts can be shared among many developers). At a certain point, FLOSS-like projects can end up like a snowball rolling down the hill; they gain so much momentum that even large sums of money — or being the first — aren’t enough to counter them. As a result, even proprietary companies with massive cash resources do not always win. In summary, it doesn’t matter if you have lots of money; if your product costs more and does less (from the user’s point of view), you must change that circumstance, eliminate all competition, or suffer failure of the product.
path: /oss | Current Weblog | permanent link to this entry
If you’ve written (or started to write) some Free/Libre/Open Source Software (FLOSS), please follow the time-tested community standards for releasing FLOSS software when you want people to be able to install it from source code. Unfortunately, a lot of people don't seem to be aware of what these conventions are. This really hit me in my recent OpenProofs work; we're trying to make it easy to install programs by pre-packaging them, and we've found that some programs are a nightmare to package or install because their developers did not follow the standard conventions.
So I've released a brief article: Releasing Free/Libre/Open Source Software (FLOSS) for Source Installation, to help people learn about them. For the details, I point to the GNU Coding Standards (especially the release process chapter) and the Software Release Practice HOWTO. I also point out some of the most important conventions that will make building and installing your software much easier for your users:
To learn more, see the whole article: Releasing Free/Libre/Open Source Software (FLOSS) for Source Installation.
path: /oss | Current Weblog | permanent link to this entry
Fixing Unix/Linux/POSIX Filenames
Traditionally, Unix/Linux/POSIX filenames can be almost any sequence of bytes, and their meaning is unassigned. The only real rules are that "/" is always the directory separator, and that filenames can't contain byte 0 (because this is the terminator). Although this is flexible, this creates many unnecessary problems. In particular, this lack of limitations makes it unnecessarily difficult to write correct programs (enabling many security flaws), makes it impossible to consistently and accurately display filenames, and it confuses users.
So for those of you who understand Unix/Linux/POSIX, I've just released a new technical article, Fixing Unix/Linux/POSIX Filenames.
This article will try to convince you that adding some limitations on legal Unix/Linux/POSIX filenames would be an improvement. Many programs already presume these limitations, the POSIX standard already permits such limitations, and many Unix/Linux filesystems already embed such limitations - so it'd be better to make these (reasonable) assumptions true in the first place. The article discusses, in particular, the problems of control characters in filenames, leading dashes in filenames, the lack of a standard encoding scheme (vs. UTF-8), and special metacharacters in filenames. Spaces in filenames are probably hopeless in general, but resolving some of the other issues will simplify their handling too. This article will then briefly discuss some methods for solving this long-term, though that's not easy - if I've convinced you that this needs improving, I'd like your help figuring out how to do it!
So - take a peek at Fixing Unix/Linux/POSIX Filenames. If you have ideas on how to help, I'd love to know.
path: /oss | Current Weblog | permanent link to this entry
2009 UK Action Plan for Open Source Software
A new report from the UK titled Open Source, Open Standards and Re–Use: Government Action Plan is in the news; it's been reported by the BBC, Times Online, and Ars Technica (among many others).
Here's the first paragraph of its foreword: "Open Source has been one of the most significant cultural developments in IT and beyond over the last two decades: it has shown that individuals, working together over the Internet, can create products that rival and sometimes beat those of giant corporations; it has shown how giant corporations themselves, and Governments, can become more innovative, more agile and more cost-effective by building on the fruits of community work; and from its IT base the Open Source movement has given leadership to new thinking about intellectual property rights and the availability of information for re–use by others."
In the policy section, it says that (note the last point):
Remarkable stuff.
path: /oss | Current Weblog | permanent link to this entry
Open Proofs: New site and why we need them
There's a new website in town: http://www.openproofs.org. This site exists to define the term "open proofs" and encourage their development. What are open proofs, you ask? Well, let's back up a little...
The world needs secure, accurate, and reliable software - but most software isn't. Testing can find some problems, but testing by itself is inadequate. In fact, it's completely impractical to fully test real programs. For example, completely testing a trivial program that only add three 64-bit numbers, using a trillion superfast computers, would take about 49,700,000,000,000,000,000,000,000,000 years! Real programs, of course, are far more complex.
There is actually an old, well-known approach that can give much more confidence that some software will do what it's supposed to do. These are often called "formal methods", which apply mathematical proof techniques to software. These approaches can produce verified software, where you can prove (given certain assumptions) that the software will (or won't) do something. There's been progress made over the last several decades, but they're not widely used, even where it might make sense to use them. If there's a need, and a technology, why hasn't it matured faster and become more common?
There are many reasons, but I believe that one key problem is that there are relatively few fully-public examples of verified software. Instead, verified software is often highly classified, sensitive, and/or proprietary. Many of the other reasons are actually perpetuated by this. Existing formal methods tools need more maturing, true, but it's rediculously hard for tool developers to mature the tools when few people can show or share meaningful examples. Similarly, software developers who have never used them do not believe such approaches can be used in "real software development" (since there are few examples) and/or can't figure out how to apply them. In addition, they don't have existing verified software that they can build on or modify to fit their needs. Teachers have difficulty explaining them, and students have difficulty learning from them. All of this ends up being self-perpetuating.
I believe one way to help the logjam is to encourage the development of "open proofs". An "open proof" is software or a system where all of the following are free-libre / open source software (FLOSS):
Open proofs do not solve every possible problem, of course. I don't expect formal methods techologies to become instantly trivial to use just because a few open proofs show up. And formal methods are always subject to limitations, e.g.: (1) the formal specification might be wrong or incomplete for its purpose; (2) the tools might be incorrect; (3) one or more assumptions might be wrong. But they would still be a big improvement from where we are today. Many formal method approaches have historically not scaled up to larger programs, but open proofs may help counter that by enabling tool developers to work with others. In any case, I believe it's worth trying.
So please take a look at: http://www.openproofs.org. For example, for open proofs to be easily created and maintained, we need for FLOSS formal methods tools to be packaged up for common systems so they can be easily installed and used; the web site has a page on the packaging status of various FLOSS tools. Please feel welcome to join us.
path: /oss | Current Weblog | permanent link to this entry
Automating DESTDIR for Packaging
Today's users of Linux and Unix systems (including emulation systems like Cygwin) don't want to manually install programs - they want to easily install pre-packaged software. But that means that someone has to create those packages.
When you're creating packages, an annoying step is handling "make install" if the original software developer doesn't support the DESTDIR convention. DESTDIR support is very important, because two of the most common packaging formats - Debian's .deb (used by Debian and Ubuntu) and RPM (used by Fedora, Red Hat, and SuSE/Novell) - both require actions (redirection of writes) that DESTDIR enables. Unfortunately, many software developers don't include support for DESTDIR, and it's sometimes a pain to add DESTDIR support. Indeed, it's often trivial to create packages except for having to make the modifications for DESTDIR support. Yes, adding DESTDIR support isn't hard compared to many other tasks, but since it applies to every program, why not automate this instead?
So, I've written a little essay about Automating DESTDIR for Packaging. In it, I identify some of the ways I've identified for automating DESTDIR. If there are more - great! (Please let me know!). In any case, I'd love to see more automation, so that software will become easier to package and install.
Here's the link, again: Automating DESTDIR for Packaging.
path: /oss | Current Weblog | permanent link to this entry
Oh, quick update - the URL for feedback to Apple is http://www.apple.com/feedback - I gave the wrong URL in my last post. My thanks to Steve Hoelzer, who was the first to send me a correction! Again - please ask them to support Ogg.
path: /oss | Current Weblog | permanent link to this entry
Ask Apple to Support Ogg on iPod/iTunes
Please ask Apple to support Ogg on their iPods, iPhones, and iTunes! It wouldn't hurt to also sign this petition (and maybe this one), though I don't know how strongly they'd influence Apple. Here's why, as good news and bad news.
Bad news: Some of the most common formats for audio (like MP3 and AAC) are patent-encumbered, and thus not open standards. Because they're patent-encumbered they are harder and more expensive to support. Many organizations like Wikipedia forbid the use of patent-encumbered standards, and they can't be directly implemented in FLOSS products used in the U.S. and some other countries.
Good news: Ogg (as maintained by the Xiph.org foundation) is available! Ogg is a "container format" that can contain audio, video, and related material using one of several encodings. Usually audio is encoded with "Vorbis" (the combination is "Ogg Vorbis"); perfect sound reproductions can be created with FLAC. This format is already the required audio format for Wikipedia, and the next version of Mozilla's Firefox will include Ogg built in. Many people already have huge music collections in Ogg format, and both many people report that Ogg is an important requirement for a player. See my older blog entry on playing Ogg Vorbis and Theora for more information.
Bad news: Apple's iPods do not directly support Ogg. That's really unfortunate for iPod users, and it also makes it harder to release files in Ogg. So please, ask Apple to add support for Ogg. People have been asking for this for some time, so it's not true that "no one's asking for it". Some people have even taken radical efforts and rewritten the iPod software - but although that shows there's a real interest, that's an extreme measure that normal people shouldn't have to do. There's already software available to Apple to implement Ogg at no charge, and even the original iPods have enough horsepower to implement Ogg. Thus, it will cost Apple very little to add support for Ogg - and there are people who want it.
path: /oss | Current Weblog | permanent link to this entry
Updating cheap websites with rsync+ssh
I've figured out how to run and update cheap, simple websites using rsync and ssh and Linux. I thought I'd share that info here, in case you want to copy my approach.
My site (www.dwheeler.com) is an intentionally simple website. It's simply a bunch of directories with static files; those files may contain Javascript and animated GIF, but site visitors aren't supposed to cause them to change. Programs to manage my site (other than the web server) are run before the files are sent to the server. Most of today's sites can't be run this way... but when you can do this, the site is much easier to secure and manage. It's also really efficient (and thus fast). Even if you can't run a whole site this way, if you can run a big part of it this way, you can save yourself a lot of security, management, and performance problems.
This means that I can make arbitrary changes to a local copy of the website, and then use rsync+ssh to upload just those changes. rsync is a wonderful program, originally created by Andrew Tridgell, that can copy a directory tree to and from remote directory trees, but only send the changes. The result is that rsync is a great bandwidth-saver.
This approach is easy to secure, too. Rsync uses ssh to create the connection, so people can't normally snoop on the transfer, and redirecting DNS will be immediately noticed. If the website is compromised, just reset it and re-send a copy; as long as you retain a local copy, no data can be permanently lost. I've been doing this for years, and been happy with this approach.
On a full-capability hosting service, using rsync this is easy. Just install rsync on the remote system (typically using yum or apt-get), and run:
rsync -a LOCALDIR REMOTENAME@REMOTESITE:REMOTEDIR
Unfortunately, at least some of the cheap hosting services available today don't make this quite so easy. The cheapest hosting services are "shared" sites that share resources between many users without using full operating system or hardware virtualization. I've been looking at a lot of the cheap Linux web hosting services like these such as WebhostGIANT, Hostmonster, Hostgator, and Bluehost. It appears that at least some of these hosting companies improve their security by greatly limiting the access granted to you via the ssh/shell interface. I know that WebhostGIANT is an example, but I believe there are many such examples. So, even if you have ssh access on a Linux system, you may only get a few commands you can run like "mv" and "cp" (and not "tar" or "rsync"). You could always ask the hosting company to install programs, but they're often reluctant to add new ones. But... it turns out that you can use rsync and other such services, without asking them to install anything, at least in some cases. I'm looking for new hosting providers, and realized (1) I can still use this approach without asking them to install anything, but (2) it requires some technical "magic" that others might not know. So, here's how to do this, in case this information/example helps others.
Warning: Complicated technical info ahead.
I needed to install some executables, and rather than recompiling my own, I grabbed pre-compiled executables. To do this, I found out the Linux distribution used by the hosting service (in the case of WebhostGIANT, it's CentOS 5, so all my examples will be RPM-based). On my local Fedora Linux machine I downloaded the DVD ".iso" image of that distro, and did a "loopback mount" as root so that I could directly view its contents:
cd /var/www # Or wherever you want to put the ISO. wget ...mirror location.../CentOS-5.2-i386-bin-DVD.iso mkdir /mnt/centos-5.2 mount CentOS-5.2-i386-bin-DVD.iso /mnt/centos-5.2 -o loop # Get ready to extract some stuff from the ISO. cd mkdir mytemp cd mytemp
Now let's say I want the program "nice". On a CentOS or Fedora machine you can determine the package that "nice" is in using this command:
rpm -qif `which nice`Which will show that nice is in the "coreutils" package. You can extract "nice" from its package by doing this:
rpm2cpio /mnt/centos-5.2/CentOS/coreutils-5.97-14.el5.i386.rpm | \ cpio --extract --make-directoriesNow you can copy it to your remote site. Presuming that you want the program to go into the remote directory "/private/", you can do this:
scp -p ./usr/bin/rsync MY_USERID@MY_REMOTE_SITE:/private/
Now you can run /private/nice, and it works as you'd expect. But what about rsync? Well, when you try to do this with rsync and run it, it will complain with an error message. The error message says that rsync can't find another library (libpopt in this case). The issue is that and cheap web hosting services often don't provide a lot of libraries, and they won't let you install new libraries in the "normal" places. Are we out of luck? Not at all! We could just recompile the program statically, so that the library is embedded in the file, but we don't even have to do that. We just need to upload the needed library to a different place, and tell the remote site where to find the library. It turns out that the program "/lib/ld-linux.so" has an option called "--library-path" that is specially designed for this purpose. ld-linux.so is the loader (the "program for running programs"), which you don't normally invoke directly, but if you need to add library paths, it's a reasonable way to do it. (Another way is to use LD_LIBRARY_PATH, but that requires that the string be interpreted by a shell, which doesn't always happen.) So, here's what I did (more or less).
First, I extracted the rsync program and necessary library (popt) on the local system, and copied them to the remote system (to "/private", again):
rpm2cpio /mnt/centos-5.2/CentOS/rsync-2.6.8-3.1.i386.rpm | \
cpio --extract --make-directories
# rsync requires popt:
rpm2cpio /mnt/centos-5.2/CentOS/popt-1.10.2-48.el5.i386.rpm | \
cpio --extract --make-directories
scp -p ./usr/bin/rsync ./usr/lib/libpopt.so.0.0.0 \
MY_USERID@MY_REMOTE_SITE:/private/
Then, I logged into the remote system using ssh, and added
symbolic links as required by the normal Unix/Linux library conventions:
ssh MY_USERID@MY_REMOTE_SITE cd /private ln -s libpopt.so.0.0.0 libpopt.so ln -s libpopt.so.0.0.0 libpopt.so.0
Now we're ready to use rsync! The trick is to tell the local rsync where the remote rsync is, using "--rsync-path". That option's contents must invoke ld-linux.so to tell the remote system where the additional library path (for libopt) is. So here's an example, which copies files from the directory LOCAL_HTTPD to the directory REMOTE_HTTPDIR:
rsync -a \ --rsync-path="/lib/ld-linux.so.2 --library-path /private /private/rsync" \ LOCAL_HTTPDIR REMOTENAME@REMOTESITE:REMOTE_HTTPDIR
There are a few ways we can make this nicer for everyday production use. If the remote server is a cheap shared system, we want to be very kind on its CPU and bandwidth use (or we'll get thrown off it!). The "nice" command (installed by the steps above) will reduce CPU use on the remote web server when running rsync. There are several rsync options that can help, too. The "--bwlimit=KBPS" option will limit the bandwidth used. The "--fuzzy" option will reduce bandwidth use if there's a similar file already on the remote side. The "--delete" option is probably a good idea; this means that files deleted locally are also deleted remotely. I also suggest "--update" (this will avoid updating remote files if they have a newer timestamp) and "--progress" (so you can see what's happening). Rsync is able to copy hard links (using "-H"), but that takes more CPU power; I suggest using symbolic links and then not invoking that option. You can enable compression too, but that's a trade-off; compression will decrease bandwidth but increase CPU use. So our final command looks like this:
rsync -a --bwlimit=100 --fuzzy --delete --update --progress \ --rsync-path="/private/nice /lib/ld-linux.so.2 --library-path /private /private/rsync" \ LOCAL_HTTPDIR REMOTENAME@REMOTESITE:REMOTE_HTTPDIR
Voila! Store that script in some easily-run place. Now you can easily update your website locally and push it to the actual webserver, even on a cheap hosting service, with very little bandwidth and CPU use. That's a win-win for everyone.
path: /misc | Current Weblog | permanent link to this entry
Moving hosting service at end of January 2009
I will be moving to a new hosting service at the end of January 2009. (I haven't determined which hosting service yet.) In theory, there should be very little downtime, but it's possible the site will be off for a little while. But if that happens, it will be very temporary - I'll get the site back up as soon as I can.
path: /website | Current Weblog | permanent link to this entry
The Center for Strategic and International Studies (CSIS) has just released an interesting new report titled "Securing Cyberspace for the 44th Presidency: A Report of the CSIS Commission on Cybersecurity for the 44th Presidency". The project was co-chaired by Representative James R. Langevin, Representative Michael T. McCaul, Scott Charney, and Lt. General Harry Raduege, USAF (Ret). If you're interested in getting our computer infrastructure more secure, I think this is worth looking at.
The three major findings were: (1) cybersecurity is now a major national security problem for the United States, (2) decisions and actions must respect privacy and civil liberties, and (3) only a comprehensive national secuirty strategy that embraces both the domestic and international aspects of cybersecurity will make us more secure.
Among their recommendations, they suggest "Regulate cyberspace. Voluntary action is not enough. The U.S. must ... set minimum standards in order to ensure that the delivery of critical services in cyberspace continues if the U.S. is attacked... [avoid] prescriptive mandates [and] overreliance on market forces, which are ill-equipped to meet national security and public safety requirements". I agree that market forces, without any help, aren't well-equipped to deliver security, but the challenge is in the details... it's difficult to strike that balance well.
They recommend conducting research and development for cybersecurity - I'm glad they do, that's vitally important. (I just saw the video The Science of Victory, which briefly discusses the importance of research to U.S. national defense.) CSIS also recommends not starting over - instead, they recommend building on and refining the existing "Comprehensive National Cybersecurity Initiative".
In any case, computers and computer networks are no longer interesting toys, they are vital services. We need to improve how we protect them.
path: /security | Current Weblog | permanent link to this entry
Use "FLOSS" instead of "FOSS" or "OSS/FS" as Universal term for Open Source Software / Free Software
Below is my brief attempt to untangle some terminology. In short, I suggest using "FLOSS" instead of "FOSS" or "OSS/FS" for software which meets the Free Software Definition and Open Source Definition.
There are many alternative terms for "Free software" in the sense of the Free Software Definition. Examples of such alternatives are "open source software", "libre software", "FOSS" or "F/OSS" (free/open-source software"), OSS/FS (open source software / free software), "freed software", and "unfettered software". Wikipedia's article on alternative terms for free software discusses alternative terminology further.
For someone (like me) who tries to write about software under these kinds of licenses, having multiple different names is annoying. What's worse, the term "Free software" (the original term) is really misleading; people who hear that term presume that you mean "no cost", which is not related to the intended meaning of "freedom". Yes, I know that the "Free" means "freedom" (aka "Free as in speech" or "Free market"), and I'm well aware that you can charge and pay for Free software. But you have to re-teach everyone who knows English, and you're fighting a losing battle against search engines (which will mix results together when you search for a phrase with two common meanings). Even the FSF admits that the term "Free software" is widely misunderstood. Years ago I suggested to Richard Stallman that he use the term "freed software" instead of "free software", so that the term would be different but the acronyms could stay the same; obviously he didn't accept that suggestion.
The term "open source software" is the most widely used term in English, and the term's creators intentionally tried to include everyone regardless of their motivations. I'm happy to use the term "open source software"; I think it's a reasonable term and it's widely accepted. So, in groups which already use that term, I'll gladly use "open source software" as the "universal" term that covers all such software, regardless of the motivations of the developers.
Unfortunately, many of the developers of such software strongly object to the term "open source software" as the universal term. Their objection is that many people who use the term "open source software" only emphasize engineering or business advantages, while the FSF emphasizes freedom for end-users and objects to a term that doesn't note that possibility somehow.
This objection causes problems for people like me. I'm usually not trying to exclude those who object to the term "open source software". Instead, I often want an inclusive term to describe such software, regardless of the motivations of its developers. Different developers often have different motivations - and even the same developer may have different motivations over time or on different projects. So, is there some term that most can accept?
One inclusive term is "OSS/FS", which you can blame me for. I starting writing about open source software / free software (OSS/FS) many years ago, when such writings were much less common. So for example, look at the title of my massive paper "Why open source software / free software (OSS/FS)? Look at the Numbers!" At the time, there wasn't an obvious "universal" term, so I chose to use "OSS/FS", which was an obvious way to combine the two most common terms. "OSS/FS" takes too long to pronounce, though, so it hasn't really caught on.
Among the other terms, "FLOSS" (Free/Libre/Open-Source Software) seems to have won the popularity contest as a "universal" English term that nearly all can accept. Google reports the approximate number of pages a phrase will return, so it's a reasonable way to determine how popular a phrase is. A quick search on Google (using English on 2008-12-08) shows these popularity figures: "FLOSS software" gets 1,570,000 hits, "Libre software" gets 596,000 hits, "FOSS software" gets 595,000 hits, "OSS/FS" gets 193,000 hits, and "F/OSS software" gets 66,200 hits. Note that FLOSS adds the term "libre"; "libre software" or "livre software" is widely used as the universal term in Romance languages, so adding it helps clarify which sense of "free" is meant. The term "FLOSS" also hides the fact that some people prefer the original spelling "open source software", while others prefer to hyphenate it as "open-source software". In context, "FLOSS" is unlikely to be confused with dental floss. If you're an advocate who objects to that similarity, just imagine that "FLOSS cleans the gunk out" :-).
So, when speaking to a group that already uses "open source software" as the universal term for such software, I'll use "open source software". I have no objection to "open source software" as the universal term; it's a reasonable term, and my primary goal is understanding by my hearers. I don't see a problem with using separate terms to describe people's motivations, as opposed to terms for the software that they (co-)develop. Thus, I can glibly say "open source software is co-developed by people and organizations who often have different motivations to do so; those who develop it to promote freedom for end-users typically term their members the 'Free Software Movement', and those who develop it for engineering and business (cost-saving) reasons may be referred to as the 'Open Source Movement'". I think that's perfectly acceptable as terminology; it's certainly clear that different people and organizations can have different motivations and yet can work together. Many, many people use "open source software" as the universal term for such software.
But not all groups accept the term "open source software" as the universal term for such software, and my writings on my website cater to a variety of people. You can't please everyone, but I'd like to avoid unnecessarily alienating people. At the least, I'd rather people object to the substance of my writing instead of my word choice :-).
So: I suggest using "FLOSS" (Free/Libre/Open-source software) instead of "FOSS" or "F/OSS" or "OSS/FS" as the universal term for such software (in English). It's easy to say, inclusive, and it seems to be the most popular of those alternatives. Many people use "open source software" or "Free software" (with the funny capitalization) as the universal term for such software, and that's fine by me. But for material on my website, I intend to slowly migrate towards FLOSS instead of my older term OSS/FS. I'll leave OSS/FS in a few places (including titles) so that people searching for it will still find it, but at this point, I think the OSS/FS acronym is a fossil. I'll tend to leave "open source software" where I use that term; typically those are written for audiences where it's established practice to use that term.
Strictly speaking, "Free software" in the FSF sense is defined by the Free Software Definition (FSD), and "open source software" is defined by the Open Source Definition (OSD). In practice, a software license will typically either meet, or fail to meet, both definitions. This is hardly surprising; different dictionaries have different descriptions of much simpler words. The Free Software Definition is much simpler and easier to understand, and it gives a better understanding of why an end-user might want such a license. So when describing what FLOSS is, I generally prefer to use the simpler and clearer Free Software Definition. If I'm doing a technical analysis of a software license, I require that both definitions be met. The FSD is better at explaining the overall concept, but the OSD has additional information that helps clarify it. In other words, a software license must meet both definitions, or it's not a FLOSS license.
path: /oss | Current Weblog | permanent link to this entry