David A. Wheeler's Blog

Bilski: Information is physical!?

Fri, 09 May 2008 10:53 GMT

The US Court of Appeals for the Federal Circuit in Washington, DC just heard arguments in the Bilski case, where the appellant (Bilski) is arguing that a completely mental process should get a patent. The fact that this was even entertained demonstrates why the patent system has truly descended into new levels of madness. At least the PTO rejected the application; the problem is that the PTO now allows business method patents and software patents. Once they allowed them, there's no rational way to say "stop! That's rediculous!" without being arbitrary.

Mr. David Hanson (Webb Law Firm) argued for the appellant (Bilski), and got peppered with questions. "Is a curve ball patentable?", for example. At the end, he finally asked the court to think of "information as physical"; it is therefore tangible and can be transformed.

That is complete lunacy, and it clearly demonstrates why the patent office is in real trouble.

Information is not physical, it is fundamentally different, and that difference has been understood for centuries. If I give you my car, I no longer have that car. If I give you some information, I still have the information. That is a fundamental difference in information, and always has been. The fact that Bilski's lawyer can't understand this difference shows why our patent office is so messed up.

This fundamental difference between information and physical objects was well-understood by the U.S. founding fathers. Here's what Thomas Jefferson said: "That ideas should freely spread from one to another over the globe, for the moral and mutual instruction of man, and improvement of his condition, seems to have been peculiarly and benevolently designed by nature, when she made them, like fire, expansible over all space, without lessening their density at any point, and like the air in which we breath, move, and have our physical being, incapable of confinement or exclusive appropriation. Inventions then cannot, in nature, be a subject of property." Thomas Jefferson was a founder, and an inventor. No, they didn't have computers then, but computers merely automate the processing of information; the essential difference between information and physical/tangible objects was quite clear then.

Our laws need to distinguish between information and physical objects, because they have fundamentally different characteristics.

Basically, by failing to understand the differences, the PTO let in software patents and business method patents, which have been grossly harmful to the United States.

Even if you thought they were merely "neutral", that's not enough. There's a famous English speech about the trade-offs of copyright law, whose principles also apply here: "It is good that authors should be remunerated; and the least exceptionable way of remunerating them is by a monopoly. Yet monopoly is an evil. For the sake of the good we must submit to the evil; but the evil ought not to last a day longer than is necessary for the purpose of securing the good." - Thomas Babbington Macaulay, speech to the House of Commons, February 5, 1841.

I believe that software patents need to be abolished, pronto. As I've discussed elsewhere, software patents harm software innovation, not help it.

But here in the Bilski case we see why some some people have managed to sneak software patents into the patent process. In short, too many people do not understand the fundamental differences between information and physical objects. People whose thinking is that fuzzy are easily duped. Though clearly many people aren't as confused as Bilski's lawyer, I think too many people in the patent process have become so confused about the difference between physical objects and information that they don't understand why software patents are a serious problem. Patents should only apply to processes that directly change physical objects, and their scope should only cover the specifics of those changes. I add that latter part because yes, changing the number on a display does change something physical, but that is irrelevant. If you have a wholly new process for making displays (say, using a new chemical compound), that could be patentable, but changing a "5" to a "6" should not be patentable because "changing a 5 to a 6" is not fundamentally a change in nature. Taking something unpatentable and adding the phrase "doing it with a computer" should not change an unpatentable invention into a patentable one; the Supreme Court understood that, but the PTO still fails to understand that.

I think pharmaceutical companies are afraid of any patent reform laws, because they're afraid that a change in the patent system might hurt them. But if the patent system isn't fixed - by eliminating business method patents and software patents - the entire patent system might become too overwhelmed to function, and thus eventually scrapped. I don't know if pharma patents are more help than hinderance; I'm not an expert in that area. But I make my living with software, and it's obvious to me (and most other software practitioners) that software patents and business patents are becoming a massive drag on innovation. If we can't fix the patent system, we'll have to abolish the patent system completely. A lot of lawyers will be unhappy if the patent system is eliminated, but there are more non-lawyers than lawyers. If the pharma companies want to have a working patent system, then they'll need to help reign in patents in other areas, or the whole system may collapse.

Open Source Computer Emergency Response Team (oCERT)

Thu, 08 May 2008 11:01 GMT

Here's something new and interesting: the Open Source Computer Emergency Response Team (oCERT). Here's how they describe themselves: "The oCERT project is a public effort providing security handling support to Open Source projects affected by security incidents or vulnerabilities...".

They promise to keep things moving. They do permit embargo periods (where vulnerabilities are not publicly disclosing, giving time for developers to fix the problem first). More importantly, though, they have a maximum embargo time of two months; I think that's great, and important, because a lot of suppliers have abused embargo periods and failed to fix critical vulnerabilities as long as they're embargoed. These abuses often resulted in customers being exploited through mechanisms that the supplier knew about, but refused to fix in a timely manner.

Google is backing oCERT, which is certainly encouraging. Google even mentions my "three conditions" for securing software (thanks!):

people need to actually review the code
developers/reviewers need to know how to write secure code
once found, security problems need to be fixed quickly, and their fixes distributed quickly

Clearly, something like oCERT could help with these.

This ComputerWorld article on oCERT makes some interesting points. One minor point: They worry that oCERT is using the term "CERT" without permission, but oCERT reports that they do indeed have that permission.

Securing Open Source Software (OSS)

Tue, 06 May 2008 21:17 GMT

I've just posted my presentation titled "Securing Open Source Software (OSS or FLOSS), which is to be presented at the 8th Semi-Annual Software Assurance Forum, May 6-8, 2008, Sheraton Premiere, Tyson's Corner in Vienna, Virginia. In it, I discuss how to improve the security of an OSS component by modifying its environment, as well as securing the OSS component itself (by selecting a secure component, building a secure component from scratch, or modifying an existing component). I include a number of examples; they're necessarily incomplete, but I hope it will help people who are developing or deploying systems. (Here is "Securing Open Source Software (OSS or FLOSS)" in OpenDocument format.) Enjoy!

Microsoft Office XML (OOXML) massively defective

Fri, 21 Mar 2008 18:48 GMT

Robert Weir has been analyzing Microsoft's Office XML spec (aka OOXML) to determine how defective it is, with disturbing results.

Most standards today are relatively small, build on other standards, and are developed publicly over time with lots of opportunity for correction. Not OOXML; Emca submitted Office Open XML for "Fast Track" as a massive 6,045 page specification, developed in an absurdly rushed way, behind closed doors, using a process controlled by a single vendor. It's huge primarily because does everything in a non-standard way, instead of referring to other standards where practical as standards are supposed to do (e.g., for mathematical equations they created their own incompatible format instead of using the MathML standard). All by itself, its failure to build on other standards should have disqualified OOXML, but it was accepted for review anyway, and what happened next was predictable.

No one can seriously review such a massive document in a short time, though ISO tried; ISO's process did find 3,522 defects. It's not at all clear that the defects were fixed - there's been no time to really check, because the process for reviewing the standard simply wasn't designed to handle that many defects. But even if they were fixed - a doubtful claim - Robert Weir has asked another question, "did they find nearly all of the defects?". The answer is: Almost all of the original defects remain. By sampling pages, he's found error after error, none of which were found by the ISO process. The statistics from the sample are very clear: practically all serious errors have not been found. It's true that good standards sometimes have a few errors left in them, after review, but this isn't "just a few errors"; these clearly show that the specification is intensely defect-ridden. Less than 2% of the defects have been found, by the data we have so far, which suggests that there are over 172,000 important defects (49x3522) left to find. That's rediculous.

Want more evidence that it's defect-ridden? Look at Inigo Surguy's "Technical review of OOXML", where he examines just the WordProcessingML section's 2300 XML examples. He wrote code to check for well-formedness and validation errors, and found that more than 10% (about 300) were in error even given this trivial test. Conclusion? "While a certain number of errors is understandable in any large specification, the sheer volume of errors indicates that the specification has not been through a rigorous technical review before becoming an Ecma standard, and therefore may not be suitable for the fast-track process to becoming an ISO standard." This did not include the other document sections, and this is a lower bound on accuracy (XML could validate and still be in error). (He also confirmed that Word 2007 does not implement the extensibility requirements of the Ecma specification, so as a result it would be hard to "write an interoperable word processor with Word" using OOXML.)

I think that all by itself, these vast number of errors in OOXML prove that the "Fast Track" process is completely inappropriate for OOXML. The "Fast Track" process was intended to be used when there was already a widely-implemented, industry-accepted standard that had already had its major problems addressed. That's just not the case here.

These huge error rates were predictable, too. The committee for creating OOXML wasn't even created until OpenDocument was complete, so they had to do a massive rush job to produce anything. ( Doug Mahugh admitted that "Microsoft... had to rush this standard through.") They didn't reuse existing mature standards, so they ended up creating much more work for themselves. Most developers (who could have helped find and fix the defects) stayed away from the Ecma process in the first place; its rules gave one vendor complete control over what was allowed, and there was already a vendor-independent standard in place, which gave most experts no reason to participate. The Ecma process was also almost entirely closed-door (OpenDocument's mailing lists are public, in contrast), which predictably increased the error rate too.

The GNOME Foundation has been involved in OOXML's development, and here's what they say in the GNOME Foundation Annual Report 2007: "The GNOME Foundation’s involvement in ECMA TC45-M (OOXML) was the main discussion point during the last meeting.... [the] Foundation does not support this file format as the main format or as a standard..." I don't think this is as widely touted as it should be. Here's an organization directly involved in OOXML development, and it thinks OOXML should not be a standard at all.

India has already voted "no" to OOXML. I hope others do the same. Countries with the appropriate rights have until March 29 to decide. It's quite plausiable that the final vote will be "no", and indeed, based on what's published, it should be "no". Open Malaysia reported on the March 2008 BRM meeting, for example. It reports that everybody "did their darnest to improve the spec... The final day was absolute mayhem. We had to submit decisions on over 500 items which we hadn't [had] the time to review. All the important issues which have been worked on repeatedly happened to appear on this final day. So it was non-stop important matters... It was a failure of the Fast Track process, and Ecma for choosing it. It should have been obvious to the administrators that submitting a 6000+ page document which failed the contradiction period, the 5 month ballot vote and poor resolution dispositions, should be pulled from the process. It should have been blatantly obvious that if you force National Bodies to contribute in the BRM and end up not deliberating on over 80% of their concerns, you will make a lot of people very unhappy... judging from the reactions from the National Bodies who truly tried to contribute on a positive manner, without having their concerns heard let alone resolved, they leave the BRM with only one decision in their mind come March 29th. The Fast Tracking process is NOT suitable for ISO/IEC DIS 29500. It will fail yet again. And this time it will be final."

In my opinion, the OOXML specification should not become an international standard, period. I think it clearly doesn't meet the criteria for "fast track" - but more importantly, it doesn't meet the needs for being a standard at all. It completely contradicts the goal of "One standard, one test - Accepted everywhere", and it simply is not an open standard. I've blogged before that having multiple standards for office documents is a terrible idea. There's nothing wrong with a vendor publishing their internal format; in fact, ISO's "type 2 technical report" or "ISO agreement" are pre-existing mechanisms for documenting the format of a single vendor and product line specification. But when important data is going to be exchanged between parties, it should be exchanged using an open standard. We already have an open standard for office documents that was developed by consensus and implemented by multiple vendors: OpenDocument (ISO/IEC 26300). For more clarification about what an open standard is, or why OpenDocument is an open standard, see my essay "Is OpenDocument an Open Standard? Yes!" OpenDocument works very well; I use it often. In contrast, it seems clear that OOXML will never be a specification that everyone can fully implement. Its technical problems alone are serious, but even more importantly, the Software Freedom Law Center's "Microsoft's Open Specification Promise: No Assurance for GPL" makes it clear that OOXML cannot be legally implemented by anyone using any license. And this matters greatly.

Andy Updegrove calls for recognition of "Civil ICT Standards", which I think helps puts this technical stuff into a broader and more meaningful context. He notes that in our new "interconnected world, virtually every civic, commercial, and expressive human activity will be fully or partially exercisable only via the Internet, the Web and the applications that are resident on, or interface with, them. And in the third world, the ability to accelerate one’s progress to true equality of opportunity will be mightily dependent on whether one has the financial and other means to lay hold of this great equalizer... [and thus] public policy relating to information and communications technology (ICT) will become as important, if not more, than existing policies that relate to freedom of travel (often now being replaced by virtual experiences), freedom of speech (increasingly expressed on line), freedom of access (affordable broadband or otherwise), and freedom to create (open versus closed systems, the ability to create mashups under Creative Commons licenses, and so on)... This is where standards enter the picture, because standards are where policy and technology touch at the most intimate level. Much as a constitution establishes and balances the basic rights of an individual in civil society, standards codify the points where proprietary technologies touch each other, and where the passage of information is negotiated... what will life be like in the future if Civil ICT Rights are not recognized and protected, as paper and other fixed media disappear, as information becomes available exclusively on line, and as history itself becomes hostage to technology? I would submit that a vote to adopt OOXML would be a step away from, rather than a way to advance towards, a future in which Civil ICT Rights are guaranteed".

Ms. Geraldine Fraser-Moleketi, Minister of Public Service and Administration, South Africa, gave an interesting presentation at the Idlelo African Conference on FOSS and the Digital Commons. She said, "The adoption of open standards by governments is a critical factor in building interoperable information systems which are open, accessible, fair and which reinforce democratic culture and good governance practices. In South Africa we have a guiding document produced by my department called the Minimum Interoperability Standards for Information Systems in Government (MIOS). The MIOS prescribes the use of open standards for all areas of information interoperability, including, notably, the use of the Open Document Format (ODF) for exchange of office documents... It is unfortunate that the leading vendor of office software, which enjoys considerable dominance in the market, chose not to participate and support ODF in its products, but rather to develop its own competing document standard which is now also awaiting judgement in the ISO process. If it is successful, it is difficult to see how consumers will benefit from these two overlapping ISO standards... The proliferation of multiple standards in this space is confusing and costly." She also said, "One cannot be in Dakar without being painfully aware of the tragic history of the slave trade... As we find ourselves today in this new era of the globalised Knowledge Economy there are lessons we can and must draw from that earlier era. That a crime against humanity of such monstrous proportions was justified by the need to uphold the property rights of slave owners and traders should certainly make us more than a little cautious about what should and should not be considered suitable for protection as property."

You can get more detail from the Groklaw ODF-MSOOXML main page, but I think the point is clear. The world doesn't need the confusion of a specification controlled by a single vendor being labelled as an international standard. NoOOXML has a list of reasons to reject OOXML.

Twisted Mind of the Security Pro

Fri, 21 Mar 2008 10:08 GMT

Bruce Schneier's "Inside the Twisted Mind of the Security Professional" is highly-recommended reading - he explains the different kind of thinking required to be good at making things secure. Security pros are able to see the bigger picture, and in particular, they are able to see things from from an attacker's perspective.

For example, "SmartWater is a liquid with a unique identifier linked to a particular owner. 'The idea is for me to paint this stuff on my valuables as proof of ownership,' I wrote when I first learned about the idea. 'I think a better idea would be for me to paint it on your valuables, and then call the police.'" Similarly, on opening up an ant farm, his friend was surprised that the manufacturer would send you ants by mail; Bruce thought it was interesting that "these people will send a tube of live ants to anyone you tell them to."

Being able to think like an attacker is so important that in my book on writing secure programs, I gave it its own heading: paranoia is a virtue. It's still true. My thanks to Bruce Schneier for expressing this need so eloquently.

We would live in a better world if all of us could see the world as attackers do - or at least make the effort to try. In particular, we'd stop doing many foolish things in the name of "security", and instead do things that actually secured our world.

OSS and the U.S. DoD - Questions and Answers

Tue, 11 Mar 2008 20:19 GMT

I've just posted Questions and Answers for 2008 "Open Source Software and DoD" Webinar. These are my attempts to answer the questions people sent me at my February "Open Source Software (OSS) and the U.S. Department of Defense (DoD)" Some of the questions were easy to answer, but some were surprisingly difficult. In some cases, I asked lawyers and got conflicting answers. But this is the best information that I could find on the topic.

For example, I explain in detail why In particular, it appears fairly clear that both the government and government contractors can release their results as open source software under the default DoD contract terms for software development (DFARS contracting clause 252.227-7014):

The government can release software as OSS once it receives "unlimited rights" to it. Unless other arrangements are made, the government has unlimited rights to software components when (1) it pays entirely for their development, or (2) five years after contract signature if it partly paid for their development. Before award, a contractor may identify the components that will have more restrictive rights (e.g., so the government can prefer proposals that give the government more rights). Where possible, software developed partly by government funds should broken into a set of smaller components at the "lowest practicable level" so the rules can be applied separately to each one. Of course, the software can only be released to the public as OSS if other laws are also met (such as classification, export control, patent law, and trademark law).
Normally a DoD contractor can release the software as OSS at any time, since it holds the copyright. This default can be overridden by the contract, e.g., DFARS 252.227-7020 assigns copyright to the government, and is included in some contracts. Again, this release can only occur if other laws are also met (such as classification, export control, patent law, and trademark law).

These are the usual defaults; negotiations can change things, so read the contract to see if the contract changes these defaults. For example, sometimes the government has copyright assigned to it, in which case it can release the software simply because it has the copyright.

I also point out that even when the government isn't the copyright holder, if it releases software under an OSS license it can still enforce its license. That's because, even when it's not the copyright holder, it can still enforce the license... and because the doctrine of unclean hands will impact those who refuse to obey the license.

Several people had questions about software developed by a government employee (which can't be copyrighted in the U.S.) and how that impacts OSS. The short impact is that there's no problem; government employees can still contribute to OSS projects, for example. I also discuss some of the export control issues (especially ITAR), and how to address them.

If there are mistakes, please let me know. Thanks!

OSS and the U.S. DoD - Webinar

Mon, 28 Jan 2008 17:59 GMT

I'm going to present a webinar on "Open Source Software (OSS) and the U.S. Department of Defense (DoD)" on Feb 11, 2008, 3:00-4:30pm EST. It is open to the public, at no charge. To find out how to sign up, see http://www.dwheeler.com/oss-dod-webinar2008.html.

Here's the summary: "Open source software (OSS) has become widespread, but there are many misconceptions about it - resulting in numerous missed opportunities. This presentation will clarify what OSS is (and isn't), rebut common misunderstandings about OSS, discuss the relationship of OSS and security, discuss how to find and evaluate OSS, and explain OSS licensing (including how to combine products and select a license). It will show why nearly all extant OSS is COTS software, and thus why it's illegal (as well as foolish) to ignore OSS options."

This presentation is hosted by the Data & Analysis Center for Software (DACS), which is technically managed by the Air Force Research Laboratory - Information Directorate (AFRL/IF).

Please sign up quickly, if you're interested. There were 45 registrants in the first half hour of its announcement.

Readable s-expressions (sweet-expressions) draft 0.2 for Lisp-like languages

Thu, 06 Dec 2007 22:39 GMT

Back in 2006 I posted my basic ideas about "sweet-expressions". Lisp-based programming languages normally represent programs as s-expressions, and though they are regular, most people find them hard to read. I hope to create an alternative to s-expressions that have their advantages, and not their disadvantages. You can see more at my readable Lisp page. I've gotten lots of feedback, based on both my prototype of the idea, as well as on the mailing list discussing it.

I've just posted a a draft of version 0.2 of sweet-expressions. This takes that feedback into account, in particular, it's now much more backwards-compatible. There's still a big question about whether or not infix should be a default; see the page for more details.

Here are the improvements over version 0.1:

This version is much more compatible with existing Lisp code. The big change is that an unprefixed "(" immediately calls the underlying s-expression reader. This way, people can quietly replace their readers with a sweet-reader, without harming most existing code. In fact, many implementations could quietly switch to a sweet-reader and users might not notice until they use the new features. Instead of using (...), this uses {..} and [...] for grouping expressions without disabling sweet-expressions.
It can work more cleanly with macros that provide infix precedence (for those who want precedence rules).
It extends version 0.1's "name-prefixing" into "term-prefixing". This is not only more general, it also makes certain kinds of functional programming much more pleasant.
It adds syntax for the common case of accessing maps (such as indexed or associative arrays) - now a[j] is translated into (bracketaccess a j).
Infix default supports arbitrarily-spelled infix operators, and it automatically accepts "and" and "or".

Here's an example of (ugly) s-expressions:

 (defun factorial (n)
   (if (<= n 1)
       1
       (* n (factorial (- n 1)))))

Here's sweet-expressions version 0.1:

 defun factorial (n)
   if (n <= 1)
       1
       n * factorial(n - 1)

Here is sweet-expressions version 0.2 (draft), with infix default (it figures out when you have an infix operation from the spelling of the operation):

 defun factorial (n)
   if {n <= 1}
       1
       n * factorial(n - 1)

Here is sweet-expressions version 0.2 (draft), with infix non-default (you must surround every infix operator with {...}):

 defun factorial (n)
   if {n <= 1}
       1
       {n * factorial{n - 1}}

I'm still taking comments. If you're interested, take a look at http://www.dwheeler.com/readable. And if you're really interested, please join the readable-discuss mailing list.

Please point me to High-Assurance Free-Libre / Open Source Software (FLOSS) Components

Mon, 05 Nov 2007 14:19 GMT

I'm looking for High Assurance and Free-Libre / Open Source Software (FLOSS) components. Can anyone point me to ones I don't know about? A little context might help, I suppose...

A while back I posted a paper, High Assurance (for Security or Safety) and Free-Libre / Open Source Software (FLOSS). For purposes of the paper, I define “high assurance software” as software where there’s an argument that could convince skeptical parties that the software will always perform or never perform certain key functions without fail. That means you have to show convincing evidence that there are absolutely no software defects that would interfere with the software’s key functions. Almost all software built today is not high assurance; developing high assurance software is currently a specialist’s field.

High assurance and FLOSS are potentially a great match. We achieve high assurance in scientific analysis and mathematical proofs by subjecting them to peer review, and then worldwide review. FLOSS programs, unlike proprietary programs, can receive similar kinds of review, and many FLOSS programs achieve really good reliability figures for medium assurance. So it can be easily argued that stepping up to high assurance should be easier for FLOSS than for proprietary software. In addition, there are a vast number of FLOSS tools that support developing high assurance components, including PVS, ACL2, Isabelle/HOL, prover9, and Alloy (there's lots more, see the paper for more details).

Yet it's hard to find High Assurance Free-Libre / Open Source Software (FLOSS) components. To be fair, high assurance software components are exceedingly rare in the non-FLOSS world as well. But I suspect there are more than I've found, and I hope that people will help me by pointing them out to me. I'd like to know about such components for direct use, and also simply for use as demonstrations of how to actually develop high assurance software. Today, it's nearly impossible to explain how to develop high assurance software, because there are almost no fully-published examples. Existing "formal methods successes" papers generally don't publish everything including their specs, proofs, and code... which makes it impossible to really learn how to do it. And no, I don't require that proofs prove every line of machine code; but showing how correspondance demonstrations are made would be valuable, and that's currently not well demonstrated by working examples.

If you're curious about the general topic, take a peek at High Assurance (for Security or Safety) and Free-Libre / Open Source Software (FLOSS). I've collected lots of interesting information; hopefully it will be useful to some of you. And let me know of high-assurance FLOSS components that I don't already know about.

Added "MapReduce" to the "Software Innovations" list

Sun, 04 Nov 2007 00:08 GMT

Ken Krugler's recent blog said that my article of The Most Important Software Innovations was "very good", but he was surprised that I hadn't included MapReduce as an important software innovation. Basically, MapReduce makes writing certain kinds of programs that process huge amounts of data, on vast distributed clusters, remarkably easy and efficient. (Wikipedia explains MapReduce, including links to alternative implementations like the open source Hadoop.)

It's not because I didn't know about MapReduce; I read about it almost immediately after it got published. I thought it was very promising, and even forwarded the original paper to some co-workers. I think MapReduce is especially promising because, now that we have cheap commodity computers, having a way to easily exploit their capabilities is really valuable. But even with something this promising, I didn't want to add it to my list of innovations right away - after all, maybe after a little while it turns out to be not so helpful.

Currently, there's aren't many who have Google-sized clusters of computers available. But it's clear that this approach is useful in many other circumstances as well. It's new, but I think it's stood the test of time enough that it's a worthy addition... so I've just added it.

One interesting issue is that the MapReduce framework is itself built primarily on the "map" and "reduce" functions, which are far, far older. So, is MapReduce really a new idea, or is it just a high-quality implementation of an old idea? I'll accept that it's a new idea, but that can be difficult to judge. This judgment doesn't really matter unless you think software patents are a good idea (since every software patents in theory prevents progress for 20 years). But I think it's quite clear that software patents are a foolish idea, and it's clear that others have come to the same conclusion. Eric S. Maskin, an economist who has long criticised the patenting of software, recently received the 2007 Nobel Prize for Economics. Here's a nice quote: "... when patent protection was extended to software in the 1980s, [...] standard arguments would predict that R&D intensity and productivity should have increased among patenting firms. Consistent with our model, however, these increases did not occur." Someone who correctly predicted that software patents were harmful to innovation just received a Nobel prize. I hope to someday see people receive other prizes because they ended software patenting in the United States.

Donald Macleay

Mon, 22 Oct 2007 23:06 GMT

Don Macleay was my mentor and friend, and he just passed away (Oct. 15, 2007). So, this is a small blog entry in his honor.

Here's what I said at his funeral: "In 1980, Don was the manager of a computer store. I was only 15, but he took a chance on employing me, and I'm grateful. He taught me much, in particular, showing by example that you could be in business (even as a salesman!) and be an honest person. He later moved to other companies, and I moved twice with him, because I found that good bosses were hard to find. Don was honest, reliable, a good friend, and an inspiration to me. I will miss him, and I look forward to seeing him again in heaven." I should add that he spoke at my Eagle scout ceremony. Later on, when he moved out to the country, it was always a pleasure to visit him and his family.

Here's a part of his biography, as printed in the funeral bulletin: "Born in Washington, D.C., on October 27, 1934, Donald Macleay was raised in Falls Church. He attained the rank of Eagle Scout and graduated at the top of the first class of St. Stephen's School in 1952. In 1956, he graduated with a Bachelor of Arts in English from the Virginia Military Institute (VMI).

After serving as a Marine Corps officer, Donald Macleay spent many years in the business world before becoming a Parole Officer for the Department of Juvenile Justice in Stafford County. As well, in 1992, he was a candidate for the U.S. Congress as an Independent."

The biography goes on to note that he "valued being a Christian, a husband, a father and grandfather, and a friend." Much of his last years were spent helping troubled youth in his area (Fredericksburg, VA), and from all accounts he was extraordinarily successful at helping them and their families.

Readable s-expressions (sweet-expressions) for Lisp-like languages

Thu, 11 Oct 2007 23:10 GMT

Back in 2006 I posted my basic ideas about "sweet-expressions". Here's a basic recap, before I discuss what's new. Lisp-based programming languages normally represent programs as s-expressions, where an operation and its parameters are surrounded by parentheses. The operation to be performed is identified first, and each parameter afterwards is separated by whitespace. So the traditional “2+3” is written as “(+ 2 3)” instead. This is regular, but most people find this hard to read. Here's a longer example of an s-expression - notice the many parentheses and the lack of infix operations:

 (defun factorial (n)
   (if (<= n 1)
       1
       (* n (factorial (- n 1)))))

Lisp-based systems are very good at symbol manipulation tasks, including program analysis. But many software developers avoid Lisp-based languages, even in cases where they would be a good tool to use, because most software developers find s-expressions really hard to read. I think I've found a better solution, which I call "sweet-expressions". Here's that same program be written using sweet-expressions:

 defun factorial (n)         ; Parameters can be indented, but need not be
   if (n <= 1)               ; Supports infix, prefix, & function <=(n 1)
       1                     ; This has no parameters, so it's an atom.
       n * factorial(n - 1)  ; Function(...) notation supported

Sweet-expressions add the following abilities:

Indentation. Indentation may be used instead of parentheses to start and end expressions: any indented line is a parameter of its parent, later terms on a line are parameters of the first term, lists of lists are marked with GROUP, and a function call with 0 parameters is surrounded or followed by a pair of parentheses [e.g., (pi) and pi()]. A “(” disables indentation until its matching “)”. Blank lines at the beginning of a new expression are ignored. A term that begins at the left edge and is immediately followed by newline is immediately executed, to make interactive use pleasant.
Name-ending. Terms of the form ‘NAME(x y...)’, with no whitespace before ‘(’, are interpreted as ‘(NAME x y...)’;. Parameters are space-separated inside. If its content is an infix expression, it's considered one parameter instead (so f(2 + 3) computes 2 + 3 and passes its result, 5, to f).
Infix. Optionally, expressions are automatically interpreted as infix if their second parameter is an infix operator (by matching an “infix operator” pattern of symbols), the first parameter is not an infix operator, and it has at least three parameters. Otherwise, expressions are interpreted as normal “function first” prefix notation. To disable infix interpretation, surround the second parameter with as(...). Infix expressions must have an odd number of parameters with the even ones being the same binary infix operator. You must separate each infix operator with whitespace on both sides. Precedence is not supported; just use parens (a lot more about that in a moment). Use the "name-ending" form for unary operations, e.g., -(x) for "negate x". Thus "2 + (y * -(x))" is a valid expression, equivalent to (+ 2 (* y (- x))). "(2 + 3 + 4)" is fine too. Infix operators must match this pattern (and in Scheme cannot be =>):
```
    [+-\*/<>=&\|\p{Sm}]{1-4}|\:
```

For more information on sweet-expressions or on making s-expressions more readable in general, see my website page at http://www.dwheeler.com/readable. For example, I provide a demo sweet-expression reader in Scheme (under the MIT license), as well as an indenting pretty-printer in Common Lisp. In particular, you can see my lengthy paper about why sweet-expressions do what they do, and some plausible alternatives. You can also download some other implementation code. I've also set up a SourceForge project named "readable" to discuss options in making s-expressions more readable, and to distribute open source software to implement them (unimplemented ideas don't go far!).

Okay, but all of that was true in 2006 - what's new? What's new is a change of heart about precedence. I've been occasionally trying to figure out how to "flesh out" sweet-expressions with operator precedence, and I just kept hitting one annoying complication after another. Precedence is nearly universal among programming languages; they're very useful, and only a few infix-supporting languages (such as Smalltalk) lack them. "Everyone" knows that 2+3*4 is 14, not 20, because of years of training in math classes that you multiply before you add. They're also pretty easy to code (it's an old solved problem). But I've discovered that in the typical use cases of a Lisp-like language's expression reader, supporting precedence (in the general case) has some significant downsides that are irrelevant in other situations. Which is interesting, given how widespread they are elsewhere, so let's see why that is.

First, let's talk about a big advantage to not supporting precedence in sweet-expressions: It makes the creation of every new list obvious in the text. That's very valuable in a list processing language; the key advantage of list processing languages is that you can process programs like data, and data like programs, in a very fluid way, so having clear markers of new lists using parentheses and indentation is very valuable.

Now let me note the downsides to supporting precedence in the specific cases of a Lisp-like language, which leads me to believe that it's a bad idea for this particular use. Basically, adding precedence rules to a general-purpose list expression processor creates a slippery slope of complexity. There are two basic approaches to defining precedence: dynamic and static.

Dynamic: You can trivially create commands that dynamically create new infix operators, as well as express their precedence and direction. Coding this is not a problem, and in fact there are several implementations of this in Lisp already out there. But using this kind of dynamism for s-expressions themselves gets hideously complicated in many uses of a Lisp-like language. In a Lisp-like language, objects often shift between being data and being code. Often you're working at multiple levels and meta-levels of input - and you may not want their precedences to be the same. Yet trying to handle those different levels turns out to require very complex management of the reader routine... you have to keep passing to the various readers the various incompatible precedence states (ugh). Combining code from different sources can be a nightmare if they have different precedence values, too. In short, having precedences dynamically vary by meta-level creates complexities you just don't want to deal with.
Static: All of the above suggests that dynamic control is painful; one solution would be a completely static, pre-canned set of precedence levels. That would eliminate passing state, at least. And indeed, this is what most programming languages do; you don't get to reset the precedence of C operators (a few languages, like Prolog, are an exception here). Fine, but someone has to predefine that set. That turns out to be really hard; the meanings of various infix operators are not cast in stone, so it's difficult to preset the precedence levels meaningfully (since you don't know what the operators mean). That's particularly true if, like me, you're trying to define a single expression syntax that would work (more or less) on arbitrary Lisp-like languages. The one obvious case is that * and / have higher precedence than binary + and - basically everywhere. That particular rule could be built-in, but if that's all you build in, there's little evidence that it's worth the trouble. This kind of odd exception (there's no precedence, except for * and / over + and -) is unlikely to help development, and it's yet another rule to remember. And if we're going to have such a limited support for precedence anyway, why not completely forbid mixing the operators - the advantage there is that it would make the mapping between lists and their textual representation much clearer.

It's easier to add precedence later, if that turns out to be important after more experimentation. But after the experimentation I've done so far, it appears that precedence simply isn't worth it in this case. Precedence creates complexity in this case, and it hides where the lists begin/end. It's not hard to work without it; you can even argue that (2 + (5 * 6)) is actually clearer than (2 + 5 * 6). Precedence is great in many circumstances - I'd hate to lose it in other languages - but in this particular set of use cases, it seems to be more of a hurt than a help.

Of course, you can write code in some Lisp dialect to implement a language that includes precedence. Many programs written in Lisp, including PVS and Maxima, do just that. But when you're implementing another language, you know what the operators are, and you're probably implementing other syntactic sugar too, so adding precedence is a non-problem. Also, if you're really happy with s-expressions as they are, and just want precedence in a few places in your code, a simple macro to implement them (such as infpre) works very well. But sweet-expressions are intended to be a universal representation in Lisp-like languages, just like S-expressions are, so their role is different. In that different role, precedence causes problems that don't show up in most other uses. It think not supporting precedence turns out to be much better for this different role.

Here are some more examples, this time in Scheme (another Lisp dialect):

Sweet-expression	(Ugly) S-expression
define factorial(n) if (n <= 1) 1 n * factorial(n - 1) substring("Hello" (1 + 1) string-length("Hello")) define move-n-turn(angle) tortoise-move(100) tortoise-turn(angle) if (0 <= 5 <= 10) display("True\n") display("Uh oh\n") define int-products(x y) if (x = y) x x * int-products((x + 1) y) int-products(3 5) (2 + 3 + (4 * 5) + 7.1) (2 + 3 + (4 / (5 * 6))) *(2 3 4 5)	(define (factorial n) (if (<= n 1) 1 (* n (factorial (- n 1))))) (substring "Hello" (+ 1 1) (string-length "Hello")) (define (move-n-turn angle) (tortoise-move 100) (tortoise-turn angle)) (if (<= 0 5 10) (display "True\n") (display "Uh oh\n")) (define (int-products x y) (if (= x y) x (* x (int-products (+ x 1) y)))) (int-products 3 5) (+ 2 3 (* 4 5) 7.1) (+ 2 3 (/ 4 (* 5 6))) (* 2 3 4 5)

Sweet-expression

(Ugly) S-expression

define factorial(n)
   if (n <= 1)
       1
       n * factorial(n - 1)
substring("Hello" (1 + 1) string-length("Hello"))
define move-n-turn(angle)
   tortoise-move(100)
   tortoise-turn(angle)
if (0 <= 5 <= 10)
   display("True\n")
   display("Uh oh\n")
define int-products(x y)
  if (x = y)
    x
    x * int-products((x + 1) y)
int-products(3 5)
(2 + 3 + (4 * 5) + 7.1)
(2 + 3 + (4 / (5 * 6)))
*(2 3 4 5)

(define (factorial n)
   (if (<= n 1)
       1
       (* n (factorial (- n 1)))))
(substring "Hello" (+ 1 1) (string-length "Hello"))
(define (move-n-turn angle)
   (tortoise-move 100)
   (tortoise-turn angle))
(if (<= 0 5 10)
   (display "True\n")
   (display "Uh oh\n"))
(define (int-products x y)
  (if (= x y)
      x
      (* x (int-products (+ x 1) y))))
(int-products 3 5)
(+ 2 3 (* 4 5) 7.1)
(+ 2 3 (/ 4 (* 5 6)))
(* 2 3 4 5)

So I've modified my demo program so that it supports infix operator chaining, such as (2 + 3 + 4). Since I no longer need to implement precedence, the addition of chaining means that I now have a working model of the whole idea, ready for experimentation. My demo isn't ready for "serious use" in development yet; it has several known bugs and weaknesses. But it's good enough for experimentation, to see if the basic idea is sensible - and I think it is. You can actually sit down and play with it, and see if it has merit. There are still some whitespace rules I'd like to fiddle with, to make both long files and interactive use as comfortable as possible, but these are at the edges of the definitions... not at its core.

I'm suggesting the use of && for "logical and", and || for "logical or". These are common symbols in other languages, and using the same symbols aids readability. Now, in Common Lisp and some Scheme implementations, || is "the symbol with 0-length name". Oddly enough, this doesn't seem to be a problem; Lisps can generally bind to the symbol with the 0-length name, and print it the same way, so it works perfectly well! In Scheme this is trivially done by running this:

define(&& and)
define(|| or)

Then you can do this:

(#t && #t)
if ((a > b) || ((a * 2) < (c + d + e))) ...

Instead of the hideous s-expressions:

(and #t #t)
(if (or (> a b) (< (* a 2) (+ c d e))) ...)

Here are some quotable quotes, by the way, showing that I'm not the only one who thinks there's room for improvement:

I've used Lisp my whole programming life and I still don't find prefix math expressions natural. - Paul Graham
I have more faith that you could convince the world to use esperanto than [to use] prefix notation. - Paul Prescod

Lisp-based languages are all over the place. There are vast number of implementations of Common Lisp and Scheme. GNU guile is a Scheme implementation embedding into many other programs, for example. GNU emacs is a widely-used text editor, built on its own dialect of Lisp. AutoCAD has its own variant under the covers, too. Programs like PVS are implemented in Lisp, and interacting with it currently requires using s-expressions. It'd be great if all of these supported an alternative, simpler syntax. With sweet-expressions, typical s-expressions are legal too. So I think this is a widely-useful idea.

So if you're interested, take a look at http://www.dwheeler.com/readable.

Software Assurance 2007

Thu, 04 Oct 2007 13:22 GMT

Lots of interesting things are happening with the various efforts to eliminate or counter software vulnerabilities. The Software Security Assurance (SwA) State-of-the-Art Report (SOAR) tries to list what's going on, especially in things related to the U.S. government. As with any such document, it's incomplete, and it's only a snapshot (things keep changing!). But if you haven't been following this world, and want to know "what's going on", it's the best place I know of to start. Of course, you can also look at sites such as the U.S. DHS / CERT "build security in" site.

The U.S. National Vulnerability Database tracks specific vulnerabilities in specific products; they identify each vulnerability using the unique id defined by Common Vulnerabilities and Exposures (CVE). But if the world is going to prevent these kinds of vulnerabilities from happening in the future, we need to categorize them in a way that everyone agrees what the categories are. Informally, there are lots of ways to categorize them, but their meanings differ between people. That's a real problem when comparing tools; different tools find different problems, but without agreed-on terminology, it's hard to even describe their differences. MITRE is currently developing a way to categorize all vulnerabilities in a way that everyone can agree on, called Common Weakness Enumeration (CWE). The U.S. National Vulnerability Database and MITRE have worked out a set of CWEs that they will use to categorize vulnerabilities. The CWE is still being developed, but at least some common terminology is getting worked out.

FLOSS License Slide - Modified for LGPL

Mon, 24 Sep 2007 22:31 GMT

My "Free-Libre / Open Source Software (FLOSS) FLOSS License Slide" helps people figure out if the most widely-used FLOSS licenses are compatible (that is, if you can combine their software to produce new works). The main body of the PDF text fits all on one page, which can be handy.

My thanks to Olaf Schmidt, who found an error of omission in the previous version of my "slide". He pointed out to me that the GNU Lessor General Public License (LGPL) 2.1 is even more compatible with various versions of the GPL than the plain reading of my "slide" originally noted. That's because the LGPL has explicit text noting that you can switch its license to GPL version 2 or later, and similarly LGPL version 3 explicitly says that you can also use GPL version 3 or later. I fixed this by adding one more arrow to my diagram, which was enough to capture this. I also noted that the previous version of the LGPL is version 2.1, not 2 (previous versions of the LGPL exist but are becoming uncommon). I also added some additional text, so I hope that the LGPL-related text is even clearer now.

Enjoy!

Navy: Open Source Software IS Commercial Software

Wed, 18 Jul 2007 17:52 GMT

On June 5, 2007, the U.S. Navy released some guidance on Open Source Software. In particular, they noted that if Open Source Software (OSS) met the U.S. law definition of "commercial item", it was a commercial item. (They actually say OSS that meets that definition is commercial off-the-shelf (COTS), not just a "commercial item" - presumably because they had in mind the off-the-shelf open source software.) I'm delighted to see this guidance, because I've been saying the same thing. This Navy memo was pretty clear, yet some people seemed to have really odd interpretations of it. In particular, GCN reported that "After several years of evaluation, the Navy Department has approved the use of open-source software in all Navy and Marine Corps information technology systems." This GCN article makes it seem like there's been some big change in direction, and I think that's a terrible misunderstanding of what's going on here.

I believe this 2007 Navy memo is not a change in policy or direction. This Navy memo merely tries to counter a widespread misunderstanding that is sometimes resulting in a failure to obey U.S. law, the U.S. government's Federal Acquisition Regulations (aka the FAR), and (by implication) the U.S. Department of Defense's (DoD) Defense Federal Acquisition Regulation Supplement (DFARS). This memo also serves as a restatement that Navy policy continues to obey existing DoD and U.S. government policies regarding open source software (OSS), as were already formally established in 2003-2004.

Below are some supporting details to justify those statements. I hope they will help put this memo in context. As is usual in any blog, my conclusions are just my own opinion, not the official position of any organization. On the other hand, I think I have really good evidence! So let's see...

Years ago some people had the strange idea that OSS was prohibited in the DoD or U.S. federal government, even though there was no such prohibition. This was particularly bizarre in the DoD, since a MITRE report (final publication early 2003) found that OSS use was already widespread and very helpful to the DoD. That MITRE report concluded that "Neither the survey nor the analysis supports the premise that banning or seriously restricting [Free / Open Source Software (FOSS)] would benefit DoD security or defensive capabilities. To the contrary, the combination of an ambiguous status and largely ungrounded fears that it cannot be used with other types of software are keeping FOSS from reaching optimal levels of use.": http://www.terrybollinger.com/dodfoss/dodfoss_pdf_hyperlinked.pdf

So in May 2003 an official DoD policy memo ("OSS in DoD") was released. It affirmed that OSS was fine as long as it met the applicable DoD requirements, just as any other kind of software had to meet the applicable DoD requirements: http://www.egovos.org/rawmedia_repository/822a91d2_fc51_4e6e_8120_1c2d4d88fa06?/document.pdf

This problem was government-wide, so in July 2004, OMB released a similar policy memo (M-04-16), which explicitly stated that U.S. federal government acquisition policy was neutral about using OSS vs. proprietary software. In particular, it said that government "policies are intentionally technology and vendor neutral, and to the maximum extent practicable, agency implementation should be similarly neutral.": http://www.whitehouse.gov/omb/memoranda/fy04/m04-16.html

Yet there still seemed to be some strange misunderstandings, in spite of these 2003 and 2004 policy memos explicitly stating that U.S. DoD and federal acquisition policies were neutral on the question of using OSS vs. proprietary software (they merely had to obey the usual requirements). More recently, these misunderstandings seem to revolve around a failure to read and understand the term "commercial item" as defined by U.S. Code Title 41, Chapter 7, section 403, as well as its corresponding FAR text. These define a "commercial item" as an item "customarily used by the general public or by non-governmental entities" (i.e., they have uses not unique to a government) and have been "sold, leased, or licensed to the general public"). It would seem obvious that if OSS meets the U.S. law/FAR definition for a commercial item, it is a commercial item for government acquisition purposes. And nearly all extant OSS does indeed meet this definition; nearly all extant OSS has non-government uses and are licensed to the public. In addition, almost all already-existing OSS software also meets the definition of commercial-off-the-shelf (COTS), since they are commercial items that are ALREADY available to the public ("off the shelf").

The problem was that some acquisition programs were redefining the term "commercial item" (and COTS) to exclude OSS competitors. These redefinitions were in contradiction to the existing DoD and federal government-wide explicit policy for neutrality regarding OSS, and in contradiction to the clear definition of "commercial item" given in U.S. law, the FAR, and by implication the DFARS. The Navy memo simply tries to correct this misunderstanding, as well as re-iterating that the existing DoD and federal government policy on OSS continues. This Navy memo was signed by Department of the Navy CIO Robert J. Carey on June 5, 2007, and titled "Department of the Navy Open Source Software Guidance". You can find the Navy memo here: http://oss-institute.org/Navy/DONCIO_OSS_User_Guidance.pdf

The main implication of this definition of "commercial item" is that (as required by law and the FAR) contractors and their subcontractors at all tiers must do market research of the commercial market and consider ALL their commercial options... including the OSS options. This is certainly NOT a special preference for OSS, and ALL evaluation characteristics for software are still valid (e.g., functionality, total cost of ownership, quality, security, support, and flexibility). But in cases where the OSS option is the better option, by policy the U.S. government intends to take advantage of it.

This approach makes sense, given the major changes that are happening in the software industry. In many market segments OSS programs are the #1 or #2 product by market share, and OSS in aggregate now represents billions of dollars of development effort. Many companies are developing OSS and/or selling commercial support for OSS, including Red Hat, Novell, Sun Microsystems, IBM, and MySQL AB. Microsoft competes with some OSS programs, business models, and licenses, but in other areas Microsoft uses, develops, and encourages OSS (Microsoft's Windows includes BSD-developed networking applications; Microsoft OSS projects include WiX and IronPython; Microsoft runs the "Codeplex" website to encourage OSS development on Windows). In areas where they are appropriate some major OSS programs have received the relevant Common Criteria or FIPS 140-2 IT security certificates.

OSS potentially affects how acquisition programs acquire software, but acquisition programs should expect to be affected by changes in relevant commercial industries. This was anticipated; the DoD policy memo "Commercial Acquisitions" (Jan. 5, 2001) explains that the benefits of commercial item acquisition include "increased competition; use of market and catalog prices; and access to leading edge technology and 'non-traditional' business segments". In other words, DoD policy anticipates that there WILL be "non-traditional business segments" - and its policy is to embrace and exploit such changes where appropriate. (Given its growth and breadth, it's become increasingly difficult to argue that OSS is "non-traditional" anyway.) AT&L's "Commercial Item Handbook" (November 2001) explains that this broad definition of "commercial item" is intentional, because it "enables the Government to take greater advantage of the commercial marketplace." See: http://www.acq.osd.mil/dpap/Docs/cihandbook.pdf

In other words, U.S. contractors must consider all their options, and then select the best one. They are not allowed to arbitrarily ignore a relevant commercial industry sector, and are specifically not allowed to ignore OSS options.

If you're interested in this topic, you might also be interested in some related articles of mine, such as Open Source Software (OSS) in U.S. Government Acquisitions and “Commercial” is not the opposite of Free-Libre / Open Source Software (FLOSS): Nearly all FLOSS is Commercial.

FLOSS License Slide - Released!

Wed, 11 Jul 2007 23:17 GMT

There are a large number of Free-Libre / Open Source Software (FLOSS) licenses, but only a few are widely used. Unsurprisingly, the widely-used licenses tend to be compatible -- that is, it’s possible to combine software under the different licenses to produce a larger work. But many people have trouble figuring out when they can be combined, or how.

So I've created a little figure which I call the “FLOSS license slide” to make it easier to see if FLOSS licenses can be combined in many common cases, and if so, what the basic ramifications are. I've crafted it so that the figure and explanatory text all fit in a page, which can be handy.

You can look at the FLOSS license slide in one of three formats:

I had released a draft of this earlier, and got some nice feedback. I added "public domain"; there seemed to be enough questions about public domain software that it made sense to add that to the figure. Yes, I know that technically "public domain" isn't a license, but it's much easier to understand and explain by treating it as if it were one. Some of the earlier text was not clear enough; hopefully this one is clearer. For example, GPLv2+ and the Affero GPL 3 licenses are compatible via GPLv3, but some people had trouble understanding why. Now the text notes that they are compatible via the GPL version 3, which will hopefully make that clearer. I also added the HTML format.

I'm not a lawyer, and if you need formal legal advice you need to consult your own attorney. But for many people, this is the information you needed, in a conveniently small format, so here it is. Enjoy!

Increasing Government Interest in Free-Libre / Open Source Software (FLOSS)

Sun, 08 Jul 2007 16:10 GMT

There's more on the web now about governments and Free-Libre / Open Source Software (FLOSS), both on my own site and other sites. I think this suggests more and more interest by governments in FLOSS.

I've now posted Open Source Software (OSS) in U.S. Government Acquisitions, which is a slightly-updated version of my article which was published in the "DoD Software Tech News" issue devoted to free-libre / open source software (FLOSS). As I noted earlier, this issue has lots of good articles. I was asked to create the article (not the other way around), so I think the very existance of this issue is evidence of increasing interest, not just a personal interest of mine.

I should probably mention why I'm posting a revision. After I submitted my article, the Department of the Navy CIO Robert J. Carey signed a June 5, 2007 memorandum titled “Department of the Navy Open Source Software Guidance” which notes that FLOSS needs to be considered a commercial item when it meets the U.S. government's standard definition of a commercial item (nearly all extant FLOSS does). That is the same basic point that I raised in my presentation and paper, and I would have gladly mentioned the Navy memo... had it been released at the time. So my modified version now points to the Navy memo, as well as making a few tweaks based on other feedback. This realization that FLOSS programs are (almost always) commercial items is an important point in the U.S. government. Why? Because as noted in Linux.com, that means that extant FLOSS software must be considered when the U.S. government acquires software the same way as other commercial software is considered (i.e., it must be considered before starting a new project to write their own). The Navy memo's assertion of this makes it worth posting an update.

But think of this updated essay as a only sampler... for the rest of the articles, you'll still need to go read the "DoD Software Tech News" issue. Getting the issue does require registration, but registration is free and in this case it's worthwhile.

Anyway, it's not just one issue about FLOSS in one magazine. Here's more evidence it's not just me - on July 5, 2007, the article Open Source Government: good-will needed was posted by Roberto Galoppini's Commercial Open Source Software blog. He points in turn to various articles, including Matt Asay's "Open source in government: Leadership needed", which then leads you back to a very interesting research paper: "Open-Source Collaboration in the Public Sector: The Need for Leadership and Value" by Michael P. Hamel. There's lots of interesting discussion in those articles about how governments can use FLOSS, and in particular how governments can use FLOSS components and approaches more effectively. I'll throw in my own two cents here. I would certainly agree that leadership is important in any project, including FLOSS projects; the leadership of Linus Torvalds of the Linux kernel is well-documented, for example. But there are different kinds and levels of leadership. Roberto Galoppini says, "I think that first we need politicians with good-will, willing to put their intellectual potential to work for the overall desires of the general public..."; while that's good, often what's important is the ability to lead concept into practice. It's easy to "lead" by saying words, but I think that often what's needed is the kind of leadership that rolls up its sleeves and makes actual projects produce useful results. Focusing on specific, measurable products can often get better results. Certainly politicians with goodwill are a good thing, but they can provide value primarily by providing "cover" for those who actually do the hands-on leading of projects, but the latter make or break such efforts.

While I can quibble with some of the stuff Hamel says, some of the his statements seem dead-on with what I see. Hamel notes that "participants in both groups also believe that the creation of value, or products that are appropriate and effective in addressing members’ wants and needs, is critical"... to which I say, Amen. Hamel concludes that "Collaborations with a strong leadership structure, and more importantly a single leader who is persistent, passionate and willing to spend a great deal of time maintaining and improving the organization are much more likely to succeed. Value is also a critical component, and requires that efforts meet the wants and needs of members and clients, whether they be in the form of software, documentation, research or even policy advocacy." Focusing on a few most useful projects is critical: "a conscious effort to focus energy on a small number of projects in early stages may be an important component in creating value for members of collaborative efforts." A FLOSS project requires collaboration to be successful, and collaboration requires that the project gain the trust of potential users/developers; "In this research I found that leadership, face-to-face contact, and the legal framework were the primary factors leading to trust. A willingness and ability to evolve, which may be tied to creating products of value to clients and members, might also be an important factor in developing a successful collaboration." Those statements, at least, seem very sound. These aren't new ideas, to be sure, but it sure is easy to lose sight of them.

I think there's a vast opportunity here for governments to use FLOSS. Which is odd in a sense, because in fact, FLOSS is already widely used in governments. MITRE determined back in 2003 that FLOSS was already critically important to the U.S. Department of Defense (DoD). But that widespread use is small compared to its potential, and to how many commercial organizations use FLOSS, and there are a lot of reasons for that. Governments have long aquisition times, often tend to avoid doing anything "risky" (aka "different than the way we did it before", abetted because there's no competitor to force them to improve), and many proprietary vendors target governments to try to prevent them from using competing FLOSS products. Which means that even when it makes sense to use FLOSS, it can often take much longer for FLOSS components and approaches to enter into government use. But it has already entered into significant use, and I expect that to accelerate over the years.

There are some other goodies on my web site if you're interested in this topic (government and FLOSS). My essay is basically a text version of my March 2007 presentation on Free-libre / Open Source Software (FLOSS). It includes snippets from much longer papers that give statistics about FLOSS, explain how to evaluate FLOSS, discuss FLOSS and security, and explain why most FLOSS is commercial software. There's even a hint of my essay Make Your Open Source Software GPL-Compatible. Or Else, since I note the widespread use of the GPL.

Again, if you have a true PHB, I can't help you. But many managers are trying to the right thing, and just need reasonable information so that they can do the right thing.

ISPs Inserting Ads Into Your Pages: Unlawful Modifications (Copyright Violation)

Sat, 23 Jun 2007 11:20 GMT

Here's a nasty trick. Apparantly some shady Internet Service Providers (ISPs) are inserting advertizements into other people's web pages without their permission.

I believe these are usually illegal modifications. Whoever creates a work automatically has a copyright on that work. (If you're doing work for someone else, or if you sell the copyright, they have the copyright.) Anyone who releases a public web page obviously grants others the right to copy that work to view it, but unless otherwise stated, you don't have the right to modify the work. U.S. law also has "fair use" provisions that let you quote another work without being sued, and so on, but those can't possibly be stretched into letting someone insert ads. The copyright owner can release works permitting such actions (that's how Free-libre / open source software works), but they have to give that permission... you can't just unilaterally modify it. What really angers me is that they might insert stuff I am morally opposed to, and sullying my reputation.

If you run a website, here's a sample letter that you can send to inform these ISPs that what they're doing is wrong.

Dear (name here),

I am concerned that you appear to be violating copyright laws,
by unlawfully modifying and then redistributing copyrighted works
of mine and many others.

In particular, your "terms of service" say... (quote them, if you can).

My website, (www.dwheeler.com), distributes many copyrighted works
at no charge, but I do _NOT_ give the right to modify many of
those works.  Many other websites do the same.  (Some can add:
In addition, I have ads on my site; by adding your own ads, you are
diluting the value of MY ads and in essence stealing from me.)

Please stop modifying my web pages immediately.  It is my hope that
this was just a misunderstanding, and that we can settle this amicably.

Thank you very much.

"DoD Software Tech News" posts open source software issue

Tue, 19 Jun 2007 12:44 GMT

The U.S. Department of Defense (DoD)'s software technology magazine "DoD Software Tech News" has posted a whole issue devoted to free-libre / open source software (FLOSS). If you're trying to get FLOSS seriously considered by acquisition or management people, this may be what you need. This issue includes essays by David A. Wheeler (that's me!), Terry Bollinger, John M. Weathersby, Mark Lucas (on Geospatial FLOSS), Peter Gallagher, Matt Asay (Alfresco), and Andrew Gordon. Free registration required. My essay is basically a text version of my March 2007 presentation on Free-libre / Open Source Software (FLOSS); it includes snippets from my papers that give statistics about FLOSS, explain how to evaluate FLOSS, discuss FLOSS and security, and explain why most FLOSS is commercial software. There's even a hint of my essay Make Your Open Source Software GPL-Compatible. Or Else, since I note the widespread use of the GPL.

After I submitted my article, but before it got published, the Department of the Navy CIO Robert J. Carey signed a June 5, 2007 memorandum titled “Department of the Navy Open Source Software Guidance” which notes that FLOSS needs to be considered a commercial item when it meets the U.S. government's standard definition of a commercial item (and nearly all extant FLOSS meets that definition). That is the same basic point that I raised in my presentation. It feels nice to have made a key statement, and then find that the U.S. Navy officially confirms it. As noted in Linux.com, that means that extant FLOSS software must be considered when the U.S. government acquires software, the same way as other commercial software is considered (i.e., it must be considered before starting a new project to write their own).

If you have a true PHB, I can't help you. But many managers just need some honest information, and this is at least one way to get it to them.

Reviews of Books, Movies, etc.

Mon, 11 Jun 2007 22:54 GMT

I read voraciously, and occasionally I've even been known to see a movie. So I've started to write down Reviews of Books, Movies and Other Stuff. There you can see what I think of various books, movies, etc. It's not huge yet, but I intend to make it grow.

In many cases you can click on it to get your own copy via Amazon.com. If you buy something that way, I even get a small cut. I won't get rich that way, but if you choose to buy something by doing that, thanks very much!