Is it possible for software to be both commercially viable and truly open source?
Obviously your answer to this question will depend on what you mean by those notoriously slippery words "open source." If you mean "anything covered by the GPL" then the question is easy. Red Hat is GPL'd, and no one can doubt that it's making money hand over fist, therefore we can answer with a resounding "yes" since at least one famous example meets the condition.
But paradoxically, Linux is not a typical example of open source. Why not? Because it owes its existence to huge subsidies – both financial and technical – from big IT vendors who have a stake in seeing Microsoft taken down a notch. For example, IBM, Oracle, Intel, Novell, HP, Fujitsu, Hitachi and NEC each fork over $500K per year to the Linux Foundation. AMD, Cisco, EMC, Motorola, NetApp, NTT, Siemens and Sun also chip in a not so measly $100K each. For good measure, another 16 IT players including Google and Red Hat toss in $20K per year. Admittedly that's chump change for them (Red Hat, you cheapskates!). But hey, it's all green.
All told, these companies pump more than $5 million a year into the Foundation, which among other good works pays Linus Torvalds' salary. And of course this is only the proverbial tip of the iceberg. Who can forget IBM's oft-repeated claim to have spent "billions" on Linux? It's probably safe to assume that most of this money went to supporting proprietary IBM products that only leverage Linux, but still – you get the idea.
So let's rephrase the question to ask, how many software packages are out there other than Linux that are both commercially viable and truly open source? One example that comes to mind is Firefox. Now that the famous browser is available under the GPL in addition to the somewhat controversial and GPL-unfriendly Mozilla Public License, its open source bona fides can't really be doubted. Although the Mozilla Foundation's tight control over the Firefox name has made some open sourcers unhappy (hence Debian's rebranding of Firefox as "Iceweasel" and the GNU fork of Firefox now known as "IceCat"). Also, Firefox is certainly commercially viable, since its Google search bar tie-in brought in roughly $60 million in 2006 and will no doubt be worth quite a bit more this year.
But Firefox belongs more in the consumer than the enterprise bin. What about purely enterprise-oriented applications? How many of them can we find that are truly open source and commercially viable? As a thought experiment I decided to check out some of the vendors participating in Red Hat's Exchange web store (aka RHX) to see how they stacked up on this question. The results are pretty interesting.
First we should ask whether Red Hat claims to be selling only open source software on RHX. The company says the following on its web site:

Q: Are you only accepting open source ISVs into RHX?
A: The initial set of participating ISVs all have an open source focus. We realize that there is debate about which companies are truly open source. To make it transparent to users, RHX includes information about each ISV's license approach. Longer term, we may introduce proprietary applications that are friendly with open source applications.

So the answer appears to be "yes." But as we shall see, in some cases "yes" actually means "not really." As Bill Clinton would say, it all depends on what the meaning of "have an open source focus" is.
The dozen or so software packages for sale on RHX cover a wide range of application and middleware categories: Compiere for ERP, Sugar for CRM, Alfresco for Contact Management, MySQL and the Postgres-based EnterpriseDB for database, Jaspersoft and Pentaho for business intelligence, Zimbra for e-mail, etc. However, figuring out what these vendors' open source policies are quickly proves to be a daunting task. Their license FAQs are full of artful evasions, empty marketing-speak or – in a few cases – outright prevarication. Red Hat itself offers little guidance. True, the distro vendor does offer a series of web pages which purport to summarize the license terms of each partner vendor. But these pages, with their use of cryptic symbols and repetitive boilerplate prose, appear to have been crafted as much to obfuscate as to inform – either that or they were designed by a deranged five-year-old (in striking contrast to the rest of the Red Hat site, which is crisp and clean).
Parsing the Red Hat summary pages and the vendor FAQs, it seems that only business intelligence vendor Pentaho offers its complete product under GPL terms. The other vendors seem to offer only limited "community" editions of their software under the GPL while reserving their full-featured versions for some type of restricted "commercial" or "enterprise" license. Exactly what these licenses entail is usually far from clear. Often it's amazingly hard to tell if the actual source code for the full version of the product is available or not. Try this SugarCRM license agreement for its Enterprise edition if you don't believe me. Frequently the marketing blurb on the web page implies one thing while the actual license text says another. A few of the RHX vendors are clearly not open source in any defensible sense of the term. EnterpriseDB, for example, sells a closed-source proprietary product that is merely "based on" the BSD-licensed Postgres open source database. Nothing wrong with that, as long as you don't call yourself an open source company, which in all honesty EnterpriseDB no longer does (but you won't find that out from Red Hat).
It's pointless to debate which license "truly" defines open source. Some purists say GPL only, but most people would probably include at least the Apache and BSD licenses. Personally I prefer to define open source in operational terms rather than legalistic ones. My test is simple: a piece of software is truly open source if any competent person can take the freely distributed source code and compile it to make a working binary clone of the original product. Red Hat clearly passes this test, because we have CentOS and plenty of other RHEL clones. But as far as I can tell, none of the RHX vendors except maybe Pentaho do.
So here we come to the crux of the matter. Too many open source enthusiasts are hung up on words and legalisms. Hence they are inclined to take vendor claims to be open source good guys at face value, especially when these claims are backed up by the apparent authority of the near-sacred GPL. I say horse pucky. What counts is what you can do with the stuff. If the software provider has imposed restrictions that make it difficult or impossible to clone a fully-functional binary from freely available source code for the complete product and not just some "community" subset, then it doesn't matter what the license is or whether you call it "open source." It is a proprietary product whose corporate owners believe that restricting access to their intellectual property is an essential component of their business model.
It is worth pointing out that all of these vendors (as far as I can tell) assert copyright over their source code and demand that would-be outside contributors sign away all ownership rights. This is a profound departure from the long-established practice of the Linux kernel, where contributors retain their copyrights. In my humble opinion, this difference pretty much reduces the holier-than-thou claims of most "commercial open source" software vendors to vapor. Don't get me wrong. I believe in the free market system, and these guys make some great software. I urge enterprises large and small to go out and test drive it, and buy it if it fits. Just make sure you don't also buy the hype about "open source."