How Open Source is Shaking Up the Software Industry

- Part I

by Steven Myers



By all accounts, 1998 was a banner year for the Open Source Software (OSS) movement. Netscape released all the source code for Communicator; Linux made huge advances in the operating systems arena, prompting a worried Microsoft to proclaim their own plans for counterattack; and by December, Sun had even announced that Java was moving to an OSS licensing scheme. In Part I of this two-part series, we discuss the general nature of the OSS model and how it has developed in Japan, and present an exclusive interview with one of the movement's major figures: the inimitable Larry Wall - software developer, violinist, author, and inventor of the Perl programming language. Next month, we'll examine the state of OSS in Japan, and finish up with a second terrific interview, this time with Tim O'Reilly, president of O'Reilly & Associates.

For the last twenty years or so, conventional business and software engineering wisdom has dictated that software development efforts be secretive closed-door affairs, limited to as few programmers as possible. Furthermore, the source code for a software project has generally been regarded as a 'sacred' asset to be placed under strict guard, seen and touched by only the privileged few. This model has firmly entrenched itself in the commercial software sector as the established paradigm for successful software development, and until recently there has been little reason for the mainstream business community to question its validity.

Over the past few years, however, as more and more businesses tasked their IT departments with the job of putting together corporate websites, an increasing number of engineers and system administrators began experimenting with "free software" - everything from the Linux operating system to the Apache Web server and Perl programming language. Along the way, they made a surprising discovery - the free software actually worked better than the expensive commercial packages. Much better. Server crashes and downtime decreased, and performance improved dramatically. How was it that this free software hacked together by part-time volunteers who had never even met each other was able to surpass - dramatically so - the work of highly-paid professional teams where the developers worked together side by side?

In his paper 'The Cathedral and the Bazaar,' OSS pioneer Eric Raymond likens the commercial software development process to a cathedral, where a small group of priests work in isolation from the rest of the world. The open source (free software) model, on the other hand, is more like a bazaar - noisy and bustling, everyone with something to say. Linux was arguably the first major software project developed bazaar-style, and the results were eye opening. Raymond posits nearly twenty reasons for the success of Linux and other open source projects, but the three major points can be summarized as follows:

1) The developers are all working on a project that interests them deeply. Raymond notes that "every good work of software starts by scratching a developer's personal itch." Engineers who are working passionately on a project they love will always do better work than those who are just "grinding away for pay."

2) Releases come early and often, and customer feedback is valued. In 'The Business Case for Open Source' on the opensource.org website, the author notes that the bazaar-style process is similar to the strategy employed so successfully by many Japanese companies in consumer product development - they get a product to market that works but is not perfect, then modify it extremely quickly based on customer feedback. The rapid evolution of Japanese keitai phones, laptops, and PDAs are typical examples of how this process results in the right combination of features that users want and need.

3) Because of the large number of beta-testers and co-developers, almost every problem is characterized quickly, and the fix is obvious to someone. 'Given enough eyeballs, all bugs are shallow' - Raymond calls this 'Linus's Law', and anyone who has ever been involved in even a moderate-size software project can attest to the value of employing a large army of beta testers. In the case of OSS projects, often one person will find a problem, and someone else will understand and be able to fix it. Both activities are crucial to the development process, and both happen much faster in the OSS model, where the beta testing troops include not only sharp problem finders but also expert fixers.

The end result of these three factors is that the OSS process tends to produce software that evolves quickly and is highly reliable. In the minds of many businesspeople, however, open source still conjures up images of shoddy, amateurish code scrapped together by students and hobbyists. But this notion is steadily beginning to change as more and more case studies and real-world testimonials are published in the mainstream media. The very foundations of the Internet itself are based in OSS: Linux, Apache, Perl, sendmail, and DNS are just a few examples of open source software without which the Internet and Web could not survive.

Given these overwhelming advantages and the astonishing track record of open source projects, one might wonder how the large commercial software companies are planning to respond to the clear and immediate business threat posed by open source efforts. After all, how does a company compete with a product that is free? Netscape was one of the first companies to grasp the significance of OSS, and announced in January 1998 that it would soon release the Communicator source code. Last summer, both Oracle and Sybase announced they would be developing Linux versions of their major database products. IBM put out a press release saying it would sell and support Apache as part of its WebSphere suite. In September, both Intel and Netscape acquired minority stakes in Red Hat, the leading Linux distributor.

Microsoft, however, seemed strangely quiet about the whole OSS issue throughout most of 1998, until publication of the now-infamous Halloween document, an internal strategy memorandum that was circulated around the company in August and inadvertently leaked in late October. In the document, the author acknowledged the high quality of OSS products and the threat that they pose to Microsoft in the server market. The writer goes on to say that one of Microsoft's favorite marketing tactics, FUD (Fear, Uncertainty, Doubt - casting doubts about the safety and reliability of a competitor), would not work against OSS. Instead, he proposes using the 'embrace and extend' strategy of "de-commoditizing protocols," meaning that Microsoft would extend (add new features that only work on Microsoft products) the existing standard protocols, and then create new protocols of their own. This document sparked a weeklong furor in the national media and heightened the visibility of OSS among mainstream businesspeople.

Perhaps the major turning point in public awareness of OSS, however, came last December, with the announcement from Sun Microsystems that Java 2 would be released under an open source license. This time, all of the major mainstream business publications sat up and took notice, both in Japan and overseas. "We are sharing our source code with companies and individuals committed to compatible implementations of the Java platform," commented JavaSoft president Alan Baratz, noting that the changes would allow licensees to participate freely in "the improvement of the Java technology base." Open source software had at last established itself as a real force in the commercial world - a force that promises to change all the rules.

How OSS businesses make money

In addition to overcoming misconceptions regarding quality, OSS must break through a second psychological barrier as well: most people do not fully grasp the business opportunities involved in "free" software. The reactions of many managers and business executives towards OSS are similar to their general questions regarding the Web about four years ago - "Okay, we understand that this works and it's 'cool', and we're going to make money with it... tell us how?"

In his paper "Setting Up Shop," Frank Hecker of Netscape details several different business models for OSS companies. Three of the most important of these models can be summarized as follows:
  1. Support Selling - In this model, the company effectively gives the software away for free, but sells distribution, service, and integration. Red Hat is one of the most successful and visible examples of this type of business.

  2. Accessory Selling - OÕReilly & Associates, SSC, and VA Research have profited tremendously by selling OSS-related hardware with open source software pre-installed, and OSS-related books that provide free OSS CD-ROMs.

  3. Widget Frosting - Here, a company that produces computer hardware or consumer electronic products makes their software open source to reduce costs. For these companies, software is a necessary expense and a valuable part of their product, but it is not the bread and butter of their operation.


    Dick Hardt & Larry Wall:

In addition to the various business models, there exist a variety of other business-related reasons for employing some type of OSS scheme. As mentioned previously, one of the hallmarks of open source development efforts is the rapid release/ feedback cycle, which by its very nature requires the company to stay close to its customers. Companies who are in tune to their customers' needs will always have an advantage over those who are not. Secondly, development time will be faster and less expensive, since other people will be fixing bugs that would otherwise require the efforts of in-house programmers. Finally, by releasing its source, a company greatly increases its chances of receiving new extensions, features, and even ports to different platforms and locales at little or no cost. OSS developers tend to be highly committed to projects they deem interesting, and often produce unexpected ports and extensions that can significantly widen the market for a particular product. (Note: For a more detailed explanation for the non-specialist, see: http://www. opensource.org/for-suits.html.)

The OSS wave hits Japan

The "open source" label began showing up on the Japanese radar a few months after it first appeared in the US. Last fall, Japanese translations of "The Cathedral and the Bazaar" and the Halloween Documents appeared on the opensource.org site. Both the Tokyo Perl Conference and the subsequent party celebrating O'Reilly & Associates' 3rd year in Japan attracted many of the industry's heavyweights and generated a burst of media attention here for OSS. And of course, Sun's announcement about Java 2's open licensing scheme finally brought the movement the level of attention and recognition it will need to become a viable commercial force in this country.

More barriers

But how will businesses here take to this new model of software development? Interestingly, the Japanese potentially have more to gain than US software companies by embracing open source projects (after all, Microsoft's mindshare and stranglehold on the market are even more prominent than in the US), yet the psychological barriers to acceptance mentioned earlier are likely to be even stronger here in Japan. There has traditionally been a fairly strong perception among Japanese consumers that a high quality product is necessarily more expensive; therefore the higher the price, the better the quality. Additionally, Japanese tend to display on the whole a much greater willingness to pay more for that perceived additional quality (present or not). It will thus be an uphill climb for OSS proponents here to convince managers and executives that a free product is actually of higher quality and reliability than one that costs millions of yen.

In his 'On the Move' column (see: http:// www.zdnet.co.jp/pcweek/column/onthemove/index.html), Shozaburo Nakamura cites two additional factors that could prove to be more of an impediment in Japan than elsewhere. One of the major PR coups for Linux overseas has been its adoption by various American and European government agencies, both at the state and national level. The savings gained by using Linux rather than NT (no small sum, considering these agencies must network thousands of machines) are then used by politicians and government representatives as an example of how they are saving taxpayer's money. Nakamura notes ruefully that reducing costs and saving tax money is much less of a concern for Japanese government organizations, and the odds that Linux will adopted by such organizations here are minimal.

Nakamura goes on to discuss a second issue that is likely to be problematic for OSS proponents: resistance to the idea of having to pay in order to get service for a product that is free. He describes a panel session at the recent Tokyo Linux Fair 98 in which one of the members, Third Ware Ltd. president Motoharu Kubo - whose company provides support for Linux and other open source products - mentioned that clients are often surprised that they must pay for after-sale service. Kubo explained that many customers are under the assumption that if the software is free, service should be free too. Indeed, the general feeling is that the very need for service is itself indicative of the product's inferior quality, and the service fee should therefore be waved. It is no small feat trying to convince these customers that the costs for NT service would ultimately be much higher and would be incurred far more frequently.

Finally, many OSS products are likely to struggle here unless they are firmly backed by a reputable and fairly well-established company that will package, distribute, and provide all necessary support for the product. Perhaps more so than elsewhere, there exists in Japan a definite and strong preference for shrink-wrapped, packaged software and recognizable brand names. The issue of responsibility for a product (sekinin mondai - who to call when something goes wrong) is quite significant, and several software consultants here have commented on the difficulty involved in convincing clients to use a product that does not come in a shrink-wrapped box displaying the telephone and fax numbers of the company's Japan office.


An Interview with Larry Wall



For someone who has achieved near-legendary status in the software industry, Larry Wall comes across as extremely humble and unpretentious. He is easy to talk to, completely open and approachable. During the following interview, I was continually reminded that software development is really about creativity, fun, and the joy that comes from making something novel and interesting.

Q: It seems you've been doing a lot of speaking about OSS lately at seminars and conferences. What are your overall impressions of the movement in general?

Larry Wall: Well, OSS is really fitting in with what I think is the whole tenor of the times. Essentially, the Industrial Revolution has succeeded, and people are making enough money over and above what they need to live. If you go into the supermarket, you notice that the choices you have just in terms of say, fruits and vegetables, is tremendously greater than it was ten years ago. In other words, people are willing to pay a little more for customized stuff. What that means is that we're seeing the renewal of the cottage industry where someone can afford to make small runs - like small breweries, for example, and that trend is going through all society. It applies to software makers also - we can now afford to have some amount of fun and not worry so much about ROI.

Q:A lot is being said lately about the factors that motivate OSS developers. What are some of the aspects of software development that motivate you the most?

Larry Wall: Mainly a sense of artistry - I appreciate having an audience - I'm really a performer at heart. I like to make people happy. Of course, in the old days, artists had patrons - people who just paid them to concentrate on their work without having to deal with distractions, and essentially that's what O'Reilly is doing for me. You've heard of IBM Fellows? - well, O'Reilly has one fellow and that's me. Essentially, I'm paid to do whatever I want. They make suggestions every now and then, and often I follow them because they're usually very good suggestions.

Q: Back when you were developing Perl, did you ever imagine you would one day have so many users?

Larry Wall:I had a sneaky suspicion that if I liked something, other people would like it too. I don't think the enormity of it ever really sunk in, though. The first conference was a real eye-opener in that respect. I didn't anticipate the Web, of course, but that's been a large part of the growth in Perl users. And in a sense that's maybe what's supposed to happen - they always say that the mark of a good tool is that it's used for something the maker didn't anticipate, and that happened to me.

Also, I think Perl enters many organizations through the back door. One of my biggest surprises came when I was at a conference and someone from Goldman Sachs came up and told me there was a Perl book on every desk. What I didn't realize was that all the hard core stuff still goes through other channels - that's not where they use Perl. Rather, it's in all the financial analysis. They really need a lot of rapid prototyping. As soon as everyone else figures out their model, they need another model.

Q: What aspects of Perl are you personally most proud of?

Larry Wall:I It flows easily. A good glue or binder will flow into the gaps and make things fit together that weren't intended to fit together and there may be many reasons that Perl flows easily. It certainly helps that it's free. As a language, it is very malleable; you can use it for very quick and dirty stuff. On the other hand, if you need stronger glue you can add different hardeners to it and get different glues. As your program grows, you can turn up the level of discipline, and that seems to work out very well.

Q: I understand you've been doing work to enable XML and Unicode support with Perl. How has that been going?

Larry Wall: Yeah, I just wanted to make sure that as the definition of text processing changes, that Perl stays good at text processing - whatever that means. I don't even know if it will even be called text processing in twenty years. What I discovered was that XML can be covered nicely with an extension module. Of course, XML requires Unicode, and for Unicode support, we needed some substantial changes to the core. It wasn't as bad as I thought it would be, but it was a piece of work. Essentially, what I did for Unicode was to add support for wide characters with a default mapping of Unicode character properties, but the wide character support is implemented in UTF-8 and they're really just abstract numbers as far as Perl is concerned, so you can have a mapping to any kind of character set you want.

I always apply the current mapping tables from the Unicode consortium. If you say you use UTF-8, it just changes the semantics from bytes to characters, but if you use funny characters in your identifiers, it will use the mapping tables to determine whether they're letters or numbers or whatever, so that's one way that it's dependent on having a current mapping. Another example is when you have a primitive for matching a base character followed by its combining characters. That is also dependent on Unicode mappings in order to figure out what's a base character and what's a combining character, but for the most part it's a pretty clean separation between the wide character support and the character mapping. [For more information on Unicode, see www.unicode.org - Ed]

Q: Are you involved much with the XML Consortium?

Larry Wall: From the beginning, we've been in touch with Tim Bray. The XML folks actually have a notion called the "DPH", which stands for "desperate Perl hacker." Somebody's even written a paper called 'XML Programming - Can the Desperate Perl Hacker Do It?' So we met in February and talked a lot about some of the stuff that's become reality since then. I then went to work on an XML parser that is actually a parser factory object. It produces parsers and functions as a workbench where you can have a full-blown OO parser and it gives you back the whole parse tree - fully annotated. Anyway, that was a fun project and I've since handed the parser off to the OSS community, where it belongs, and the Unicode support is in the current development release of Perl.


Q: What are some of your impressions from the Perl conference here?

Larry Wall:I came with great trepidation, not knowing what to expect. As it turns out, I need not have worried - it was wonderful and everyone there was wonderful. The conference in the US was bigger; this one was more intimate, which I thought was nice. In some ways though, I think this conference might be more important than the ones in the US, because people in the States can interview me or see me if they want to, and there are a lot of Perl programmers there already and the community has lots of ties. When I came here, I got the feeling that there were quite a few Perl programmers in Japan but they tend to be isolated from each other. This was a really important step in bringing them together and I was glad to see them forming a Tokyo Perl mongers group which will continue the social and intellectual interchange.

Q: Finally, what message do you want the attendees to take away from this conference?

Larry Wall: Have the appropriate amount of fun.



Back to the table of contents