Contracted work needs to be handled carefully in free software projects. Ideally, if you hire a contractor you want her work to be accepted by the community and folded into the public distribution. In theory, it wouldn't matter who the contractor is, as long as her work is good and meets the project's guidelines. Theory and practice can sometimes match, too: a complete stranger who shows up with a good patch will generally be able to get it into the software. The trouble is, it's very hard to produce an acceptable patch for a non-trivial enhancement or new feature as a complete stranger. One must first discuss the changes with the rest of the project, and even for those who are very familiar with the project the duration of that discussion cannot be precisely predicted — for those new to the project, the margin of error will only be higher. If the contractor is paid by the hour, you may end up paying more than you expected; if she is paid a flat sum, she may end up doing more work than she can afford.
There are various ways to cope with this. You can try to make an educated guess about the length of the discussion process, based on whatever past experience you have with that community, add in some padding for error, and base the contract on that. It also helps to divide the problem into as many small, independent chunks as possible, to increase the predictability of each chunk.
Another standard technique is to contract for delivery of a patch that meets the formal upstream guidelines and for a tightly budgeted "best effort" at getting the patch integrated into the upstream project treat. The contract itself can never require that the patch be accepted by the upstream project, because that would involve selling something that's not for sale. (What if the rest of the project unexpectedly decides not to support the feature?) However, the contract can require a bona fide effort to get the change accepted by the community, and that it be committed to the repository if the community agrees with it. For example, if the project has written standards (e.g., about coding conventions, documentation, writing regression tests, submitting patches, etc), the contract can reference those standards and specify that the contracted work must meet them. In practice, this usually works out the way everyone hopes.
One tactic for successful contracting is to hire one of the project's developers — preferably a committer — as the contractor. This may seem like a form of purchasing influence, and, well, it is. But it's not as corrupt as it might seem. A developer's influence in the project is due mainly to the quality of her code and to her interactions with other developers. The fact that she has a contract to get certain things done doesn't raise her status in any way, and doesn't lower it either, though it may make people scrutinize her more carefully. Most developers would not risk their long-term position in the project by backing an inappropriate or widely disliked new feature. In fact, part of what you get, or should get, when you hire such a contractor is advice about what sorts of changes are likely to be accepted by the community. You also get a slight shift in the project's priorities. Because prioritization is just a matter of who has time to work on what, when you pay for someone's time, you cause their work to move up in the priority queue a bit. This is a well-understood fact of life among experienced open source developers, and at least some of them will devote attention to the contractor's work simply because it looks like it's going to get done, so they want to help it get done right. Perhaps they won't write any of the code, but they'll still discuss the design and review the code, both of which can be very useful. For all these reasons, the contractor is best drawn from the ranks of those already involved with the project.
(See also the section called “Hiring Open Source Developers” for the related topic of hiring open source developers as employees.)
If you have a long-term goal of increasing the project's stability and longevity, then the opposite tactic from that described above may be called for: you might want to deliberately hire a person or firm who is new to the project. While it may take them some time to find their way in the code and in the development community, once the contract is done they will now be invested in the project and may continue to participate, and even to develop new business based on the project. the section called “Foster Pools of Expertise in Multiple Places” discusses this startegy in more depth.
Both techniques described above raise a couple of questions: Should contracts ever be secret? And when they're not secret, should you worry about creating tensions in the community by the fact that you've contracted with some developers and not others?
In general, it's best to be open about contracts when you can. Otherwise, the contractor's behavior may seem strange to others in the community — perhaps she's suddenly giving inexplicably high priority to features she's never shown interest in the past. When people ask her why she wants them now, how can she answer convincingly if she can't talk about the fact that she's been contracted to write them?
At the same time, neither you nor the contractor should act as though others should treat your arrangement as a big deal. Sometimes I've seen contractors waltz onto a development list with the attitude that their posts should be taken more seriously simply because they're being paid. That kind of attitude signals to the rest of the project that the contractor regards the fact of the contract — as opposed to the code resulting from the contract — to be the important thing. But from the other developers' point of view, only the code matters. At all times, the focus of attention should be kept on technical issues, not on the details of who is paying whom. For example, one of the developers in the Subversion community handles contracting in a particularly graceful way. While discussing his code changes in chat, he'll mention as an aside (often in a private remark, or privmsg, to one of the other committers) that he's being paid for his work on this particular bug or feature. But he also consistently gives the impression that he'd want to be working on that change anyway, and that he's happy the money is making it possible for him to do that. He may or may not reveal his customer's identity, but in any case he doesn't dwell on the contract. His remarks about it are just an ornament to an otherwise technical discussion about how to get something done.
That example shows another reason why it's good to be open about contracts. There may be multiple organizations sponsoring contracts on a given open source project, and if each knows what the others are trying to do, they may be able to pool their resources. In the above case, the project's largest funder (CollabNet) was not involved with these piecework contracts, but knowing that someone else was sponsoring certain bug fixes allowed CollabNet to redirect its resources to other bugs, resulting in greater efficiency for the project as a whole.
Will other developers resent that some are paid for working on the project? In general, no, particularly when those who are paid are established, well-respected members of the community anyway. No one expects contract work to be distributed equally among all the committers. People understand the importance of long-term relationships: the uncertainties involved in contracting are such that once you find someone you can work reliably with, you would be reluctant to switch to a different person just for the sake of evenhandedness. Think of it this way: the first time you hire, there will be no complaints, because clearly you had to pick someone — it's not your fault you can't hire everyone. Later, when you hire the same person a second time, that's just common sense: you already know her, the last time was successful, so why take unnecessary risks? Thus, it's perfectly natural to have a few go-to people in the community, instead of spreading the work around evenly.
The project's community will always be important to the long-term success of contract work. Their involvement in the design and review process for sizeable changes cannot be an afterthought; It must be considered part of the work, and fully embraced by the contractor. Don't think of community scrutiny as an obstacle to be overcome — think of it as a free design board and QA department. It is a benefit to be aggressively pursued, rather than an obstacle to be overcome.
In 1995, I was one half of a partnership that provided support and enhancements for CVS (the Concurrent Versions System; see http://nongnu.org/cvs). My partner Jim Blandy and I were, informally, the maintainers of CVS by that point. But we'd never thought carefully about how we ought to relate to the existing mostly part-time and volunteer CVS development community. We just assumed that they'd send in patches, and we'd apply them, and that was pretty much how it worked.
Back then, networked CVS could be done only over a remote login
program (in those days rsh rather than ssh).
Using the same account for
CVS access as for system (shell) access was an obvious security risk, and many
organizations were put off by it. A major investment bank hired us to
add a new authentication mechanism, so they could safely use networked
CVS with their remote offices.
Jim and I took the contract and sat down to design the new authentication system. What we came up with was pretty simple (the United States had export controls on cryptographic code at the time, so the customer understood that we couldn't implement strong authentication), but as we were not experienced in designing such protocols, we still made a few gaffes that would have been obvious to an expert. These mistakes would easily have been caught had we taken the time to write up a proposal and run it by the other developers for review. But we never did so, because it didn't occur to us to think of the development list as a resource to be used to improve our contracted work. We knew that people were probably going to accept whatever we committed, and — because we didn't know what we didn't know — we didn't bother to do the work in a visible way, e.g., posting patches frequently, making small, easily digestible commits to a special branch, etc. The resulting authentication protocol was not very good, and of course, once it became established, it was difficult to improve, because of compatibility concerns.
The root of the problem was not lack of experience; we could easily have learned what we needed to know. The problem was our attitude toward the rest of the development community. We regarded acceptance of the changes as a hurdle to get over, rather than as a process by which the quality of the changes could be improved. Since we were confident that what we did would be accepted (as it was), we made little effort to get others involved.
Obviously, when you're choosing a contractor, you want someone with the right technical skills and experience for the job. But it's also important to choose someone with a track record of constructive interaction with the other developers in the community. That way you're getting more than just a single person; you're getting an agent who will be able to draw on a network of expertise to make sure the work is done in a robust and maintainable way.
If you're hiring outside contractors to create software for you, the language you put in your Requests For Information (RFIs), Requests For Proposals (RFPs), and contracts becomes crucially important.
There is one key thing you must understand at the outset: the decision makers at most large-scale software development vendors don't really want their work to be open source. (The programming staff may feel differently, of course, but the path to the executive suite is usually smoother for those with an instinct for monopoly.) Instead, the vendors would prefer that a customer hire them to produce bespoke software that, under the hood, shares many components with the other bespoke software they're producing for other customers.[73] That way the vendor can sell mostly the same product at full price many times. This is especially true of vendors to government agencies, because the needs of government agencies are so similar, and because jurisdictional boundaries create an artificial multiplicity of customers who all have pretty much the same needs. Only minor customizations may be needed for each instance, but the different customers will pay full price each time.
As a customer, then, your starting point for a successful large-scale open source project is to set clear, explicit requirements about open source development from the beginning. From the RFI or RFP stage, all the way through the contract and into delivery and maintenance, you must require behaviors and deliverables that will result in a truly open source product — meaning, among other things, a product that has the potential to be supported and customized by vendors other than the one who originally developed it. The most important of those requirements are:
Design and development must be done in the open from the very start of the project (see the section called “Be Open From Day One”
The code shall be explicitly licensed for open source distribution, from the start of development through delivery and deployment.
If the same vendor is both writing the software and deploying the production instances, require that deployed code must match the open source code. Don't let proprietary tweaks — and thus vendor lock-in — slip in via the back door through deployment customizations.
The product should have no dependencies on proprietary software modules; written permission from you must be obtained before any such dependencies are introduced.
Documentation must be sufficient to allow third parties to understand, configure, and deploy the software. (Some customers even test this by hiring an independent third party to perform a deployment and submit any needed improvements to the installation documentation — via the open source project's usual contribution channels, of course.) documentation must be in formats typically used by open source projects, e.g., plaintext, Markdown, Asciidoc, DocBook, etc.[74]
The vendor's engagement with third parties who become involved in the project should be anticipated and budgeted for. If it is a successful open source project, there will eventually be community management overhead, so plan for it: e.g., specify that the vendor must establish a participation workflow, review and prioritize contributions, etc.
Set clear expectations about the extent to which the vendor will participate in publicity about the project, both among technical developer communities and among potential users.
You, the customer, should be the copyright owner of the code written by the vendor.
For any patents controlled by the vendor and affecting the project, there must be an unambiguous, non-restrictive patent grant not just to you but to everyone who receives the code under its open source license.
If the vendor has little or no experience running or at least participating open source projects, bring in a separate Open Source Quality Assurance (OSQA) vendor to provide assistance and oversight (see the section called “Open Source Quality Assurance (OSQA)”).
Although this is not a complete list — every project is different — it should give you some idea of how to set expectations with your partners. The ability to recognize whether these expectations are being met, in spirit not just in letter, is also important of course, and is the subject of the next section.
When a vendor whose normal mode is proprietary development is hired to do open source, the result is usually a product that is not truly open source and that no third party can actually deploy[75]. This section is about how to avoid that problem. While in some instances the vendor — or at least factions within the vendor — may be actively resistant to open source, more often the problem is that they simply don't know what they don't know. The fastest solution is to bring in that knowledge from the outside: have a separate contract with a different company, one entirely independent of the primary vendor, to play the role of third-party open source participant.
There is a long tradition of such outside review in technical contracting, where it's known as IV&V, for "Independent Verification and Validation".[76] It ensures that the deliverables meet the necessary standards by having an independent party verify this. The independent reviewer reports to the customer, not to the primary development contractor.
My colleague James Vasile came up with the name Open Source Quality Assurance (OSQA) for the corresponding role in open source software development efforts. I like that name much better than "Open Source IV&V" because it emphasizes the interactive and collaborative nature of the independent vendor's role. In an open source project, the deliverables include not just the code, but the development process itself and the resultant potential for third-party participation. Assistance from a separate OSQA vendor can make the difference between a project that is open source in name only and a project that is truly open source, in the sense that it is possible for parties other than its orginal developers to deploy, maintain, and improve it.
During development, an OSQA reviewer participates the way any third party would, posting in the project's public discussion forums, using the installation documentation to try to get the software up and running, reporting bugs via the public tracker, submitting pull requests, and so on. As the project reaches the alpha or beta stage, the reviewer confirms that the software can be deployed as documented, without reliance on proprietary dependencies or vendor-specific environmental conditions; that necessary per-deployment configurations can be made; that sample data can be loaded; that there exist documented paths by which third parties can participate in the project; and so on — in other words, that all the expectations one would have of an open source project are truly met.
But the reviewer's job is not just to review. The reviewer is there to help the primary vendor meet these expectations throughout development, and to report back to the customer as to whether the vendor is doing so. In far too many cases, I have seen a nominally open source project be contracted for and developed, only for the customer to discover at the end — too late to do anything about it — that no party besides than the original vendor can actually deploy, maintain, or extend the software, because the vendor never came close to meeting normal open source standards. Had parallel, independent review been built into the process from the start, the problems would have been detected early and the unsatisfactory outcome prevented. (Relatedly, see the section called “Be Open From Day One”.)
Note that the primary vendor may often be quite unconscious that anything is wrong. In their mind, they developed and delivered software the way they usually do, so what's the problem? The fact that no one other than them can deploy or modify the end result doesn't register as a failure, because in all their other projects third-party deployability was not a goal in the first place. The fact that the contract requires it is meaningless unless the customer has some way to test and enforce that requirement. Since most customers do not have the in-house technical capability to do so, the open source clauses in the contract are effectively void unless there is some kind of external review process.
Independent review is not merely a sort of open source insurance, however, though it would be worthwhile even if it were only that. It is also an investment in the success of future partnerships with the primary vendor. The vendor becomes more inherently capable of performing quality open source work in the future, because the OSQA process provides a practical education in open source development. Thus, done right, third-party review results in both a healthier open source project and a healthier long-term relationship with the primary vendor.
It also helps foster concentrations of expertise outside that primary contractor right from the start, as discussed in the section called “Foster Pools of Expertise in Multiple Places”. Ideally, at the end of development for a new open source product, you should have at least two independent commercial entities able to deploy and support the software: the primary development vendor and the OSQA vendor. That's already twice as much supplier diversity as most projects have coming out of the gate, and it's much easier to add a third vendor than a second.
The key to successful OSQA is that the reviewer is responsible to the customer, not to the primary development vendor. That part is crucial: even if the two vendors are contracting through the same prime vehicle, or one is a subcontractor to the other, it must be clear in the contracts that the reviewer reports directly to the client, interacting with the primary development vendor only to perform the OSQA function.
The cost of OSQA review is much smaller than the cost of the main contract — generally, expect on the order of 5% to 10% — and the benefit is large: the difference between an end product that is not useably open source and one that is truly open source, able to be deployed and supported by anyone.
Corporate lawyers (and to a lesser degree lawyers in the non-profit world and in government) sometimes have an uneasy relationship with free software. They have often spent their careers diligently seeking to maximize the control and exclusivity their clients have over everything the clients produce — including software. A good lawyer will understand why their client is choosing to deliberately give up that control for some larger purpose, when it is explained, but even then may still be unfamiliar with the factors that go into choosing an open source license for the project, the interaction of the license with trademarks and patents, the legal technicalities of how to accept contributed code such that it can be redistributed, etc. (See Chapter 9, Legal Matters: Licenses, Copyrights, Trademarks and Patents for a deeper discussion of legal issues.)
The ideal course is to make sure your lawyers first understand why you are running an open source project, and give them a chance to familiarize themselves with open source in general, before you bring the particulars of the project to them. If the lawyers are good, they will know when they should seek help from outside advisors and will not hesitate to do so. By the time the project is under way, the lawyers should have enough familiarity with open source legal issues to make basic decisions with confidence, and to know when and where they need help.
Do not assume that open source is part of a standard legal education. It is not, at least as of this writing in 2022. If you wait until development is already under way and code is starting to be published before consulting your legal team, they may be forced to scramble and make under-researched decisions hastily. This will not be good for either the project or the organization, in the long run.
[73] By the way, those common components are quite often open source libraries themselves. These days, it's typical for a proprietary software product to contain a lot of open source code, with a layer of proprietary custom code wrapped around the outside.
[74] Microsoft Word format is almost never seen in open source projects, among other reasons because it is not amenable to auditable spot-changes by contributors. You will need to make sure your vendor knows this, or else you are likely to end up with a lot of .docx files in the repository.
[75] While some selection bias no doubt informs my experience — after all, the consultant tends to get brought in when things are going wrong, not when they're going right — my assertion that proprietary vendors don't get open source right if left to their own habits is based not just on my own experiences but also on talking to many other people, who report the same finding with remarkable consistency.
[76] For a more general discussion of IV&V, see https://en.wikipedia.org/wiki/Verification_and_validation and https://en.wikipedia.org/wiki/Software_verification_and_validation. Note that neither of those discusses open source specifically, however.