In free software, there is a fairly smooth continuum between purely internal discussions and public relations statements. This is partly because the target audience is always ill-defined: given that most or all posts are publicly accessible, the project doesn't have full control over the impression the world gets. Someone—say, a Hacker News poster or slashdot.org editor—may draw millions of readers' attention to a post that no one ever expected to be seen outside the project. This is a fact of life that all open source projects live with, but in practice, the risk is usually small. In general, the announcements that the project most wants publicized are the ones that will be most publicized, assuming you use the right mechanisms to indicate relative newsworthiness to the outside world.
For major announcements, there tend to be a few main channels of distribution, on which announcements should be made as nearly simultaneously as possible:
Your project's front page is probably seen by more people than any other part of the project. If you have a really major announcement, put a blurb there. The blurb should be a very brief synopsis that links to the press release (see below) for more information.
At the same time, you should also have a "News" or "Press Releases" area of the web site, where the announcement can be written up in detail. Part of the purpose of a press release is to provide a single, canonical "announcement object" that other sites can link to, so make sure it is structured accordingly: either as one web page per release, as a discrete blog entry, or as some other kind of entity that can be linked to while still being kept distinct from other press releases in the same area.
Make sure the announcement gets broadcast by any relevant Twitter handles, and goes out on any news channels or RSS feeds. (The latter may happen automatically when you publish announcement, depending on how things are set up at your site.)
Send a mail to your project's announcement mailing
list. This list's name should actually be "announce",
because that's a fairly standard convention now, and the
list's charter should make it clear that it is very
low-traffic, reserved for major project announcements.
Most of those announcements will be about new releases of
the software, but occasionally other events, such as a
fundraising drive, the discovery of a security
the section called “Announcing Security Vulnerabilities”)
later in this chapter, or a
major shift in project direction may be posted there as
well. Because it is low traffic and used only for
important things, the
typically has the highest subscribership of any mailing
list in the project (of course, this means you shouldn't
abuse it—consider carefully before posting). To
avoid random people making announcements, or worse, spam
getting through, the
announce list must
always be moderated.
Try to make the announcements in all these places at the same time, as nearly as possible. People might get confused if they see an announcement on the mailing list but then don't see it reflected on the project's home page or in its press releases area. If you get the various changes (emails, web page edits, etc.) queued up and then send them all in a row, you can keep the window of inconsistency very small.
For a less important event, you can eliminate some or all of the above outlets. The event will still be noticed by the outside world in direct proportion to its importance. For example, while a new release of the software is a major event, merely setting the date of the next release, while still somewhat newsworthy, is not nearly as important as the release itself. Setting a date is worth an email to the daily mailing lists (not the announce list), and an update of the project's timeline or status web page, but no more.
However, you might still see that date appearing in discussions elsewhere on the Internet, wherever there are people interested in the project. People who are lurkers on your mailing lists, just listening and never saying anything, are not necessarily silent elsewhere. Word of mouth gives very broad distribution; you should count on it, and construct even minor announcements in such a way as to encourage accurate informal transmission. Specifically, posts that you expect to be quoted should have a clearly meant-to-be-quoted portion, just as though you were writing a formal press release. For example:
Just a progress update: we're planning to release version 2.0 of Scanley in mid-August 2005. You can always check http://www.scanley.org/status.html for updates. The major new feature will be regular-expression searches.
Other new features include:†... There will also be various bugfixes, including:†...
The first paragraph is short, gives the two most important pieces of information (the release date and the major new feature), and a URL to visit for further news. If that paragraph is the only thing that crosses someone's screen, you're still doing pretty well. The rest of the mail could be lost without affecting the gist of the content. Of course, sometimes people will link to the entire mail anyway, but just as often, they'll quote only a small part. Given that the latter is a possibility, you might as well make it easy for them, and in the bargain get some influence over what gets quoted.
Handling a security vulnerability is different from handling any other kind of bug report. In free software, doing things openly and transparently is normally almost a religious credo. Every step of the standard bug-handling process is visible to all who care to watch: the arrival of the initial report, the ensuing discussion, and the eventual fix.
Security bugs are different. They can compromise users' data, and possibly users' entire computers. To discuss such a problem openly would be to advertise its existence to the entire world—including to all the parties who might make malicious use of the bug. Even merely committing a fix effectively announces the bug's existence (there are potential attackers who watch the commit logs of public projects, systematically looking for changes that indicate security problems in the pre-change code). Most open source projects have settled on approximately the same set of steps to handle this conflict between openness and secrecy, based on the these basic guidelines:
Don't talk about the bug publicly until a fix is available; then supply the fix (packaged as a release) at exactly the same moment you announce the bug.
Come up with that fix as fast as you can—especially if someone outside the project reported the bug, because then you know there's at least one person outside the project who is able to exploit the vulnerability.
In practice, those principles lead to a fairly standardized series of steps, which are described in the sections below.
Obviously, a project needs the ability to receive security bug reports from anyone. But the regular bug reporting channels won't do, because they can be watched by anyone too. Therefore, have a separate mailing list or contact form for receiving security bug reports. That forum must not have publicly readable archives, and its subscribership must be strictly controlled—only long-time, trusted developers can be on the list, and people whom such developers have consensus that they trust. If you need a formal definition of "trusted developer", you can use "anyone who has had commit access for two years or more" or something like that, to avoid favoritism. This is the group that will handle security bugs.
Ideally, that reporting gateway should not be spam-protected or moderated, since you don't want an important report to get filtered out or delayed just because no moderators happened to be online that weekend. If you do use automated spam-protection software, try to configure it with high-tolerance settings; it's better to let a few spams through than to miss a vulnerability report.
The submission mechanism should itself be secure. That is, if
it is a contact form, it should be on an
(TLS-protected) page, or if it is an email address, there should be a
well-advertised public key (digitally signed by as many of the core
developers as possible) so people can send encrypted mails to that
address. A web form submission
or an email sent to your project may travel over many Internet hops on
its way there; you have no reason to trust whoever runs those
intermediate servers, and there is a flourishing market for new
security vulnerabilities. Assume the worst and design
So what does the security list do when it receives a report? The first task is to evaluate the problem's severity and urgency:
How serious is the vulnerability? Does it allow a malicious attacker to take over the computer of someone who uses your software? Or does it, say, merely leak information about the sizes of some of their files?
How easy is it to exploit the vulnerability? Can an attack be scripted, or does it require circumstantial knowledge, educated guessing, and luck?
Who reported the problem to
you? The answer to this question doesn't change the
nature of the vulnerability, of course, but it does give
you an idea of how many other people might know about it.
If the report comes from one of the project's own
developers, you can breathe a little easier (but only a
little), because you can trust them not to have told
anyone else about it. On the other hand, if it came in an
then you'd better act as fast as you can. The person did
you a favor by informing you of the problem at all, but you
have no idea how many other people she's told, or how long
she'll wait before exploiting the vulnerability on live
Note that the difference we're talking about here is often just a narrow range between urgent and extremely†urgent. Even when the report comes from a known, friendly source, there could be other people on the Net who discovered the bug long ago and just haven't reported it. The only time things aren't urgent is when the bug inherently does not compromise security very severely.
is not facetious, by the way. You really may get bug reports from
identity-cloaked people who, by their words and behavior, never quite
clarify whether they're on your side or not. It doesn't matter: if
they've reported the security hole to you, they'll feel they've done
you a good turn, and you should respond in kind. Thank them for the
report, give them a date on or before which you plan to release a
public fix, and keep them in the loop. Sometimes they may
give you a date—that is, an implicit threat
to publicize the bug on a certain date, whether you're ready or not.
This may feel like a bullying power play, but it's more likely a
preŽmptive action resulting from past disappointment with
unresponsive software producers who didn't take security reports
seriously enough. Either way, you can't afford to tick this person
off. After all, if the bug is severe, she has knowledge that could
cause your users big problems. Treat such reporters well, and hope
that they treat you well.
Another frequent reporter of security bugs is the security professional, someone who audits code for a living and keeps up on the latest news of software vulnerabilities. These people usually have experience on both sides of the fence—they've both received and sent reports, probably more than most developers in your project have. They too will usually give a deadline for fixing a vulnerability before going public. The deadline may be somewhat negotiable, but that's up to the reporter; deadlines have become recognized among security professionals as pretty much the only reliable way to get organizations to address security problems promptly. So don't treat the deadline as rude; it's a time-honored tradition, and there are good reasons for it. Negotiate if you absolutely must, but remember that the reporter holds all the cards.
Once you know the severity and urgency, you can start working on a fix. There is sometimes a tradeoff between doing a fix elegantly and doing it speedily; this is why you must agree on the urgency before you start. Keep discussion of the fix restricted to the security list members, of course, plus the original reporter (if she wants to be involved) and any developers who need to be brought in for technical reasons.
Do not commit the fix to any public repository before the go-public date. If you were to commit it publicly, even with an innocent-looking log message, someone might notice and understand the change. You never know who is watching your repository and why they might be interested. Turning off commit emails wouldn't help; first of all, the gap in the commit mail sequence would itself look suspicious, and anyway, the data would still be in the repository. Just do all development in some private place known only to the people already aware of the bug.
You may have seen a CVE number associated with a particular security problems†—†e.g., a number like "CVE-2014-0160", where the first numeric part is the year, and the second is an increasing sequence number (it may exceed four digits if more than 10,000 numbers are handed out in a given year).
A CVE number is an entry in the "Common Vulnerabilities and Exposures" list maintained at cve.mitre.org. The purpose of the list is to a provide standardized name for every known computer security problem, so that everyone has a unique, canonical name to use when discussing it, and a central place to go to find out more information.
A CVE entry does not itself contain a full description of the bug and how to protect against it. Instead, it contains a brief summary, and a list of references to external resources (such as a announcement post from the project in question) where people can go to get more detailed information. The real purpose of cve.mitre.org is to provide a well-organized space in which every vulnerability has a single name, and people have a clear route to get more data about it. See cve.mitre.org/cgi-bin/cvename.cgi?name=2014-0160 for an example of an entry. Note that the references can be very terse, with sources appearing as cryptic abbreviations. A key to those abbreviations is at cve.mitre.org/data/refs/refkey.html.
If your vulnerability meets the criteria, you may wish to obtain a CVE number for it. You can request one using the instructions at cve.mitre.org/cve/request_id.html, but if there is someone in your project who has already obtained CVE numbers, or who knows someone who has, let them do it. The CVE Editorial Board gets a lot of submissions, many of them spurious or poorly written submissions; by approaching them through a trusted source, you are saving them time and possibly getting your CVE number assigned more quickly. The other advantage of doing it this way is that somewhere along the chain, someone may know enough to tell you that a) it wouldn't count as a vulnerability or exposure according to MITRE's criteria, so there is no point submitting it, or b) the vulnerability already has a CVE number. The latter can happen if the bug has already been published on another security advisory list, for example at www.cert.org or on the BugTraq mailing list at www.securityfocus.com. (If that happened without your project hearing about it, then you should worry what else might be going on that you don't know about.)
If you get a CVE number at all, you usually want to get it in the early stages of your bug investigation, so that all further communications can refer to that number. CVE entries are embargoed until the go-public date: the number will be reserved, but MITRE will allow some time (within reason) before revealing information about the vulnerability†—†so make sure you or your intermediary communicate clearly with the CVE Editorial Board about how long you need before publicly announcing.
See cve.mitre.org for more information about the CVE process. See also debian.org/security/cve-compatibility for a particularly clear exposition of one open source project's use of CVE numbers, and see securityblog.redhat.com/2013/01/30/a-minimal-security-response-process for a good writeup of a minimal security response process, from a security engineer at RedHat.
Once your security response team (that is, those developers who are on the security mailing list, or who have been brought in to deal with a particular report) has a fix ready, you need to decide how to distribute it.
If you simply commit the fix to your repository, or otherwise announce it to the world, you effectively force everyone using your software to upgrade immediately or risk being hacked. It is sometimes appropriate, therefore, to do pre-notification for certain important users. This is particularly true with client/server software, where there may be well-known services using your software and who are tempting targets for attackers. Those service's administrators would appreciate having an extra day or two to do the upgrade, so that they are already protected by the time the exploit becomes public knowledge.
Pre-notification simply means sending mails to those administrators before the go-public date, telling them of the vulnerability and how to fix it. You should send pre-notification only to people you trust to be discreet with the information, and with whom you can communicate securely. That is, the qualification for receiving pre-notification is threefold: the recipient must run a large, important service where a compromise would be a serious matter; the recipient must be known to be someone who won't blab about the security problem before the go-public date; and you must have a way (such as via GnuPG-encrypted email) to communicate securely with the recipient, so that any eavesdroppers between you and your recipient can't read the message.
Pre-notification should be done via secure means. If email, then encrypt it, for the same reasons explained in the section called “Receive the report”, but if you have a phone number or other out-of-band secure way to contact the administrator, use that. When sending encrypted pre-notification emails, send them individually (one at a time) to each recipient. Do not send to the entire list of recipients at once, because then they would see each others' names—meaning that you would essentially be alerting each recipient to the fact that each other recipient may have a security hole in her service. Sending it to them all via blind CC (BCC) isn't a good solution either, because some admins protect their inboxes with spam filters that either block or reduce the priority of BCC'd mail, since so much spam is sent via BCC.
Here's a sample pre-notification mail:
From: Your Name Here To: email@example.com Reply-to: Your Name Here (not the security list's address) Subject: Confidential important notification. [[[ BEGIN ENCRYPTED MAIL ]]] This email is a confidential pre-notification of a security alert in the Scanley server software. Please *do not forward* any part of this mail to anyone. The public announcement is not until May 19th, and we'd like to keep the information embargoed until then. You are receiving this mail because (we think) you run a Scanley server, and would want to have it patched before this security hole is made public on May 19th. References: =========== CVE-2015-892346: Scanley stack overflow in queries Vulnerability: ============== The server can be made to run arbitrary commands if the server's locale is misconfigured and the client sends a malformed query. Severity: ========= Very severe; can involve arbitrary code execution on the server. Workarounds: ============ Setting the 'natural-language-processing' option to 'off' in scanley.conf closes this vulnerability. Patch: ====== The patch below applies to Scanley 3.0, 3.1, and 3.2. A new public release (Scanley 3.2.1) will be made on or just before May 19th, so that it is available at the same time as this vulnerability is made public. You can patch now, or just wait for the public release. The only difference between 3.2 and 3.2.1 will be this patch. [...patch goes here...]
If you have a CVE number, include it in the pre-notification (as shown above), even though the information is still embargoed and therefore the corresponding MITRE page will show nothing at the time of pre-notification. Including the CVE number allows the recipient to know with certainty that the bug they were pre-notified about is the same one they later hear about through public channels, so they don't have to worry whether further action is necessary or not, which is precisely the point of CVE numbers.
The last step in handling a security bug is to distribute the fix publicly. In a single, comprehensive announcement, you should describe the problem, give the CVE number if any, describe how to work around it, and how to permanently fix it. Usually "fix" means upgrading to a new version of the software, though sometimes it can mean applying a patch, particularly if the software is normally run in source form anyway. If you do make a new release, it should differ from some existing release by exactly the security patch. That way, conservative admins can upgrade without worrying about what else they might be affecting; they also don't have to worry about future upgrades, because the security fix will be in all future releases as a matter of course. (Details of release procedures are discussed in the section called “Security Releases” in Chapter†7, Packaging, Releasing, and Daily Development.)
Whether or not the public fix involves a new release, do the
announcement with roughly the same priority as you would a new
release: send a mail to the project's
list, make a new press release, etc. While you should never try to
play down the existence of a security bug out of concern for the
project's reputation, you may certainly set the tone and prominence of
a security announcement to match the actual severity of the problem.
If the security hole is just a minor information exposure, not an
exploit that allows the user's entire computer to be taken over, then
it may not warrant a lot of fuss. You may even decide not to distract
announce list with it. After all, if the
project cries wolf every time, users might end up thinking the
software is less secure than it actually is, and also might not
believe you when you have a really big problem to announce. See
cve.mitre.org/about/terminology.html for a good introduction
to the problem of judging and describing severity.
In general, if you're unsure how to treat a security problem, find someone with experience and talk to them about it. Assessing and handling vulnerabilities is very much an acquired skill, and it's easy to make missteps the first few times.
See also the Apache Software Foundation guidelines on handling security vulnerabilities at apache.org/security/committers.html. They are an excellent checklist you can compare against to see if you're doing everything carefully.
 E.g., a release manager who maybe isn't a core developer but who is already trusted to roll releases anyway. I've seen cases where companies who had been long involved in a project had managers as members of its security group, even though those managers had never committed a line of code, because by common consent the project's maintainers trusted them and felt it was to the project's benefit for them to see vulnerability reports as soon as possible. There is no one rule that will be appropriate for all projects, but in general, the core maintainers should follow the principle that anyone who receives security reports must be trustable both in terms of intention and in terms of their technical ability to not accidentally leak information (e.g., someone whose email gets hacked regularly should probably not be on the security list).
 If you don't know what all of these terms mean, find people you trust who do and get them to help your project. Handling security vulnerablities competently requiries a working knowledge of these concepts.
 In the past, a CVE number would start out as a CAN number ("CAN" for "candidate") until it was approved for inclusion in the official list, at which point the "CAN" would be replaced with "CVE" while the number portion remained the same. However, nowadays they are just assigned a "CVE-" prefix from the start, although that prefix does not guarantee that the vulnerability will be included in the official list. (For example, it might be later discovered to be a duplicate of an existing CVE, in which case the earlier one†—†the lower number†—†should be used.)
 Remember that Subject lines in emails aren't encrypted, so don't put too much information about the vulnerability in the Subject line.