In free software, there is a fairly smooth continuum between purely internal discussions and public relations statements. This is partly because the target audience is not strictly bounded: because posts are publicly accessible, the project doesn't have full control over the impression the world gets. Someone — say, a https://news.ycombinator.com/ poster or https://slashdot.org/ editor — may draw millions of readers' attention to a post that no one ever expected to be seen outside the project. This is a fact of life that all open source projects live with, but in practice, the risk is usually small. In general, the announcements that the project wants most publicized most are the ones that will be most publicized, assuming you use the right mechanisms to indicate relative newsworthiness to the outside world.
For major announcements, there tend to be a few main channels of distribution, in which announcements should be made as nearly simultaneously as possible:
Your project's front page is probably seen by more people than any other part of the project. If you have a really major announcement, put a blurb there. The blurb should be a very brief synopsis that links to the press release (see below) for more information.
At the same time, you should also have a "News" or "Press Releases" area of the web site, where the announcement can be written up in detail. Part of the purpose of a press release is to provide a single, canonical "announcement object" that other sites can link to, so make sure it is structured accordingly: either as one web page per release, as a discrete blog entry, or as some other kind of entity that can be linked to while still being kept distinct from other press releases in the same area.
Make sure the announcement gets broadcast by any relevant Twitter or other microblog handles, and goes out on any news channels and RSS feeds. (The latter should happen automatically when you publish the announcement, if things are set up properly.)
Post to forums as appropriate, in the manner described in the section called “Announcing”).
Send a mail to your project's announcement mailing
list. This list's name should actually be "announce",
that is, announce@yourprojectdomain.org,
because that's a fairly standard convention now, and the
list's charter should make it clear that it is very
low-traffic, reserved for major project announcements.
Most of those announcements will be about new releases of
the software, but occasionally other events, such as a
fundraising drive, the discovery of a security
vulnerability (see
the section called “Announcing Security Vulnerabilities”), or a
major shift in project direction may be posted there as
well. Because it is low traffic and used only for
important things, the announce list
typically has the highest subscribership of any mailing
list in the project (of course, this means you shouldn't
abuse it — consider carefully before posting). To
avoid random people making announcements, or worse, spam
getting through, the announce list must
always be moderated.
Try to make the announcements in all these places at the same time, as nearly as possible. People might get confused if they see an announcement on the mailing list but then don't see it reflected on the project's home page or in its press releases area. If you get the various changes (emails, web page edits, etc) queued up and then send them all in a row, you can keep the window of inconsistency very small.
For a less important event, you can eliminate some or all of the above outlets. The event will still be noticed by the outside world in direct proportion to its importance. For example, while a new release of the software is a major event, merely setting the date of the next release, while still somewhat newsworthy, is not nearly as important as the release itself. Setting a date is worth an email to the daily mailing lists (not the announce list), and an update of the project's timeline or status web page, but no more.
However, you might still see that date appearing in discussions elsewhere on the Internet, wherever there are people interested in the project. People who are lurkers on your mailing lists, just listening and never saying anything, are not necessarily silent elsewhere. Word of mouth gives very broad distribution; you should count on it, and construct even minor announcements in such a way as to encourage accurate informal transmission. Specifically, posts that you expect to be quoted should have a clearly meant-to-be-quoted portion, just as though you were writing a formal press release. For example:
Just a progress update: we're planning to release version 2.0 of Scanley in mid-August 2022. You can always check http://www.scanley.org/status.html for updates. The major new feature will be regular-expression searches.
Other new features include: ... There will also be various bugfixes, including: ...
The first paragraph is short, gives the two most important pieces of information (the release date and the major new feature), and a URL to visit for further news. If that paragraph is the only thing that crosses someone's screen, you're still doing pretty well. The rest of the mail could be lost without affecting the gist of the content. Of course, sometimes people will link to the entire mail anyway, but just as often, they'll quote only a small part. Given that the latter is a possibility, you might as well make it easy for them, and in the bargain get some influence over what gets quoted.
Handling a security vulnerability is different from handling any other kind of bug report. In free software, doing things openly and transparently is normally almost a religious credo. Every step of the standard bug-handling process is visible to all who care to watch: the arrival of the initial report, the ensuing discussion, and the eventual fix.
Security bugs are different. They can compromise users' data, and possibly users' entire computers. To discuss such a problem openly would be to advertise its existence to the entire world — including to all the parties who might make malicious use of the bug. Even merely committing a fix effectively announces the bug's existence (there are organizations who watch the commit logs of public projects, systematically looking for changes that indicate security problems in the pre-change code; these organizations do not have your users' interests in mind).
Most open source projects have settled on approximately the same set of steps to handle this conflict between openness and secrecy, based on these basic guidelines:
Don't talk about the bug publicly until a fix is available, and then supply the fix publicly at the same instant you announce the bug.
It may make sense to supply the fix by packaging it as a release, or it may be enough to just commit it to the project's public repository. Whichever of those you you do, doing it effectively announces the vulnerability, so your formal announcement should go out in tandem with that fix.
Come up with that fix as fast as you can — especially if someone outside the project reported the bug, because then you know there's at least one person outside the project who is able to exploit the vulnerability.
In practice, those principles lead to a fairly standardized series of steps, which are described in the sections below.
Obviously, a project needs the ability to receive security bug reports from anyone. But the regular bug reporting channels won't do, because they can be watched by anyone too. Therefore, have a separate mailing list or contact form for receiving security bug reports. That forum must not have publicly readable archives, and its subscribership must be strictly controlled — only long-time, trusted developers can be on the list, and people whom such developers have consensus that they trust.[99] (If you need a formal definition of "trusted developer", you can use "anyone who has had commit access for two years or more" or something like that, to avoid favoritism.) This is the group that will handle security bugs.
Ideally, that reporting gateway should not be spam-protected or moderated, since you don't want an urgent report to get filtered out or delayed just because no moderators happened to be online that weekend. If you do use automated spam-protection software, try to configure it with high-tolerance settings; it's better to let a few spams through than to miss a vulnerability report.
The submission mechanism should itself be secure. That is, if
it is a contact form, it should be on an https://
(TLS-protected) page, or if it is an email address, there should be a
well-advertised public key (digitally signed by as many of the core
developers as possible) so people can send encrypted mails to that
address.[100] A web form submission
or an email sent to your project may travel over many Internet hops on
its way there; you have no reason to trust whoever runs those
intermediate servers, and there is a flourishing market for new
security vulnerabilities. Assume the worst and design
accordingly.
So what does the security list do when it receives a report? The first task is to evaluate the problem's severity and urgency:
How serious is the vulnerability? Does it allow a malicious attacker to take over the computer of someone who uses your software? Or does it, say, merely leak information about the sizes of some of their files?
How easy is it to exploit the vulnerability? Can an attack be scripted, or does it require circumstantial knowledge, educated guessing, and luck?
Who reported the problem to
you? The answer to this question doesn't change the
nature of the vulnerability, of course, but it does give
you an idea of how many other people might know about it.
If the report comes from one of the project's own
developers, you can breathe a little easier (but only a
little), because you can trust them not to have told
anyone else about it. On the other hand, if it came in an
email from anonymous14@globalhackerz.net,
then you'd better act as fast as you can. The person did
you a favor by informing you of the problem at all, but you
have no idea how many other people she's told, or how long
she'll wait before exploiting the vulnerability on live
installations.
Note that the difference we're talking about here is often just a narrow range between urgent and extremely urgent. Even when the report comes from a known, friendly source, there could be other people on the Net who discovered the bug long ago and just haven't reported it. The only time things aren't urgent is when the bug inherently does not compromise security very severely.
The "anonymous14@globalhackerz.net" example
is not facetious, by the way (though that particular email address is).
You really may get bug reports from
identity-cloaked people who, by their words and behavior, never quite
clarify whether they're on your side or not. It doesn't matter: if
they've reported the security hole to you, they'll feel they've done
you a good turn, and you should respond in kind. Thank them for the
report, give them a date on or before which you plan to release a
public fix, and keep them in the loop. Sometimes they may
give you a date — that is, an implicit threat
to publicize the bug on a certain date, whether you're ready or not.
This may feel like a bullying power play, but it's more likely a
preëmptive action resulting from past disappointment with
unresponsive software producers who didn't take security reports
seriously enough. Either way, you can't afford to tick this person
off. After all, if the bug is severe, she has knowledge that could
cause your users big problems. Treat such reporters well, and hope
that they treat you well.
Another frequent reporter of security bugs is the security professional, someone who audits code for a living and keeps up on the latest news of software vulnerabilities. These people usually have experience on both sides of the fence — they've both received and sent reports, probably more than most developers in your project have. They too will usually give a deadline for fixing a vulnerability before going public. The deadline may be somewhat negotiable, but that's up to the reporter; deadlines have become recognized among security professionals as pretty much the only reliable way to get organizations to address security problems promptly. So don't treat the deadline as rude; it's a time-honored tradition, and there are good reasons for it. Negotiate if you absolutely must, but remember that the reporter holds all the cards.
Once you know the severity and urgency, you can start working on a fix. There is sometimes a tradeoff between doing a fix elegantly and doing it speedily; this is why you must agree on the urgency before you start. Keep discussion of the fix restricted to the security list members, of course, plus the original reporter (if she wants to be involved) and any developers who need to be brought in for technical reasons.
Do not commit the fix to any public repository before the go-public date. If you were to commit it publicly, even with an innocent-looking log message, someone might notice and understand the change. You never know who is watching your repository and why they might be interested. Turning off commit emails wouldn't help; first of all, the gap in the commit mail sequence would itself look suspicious, and anyway, the data would still be in the repository. Just do all development in some private place known only to the people already aware of the bug.
You may have seen a CVE number associated with a particular security problems — e.g., a number like "CVE-2014-0160", where the first numeric part is the year, and the second is an increasing sequence number (it may exceed four digits if more than 10,000 numbers are handed out in a given year).
A CVE number is an entry in the "Common Vulnerabilities and Exposures" list maintained at https://www.cve.org/.[101] The purpose of the list is to a provide standardized name for every known computer security problem, so that everyone has a unique, canonical name to use when discussing it, and a central place to go to find out more information.[102]
A CVE entry does not itself contain a full description of the bug and how to protect against it. Instead, it contains a brief summary, and a list of references to external resources (such as a announcement post from the project in question) where people can go to get more detailed information. The real purpose of https://www.cve.org/ is to provide a well-organized space in which every vulnerability has a single name, and people have a clear route to get more data about it. See https://www.cve.org/CVERecord?id=CVE-2014-0160 for an example of an entry.
If your vulnerability meets the criteria, you may wish to obtain a CVE number for it. You can request one using the instructions at https://www.cve.org/ResourcesSupport/ReportRequest, but if there is someone in your project who has already obtained CVE numbers, or who knows someone who has, let them do it. The CVE Program gets a lot of submissions, many of them spurious or poorly formed; by submitting through an experienced and trusted source, you will save everyone time and possibly get your CVE number assigned more quickly. The other advantage of doing it this way is that somewhere along the chain, someone may know enough to tell you that a) it wouldn't count as a vulnerability or exposure according to the official criteria, so there is no point submitting it, or b) the vulnerability already has a CVE number. The latter can happen if the bug has already been published on another security advisory list (and if that happened without your project hearing about it, then you should worry what else might be going on that you don't know about.)
If you get a CVE number at all, you usually want to get it in the early stages of your bug investigation, so that all further communications can refer to that number. The entry won't become public right away — see https://www.cve.org/About/Process for how and when public disclosure happens — but the number will be reserved, and that in itself makes coordination and management of the vulnerability easier.
Describing the severity of a vulnerability accurately is actually a difficult task. Does the vulnerability require physical access to the computer, or is network access enough? Does it require an authenticated user or not? Is it technically difficult to exploit, or can any bored teenager with some coding skills run it? Does it affect data integrity? Does it cause the software to crash?
Therefore, don't try to improvise language for expressing severity. Instead, use the Common Vulnerability Scoring System (CVSS) developed by the National Vulnerability Database at the U.S. National Institute of Standards: https://nvd.nist.gov/vuln-metrics/cvss
NVD has thought very carefully about how to accurately and completely characterize severity for digital vulnerabilities, and their standardized expression format has become a standard in computer security. You can see an example in the "Severity:" section of the sample pre-notification email in the section called “Pre-Notification” below.
Once your security response team (that is, those developers who are on the security mailing list, or who have been brought in to deal with a particular report) has a fix ready, you need to decide how to distribute it.
If you simply commit the fix to your repository, or otherwise announce it to the world, you effectively force everyone using your software to upgrade immediately or risk being hacked. It is sometimes appropriate, therefore, to do pre-notification for certain important users.
Pre-notification is somewhat controversial, because it privileges some users over others. I personally think there are some circumstances where it is the right choice, particularly when there are well-known online services that use the software and that are tempting targets for attackers (perhaps because those services hold a lot of commercial or personal data about users). Those service's administrators would appreciate having an extra day or two to do the upgrade, so that they are already protected by the time the exploit becomes public knowledge — and their users, if they knew about this at all, would appreciate it too.
Pre-notification simply means contacting those administrators privately before the go-public date, telling them of the vulnerability and how to fix it. You should send pre-notification only to people you trust to be discreet with the information, and with whom you can communicate securely. That is, the qualification for receiving pre-notification is threefold: the recipient must run a large, important service where a compromise would be a serious matter; the recipient must be known to be someone who won't blab about the security problem before the go-public date; and you must have a way to communicate securely with the recipient, so that any eavesdroppers between you and your recipient can't read the message.[103]
Pre-notification should be done via secure means. If email, then encrypt it, for the same reasons explained in the section called “Receive the Report”, but if you have a phone number or other out-of-band secure way to contact the administrator, use that. When sending encrypted pre-notification emails, send them individually (one at a time) to each recipient. Do not send to the entire list of recipients at once, because then they would see each others' names — meaning that you would essentially be alerting each recipient to the fact that each other recipient may have a security hole in her service. Sending it to them all via blind CC (BCC) isn't a good solution either, because some admins protect their inboxes with spam filters that either block or reduce the priority of BCC'd mail, since so much spam is sent via BCC.
Here's a sample pre-notification mail:
From: Your Name Here To: admin@large-famous-server.com Reply-to: Your Address Here (not the security list's address) Subject: Confidential notification regarding a security vulnerability. [[[ BEGIN ENCRYPTED AND DIGITALLY-SIGNED MAIL ]]] This email is a confidential pre-notification of a security alert in the Scanley server software. Please *do not forward* any part of this mail to anyone. The public announcement is not until May 19th, and we'd like to keep the information embargoed until then. You are receiving this mail because (we think) you run a Scanley server, and would want to have it patched before this security hole is made public on May 19th. References: =========== CVE-2022-892346: Scanley stack overflow in queries Vulnerability: ============== The server can be made to run arbitrary commands if the server's locale is misconfigured and the client sends a malformed query. Severity: ========= CVSSv2 Base Score: 9.0 CVSSv2 Base Vector: AV:N/AC:L/Au:N/C:C/I:C/A:C (See https://nvd.nist.gov/CVSS/Vector-v2.aspx for how to interpret these expressions.) Workarounds: ============ Setting the 'natural-language-processing' option to 'off' in scanley.conf closes this vulnerability. Patch: ====== The patch below applies to Scanley 3.0, 3.1, and 3.2. A new public release (Scanley 3.2.1) will be made on or just before May 19th, so that it is available at the same time as this vulnerability is made public. You can patch now, or just wait for the public release. The only difference between 3.2 and 3.2.1 will be this patch. [...patch goes here...]
If you have a CVE number, include it in the pre-notification (as shown above), even though the information is still embargoed and therefore the corresponding MITRE page will show nothing at the time of pre-notification. Including the CVE number allows the recipient to know with certainty that the bug they were pre-notified about is the same one they later hear about through public channels, so they don't have to worry whether further action is necessary or not, which is precisely the point of CVE numbers.
The last step in handling a security bug is to distribute the fix publicly. In a single, comprehensive announcement, you should describe the problem, give the CVE number if any, describe how to work around it, and how to permanently fix it. Usually "fix" means upgrading to a new version of the software, though sometimes it can mean applying a patch, particularly if the software is normally run in source form anyway. If you do make a new release, it should differ from some existing release by exactly the security patch. That way, conservative admins can upgrade without worrying about what else they might be affecting; they also don't have to worry about future upgrades, because the security fix will be in all future releases as a matter of course. (Details of release procedures are discussed in the section called “Security Releases”.)
Whether or not the public fix involves a new release, do the
announcement with roughly the same priority as you would a new
release: send a mail to the project's announce
list, make a new press release, etc. While you should never try to
play down the existence of a security bug out of concern for the
project's reputation, you may certainly set the tone and prominence of
a security announcement to match the actual severity of the problem.
If the security hole is just a minor information exposure, not an
exploit that allows the user's entire computer to be taken over, then
it may not warrant a lot of fuss. See https://www.cve.org/ResourcesSupport/Glossary for a good
introduction to how to think about and discuss vulnerabilities.
In general, if you're unsure how to treat a security problem, find someone with experience and talk to them about it. Assessing and handling vulnerabilities is very much an acquired skill, and it's easy to make missteps the first few times.
https://www.cve.org/ is the official source of information about the CVE process.
https://www.debian.org/security/cve-compatibility is a particularly clear exposition of one open source project's use of CVE numbers.
The post "A minimal security response process" at https://access.redhat.com/blogs/766093/posts/1975833 is a good writeup from a security engineer at RedHat.
The Apache Software Foundation's guidelines on handling security vulnerabilities at https://www.apache.org/security/committers.html are are an excellent checklist that you can compare against to see if you're doing everything carefully.
[99] E.g., a release manager who maybe isn't a core developer but who is already trusted to roll releases anyway. I've seen cases where companies who had been long involved in a project had managers as members of the project's security group, even though those managers had never committed a line of code, because by common consent the project's maintainers trusted them and felt it was to the project's benefit for them to see vulnerability reports as soon as possible. There is no one rule that will be appropriate for all projects, but in general, the core maintainers should follow the principle that anyone who receives security reports must be trustable both in terms of intention and in terms of their technical ability to not accidentally leak information (e.g., someone whose email gets hacked regularly should probably not be on the security list).
[100] If you don't know what all of these terms mean, find people you trust who do and get them to help your project. Handling security vulnerablities competently requires a working knowledge of these concepts.
[101] Formerly at https://cve.mitre.org/, so if you're accustomed to the older URL, note that the site is transitioning to www.cve.org. The transition started in late September 2021 and is planned to be completed within one year.
[102] In the past, a CVE number would start out as a CAN number ("CAN" for "candidate") until it was approved for inclusion in the official list, at which point the "CAN" would be replaced with "CVE" while the number portion remained the same. However, nowadays they are just assigned a "CVE-" prefix from the start, although that prefix does not guarantee that the vulnerability will be included in the official list. (For example, it might be later discovered to be a duplicate of an existing CVE, in which case the earlier one — the lower number — should be used.)
[103] Remember that Subject lines in emails aren't encrypted, so don't put too much information about the vulnerability in a Subject line.