Although this book is mainly about how to launch and run new open source projects, that topic is inextricably linked to the problem of evaluating existing open source projects. You can't know whether you need to start a new one until you've evaluated what's out there (as explained in the section called “But First, Look Around”). Furthermore, even in a new project, you'll usually still be building on existing open source components, and will often be in the position of choosing between different projects that implement the same basic functionality. That is not just a technical choice; it's also about social health and general level of project maturity. How large and diverse are their developer communities? Do they get new contributors on a regular basis? Do they handle incoming bug reports in a reasonable way? Do they make stable releases frequently enough for your needs? And so on.
Evaluating open source projects is certainly an art, not a science. However, there are some shortcuts that experienced people use. Below is what has worked for me, and by "worked", I mean that when I have applied these evaluation techniques to a project and then checked in with that project months or years later, I have generally found that its current circumstances are in line with what the evaluation predicted.
The most reliable signals of project health can usually be found in the bug tracker. Look at the rate of issue filings and the number of unique filers (because that's a proxy for the size and level of engagement of the user base), and look at how often project developers respond in bug tickets — and at how they respond: are they constructive? Do they interact well with both the reporter and with other developers? Is it always the same developer responding, or is responsiveness well-distributed throughout the development team? More bug reports is better, by the way, as discussed in the section called “Version Control and Bug Tracker Access”. The rate at which bug reports are closed is not as important as you might think; in a healthy project with an active user base, bug reports will probably be filed faster than the development team can close them. The important question is not rate of resolution, but how the project responds to and organizes the influx of reports.
Look at the distribution of commits across committers, not just at the raw frequency of commits. Does the project have a variety of people working together in a sustained way? Too often, evaluators look just at the commit rate, but the rate isn't very informative — knowing the number of commits per week doesn't tell you anything except that (perhaps) someone keeps making typos and keeps correcting them in new commits. If have time to look at the content of individual commits, then look at how often one developer's commit is a response to (refers to) some other developer's previous commit. This tells you that group code review is going on, and the more of that you see, the better the project is doing.
In addition to looking for a variety of individual identities, see if you can tell how many different organizations are participating in the project — in particular, commercial organizations. If a number of different sources of money are all investing in a project, that's a sign that that project is going to be around for the long term. (See the discussion of "bus factor" in Chapter 4, Social and Political Infrastructure.)
If the project has discussion forums, scan them quickly looking for signs of a functional community. Specifically, whenever you see a long thread, spot check responses from core developers coming late in the thread. Are they summarizing constructively, and taking steps to bring the thread to a decision while remaining polite? If you see a lot of flame wars going on, that's often a sign that energy is going into argument instead of into development.
Any project that is functioning well will usually have made announcements within the past few months. Check the project's front page, news feed, Twitter or other microblog accounts, etc. If things are quiet on stage, they're probably quiet backstage too.
This is only a quick introduction to the art of evaluating projects, but even using just the steps above can save you a lot of trouble. I have found them particularly useful when evaluating the two sides of a recent fork. Even in a recent fork, it is often possible to tell, just by looking at some of the signals described above, which side will flourish over the long term.