For reasons that I don't understand [*], assessing patents is a hot topic this week. Two new companies are out with assessment tools: Patent Site and Patent Board.
Assessing patent quality is much like assessing the quality of research articles - any logical approach is fraught with loopholes and errors. The most common basis for the analysis is to look at the citations of other patents, as these citations establish links to other relevant work. The more times a patent is cited, the better it must be (which implicitly assumes that all patent examiners cite all the relevant patents and no irrelevant ones). Certainly this is true for breakthrough patents, but for other patents that represent smaller increments of improvement, that would not be the case. And with patents, far more than with research reports, there is an economic aspect that outweighs all other aspects. Patents for blockbuster drugs are not cited nearly as much as their economic impact on the company would suggest.
An additional challenge that is becoming more important all the time is that patents can be assigned to others (exclusively or not). Patent Site claims to address this, but I don't see how that can be done given that most of these agreements are not public.
Lastly, I'm not surprised at all that these two companies come up with different results. Patent Site lists the top ten chemical companies as
while Patent Boards list is
5) Nitto Denko
10) Air Products
DuPont, Dow and BASF are near the top of both lists, but the other 7 candidates aren't even the same! Obviously, there field still needs additional work.
[*] I'd would guess that companies probably will use this as a PR tool to push the value of their portfolios, and since this analysis is so much more sophisticated than just counting up patents, it can used that much more so.