By Aaron Tay, Head, Data Services
Last month, we introduced our webinar series on research impact beyond traditional metrics. This series included
The third part of our series was conducted by Sean Rife of Scite.ai on citation sentiment
The first part of Sean's talk was about scite.ai (SMU has an institutional subscription) and how it was developed.
The second part of the talk focused on scite's new metric. First, he introduced the unweighted Scite Index (USI).
As you can see above, USI is a simple ratio of % of supporting citations out of sum of supporting and contrasting citations.
Technical note : FT50 Research Insights just counts the number of positive citations (supporting cites) from a curated list and does not use USI
While this makes sense, Sean shows that when you rank institutions this way, you get lesser-known institutions ranking on top. (See current list of institutions ranked by USI - SMU is not on the list because our publications did not receive enough supporting or contrasting cites)
Part of the issue is some works are more straight forward to replicate and if you just rank with USI you end up with narrow technical universities ranked on top.
That said, Sean argues USI has some nice properties, it can be applied to any type of entity not just institution but also journal, department but scite is advising caution when applying it to authors.
In particular, when USI is applied to journal, it is mostly not-correlated with Journal Impact Factor, which can be seen as a plus because it captures another dimension of quality not captured by JIF.
USI is also clearly field dependent, as you can see below, USI is generally higher for fields like Math and Computer Science that is easier to replicate vs Social Science and Humanities.
So clearly just using USI alone may not be ideal. As such, currently the Scite Index (SI), incorporates both the signal from USI and raw citations by
- Multiplying raw references by USI squared
- Do a log of the product.
Technical note here, the metrics from USI are from the scite index which is limited by access to citation statements available (from full text) while the raw references are more generally available e.g. from Crossref
Sean shows that the SI ranking of institutions gives more expected results.
In terms of limitations, SI has the usual limitations of citation-based metrics. For example, the metric might be gameable and scite is considering filtering out self cites, and they are experimenting with changing the exponential (there is nothing special about using 2 or square) to weight more or less the signal from USI
The SI specific limitations relate to
- lack of full text access for scite to detect supporting & contrasting citations
- the accuracy of the classification of supporting & contrasting citations
- The concept and expectations of supporting or contrasting cites differ between fields
- The fact that most citations are neutral (or "mentioning") so the signal from USI might be low (range of USI for institutions isn't huge)
Conclusion
USI and SI are cutting edge metrics that incorporate citation sentiment on a large scale but are clearly still experimental. Recording of talk is available here.