By Aaron Tay, Lead, Data Services
The UN Sustainability Goals or UN SDGs was first adopted in 2015 and consists of 17 goals adopted by all United Nations Member States to achieve by 2030.
Recently there have been a lot of activities around mapping scientific literature to each UN SDG goal.
Academic discovery services and tools like Digital Science’s Dimensions (Free for personal use) and Elsevier’s Scival (An add-on service to Scopus that we subscribed to) have started adding filters and metrics around such SDG mappings.
On the policy documents front, providers like OECD (See their award winning pathfinder site) and Overton have also mapped their policy documents to UN SDG classifications.
More interestingly, rankings are starting to emerge that use some of this metrics. For instance, while the THE (Times Higher Education) World University Rankings is widely known, lesser known is the THE Impact rankings. It claims to be
“the only global performance tables that assess universities against the United Nations’ Sustainable Development Goals (SDGs). We use carefully calibrated indicators to provide comprehensive and balanced comparisons across three broad areas: research, outreach and stewardship."
The rankings are already in in its second edition in 2020 and includes 768 universities from 85 countries, however it is still not as comprehensive as the more well known THE World University rankings which includes 1,500 universities. Entry into such rankings is voluntary and currently no Singapore institution is included.
For each UN SDG goal, universities are assessed on a variety of metrics to measure impact. For example, for UN SDG 5 Gender equality, metrics measuring gender equality like % of female academics are used.
However, for the research components for most UN SDGs, this is generally done by mapping publications to each of the UN SDG areas and then measuring both volume and impact (proportion of papers in the top 10 per cent of cited journals) with data provided by Elsevier’s Scopus.
How is mapping literature papers to UN SDGs done?
The most popular way of mapping literature to UN SDGs is by searching with structured advanced Boolean search strategies (similar to those done in systematic reviews) to identify papers.
For example, Scopus has put their 2019 methodology and search strategies freely available here.
Below shows the search string used in Scopus for UN SDG 1 – Reduce poverty.
Not to be outdone, Clarivate which owns Web of Science – the main rival to Scopus in the citation metrics space, have also produced their own report on mapping publications to UN SDG using the same methods I.e. Boolean searches.
The other major approach is to use supervised machine learning (by using keywords to generate initial high quality datasets for training) to assign papers and this is the approach taken by Dimensions.
Supplementing the mapping with machine learning seems to be an approach many are trying. Scopus for example are trying machine learning as well in 2021 to classify papers to UN SDGs.
There is some concern such approaches are less transparent of course.
Are such mapping methods consistent?
While it is easy to construct mapping techniques, it is tricky to assess if they are working well. Are we really measuring something “real”? Are the methods to identify such papers robust?
Unfortunately, it seems that translating research contributions to SDG into search queries is far from straightforward.
For example, there can be disagreement on what technologies are really helpful. Take contributions to the UN SDG on providing affordable clean and affordable energy, should papers on nuclear energy count? Also, even if one should agree on those, should only research on applications count or can theory papers count as well?
Empirical research is emerging that the various existing studies on mapping publications to UN SDGs are showing very inconsistent results.
For example, in “Mapping scholarly publications related to the Sustainable Development Goals: Do independent bibliometric approaches get the same results?”, the authors show there is very little overlap in the papers identified by different methods even when you took into account the differences in source used (Scopus vs Web of Science).
This paper which incidentally placed second in the International Society for Scientometrics and Informetrics (ISSI) Paper of the Year contest, warned that
“The results suggest that it would be premature to trust commercial SDG analyses for anything other than exploratory purposes at this stage in their development”.
Obviously, such issues need to be resolved, but trying to measure impact on UN SDGs seems to be a worthwhile endeavor and there is research currently underway to resolve these issues such as the STRINGS (Steering Research and Innovation for Global Goals) project which is a “consortium of seven leading universities, research centers and the UNDP ... working together to better understand the ways in which science, technology and innovation contribute, or not, to meeting the United Nations Sustainable Development Goals (SDGs) in Low and Middle Income Countries" (STRINGS, n.d., para. 2).
References
Armitage, C. S., Lorenz, M., & Mikki, S. (2020). Mapping scholarly publications related to the Sustainable Development Goals: Do independent bibliometric approaches get the same results? Quantitative Science Studies, 1(3), 1092–1108. https://doi.org/10.1162/qss_a_00071
Times Higher Education. (2020, March 4).Impact Rankings 2020—Times Higher Education (THE). Times Higher Education (THE). https://www.timeshighereducation.com/rankings/impact/2020/overall
STRINGS. (n.d.). STRINGS Science technology research and innovations for the global goals. Retrieved July 26, 2021, from http://strings.org.uk/