Test with FAIR assessment tools
After performing FAIRification activities, the FAIR status of the modified dataset (or in general digital object) should be assessed to confirm, that all measures result in the intended improvements. For this, several evaluation tools have been developed in the FAIR community. The compliance of data resources with the FAIR principles can be evaluated using checklists, questionnaires and (semi-)automated evaluators. The tools sometimes also offer support in how to improve the current status. Some of the tools offer a scoring to quantify the FAIR status. Note however, that there is no agreement in the community on an absolute scoring based on scientific grounds. Currently, the community is striving for a harmonization of the assessment tools. For an application of the tools in the low-carbon energy research domain, see Schwanitz et al. 2022 [1]. The supplementary material illustrates typical results from a manual and an automated assessment.
Contents
(Semi-)automatic tools
Manual assessment tools
The following tools have been used during the EERAdata project to conduct a manual assessment of research data
| Method/Tool | Comment |
|---|---|
| ARDC: ANDS-NECTAR-RDS-FAIR data assessment tool | Online assessment tool, developed by the Australian Research Data Commons (ARDC). Designed for manual assessment by the user, following all FAIR data principles of Wilkinson et al. 12 technical and non-technical questions2. Users choose the answers from a drop-down menu. Contains short explanations of FAIR principles and terms. Results from the test are indicated with a progress bar as a new answer is given. Scores are not made explicit. However, one can infer them from the source code. The authors note that the test only serves as an orientation. |
| SATIFYD: the DANS Self-Assessment Tool | Online assessment tool, developed by Data Archiving and Networked Services (DANS) and designed for manual assessment by the user. Does not follow the FAIR data principles, only touching upon them. Consists of different questions, e.g., regarding the use of standards to describe data (controlled vocabularies, taxonomies, ontologies). Consists of 12 non-technical questions, thus, easier to understand and answer. Contains short explanations of assessment principles, questions and terms. Indicates rating and scoring by percentages. |
| Fair enough? [ checklist] | Online checklist developed by Data Archiving and Networked Services (DANS). Designed for manual assessment of FAIRness of data(sets) and the trustworthiness of the selected repository. Does not follow the FAIR data principles, only touching upon them. Consists of 11 non-technical questions, thus, easier to understand and answer. Contains explanations of terms/questions for corresponding paragraphs. Does not provide a rating for FAIR and scoring of overall FAIRness. |
| OzNome: The CSIRO 5-star Data Rating tool | Online assessment tool, developed by Commonwealth Scientific and Industrial Research Organization, Australia. Designed for manual assessment of FAIRness of data(sets). Follows the FAIR data principles. Additionally includes questions on the openness of data (e.g., groups of users with full or limited access to the actual data). Consists of 14 technical and non-technical questions. Contains short explanations of terms/questions. Displays a chart using a 5-star data rating. |
| Stewardship Maturity Mix “Scientific Data Stewardship Maturity Assessment Model Template” (template) | Online self assessment template developed by North Carolina Institute for Climate Studies (CICS-NC), the National Centers for Environmental Information (NCEI), and domain experts. Designed for measuring data stewardship practices and leveraging community best practices and standards. Does not follow the FAIR data principles. Key components are: preservability, accessibility, usability, production sustainability, data quality assurance, data quality control & monitoring, data quality assessment, transparency & traceability, and data integrity. Technical and non-technical checks of the components. Contains short explanations of terms/questions. Indicates a self assessment of the maturity level using a 5-level grading scheme from 1 (Ad hoc, not managed) to 5 (optimal/well managed). Does not include scoring relating to FAIR criteria. |
| Data Stewardship Wizard - questionnaire | Online questionnaire to create smart Data Management Plans (DMP) for FAIR Open Science tool, developed by Data Stewardship Wizard (DSW) in cooperation with six research organizations. Designed for DMP development for projects. Does not follow all FAIR data principles. Consists of 189 technical and non-technical questions, pertaining to experiment design, data design and planning, capture/ measurement, processing and curation, integration, interpretation, information and insights Does not contain short explanations of terms/questions. Percentage scoring relating to FAIR criteria, “Good DMP Practice” and “Openness”. |
| Checklist for Evaluation of Dataset Fitness for Use | Online checklist developed in cooperation with ICSU World Data System and Research Data Alliance. Designed for manual estimation of FAIRness of data(sets) and data curation. Dataset to be evaluated should be stewarded within a CoreTrustSeal-certified repository. Does not follow strictly FAIR data principles, only touching upon them. Consists of 20 non-technical questions, but it is not easy to follow the documentations and checklist. Some criteria are marked with asterisks denoting that assessing will require domain/discipline specific knowledge. Does not contain short explanations of terms/questions. Does not indicate ratings or scoring. |
| RDA-SHARC Evaluation, presentation | Poster presentation created by Research Data Alliance (RDA) and SHAring Rewards & Credits (SHARC). Designed for manual assessment by the user as a decision tree in each FAIR Principle. Follows the FAIR data principles and Open Science Career Assessment Matrix designed by EC Working Group on Reward under Open Science. Details on questions in the poster are limited. Unclear if the tool is ready for use. Details are limited. Details are limited. |
| WMO-Wide Stewardship Maturity Matrix for Climate Data, tool | Online assessment tool developed by WMO Stewardship Maturity Matrix for Climate Data (SMM-CD) Working Group. Designed for manual assessment by the user. Follows “internationally-validated data stewardship best practices”. Consists of 11 questions. Contains explanations/notes below each question. Indicates rating by maturity level 1 to 5 for the different questions, no overall score. |
| Data Use and Services Maturity Matrix, tool | Online assessment tool developed by MM-Serv Working Group adopting the approach of the NCEI (National Centers for Environmental Information)/CICS-NC Data Stewardship Maturity Matrix (DSMM). Designed for the ESIP-DSC (Data Stewardship Committee)-wide review of the MM-Serv to ensure and improve its quality. Follows MM-Serv Key Components and Maturity Level Criteria. Consists of 9 Key Components, each being associated with maturity levels 1 to 5. Contains explanations of terms for corresponding maturity levels. Indicates rating based on maturity levels, no overall score. |
Aggregated summary for manual vs. machine assessments
| Name of database | Manual assessment | Machine assessment |
|---|---|---|
| Satellite Application Facility on Climate Monitoring (CMSAF) | F: 82%, A: 80%, I: 63%, R: 29% Total FAIR: 64%; O: 67% | F: 74%, A: 33%, I: 88%, R: 0% Total FAIR: 48% |
| CommONEnergy Data Mapper | F: 41%, A: 70%, I: 38%, R: 43% Total FAIR: 48%; O: 67% | F: 50%, A: 33%, I: 13%, R: 0% Total FAIR: 24% |
| Copernicus Space Component Data Access system (CSCDA) | F: 65%, A: 90%, I: 88%, R: 86% Total FAIR: 82%; O: 100% | F: 0%, A: 0%, I: 0%, R: 0% Total FAIR: 0% |
| Covenant of Mayors | F:47%, A: 70%, I: 50%, R: 29% Total FAIR: 49%; O: 67% | F: 50%, A: 33%, I: 88%, R: 0% Total FAIR: 43% |
| DataCite | F: 59%, A: 90%, I: 75%, R: 57%, Total FAIR: 70%; O: 33% | F: 50%, A: 33%, I: 13%, R: 0%
Total FAIR: 24% |
| EMODnet - THE EUROPEAN MARINE OBSERVATION AND DATA NETWORK | F: 53%, A: 40%, I: 50%, R: 71%, Total FAIR: 54%; O: 67% | F: 24%, A:33%, I: 0%, R: 0%, Total FAIR: 14% |
| Policies to Enforce the Transition to nearly zero energy buildings in EU-27 (Project: Entranze) | F: 47%, A: 70%, I: 38%, R: 43% Total FAIR: 49.5%; O: 67% | F: 24%, A:33%, I: 0%, R: 0%, Total FAIR: 14% |
| ENTSO-E transparency platform | F: 41%, A: 70%, I: 0%, R: 0% Total FAIR: 28%; O: 67% | F: 50%, A:33%, I: 25%, R: 0%, Total FAIR: 27% |
| EUR-Lex (Document: Directive 95/46/EC) | F: 47%, A: 80%, I: 63%, R: 57% total FAIR: 62%; O: 67% | F: 50%, A:33%, I: 25%, R: 0%, Total FAIR: 27% |
| Global Buildings Performance Network (GBPN) | F: 70%, A: 70%, I: 50%, R: 14% Total FAIR: 51%; O: 67% | F: 50%, A:33%, I: 25%, R: 0%, Total FAIR: 27% |
| The open source mapping and planning tool for cooling and heating (project: Hotmaps) | F: 47%, A: 70%, I: 50%, R: 86% Total FAIR: 63%; O: 67% | F: 24%, A:33%, I: 0%, R: 0%, Total FAIR: 14% |
| IEA Policy database | F: 47%, A: 50%, I: 38%, R: 57% Total FAIR: 48%; O: 100% | F:50%, A: 33%, I: 13%, R:0%, Total FAIR: 24 % |
| IEA World Energy Statistics and Balances | F: 47%, A:50%, I: 38%, R: 57%, Total FAIR: 48%; O: 0% | F:74%, A: 33%, I: 88%, R:0%, Total FAIR: 48 % |
| JRC data catalogue | F: 53%, A: 70%, I: 63%, R: 57% Total FAIR: 61%; O: 100% | F:62%, A: 78%, I: 88%, R:0%, Total FAIR: 57 % |
| JRC data catalogue - data collection | F: 47%, A:100%, I: 38%, R: 43%, Total FAIR: 57%, O: 100% | F:50%, A: 33%, I: 13%, R:0%, Total FAIR: 24 % |
| JRC data catalogue - data set | F: 47%, A:80%, I: 25%, R: 43%, Total FAIR: 49%, O:67% | F:62%, A: 78%, I: 88%, R:0%, Total FAIR: 57% |
| Mesures d'Utilisation Rationnelle de l'Energie (Mure database) | F: 47%, A: 70%, I: 63%, R: 29% Total FAIR: 52%, O: 67% | F:50%, A: 33%, I: 13%, R:0%, Total FAIR: 24% |
| Kaggle Repository (data set) | F:41%, A: 80%, I: 25%, R: 43% Total FAIR: 47%; O: 67% | F: 74%, A: 78%, I: 88%, R: 0% Total FAIR: 60% |
| Simulations of hourly power output from wind and solar PV farms (Renewables.Ninja) | F: 76%, A: 70%, I: 75%, R: 71% Total FAIR: 73%; O: 67% | F: 24%, A: 33%, I: 0%, R: 0% Total FAIR: 14% |
| U.S. National Renewable Energy Laboratory (NREL) data catalog | F: 100%, A: 90%, I: 50%, R: 29%
Total FAIR: 67%; O: 100% || F: 62%, A: 33%, I: 88%, R: 0% Total FAIR: 45% | |
| Scenarios of market transition to nearly Zero Energy Buildings (nZEB) | F: 53%, A: 60%, I: 38%, R: 0% Total FAIR: 38%; O: 67% | F: 24%, A: 33%, I: 0%, R: 0% Total FAIR: 14% |
| OECD hub (energy data) | F: 70%, A: 90%, I: 75%, R: 100% Total FAIR: 84%; O: 100% | F: 50%, A: 33%, I: 13%, R:0%, Total FAIR: 24% |
| OECD hub (data set) | F: 77%, A: 70%, I: 38%, R: 29% Total FAIR: 53%; O: 67% | F: 50%, A: 33%, I: 13%, R:0%; Total FAIR: 24% |
| Heat Roadmap Europe (PETA) | F: 59%, A: 60%, I: 38%, R: 0% Total FAIR: 39%; O: 67% | F: 50%, A: 33%, I: 50%, R:0%, Total FAIR: 33% |
| Database for support schemes, grid issues and policies about RE sources in EU-28, EFTA etc. (RES LEGAL) | F: 47%, A: 80%, I: 38%, R: 29% Total FAIR: 49%; O: 100% | F: 24%, A: 33%, I: 0%, R: 0% Total FAIR: 14% |
| EU Building Stock Monitoring (Project: EPISCOPE) | F: 47%, A: 70%, I: 25%, R: 43% Total FAIR: 46%; O: 67% | F: 24%, A: 33%, I: 0%, R: 0% Total FAIR: 14% |
| Zenodo Repository (Data set ‘Greco’) | F: 94%, A: 90%, I:88%, R: 100%, Total FAIR: 93%; O: 100% | F:62%, A: 78%, I: 88%, R:100%, Total FAIR: 82% |
| Zenodo Repository (Data set ‘Sharewind’) | F: 100%, A: 80%, I: 63%, R:57% Total FAIR: 75%; O: 100% | F: 38%, A: 33%, I: 88%, R:100%; Total FAIR: 65% |
Further reading on FAIR Evaluation
Articles in scientific journals
- Comment: A design framework and exemplar metrics for FAIRness
- Evaluating FAIR maturity through a scalable, automated, community- governed framework
- FAIR Principles: Interpretations and Implementation Considerations
Reports and grey literature
- FAIR Assessment Tools: Towards an “Apples to Apples” Comparisons
- Testing the FAIR metrics on data catalogs
Opinion pieces
Webinars
- EOSC Nordic - FAIRification study testing FAIRsFAIR F-UJI tool
- EOSC Nordic - FAIRification STEP 5 webinar: Value and limitations of FAIR assessment tools
References
- ↑ V.J. Schwanitz et al. Current state and call for action to accomplish findability, accessibility, interoperability, and reusability of low carbon energy data. Scientific Reports 12:5208 (2022)
