Data provided by National Institute of Standards and Technology
The open dataset, software, and other files accompanying the manuscript "An Open Combinatorial Diffraction Dataset Including Consensus Human and Machine Learning Labels with Quantified Uncertainty for Training New Machine Learning Models," submitted for publication to Integrated Materials and Manufacturing Innovations.Machine learning and autonomy are increasingly prevalent in materials science, but existing models are often trained or tuned using idealized data as absolute ground truths. In actual materials science, "ground truth" is often a matter of interpretation and is more readily determined by consensus. Here we present the data, software, and other files for a study using as-obtained diffraction data as a test case for evaluating the performance of machine learning models in the presence of differing expert opinions. We demonstrate that experts with similar backgrounds can disagree greatly even for something as intuitive as using diffraction to identify the start and end of a phase transformation. We then use a logarithmic likelihood method to evaluate the performance of machine learning models in relation to the consensus expert labels and their variance. We further illustrate this method's efficacy in ranking a number of state-of-the-art phase mapping algorithms. We propose a materials data challenge centered around the problem of evaluating models based on consensus with uncertainty. The data, labels, and code used in this study are all available online at data.gov, and the interested reader is encouraged to replicate and improve the existing models or to propose alternative methods for evaluating algorithmic performance.
About this Dataset
Updated: 2025-04-06
Metadata Last Updated:
2020-09-22 00:00:00
Date Created:
N/A
Data Provided by:
Dataset Owner:
N/A
| Title | Dataset: An Open Combinatorial Diffraction Dataset Including Consensus Human and Machine Learning Labels with Quantified Uncertainty for Training New Machine Learning Models |
|---|---|
| Description | The open dataset, software, and other files accompanying the manuscript "An Open Combinatorial Diffraction Dataset Including Consensus Human and Machine Learning Labels with Quantified Uncertainty for Training New Machine Learning Models," submitted for publication to Integrated Materials and Manufacturing Innovations.Machine learning and autonomy are increasingly prevalent in materials science, but existing models are often trained or tuned using idealized data as absolute ground truths. In actual materials science, "ground truth" is often a matter of interpretation and is more readily determined by consensus. Here we present the data, software, and other files for a study using as-obtained diffraction data as a test case for evaluating the performance of machine learning models in the presence of differing expert opinions. We demonstrate that experts with similar backgrounds can disagree greatly even for something as intuitive as using diffraction to identify the start and end of a phase transformation. We then use a logarithmic likelihood method to evaluate the performance of machine learning models in relation to the consensus expert labels and their variance. We further illustrate this method's efficacy in ranking a number of state-of-the-art phase mapping algorithms. We propose a materials data challenge centered around the problem of evaluating models based on consensus with uncertainty. The data, labels, and code used in this study are all available online at data.gov, and the interested reader is encouraged to replicate and improve the existing models or to propose alternative methods for evaluating algorithmic performance. |
| Modified | 2020-09-22 00:00:00 |
| Publisher Name | National Institute of Standards and Technology |
| Contact | mailto:[email protected] |
| Keywords | machine learning models , X-ray diffraction , human labeling , combinatorial methods , V-Nb-O thin films , quantified uncertainyy , open data challenge |
{
"identifier": "ark:\/88434\/mds2-2301",
"accessLevel": "public",
"contactPoint": {
"hasEmail": "mailto:[email protected]",
"fn": "Brian DeCost"
},
"programCode": [
"006:045"
],
"landingPage": "https:\/\/data.nist.gov\/od\/id\/mds2-2301",
"title": "Dataset: An Open Combinatorial Diffraction Dataset Including Consensus Human and Machine Learning Labels with Quantified Uncertainty for Training New Machine Learning Models",
"description": "The open dataset, software, and other files accompanying the manuscript \"An Open Combinatorial Diffraction Dataset Including Consensus Human and Machine Learning Labels with Quantified Uncertainty for Training New Machine Learning Models,\" submitted for publication to Integrated Materials and Manufacturing Innovations.Machine learning and autonomy are increasingly prevalent in materials science, but existing models are often trained or tuned using idealized data as absolute ground truths. In actual materials science, \"ground truth\" is often a matter of interpretation and is more readily determined by consensus. Here we present the data, software, and other files for a study using as-obtained diffraction data as a test case for evaluating the performance of machine learning models in the presence of differing expert opinions. We demonstrate that experts with similar backgrounds can disagree greatly even for something as intuitive as using diffraction to identify the start and end of a phase transformation. We then use a logarithmic likelihood method to evaluate the performance of machine learning models in relation to the consensus expert labels and their variance. We further illustrate this method's efficacy in ranking a number of state-of-the-art phase mapping algorithms. We propose a materials data challenge centered around the problem of evaluating models based on consensus with uncertainty. The data, labels, and code used in this study are all available online at data.gov, and the interested reader is encouraged to replicate and improve the existing models or to propose alternative methods for evaluating algorithmic performance.",
"language": [
"en"
],
"distribution": [
{
"downloadURL": "https:\/\/data.nist.gov\/od\/ds\/mds2-2301\/VO2%20-Nb2O3%20XRD%20Combiview.txt",
"mediaType": "text\/plain"
},
{
"downloadURL": "https:\/\/data.nist.gov\/od\/ds\/mds2-2301\/VO2%20-Nb2O3%20XRD%20Combiview.txt.sha256",
"mediaType": "text\/plain"
},
{
"downloadURL": "https:\/\/data.nist.gov\/od\/ds\/mds2-2301\/VO2%20-%20Nb2O3%20Composition%20and%20temp%20Combiview.txt",
"mediaType": "text\/plain"
},
{
"downloadURL": "https:\/\/data.nist.gov\/od\/ds\/mds2-2301\/VO2%20-%20Nb2O3%20Composition%20and%20temp%20Combiview.txt.sha256",
"mediaType": "text\/plain"
},
{
"downloadURL": "https:\/\/data.nist.gov\/od\/ds\/mds2-2301\/Human%20Labels.xlsx",
"mediaType": "application\/vnd.openxmlformats-officedocument.spreadsheetml.sheet"
},
{
"downloadURL": "https:\/\/data.nist.gov\/od\/ds\/mds2-2301\/Human%20Labels.xlsx.sha256",
"mediaType": "text\/plain"
},
{
"downloadURL": "https:\/\/data.nist.gov\/od\/ds\/mds2-2301\/Compare%20ML%20Labels.csv",
"mediaType": "application\/vnd.ms-excel"
},
{
"downloadURL": "https:\/\/data.nist.gov\/od\/ds\/mds2-2301\/Compare%20ML%20Labels.csv.sha256",
"mediaType": "text\/plain"
},
{
"downloadURL": "https:\/\/data.nist.gov\/od\/ds\/mds2-2301\/cluster_assignment_loglik_all.csv",
"mediaType": "application\/vnd.ms-excel"
},
{
"downloadURL": "https:\/\/data.nist.gov\/od\/ds\/mds2-2301\/cluster_assignment_loglik_all.csv.sha256",
"mediaType": "text\/plain"
},
{
"downloadURL": "https:\/\/data.nist.gov\/od\/ds\/mds2-2301\/Open%20Data%20Challenge%20Notebook%20Human%20Labels%20and%20Plots.py",
"mediaType": "application\/octet-stream"
},
{
"downloadURL": "https:\/\/data.nist.gov\/od\/ds\/mds2-2301\/Open%20Data%20Challenge%20Notebook%20Human%20Labels%20and%20Plots.py.sha256",
"mediaType": "text\/plain"
},
{
"downloadURL": "https:\/\/data.nist.gov\/od\/ds\/mds2-2301\/Open%20Data%20Challenge%20Notebook%20Machine%20Labels%20and%20Plots.py",
"mediaType": "application\/octet-stream"
},
{
"downloadURL": "https:\/\/data.nist.gov\/od\/ds\/mds2-2301\/Open%20Data%20Challenge%20Notebook%20Machine%20Labels%20and%20Plots.py.sha256",
"mediaType": "text\/plain"
},
{
"downloadURL": "https:\/\/data.nist.gov\/od\/ds\/mds2-2301\/Condensing%20Write-Ups%20of%20Human%20and%20Machine%20Labeling%20Metrics.docx",
"mediaType": "application\/vnd.openxmlformats-officedocument.wordprocessingml.document"
},
{
"downloadURL": "https:\/\/data.nist.gov\/od\/ds\/mds2-2301\/Condensing%20Write-Ups%20of%20Human%20and%20Machine%20Labeling%20Metrics.docx.sha256",
"mediaType": "text\/plain"
},
{
"downloadURL": "https:\/\/data.nist.gov\/od\/ds\/mds2-2301\/Readme.txt",
"mediaType": "text\/plain"
},
{
"downloadURL": "https:\/\/data.nist.gov\/od\/ds\/mds2-2301\/Readme.txt.sha256",
"mediaType": "text\/plain"
},
{
"accessURL": "https:\/\/doi.org\/10.18434\/mds2-2301",
"title": "DOI Access for Dataset: An Open Combinatorial Diffraction Dataset Including Consensus Human and Machine Learning Labels with Quantified Uncertainty for Training New Machine Learning Models"
}
],
"bureauCode": [
"006:55"
],
"modified": "2020-09-22 00:00:00",
"publisher": {
"@type": "org:Organization",
"name": "National Institute of Standards and Technology"
},
"theme": [
"Information Technology:Data and informatics",
"Materials:Modeling and computational material science",
"Materials:Materials characterization"
],
"keyword": [
"machine learning models",
"X-ray diffraction",
"human labeling",
"combinatorial methods",
"V-Nb-O thin films",
"quantified uncertainyy",
"open data challenge"
]
}
