Dioptra Test Platform

Source code, documentation, and examples of use of the source code for the Dioptra Test Platform.Dioptra is a software test platform for assessing the trustworthy characteristics of artificial intelligence (AI). Trustworthy AI is: valid and reliable, safe, secure and resilient, accountable and transparent, explainable and interpretable, privacy-enhanced, and fair - with harmful bias managed1. Dioptra supports the Measure function of the NIST AI Risk Management Framework by providing functionality to assess, analyze, and track identified AI risks.Dioptra provides a REST API, which can be controlled via an intuitive web interface, a Python client, or any REST client library of the user's choice for designing, managing, executing, and tracking experiments. Details are available in the project documentation available at https://pages.nist.gov/dioptra/.Use CasesWe envision the following primary use cases for Dioptra:- Model Testing: -- 1st party - Assess AI models throughout the development lifecycle -- 2nd party - Assess AI models during acquisition or in an evaluation lab environment -- 3rd party - Assess AI models during auditing or compliance activities- Research: Aid trustworthy AI researchers in tracking experiments- Evaluations and Challenges: Provide a common platform and resources for participants- Red-Teaming: Expose models and resources to a red team in a controlled environmentKey PropertiesDioptra strives for the following key properties:- Reproducible: Dioptra automatically creates snapshots of resources so experiments can be reproduced and validated- Traceable: The full history of experiments and their inputs are tracked- Extensible: Support for expanding functionality and importing existing Python packages via a plugin system- Interoperable: A type system promotes interoperability between plugins- Modular: New experiments can be composed from modular components in a simple yaml file- Secure: Dioptra provides user authentication with access controls coming soon- Interactive: Users can interact with Dioptra via an intuitive web interface- Shareable and Reusable: Dioptra can be deployed in a multi-tenant environment so users can share and reuse components

                                            [
  "006:55"
]
                                        

                                            {
  "fn": "Harold Booth III",
  "hasEmail": "mailto:harold.booth@nist.gov"
}
                                        

                                            [
  {
    "title": "Dioptra Github Repository",
    "format": "Github Repository",
    "accessURL": "https://github.com/usnistgov/dioptra",
    "description": "The USNIST Github location where source code is located."
  }
]
                                        

                                            [
  "AI; Trustworthy AI; Test; Evaluation; Adversarial Machine Learning; Machine Learning; TEVV"
]
                                        

                                            [
  "en"
]
                                        

                                            [
  "006:045"
]
                                        

                                            {
  "name": "National Institute of Standards and Technology",
  "@type": "org:Organization"
}
                                        

                                            [
  "Information Technology:Cybersecurity"
]