Welcome to Perceptron Robustness Benchmark’s page!¶
Perceptron is a benchmark to test safety and security properties of neural networks for perceptual tasks.
It comes with support for many frameworks to build models including
- Cloud API
See currently supported evaluation metrics, models, adversarial criteria, and verification methods in Summary.
See current Leaderboard.
perceptron benchmark improves upon the existing adversarial
toolbox such as
advbox in three important aspects:
- Consistent API design that enables easy evaluation of models across different deep learning frameworks, computer vision tasks, and adversarial criterions.
- Standardized metric design that enables DNN models’ robustness to be compared on a large collection of security and safety properties.
- Gives verifiable robustness bounds for security and safety properties.
You can run evaluation against DNN models with chosen parameters using
python perceptron/launcher.py \ --framework keras \ --model resnet50 \ --criteria misclassification\ --metric carlini_wagner_l2 \ --image example.png
In above command line, the user lets the framework as
keras, the model as
the criterion as
misclassification (i.e., we want to generate an adversary which is
similar to the original image but has different predicted label), the metric as
carlini_wagner_l2, the input image as
You can try different combinations of frameworks, models, criteria, and metrics. To see more options using -h for help message.
python perceptron/launcher.py -h
We also provide a coding example which serves the same purpose as above command line. Please refer to Examples for more details.