We have released our testbed
, which comes with all our code and data, to allow researchers to use our framework to develop and evaluate new models and distribution shifts. To this end, we have made it very simple to add new models with a few lines of code (resnet50 example):
from torchvision.models import resnet50
from registry import registry
from models.model_base import Model, StandardTransform, StandardNormalization
name = 'resnet50',
transform = StandardTransform(img_resize_size=256, img_crop_size=224),
normalization = StandardNormalization(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]),
classifier_loader = lambda: resnet50(pretrained=True),
eval_batch_size = 256,
It is similarly easy to add new datasets; check the repository for more information.
Running evaluations is also simple (either with existing models/datasets or ones you've added):
python eval.py --gpus 0 1 --models resnet50 densenet121 --eval-settings val imagenetv2-matched-frequency
Lastly, all our plotting code to visualize results is available. We hope you find these tools useful!