Hard-Bench

We are delighted to introduce Hard-Bench, a benchmark that carefully curates training data from original datasets, which provides a greater challenge than simply random sampling from typical datasets. Our benchmark draws on data from a variety of areas, including NLP and CV, such as the eight tasks from the GLUE benchmark, CIFAR-10, CIFAR-100, and the ImageNet collection.

In Hard-Bench (GradNorm), we create a k-shot dataset by selecting the k data points with the highest gradient norms for each label.
Specifically, for each data point, we compute the gradient vector of the predictor being trained and calculate its Euclidean norm. We then sort the data points by their gradient norms within each label and select the top k data points to form the k-shot dataset. This method allows us to construct a challenging dataset that provides a rigorous evaluation of model performance.

In Hard-Bench (Loss), we construct a new k-shot dataset by selecting the k data points of each label with the highest loss value.
Specifically, for each data point in the original dataset, we calculate the loss value of the predictor being trained. We then sort the data points by their loss values within each label and select the top k data points to form the k-shot dataset. This approach enables us to construct a challenging dataset that provides a more comprehensive evaluation of model performance.