Automating Submission Grading

How to use the Auto Grade feature of Dockship.

What is Auto Grade?

Turning this feature on will automatically grade the solutions submitted by the participants of your challenge. Auto grading currently works for classification, multiclass classification, object detection, and time series prediction (RMSE) problems Here’s how it works:

1. Break your dataset into train and test (80:20 ratio is recommended).

2. Upload a CSV file [follow the guidelines below] in the adjacent "Upload Answer Key" section containing the correct labels for only the test files/images in the dataset.

3. Only supply labels for your training dataset when you upload the dataset in a zip file.

4. Ask the participants to train their model on the training dataset and evaluate it on the test dataset (that you have created).

5. Participants will submit their solutions in the form of CSV which will be graded automatically.

Please follow the guidelines for creating your answers CSV:

A. Classification

This type of challenge takes as input a CSV consisting of two columns “filename” and “label” in the same order, changing the order of columns will lead to wrong results. The evaluation is done as a key-value pair, hence duplicate filenames are discarded. The label class may be either string for example “cat” or numeric based labels such as 0-9 for 10 classes.

Sample structure of the CSV.

B. Multiclass Classification

This type of challenge takes input as a CSV consisting of two columns “filename” and “labels” in the same order. Changing the order of columns will lead to wrong results. The label is a string for example “car” for single class or “car, automobile” for multiple classes (comma separated).

Sample structure of the CSV.

C. Object Detection

This involves matching of the detected bounding boxes and the corresponding class of images. Each line of the answer key consists of these attributes “filename”, “width”, “height”, “class”, “xmin”, “ymin”, “xmax”, “ymax”

If there are multiple objects within in an image, they can be provided in different rows, hence the filename may be repeated. The submission file also includes the same attributes with an additional “conf” (confidence) attribute which implies the confidence of the prediction.

The evaluation criteria is mAP(mean Accuracy Precision) based on Intersection Over Union (IoU)

Sample structure of the CSV.

D. Time Series Prediction (RMSE)

This challenge involves the prediction of data based on time series data. The answer and submission files have to be structured in a CSV. There is one fixed attribute in this challenge such as “date/datetime” as the first attribute. Additional attributes (columns) may be of type string or numeric (at least one numeric attribute is a must).

The evaluation criteria is Root Mean Square Error (RMSE) for the numeric attributes which is added in case of multiple numeric attributes. Also, all string based attributes (datetime, and additional attributes e.g. city) must be identical with the answer key and the only variation in numeric columns is calculated.

Two screenshots are provided to help understand the possible structures of the CSV file.

E. Non Time Series Prediction (RMSE)

This challenge involves prediction of data based on time series data. The answer and submission files have to be structured in a CSV. Additional attributes (columns) may be of type string or numeric (at least one numeric attribute is a must).

The evaluation criteria is Root Mean Square Error (RMSE) for the numeric attributes which is added in case of multiple numeric attributes. Also, all string based attributes (datetime, and additional attributes e.g. city) must be identical with the answer key and the only variation in numeric columns is calculated.

Two screenshots are provided to help understand the possible structures of the CSV file.