This dialog contains information about the CoralNet automated annotation system. For technical details refer to "Towards automated annotation of benthic survey images: Variability of human experts and operational modes of automation"

• Basic information

  The principal goal of the CoralNet computer-vision backend is to facilitate faster and more accurate point annotations. Here are the basics:

  After an initial set of images have been annotated, a first classifier will be trained, assuming the source has training enabled. This classifier will then "pre-annotate" all other images uploaded to the source. The status of these images change from unannotated to unconfirmed.

  When you enter the annotation-tool for an unconfirmed image, all point annotations will have label-suggestions. These are provided along with a posterior probability estimate indicating how "confident" the classifier is that the substrate directly under the point pertains to the respective labels.

  New classifiers are trained continuously as more images are confirmed by the source admins/editors.

  Source admins can set the source "confidence threshold". When you enter the annotation tool, all points for which the classification confidence is higher than the source threshold will be automatically confirmed so that you can focus on the other points and more rapidly work through the images.

  The default confidence threshold is 100%, which effectively turns off the automatic confirmation feature, since the posterior probabilities can never be 100% or more. If you want to use automatic confirmation, the diagnostics on this page can help you choose an appropriate confidence threshold to set.

  Note: if you are using a classifier that was trained in another source, then this page's diagnostic info only reflects performance on the classifier's source's data, not this source's data.

• Classifier Performance Estimation

  The confirmed data is split in eight parts. A classifier is trained on 7/8'ths and evaluated on the last part, the "validation-set". This procedure is used to generate the confusion matrix and classifier analytics curve shown on this page.

  By inspecting the curves the source admin can set the appropriate confidence threshold balancing the amount of manual work with the decrease in performance.

  NOTE: since the classifier is compared to a single set of annotations, and that set of taken to be the "ground truth", the performance will always seem to deteriorate as more annotations are done automatically. However, in our experiments, inter and intra operator variability is significant, and a well-trained classifier operating at a threshold where around 50% of the points are done automatically may actually increase the performance compared to fully manual annotations. We encourage all project managers to do a thorough inter- and intra- operator assessment before starting large-scale annotation projects.

• Source specific classifiers

  All classifiers are source-specific. This means that it will learn only from confirmed annotations within the source. The reason for this is simple: machine learning across different sources is difficult, and it's not clear how to do this efficiently yet.

• Classifier threshold sweep

  The first plot on this page is created by trying hundreds of confidence thresholds on the validation-set. For each threshold all annotations for which the classifier confidence is lower then the threshold are discarded. Accuracy is then calculated for the remaining annotations both on the level of individual labels and the functional group level. This is plotted along with the fraction of points above the threshold.

• Confusion Matrix

  You can investigate any point on the threshold sweep by selecting a threshold and label mode using the form provided. A confusion matrix is then displayed for the selection.