Introduction

NeuralMarker through its review functionality allows QA or an Admin account holder to keep track of the number of annotated and skipped images "reviewed" by them in a training dataset.

How it Works 

PreRequisite:

NeuralMarker lets its users validate annotated as well as skipped images through multiple options. Therefore before moving forward with any of the options explained below.

Just make sure that the team of annotators has at least drawn annotations over few images or skipped irrelevant images in the dataset for you to review. 

Disclaimer : "Approve button" on image overlay as well as on image card will not appear for images which are not skipped neither annotated by the data annotation team.

Option 1: Through Approve Button on Image Card

Steps to follow :

1. Click on the preview button present on the dataset card.

2. Once you reach the preview page, you will observe an Approve button over the annotated images as well as over skipped images.

2 a. All those images which are annotated by the labelers will have a "Pending Review" tag on them.

2 b. All those images which are skipped by the annotators will have a "Skipped" tag on them.

3. On the image card of the annotated image, If you find that image correctly annotated, then just click on Approve button after which a toast message stating " Image marked as reviewed" will appear.

4. On the image card of the annotated image, the "pending review" tag will be changed to a blue tick and an Approve button will be changed to an Unapprove button.

However, if you want to get a second opinion on the annotated image you approved recently, you can click on the unapprove button, then a toast message stating " Image marked as Not reviewed" will appear.

 And then, that same image will again start showing the "Pending Review" tag over it. 

For skipped images, Only the approve button will be changed to unapprove button or vice-versa and the toast message according to which button is pressed will appear accordingly.

Option 2: Through Approve Button on Image Overlay

Steps to follow :

1. Click on the preview button present on the dataset card.

2. Once you reach the preview page, just hover and click on any of the annotated or skipped image which has the "Pending Review" tag on it. 

3. Image overlay over that image will appear. which will broadly highlight the annotation details about that image such as annotator name, total annotation, etc.

4. On the image overlay of the annotated image, If you find that image correctly annotated, then just click on Approve button after which a toast message stating " Image marked as reviewed" will appear.

And then you will be directed to the next image overlay to review. 

5. Once you come out of the image overlay, on the image card of the recently reviewed image, which you approved through the image overlay option you will observe 2 changes :

     a.  On the image card of the annotated image, the "pending review" tag will be changed to a blue tick.

b.  The Approve button will be changed to an Unapprove button.

6. However, if you want to get a second opinion on the annotated image you approved recently through image overlay, you can click on the unapprove button present on the image overlay, then a toast message stating " Image marked as Not reviewed" will appear. 

 And then, that same image will again start showing the "Pending Review" tag over it on its image card.

Extend Functionality of Review Feature

1. NeuralMarker allows its users to keep a track of already reviewed images through the "Show Reviewed" filter present in the "filter by" option of the preview page.  

2. NeuralMarker also allows its users to keep a track of already reviewed images by displaying the no of image Reviewed count on the dataset card.