Label resolution and export

Documentation outdated

This section is partially outdated. We did not delete this page, as it still provides some valid and useful information. Please get in contact if you would be interested in properly documenting and updating this section.

The Label Centre is used to consolidate labels and export the annotations. At this point, the interface is still under construction, so some features might still be missing or this guide might be out-of-sync with the platform.

Resolution vs consolidation

Note, that we may use the term "consolidation" and "resolution" interchangeably.

The general concept is, that (multi-annotations) per document and label are consolidated into a gold standard label independent of the user annotations. There multiple strategies to pre-compute the consolidated labels automatically (e.g. majority vote), which can be adjusted manually. A consolidation is tied to a single annotation scheme but can span across multiple assignment scopes.

In order to view or edit an existing consolidation, click on the icon. In order to export the data of a consolidation, click on the ??? icon. In order to create a new consolidation, click on the "Create new export" button at the bottom of the page. You can, of course, create multiple consolidations of the same (partial) (sub-)set of labels if you'd like.

Consolidation strategies⚓︎

We plan to implement different strategies to automatically resolve (conflicting) multi-coded documents. In the following subsections, we describe how to use these methods.

Majority Vote⚓︎

This automatic resolver provides several options. First, you have to set the "Resolution algorithm" to "majority vote" (default at this point) and then select an annotation scheme. Second, you can select one or more "Source assignment scopes" to indicate where to take the labels from you'd like to consolidate. You can also select a (non-empty) set of users who's labels should be included ("Annotator selector"). Once you select one or more "Scheme labels to resolve", you can additionally specify how to interpret the hierarchy and priorities.

For example, if you'd like to only include the "primary labels", only keep "1" select under "Repeats to resolve". If you have a ranking of more than four, we need to adapt the interface and resolution algorithm. If you don't want/need the order, you can check the "Ignore annotation order" box. This way, all labels for that key are evaluated equally, as if they have rank=1. In case you have a nested hierarchy and would like to resolve nested labels independent of their parent label, you can check the box for "Ignore scheme hierarchy".

When clicking the "load" button, this does not save the resolution, yet. So you can experiment with the options without the platform knowing (apart from the computational overhead). For larger sets of labels, it might take a few seconds to run the resolver.

Note: At this point, we only implemented resolution for boolean, single-choice, and multi-choice types.

Reviewing and editing consolidated labels⚓︎

Once you loaded resolved labels, you will see a big table below the configuration box.

(A) This is the document ID. When you click on it, a popover will open with the rendered document.
(B) The rounded boxes show the resolved label. You can click on it and manually adjust the consolidated label.
(C) In case there are no user annotations, this box (B) will show a question mark. You can still set a consolidated label manually.
(D) Each column header represents a label indicated by the key. You can hover it to see the values and names for the available choices. The number in the pill indicates the "repeat". If you opted to ignore the order, they are all virtually set to "1".
(E) Same as (D), but for a nested label, e.g. the child label in the annotation scheme hierarchy.
(F) These are the user annotations. You can hover one to see by which user it was made.

In this screenshot you can see how multi-choice labels are resolved. These will, for the majority vote resolver (and probably most others) just be the set of all values that were selected by any user. You can click on the "x" of a pill to remove a value from the set or "+" to add another value.

Cells with a orange background are pointing to conflicts with the annotation scheme. For example, based on the majority vote, a sub-label was assigned that is not valid in combination with the consolidated parent label.

Exporting⚓︎

Exporting labels proceeds in two stages: * First, the user can filter the entirety of documents using an NQL filter. Most simply, all documents having annotations can be retrieved putting "HAS ANNOTATION". * Second, the labels to be exported can be selected as well as the sources which are considered by filtering annotations by user name, annotation scope, resolved annotation.

Note that all items filtered in the first step will be exported. If they do not have annotations from the sources filtered by in the second step, these annotations are left empty in the exported table.

Further notes⚓︎

The consolidated labels are stored separate and independent of the user annotations as so-called BotAnnotations of type RESOLVE.

The resolvers are experimental at this point, if there are issues or there is a good reason to add other features or resolvers, they can be added.