Assigning documents

Documentation outdated

This section is partially outdated. It is mostly updated in November 2024 but would benefit from refinement. Please get in contact if you would be interested in properly documenting and updating this section.

Users with the appropriate permission see another "Annotation" entry in the navigation menu with a gear icon. This guide walks you through the process of assigning documents to be annotated by users.

Save before assignment

In theory, you should not be able to make assignments before you saved the configuration. Just to be on the save side, always click save on the top right before you trigger the generation of assignments.

Schemes and scopes screen

In order to create assignments for an annotation scheme, click the respective " Add scope to ..." button. For editing or viewing an existing scope, click on the respective icon.

Use concise and descriptive titles!

It might be tempting to not provide titles or leave them at the default value. For your own sanity (and that of colleagues), please take some time to come up with a descriptive and concise title for each assignment scope. For example "Test run for training students". Furthermore, the scope description is shown in the annotation interface, thus you may want to provide what annotators should look out for specifically in this assignment scope. The description can also be used to keep notes on this scope for yourself in order to keep the title more concise.

Assignment scope config screen

The configuration screen has three sections. In the top section, you can specify a title and description for this scope. In the middle section, you define how assignments will be distributed. Once you created assignments, you can monitor the progress or edit assignments in the third section.

The assignment settings have three columns. In the first column, you can decide how many assignments each user should get. Usually, everyone would get the same number of assignments. Leave at 0 if that user should not get any assignment.

In the second column, you can specify for how many items you would like an annotation by more than one annotator ("multi-coding"). Typically, we double-code everything. This means that each item is seen by exactly two people. When all annotators are in perfect agreement later on, it might make sense to single-code the vast majority of items with only a few overlapping assignments. The number listed will help you find a valid configuration. Note, that there is no error if the user pool can be too big, but assignments will fail when you try to assign to many papers to too little user capacity. Note also, that these numbers and the plausibility checker are not a guarantee that you created a valid configuration. Due to the way the algorithm works, there are cases where there would be a possible way to distribute assignments, but we did not find that optimal distribution.

In the third column, you can set up the pool of items to sample from. This sample might be drawn at random from the entire project if the query box is left empty or from the filtered set. Alternatively, assuming you already have a ranked list of prioritised items, you can sample from that ranking. The algorithm will then take the first N items (and keep the order in the assignments) from that list starting at the specified offset. There are no additional checks, it's up to you to ensure that the "item pool" is large enough to fit the numbers in the columns to the left.

Assignment scope progress / status

This section allows you to monitor the progress and see how assignments to documents is spread across users. Not all details are loaded by default, because it could potentially contain a lot of information. Note, that the detail view is work-in-progress and currently more targeted for smaller sets of documents (fewer than 20).

Each row in the table corresponds to a user from the pool. Each column corresponds to a document that has at least one assignment in this scope. An empty (white) cell indicates no assignment for that user-document pair. A blue cell indicates an open (unfulfilled) assignment, yellow a partially fulfilled assignment, and green a completed assignment. Above that table, you get a summarised statistic of all assignments in this scope.

Edit assignments

You may wish to transfer assignments to someone else or edit who should annotate what. Below the assignment scope details (label manager -> open assignment scope) you see the progress of the assignments of that scope. By clicking on the small list icon (in the image highlighted green), you see who is assigned to annotate what. Each row is a user, each column an item. Coloured cells indicate assignments, whereas blue means "open" (to-do), yellow means "partial" (incomplete annotation), and green means "done".

By clicking the "brush" icon, all blue cells will be dropped (you may need to refresh the page to see the result).

Alternatively, you can activate the edit mode. If you click on a white cell, you create an assignment for that user/item. If you click on a blue cell, you drop that assignment. You cannot click on yellow/green cells, as there are already annotations for that assignment, so it would be too dangerous to delete.

Caveat: In some cases, you may have specified a specific order. This order might have varied for each user, the table orders the items (columns) by first occurrence for any user. When editing (particularly creating assignments), the order will be stored as the column number. It might very well be, that this will affect the order in which the user sees this item (for various reasons beyond the scope of this documentation). If the order is critical in your use-case, make sure this feature does not affect it.

Further notes

The database schema and logic behind it does not require any assignments for an annotation. For example, you could just import annotations from elsewhere and don't need to create a respective assignment first. Consider this to be an information for advanced usage.

It is safe to edit the title and description even after assignments have been made. Theoretically, changing the configuration should be disabled, but if it isn't for some reason, do not edit those to prevent inconsistencies!