Skip to content

Glossary

Last updated: 28/08/2026

  • Annotation: A single key/value pair (e.g. "Technology" -> "BECCS") with a reference to project, document, annotation scheme, user, and assignment scope. When you annotate a document with a more complex annotation scheme, this will create multiple annotations in the database (one for each label). The Guide for Annotators provides an overview of the user interface for annotators.
  • Annotation Scheme: Describes how each document should be annotated with hints and all the (hierarchical) labels and options. For more details, check the guide.
  • Assignment Scope: A batch of documents that are assigned to users. Each annotation scheme can have multiple scopes, e.g. rounds of annotations. This guide contains notes on how to assign documents to users.
  • Assignment: An assignment is the link between a document, annotation scheme, and user. It is used to task a user to annotate a document with a given annotation scheme. If you create assignments via a new assignment scope, it will generate assignments based on the assignment strategy you chose. For machine-learning-assisted scopes, these will be created continuously or deleted/updated if a better order of operation was determined by the model. An assignment has the status FULL if all required annotations for this assignment are done, PARTIAL if some are done, or OPEN if the user has not worked on that assignment yet.

🆕⚓︎

  • Consolidation / Resolution: Producing a gold-standard label set from multiple annotations; docs note the terms may be used interchangeably.
  • Item: A single entry in the database, for instance one academic publication including its title, abstract, DOI, etc.
  • Label: A single question/field in an annotation scheme (e.g., relevance, methodology, technology). Can be a multiple choice question, numeric value, True/False boolean value, or free text.
  • Multi-coding / Double-coding: Assigning the same item to multiple annotators to measure agreement and enable later consolidation.
  • Priority screening: Using machine-learning to predict which items are most likely relevant given what one has annotated already. This, together with a statistical stopping criterion, allows one to find most of the relevant literature whilst having to manually screen only a part of it.