US Patent 10977518 Machine learning based adaptive instructions for annotation

Techniques for generating and utilizing machine learning based adaptive instructions for annotation are described. An annotation service can use models to identify edge case data elements predicted to elicit differing annotations from annotators, “bad” data elements predicted to be difficult to annotate, and/or “good” data elements predicted to elicit matching or otherwise high-quality annotations from annotators. These sets of data elements can be automatically incorporated into annotation job instructions provided to annotators, resulting in improved overall annotation results via having efficiently and effectively “trained” the annotators how to perform the annotation task.