Until recently, a majority of data annotation for training AI models was carried out manually, which invited the usual challenges that come with human intervention. Manual data annotation is prone to a variety of biases and errors and is also time-consuming.

Automated data annotation presents the opportunity to forego these limitations; it boosts the speed of annotation, promoting consistency, and ultimately helping develop more well-trained AI models. These advantages, along with the added capability of AI to work with massive datasets with little ramp-up time, make automation seem like the perfect solution for annotation.

However, the reality is not that cut and dry. Label noise, variable quality of results, and difficulty in customization are just some of the challenges that the AI-based annotation tools of today have yet to overcome. On the contrary, third-party data annotation service providers remain a much safer bet, as long as you choose a company that has set sufficient guardrails in place to ensure accuracy and the quality of the training dataset.

Let’s explore the potential challenges that AI model-driven data annotation may throw up and some effective methods we can adopt to bypass them.

The challenges of automated data annotation

Automated data annotation involves the use of an existing AI/ML model to generate labels for a dataset for various cross-industry applications. So, the data labeling process’s outcomes are affected by two moving parts: the training data and the model itself.

Here are the possible challenges that may arise in automated data annotation and how each can be overcome:

• Limited model customizability

When opting to use an automated data annotation tool, you need to manage expectations in terms of the underlying AI model’s customizability. While some models have boundless potential for customizability in their task-based datasets or adjustment of their parameters, most are very rigid and may not work well with different data domains. Additionally, the automation tool may only be able to produce certain types and formats of annotations like polygonal or semantic segmentation, but not others like bounding boxes or key points.

How to overcome: To get past this hurdle, you can go with the hybrid approach and let an annotation specialist verify your tool’s resultant annotations. Another option is to use multiple annotation tools that support various annotation types, formats, and domains. However, as this approach can get messy owing to issues of the interoperability of these tools, it is best opted for as a worst-case scenario.

• Variance in quality

One of the foremost challenges of automating data annotation is the unfettered diversity in the quality of results when your model’s target dataset is not congruent with its training data. The more the two sets of data differ, the less likely the model is to recognize the objects or classes accurately. Moreover, suppose the dataset to be labeled consists of too much variety in shapes, orientation, size, appearance, and other characteristics. In that case, the automation tool may not provide sufficient accuracy and quality in the labels.

How to overcome: The automated tool employed for annotation should be sufficiently and iteratively trained while labeling datasets. The tool’s knowledge base needs to keep expanding across various data domains it will be used for so that it is better prepared to handle diverse datasets and also maintain its efficiency and outcome accuracy. Another solution is to let these tools generate initial labels for the data which can be verified and refined by human experts.

• Maintenance struggles

When working with an auto-labeling AI solution, you may face difficulties maintaining the automation tool itself, the quality of the dataset the AI model is trained on, and the accuracy of the labels generated. Moreover, if you wish to update the automation tool to be used for a completely new data domain it hasn’t worked with before, this raises more concerns. Certain aspects of the training datasets such as ambiguity, domain complexity, extensive diversity, and bias can further complicate the maintainability of the annotation tool.

How to overcome: The automation tool must be updated with the new data domain to meet the associated labeling needs. As for the sample-labeled dataset, it needs to be cleaned, validated, and augmented sufficiently before the AI/ML model is trained on it. And, while there are many ways to do that, outsourcing data cleansing services will be the easiest and fastest for businesses who are already likely to have their hands full with their AI/ML project.

• Label noise

If the sample-labeled dataset is noisy, consists of imbalances, or is fraught with inaccuracies, the automation tool may learn from it and produce inaccurate labels leading to low-quality end results. The quality and representativeness of the training data introduce biases, and the wrong data goes downstream to the ML model, eventually leading to enterprise-wide consequences. What makes data classes more likely to get labeled incorrectly is the possibility of there being some amount of congruence between the input training data features and label noise.

How to overcome: After assessing the noise distribution, suitable learning algorithms with robust loss functions, noise-correction layers, or meta-learning methods can be applied to remove noise from the sample-labeled data.

• Class imbalance

A class imbalance occurs during data annotation when some classes or categories have more data samples than others. This could occur due to a variety of reasons and a common one is naturally imbalanced datasets such as the target classes in spam or fraud detection wherein the number of fraudulent transactions is minute when compared to the total number of transactions. The data annotation process can be further slowed down if an AI model trained on imbalanced datasets is used for annotation. This can be aggravated if the tool chooses the wrong techniques for annotation, i.e., bounding boxes or key points for datasets that are more suited for polygon or semantic segmentation. Weeding out class imbalances is essential when working with image datasets associated with critical use cases such as traffic safety or oncological screening.

How to overcome: Undersampling majority classes and oversampling minority classes can help introduce more balance in the data classes and categories. Oversampling techniques like SMOTE can be used to reduce the computational complexity without increasing the risk of noise in the resultant data.

What are some annotation tasks that circumvent these challenges?

Most data annotation tasks such as sentiment analysis, named entity recognition, or creative captioning for image annotation, or video annotation require humans to verify the resultant labels to ensure accuracy. However, not all the constituent tasks of the data annotation process pose the above challenges with automation.

Some tasks that you can almost blindly assign to automation tools without having to worry about disruptions or the need for manual verification are the following:

Bounding box annotation: Using attention maps and object co-localization, an automaton tool would be better equipped to carry out bounding box annotation without leading to an error. Pixel-level classification meant for image annotation carries this out quite efficiently and so does instance segmentation. Moreover, training the bounding box model’s object detector iteratively with small batches of data can improve annotation efficiency.

Named entity recognition: Named entity recognition with a rule-based approach based on syntactic, semantic, or contextual cues can be formulated for the automation tool to circumvent data challenges. Other lexicological methods like utilizing dictionaries and gazetteers as sample-labeled data can also help effectively automate this task. Additionally, the AI model can be trained to propose labels for new batches of data.

Semantic segmentation: Deep learning-based automation models can be opted for this as they are known to perform quite well for semantic segmentation. You can also allow your resultant annotations to pass through a post-processing algorithm that refines the labels for better accuracy. By extracting clear performance metrics of the automation models, you can formulate better training methods to select and optimize the model for specific types of datasets.

Keypoint annotation: To ensure reliable and independent keypoint annotation with automation models, first, obscured key points must be adequately handled. The training data can be pre-processed with an appropriate ML model and the number of potential key points to be annotated can be reduced with frame interpolation. You can also train a self-supervised learning model to identify the spatial relationships of the key points with scaled, cropped, or rotated images.           

Just keep this in mind: You must choose an appropriate tool that can provide the expected level of outcome quality. Certainly, with the right data annotation tool, you can automate the above-mentioned activities. But, if you fail to find one that matches your requirements closely, or find one but it is beyond your budget, you can always turn to outsourcing data annotation services as a cost-effective measure.

Why is Human-in-the-loop necessary for data annotation?

Human-in-the-loop (HITL), where the machine learning lifecycle involves continuous improvement and training by human experts, is the most feasible approach toward enhancing the end-to-end data annotation process. Manual verification of data that is annotated by AI models reduces any errors or inaccuracies in AI-implemented labels.

At the other end of this equation, training those AI models with datasets processed and enriched by humans ensures that these models are competent enough to avoid those errors and biases in the first place (although specialist supervision at this stage is just as necessary to weed out outlier cases where the machine fails). Here are some ways data annotation necessitates human intervention:

Adjusting for subjectivity: In data annotation use cases rife with subjectivity, such as labeling images for recognizing emotions from people’s faces or annotating product reviews, humans have the understanding of nuance to adjust for subjectivity. With enough reinforcement, the AI models may learn to replicate this nuance but lack the ability to produce it on their own.

Contextual understanding: In the example of a product review, there may be colloquialisms like ‘bite the bullet’ or ‘hit the hay’ which aren’t used for their literal meanings but make sense in the right context. AI models may misconstrue them for their literal meanings, tainting the resultant data labels.

Resolving edge cases: AI may overlook edge cases, especially those that have ethical ambiguity, such as in the case of flagging images as ‘potentially offensive’. While AI models are often unable to distinguish these images when the potential offense isn’t explicit, they can also be manipulated by malicious actors to completely overlook them, which can further worsen their capability.

What’s the verdict?

Supervisorless automated data annotation has its limitations and complete automation of the entire data pipeline is something that is still in development, albeit rapidly advancing. However, some of the methods discussed above are helping data scientists take significant steps toward foolproofing this technique.

While automation tends to be favored over the manual approach most of the time, you cannot completely rule out the necessity for human intervention, especially when annotating data. Given AI’s limitations, using a hybrid approach is the way to go where manual expertise can be called upon to validate the results and perform complex and critical annotation tasks.

Humans in the loop who fully understand the ins and outs of the automation systems they are working with are required to facilitate this synergy.