Labeled Training Data Volume and What It Means for Skin Diagnosis Accuracy

More labeled images should mean a smarter model. That logic holds up to a point, but it breaks down fast once you look at what actually determines whether an AI skin diagnosis tool works on the patient in front of it. Volume sets a ceiling on how good a model can get. Whether that ceiling gets reached depends on two other factors: how carefully the images were labeled, and how well the dataset represents the people who will actually use the tool. A systematic review covering 84 peer-reviewed studies, published July 2026, found that despite promising results, heavy reliance on retrospective datasets and thin clinical validation leaves real-world performance uncertain. A separate literature review in JID Innovations, published September 2026, names restricted dataset sizes, limited diagnostic diversity, selection bias, and inconsistent model evaluation as recurring problems, and none of those four are solved by adding more pictures.

What labeled volume contributes

A bigger pool of labeled images does improve a model's ability to generalize, at least for a while. More examples of a given condition help a model learn the visual range that condition can take: how it looks on different parts of the body, at different stages, under different lighting. That part of the story is real and worth granting before complicating it.

But the bottleneck in supervised dermatology AI was never the raw collection of images. Every image needs a clinically reliable label attached to it, and the gold standard for that label is dermatologist review, which is slow, limited in supply, and expensive to scale. A pipeline can ingest as many photos as a camera can produce. It cannot manufacture more dermatologist hours to check them.

This is why a dataset can be both large and unbalanced at the same time. The widely-used HAM10000 dataset contains dermoscopic images across seven common pigmented lesion types, but it shows class imbalance that can bias model predictions: the conditions represented more often in the dataset get learned more thoroughly than the ones represented less. Adding more images of the conditions already well covered doesn't help the ones that are underrepresented. It can make the problem worse, by making the model even more confident in what it already knows well.

How class imbalance turns volume into a liability for rare and visually similar conditions

Dermatology covers thousands of distinct conditions. A model trained this way learns to predict what it has seen most, and that habit turns dangerous exactly where conditions are rare or look like something else. Standard models trained on skewed data perform poorly on rare conditions, a pattern documented across reviews of deep learning in skin disease diagnosis.

The ISIC series of datasets, used across much of the field as a benchmark, is marked by extreme class imbalance. That matters beyond the datasets themselves, because ISIC data underpins many of the benchmark studies whose accuracy numbers get cited as evidence that AI dermatology works. If the underlying data is skewed, the benchmark built on it inherits that skew.

The September 2026 JID Innovations review points to a related gap: AI development in dermatology has focused far more on neoplastic disease, meaning skin cancers and tumors, than on inflammatory dermatoses, meaning conditions like rosacea, eczema, psoriasis, and hormonally triggered acne. Clinical validation for inflammatory conditions in real-world settings remains limited, with AI applied to them still mostly confined to research protocols. That gap matters enormously for any skin-focused telehealth service, because the conditions patients bring most often, day in and day out, are precisely the ones where real-world AI validation is thinnest and where class imbalance cuts deepest.

How skin-tone underrepresentation produces diagnostic disparities

Publicly available dermatology datasets skew heavily toward lighter skin tones. Models trained on that data show measurably worse diagnostic accuracy for patients with darker skin, and adding more images doesn't fix this unless those new images come specifically from the populations the dataset is missing. The mechanism is the same one driving class imbalance for rare conditions: a model reflects the distribution it was trained on, nothing more and nothing less.

The July 2026 systematic review of 84 studies names fairness across skin tones as a persistent, unresolved challenge that appears across multiple studies in the field, not an isolated edge case. The evidence available doesn't yet let you claim with confidence that these systems perform equitably across populations.

Patients with darker skin tones, patients with rare conditions, and patients who present via smartphone photos rather than a dermoscope carry the highest risk from this gap, and these three groups overlap substantially in underserved and primary care settings. The problem also feeds itself. Privacy rules, limited institutional access, and the sheer difficulty of sourcing annotated images from underrepresented groups make it hard to collect the very data that would close the gap, so the gap tends to persist on its own.

What annotation quality adds beyond volume

A label only has clinical value if it's precise and applied consistently. A dataset where different annotators label similar cases differently will degrade a model's performance no matter how many images sit in that dataset. This is why the September 2026 JID Innovations review treats inconsistent model evaluation and limited diagnostic diversity as problems standing alongside dataset size, not beneath it. Annotation quality is a peer concern to volume, not an afterthought.

Expert curation, meaning dermatologist-reviewed validation of labels, is what turns a pile of raw images into a training signal a model can actually learn from. Skinive's 2026 accuracy report offers a concrete illustration: the company describes steady improvement across key performance metrics from 2021 to 2026, and attributes part of that gain to using real-world user images captured on smartphones, specifically to make the model more robust to the lighting, angle, and quality variation that occurs outside a dermatologist's office. Choosing which images to add, and matching them to real deployment conditions, shapes what the added volume actually buys.

What specialized training data achieves that general-purpose scale cannot

A smaller dataset, curated carefully and specific to the clinical task, can beat a general-purpose model trained on far more data. A study out of Indiana University put this to a direct test. Researchers compared DermFlow, a dermatology-trained multimodal model, against Claude Sonnet 4, a general-purpose multimodal model that received only clinical images and no patient history, across 68 biopsy-proven pigmented skin lesions drawn from patients seen between February 2023 and May 2025. It was the first direct comparison of a domain-specific dermatology AI against a general-purpose large language model built to measure what specialized training actually adds.

DermFlow reached a top-diagnosis accuracy of 47.1% and an any-diagnosis accuracy of 92.6%, with sensitivity at 93.9% and specificity at 89.5%, for a balanced accuracy of 91.7%. DermFlow also recommended biopsy in a higher share of cases than Claude did, with several of those differences reaching statistical significance (p < 0.05). Still, the authors pointed specifically to settings with limited dermatologist availability as where this kind of tool would carry the most value, which is the exact access gap specialty telehealth is meant to close.

The broader pattern here is structural. A December 2025 meta-analysis found the same pattern at scale: pooled AI sensitivity across a large pool of test images was high, but specificity was substantially lower, consistent with what the DermFlow and Claude comparison shows at the level of a single model.

AI decision support performance in real-world deployment

Deploying AI as decision support inside a supervised workflow, rather than as a standalone diagnostic tool, cushions some of the limitations any one training factor introduces. But it doesn't erase the underlying data quality problems. Those problems still shape where the AI adds genuine value and where it falls short.

A real-world study of MoleMap, a nurse-led teledermoscopy screening program running across clinics in multiple countries from January 2024 to July 2025, tested whether real-time AI decision support improved malignancy detection during routine screening. Nurses examined patients, captured dermoscopic images, and forwarded selected lesions to teledermatologists for review. The AI sat only at the nursing stage, functioning as decision support rather than an autonomous diagnostician at any point in the chain. The study's scale is notable on its own: the analytic cohort covered more than one million lesions across nearly a hundred thousand patients, making it one of the largest real-world dermatology AI deployments in the current evidence base. The design is worth paying attention to not just for its size but because the AI was embedded inside a workflow that kept a clinician in the loop at every decision point, which is the structure the research community has increasingly converged on as the safer model.

Who benefits most from that structure turns out to be uneven. A JID Innovations review found that physicians with fewer than three years of experience gained the most from AI assistance, while physicians with more than ten years of experience saw minimal gains. AI decision support, in other words, expands access most in exactly the settings where specialist experience is thinnest on the ground, which happens to be where that support is needed most. AI helps the least-experienced clinicians the most, which can widen access to care, but it may also encourage reliance on the tool rather than building independent clinical judgment in settings that are already under-resourced.

What benchmark accuracy figures hide

High sensitivity paired with low specificity is the pattern the evidence keeps showing. The December 2025 systematic review and meta-analysis confirmed this across a large body of test images: pooled sensitivity was high, pooled specificity was substantially lower, and that split occurred consistently enough to call it a systematic feature of current AI dermatology performance.

For a patient, that trade-off has a direct, human cost depending on which setting they're in. The July 2026 systematic review is blunt that real-world applicability stays uncertain even where benchmark figures look strong, and the gap between how a model performs in a controlled study and how it performs once deployed is not a minor calibration issue. It traces back to the structural data problems laid out above: uneven volume, skewed classes, uneven skin-tone representation, inconsistent labeling.

Clinical validation means testing a model against real patient populations rather than retrospective datasets collected after the fact, and that is what actually closes that distance. The September 2026 JID Innovations review calls for rigorous attention to validation, dataset diversity, implementation strategy, ethical considerations, and evidence of real-world impact before any of these tools move into clinical use. Patients who understand that an AI-assisted assessment offers a ranked list of possibilities rather than a confirmed diagnosis are in a better position to use that result as the start of a conversation with a clinician, instead of either dismissing it outright or treating it as the final word.

The gap between benchmark performance and real-world reliability becomes most visible in telehealth, where patients submit photos without dermoscopic equipment and a clinician has to interpret the AI's output alongside whatever symptoms the patient describes. Nolla, a platform pairing AI-guided symptom intake with licensed clinician review, is built around that exact tension: its AI flags patterns drawn from more than three million labeled clinical cases, but a physician makes the final call on diagnosis and treatment. That structure sidesteps the assumption that model accuracy by itself is enough to guarantee clinical safety.

Designing a skin telehealth service around these constraints

The research points toward a clear design principle. AI-assisted skin diagnosis works best when the model trains on data that is large, carefully annotated, and demographically diverse; when it's deployed as decision support rather than a stand-in for a clinician; and when a licensed professional reviews its output before a patient ever sees it.

Nolla's platform runs on large multimodal models and Vision Transformers trained on more than three million labeled clinical cases, tuned to spot patterns across images and user-reported data for general and specialty skin care. A message-based consultation with a real clinician, priced at $39, means the AI's output gets reviewed and put in context by a licensed professional before a patient acts on it, which lines up with what the research identifies as the structure that makes real-world performance reliable. Treating skin health with the same clinical rigor as any other specialty also addresses a gap the research keeps finding: inflammatory dermatoses remain underserved by an AI field that has focused primarily on neoplastic disease.

Nolla isn't alone in building this way. PanDerm, DermFM-Zero, Skin Analytics, FotoFinder, SkinVision, Google Health, First Derm, and MetaOptima each operate within some version of a clinician-supervised model, and each approaches the same underlying problem from a different angle. PanDerm and DermFM-Zero represent more recent entrants built around foundation-model approaches to dermatology imaging. The specificity of the training data behind the model, how wide a range of conditions it actually covers, and how rigorously its claims have been validated against real patients rather than retrospective image sets are what separate these services from each other, since most of them already build around the principle of human review, the same three factors this piece has traced from the start.

For a patient deciding which of these services to trust with a photo of a rash or a mole, the practical takeaway is simple enough to hold onto: an AI skin assessment works best as the opening move in a conversation with a clinician, not as the final answer. A service built around that principle, whichever one a patient chooses, stands a better chance of turning a strong benchmark number into something that actually helps the person looking at the result on their phone.