Text and email labeling for machine learning: Threat detection case study
An enterprise cybersecurity provider partnered with Defined.ai to operationalize daily email labeling and text threat detection for high-volume, time-sensitive customer-reported content.
~7 min read
Our bespoke text and email labeling workflow helped our customer turn raw data into reliable training signals for automated classification.
TL;DR: Text and email labeling key takeaways
- High-throughput delivery: Received 1,650 emails and 3,750 texts per day to support continuous AI threat-detection training and operational triage.
- Total scale to date: Labeled 804,600 messages across email and text in English and Spanish.
- Time-bound pipeline: Built a workflow designed to return labeled batches within a 72-hour window, aligned to the customer’s response requirements.
- Median turnaround time: Achieved a median turnaround of 3 days per batch, with some batches completed in 4 days depending on workload and review complexity.
- Quality KPI: Operated to a batch-level approval target of 94.5% agreement with the customer’s labeling expectations.
- Multilingual coverage: Delivered in English and Spanish, using a dedicated, consistent workforce to maintain guideline adherence.
- Human-in-the-loop at scale: Combined automation for throughput with calibrated reviewers for nuanced decisions, reducing subjectivity and improving label consistency over time.
Our customer: labeling text for a global cybersecurity provider
Our customer is a global cybersecurity organization that protects consumers and enterprises from digital threats. They needed a steady stream of accurately labeled messages to support near-real-time triage of customer-reported submissions and ongoing machine learning improvement.
The customer’s internal team had deep domain expertise, but the daily volume and turnaround constraints made it difficult to label everything in-house. They needed a partner that could scale data annotation quickly, operate continuously and align tightly to a specialized taxonomy.
The challenge: high-volume text labeling under tight deadlines
The customer’s challenge combined three hard requirements:
-
High volume, every day
A consistent feed of thousands of messages arrived daily across email and SMS-like text. -
Fast turnaround for operational usefulness
Messages had to be returned within three days, placing pressure on operations. -
Subjective edge cases and evolving taxonomies
Threat classification often involves ambiguity, requiring consistent interpretation and adaptive guidelines.
Our capabilities: scalable email labeling with QA and labeling control
Defined.ai specializes in scalable data operations, including high-velocity email labeling programs.
Key capabilities:
- Workflow design for continuous delivery
- Human-in-the-loop operations
- Guideline alignment and calibration
- Automation for efficiency
- Multilingual execution
This delivered a dependable labeling dataset for ML and operational use.
The solution: a labeling dataset pipeline built for scale and accuracy
1. Intake and batching for SLA-driven execution
We implemented a daily workflow for AI data labeling to meet the 72-hour SLA. Each day processed 1,650 emails and 3,750 texts.
2. Taxonomy-based labeling for downstream ML
A multi-label taxonomy supported structured outputs for an email classification dataset.
Focus areas:
- Clear decision criteria
- Context handling
- Intent interpretation
- Edge-case tracking
3. Dedicated staffing model for consistent interpretation
- 16 contributors (English)
- 6 contributors (Spanish)
4. Quality assurance built around batch approval
QA aligned to a 94.5% approval KPI, using:
- Pre-delivery checks
- Sampling review
- Feedback loops
- Performance reporting
5. Scalable workload with strict controls
The program balanced scale and quality to deliver reliable email classification training data.
The results: daily email classification training data delivered at scale
Key outcomes:
- Daily scale: ~5,400 items/day
- Total labeled: 804,600 messages
- Turnaround: 3-day median
- Quality: 94.5% KPI
- Multilingual: English and Spanish
- Training-ready outputs
Breakdown:
- 372,500 English texts
- 186,250 Spanish texts
- 163,900 English emails
- 81,950 Spanish emails
Email labeling for machine learning FAQ
What is email labeling?
Email labeling is the AI annotation process of assigning categories to emails for ML training and triage.
What types of email labeling exist?
Includes classification, email intent classification, entity tagging and risk categorization.
How is an email classification dataset labeled?
Through structured guidelines, calibrated reviewers and QA workflows.
What is the difference between labeling and annotation?
Labeling assigns categories; annotation may include deeper metadata.
What is an email classification dataset used for?
Training and evaluating models for classification and threat detection.
When should you use annotation services?
When you need scalable, QA-backed labeling with domain expertise.