label-training-dataSetup systematic data labeling via Label Studio or similar. QC, inter- annotator agreement, labeler teams, integrate labeled data into ML pipelines. Use starting supervised ML needing labeled data, model perf limited by insufficient labels, labeling text/image/audio/video, or active learning to prioritize valuable examples.
Independent GitHub snapshot · Automated classification · Original documentation governs
See Extended Examples for complete config files + templates.
Systematically label data for supervised ML w/ QC + efficient workflows.
Setup labeling platform.
# Install Label Studio
pip install label-studio
# Or use Docker for production
docker pull heartexlabs/label-studio:latest
# Create project directory
mkdir -p labeling-project/{data,exports,config}
cd labeling-project
# Initialize Label Studio
label-studio init my_project
# Start Label Studio server
label-studio start my_project --port 8080
Access http://localhost:8080 (default creds: create on first visit).
Prod deploy w/ Docker:
# docker-compose.yml
version: '3.8'
services:
label-studio:
image: heartexlabs/label-studio:latest
ports:
- "8080:8080"
# ... (see EXAMPLES.md for complete implementation)
docker-compose up -d
→ Label Studio running + accessible, PostgreSQL DB init for prod.
If err: Port 8080 busy → change port in config. Docker fails → check daemon running. Ensure disk space for data vols. Firewall allows 8080.
Create labeling config for task type.
# labeling-project/config/labeling_config.py
"""
Label Studio configuration templates for common tasks.
"""
# Text Classification (single label)
TEXT_CLASSIFICATION = """
<View>
# ... (see EXAMPLES.md for complete implementation)
→ Interface configured w/ appropriate controls for task type, data imported, interface accessible to annotators.
If err: Validate XML config w/ Label Studio validator. Check data file format (JSON / CSV). Ensure image/audio URLs accessible if external storage. Verify API key perms.
Format data for import + prioritize for labeling.
# labeling-project/prepare_data.py
import pandas as pd
import json
import random
from typing import List, Dict
from sklearn.cluster import KMeans
import numpy as np
# ... (see EXAMPLES.md for complete implementation)
→ Data formatted for Label Studio import, sampling prioritizes informative examples, tasks include metadata for tracking.
If err: Verify JSON w/ jq / Py json.load(). Check URLs accessible if remote images. Ensure no special chars break JSON. Validate column names match config.
Setup processes to measure + improve quality.
# labeling-project/quality_control.py
import pandas as pd
import numpy as np
from sklearn.metrics import cohen_kappa_score, confusion_matrix
from typing import Dict, List, Tuple
import logging
logging.basicConfig(level=logging.INFO)
# ... (see EXAMPLES.md for complete implementation)
→ IAA measured (Cohen's Kappa > 0.6 = moderate, >0.8 = good), difficult tasks ID'd for review, annotator perf tracked.
If err: Kappa very low (<0.4) → review guidelines for clarity, retrain annotators, simplify schema, check ambiguous examples, consider expert annotators for gold std.
Export labels + prep for ML training.
# labeling-project/export_labels.py
import requests
import pandas as pd
import json
from typing import List, Dict
import logging
logger = logging.getLogger(__name__)
# ... (see EXAMPLES.md for complete implementation)
→ Annotations exported training-ready format, label distribution balanced / documented, data quality validated before training.
If err: Verify API key perms. Check export format compat w/ ML framework. Handle missing annotations gracefully. Validate JSON matches expected.
Automate workflow w/ active learning.
# labeling-project/active_learning_pipeline.py
import schedule
import time
import logging
from datetime import datetime
from prepare_data import DataSampler, prepare_label_studio_format
from export_labels import LabelStudioExporter, convert_to_training_format
import pandas as pd
# ... (see EXAMPLES.md for complete implementation)
→ Active learning selects informative examples auto, batches prep weekly, model retrained when sufficient new labels avail.
If err: Uncertainty sampling doesn't improve model → try diversity sampling. Annotators can't keep up → reduce batch size. Monitor queue length, backpressure if queue grows.
version-ml-data — version control for labeled datasetstrack-ml-experiments — track model perf as labels added