PII Removal — Fine-tuned RoBERTa for Ukrainian Personal Data

AWS Comprehend shows 14% accuracy on Ukrainian texts, Azure AI Language — 37%. Both lack training data with Cyrillic and don't understand Ukrainian identifiers RNOKPP, EDRPOU. Our model achieves 76-87% accuracy thanks to training on real Ukrainian data leaks with support for Cyrillic, transliteration per pre/post-2010 rules, recognition of specific document formats and identifiers. Cloud services cannot process mixed Cyrillic-Latin texts, don't understand variability of Ukrainian names in transliteration, and don't account for peculiarities of personal data formatting in local documents and databases. Additionally, cloud solutions have data privacy limitations: transferring personal data to external servers may violate GDPR and Ukrainian personal data protection legislation.

RoBERTa is a specialized model for NER tasks with fast inference 50-100ms on CPU. GPT is a generative model that requires more resources and is less predictable for classification. For production PII removal, we need speed and accuracy, not text generation. RoBERTa is trained on masked language modeling, making it ideal for understanding context and token classification. It consumes less memory, runs faster on CPU without GPU, has deterministic output for each input, which is critical for compliance and audit. GPT requires GPU for acceptable speed, has variable output even with temperature=0, consumes 3-5x more memory. For classification and NER tasks, RoBERTa shows better results with lower resource costs.

Instead of manually annotating thousands of examples, we used OpenAI as a teacher to generate labeled data. This accelerated dataset preparation and improved annotation quality compared to manual process. The process looks like this: take unlabeled text from real leaks, pass to OpenAI API with prompt for PII recognition, receive labeled entities, validate result, add to training dataset. This allowed creating over 10000 labeled examples in a week instead of months of manual work. Annotation quality is higher than manual because OpenAI understands context better than a human annotator without specialization. Knowledge distillation approach allows obtaining a compact model with quality close to large teacher model, but with much lower resource requirements.

Synthetic data doesn't reflect real variability: name typos, non-standard phone formats, mixed Cyrillic-Latin texts. Real leaks give the model understanding of what data looks like in production. Real data contains typos in names, incomplete addresses, phones without country codes, emails with errors, mixed date formats, transliteration per different standards, outdated document formats. Synthetic datasets generate perfect examples that don't prepare the model for the real world. Training on leaks gives the model robustness to noise, context understanding, and ability to recognize PII even in incorrectly formatted texts. This is critical for working with legacy systems and databases with inconsistent formatting.

Model sometimes clips token edges at the beginning and end of entities. This appears in short texts without context. Fix — retraining on edge cases. In longer PDF texts, the problem is less noticeable. The reason is tokenization: RoBERTa splits text into subword tokens, and if an entity starts or ends with a rare subword, the model may miss it. For example, Tkachenko may tokenize as Tka-chen-ko, and the model may miss Tka or ko. Solution: expand training dataset with edge token examples, add augmentation with context clipping, use CRF layer to account for dependencies between tokens. Also planning experiment with different tokenizers for better processing of Ukrainian words.