Preprocessing

Low-level Sinhala text processing functions used internally by Tokenizer, TypoDetector, and Romanizer.

Function

process_text

Splits a Sinhala string into phonological units (base consonant + vowel diacritics/virama).

process_text(text)

---

Function

normalize_sinhala

Cleans duplicate ZWJ/ZWNJs, canonicalizes diacritic ordering, and maps legacy diacritics to NFC.

normalize_sinhala(text)

---

Function

get_sinhala_character_ratio

Calculates the ratio of Sinhala Unicode block characters in a text string.

get_sinhala_character_ratio(text)

---

Function

remove_non_printable

Removes non-printable characters, keeping ASCII printable (U+0020-U+007E) and Sinhala block (U+0D80-U+0DFF).

remove_non_printable(text)

---

Function

remove_english_characters

Removes ASCII Latin characters (a-z / A-Z) from a text string.

remove_english_characters(text)

Code Examples

Phonological Unit Splitting

from sinlib.utils.preprocessing import process_text

process_text("ආයුබෝවන්")
# ['ආ', 'යු', 'බෝ', 'ව', 'න්']

Normalization

from sinlib.utils.preprocessing import normalize_sinhala

# Standardizes diacritics and clears ZWJ anomalies
normalize_sinhala("සිංහල")

Sinhala Character Ratio

from sinlib.utils.preprocessing import (
    get_sinhala_character_ratio
)

get_sinhala_character_ratio("මම ගෙදර ගියා.")
# 1.0

get_sinhala_character_ratio("Hello සිංහල")
# 0.5

Text Cleaning

from sinlib.utils.preprocessing import (
    remove_non_printable,
    remove_english_characters,
)

remove_non_printable("මම ගෙදර\x00")
# 'මම ගෙදර'

remove_english_characters("Hello සිංහල World")
# 'සිංහල'