BatchEncoding

BatchEncoding is the return type of Tokenizer.__call__, Tokenizer.encode_plus(), and Tokenizer.batch_encode(). It provides both attribute-style and dict-style access to tokenizer output.

Class

BatchEncoding

Dict-like container holding the outputs of the Tokenizer.

Fields

Field Name	Type	Description
input_ids	list[int] | list[list[int]] | Tensor	Phonological unit integer IDs mapping the input sequence.
attention_mask	list[int] | list[list[int]] | Tensor	Mask indicating real elements (1) vs padded indices (0).

---

Methods

to_dict

Convert the internal structure to a standard Python dictionary.

encoding.to_dict()

convert_to_tensors

Convert the internal sequence lists into tensors of a specific framework.

encoding.convert_to_tensors(tensor_type)

Argument	Type	Description
tensor_type	strRequired	Target tensor framework: “pt” (PyTorch), “tf” (TensorFlow), or “np” (NumPy).

Code Examples

Attribute and Dict Access

from sinlib import Tokenizer
tokenizer = Tokenizer.from_pretrained("Ransaka/sinlib")

encoding = tokenizer("ආයුබෝවන්")

# Access by attribute
ids = encoding.input_ids
# [4, 23, 18, 7, 12]

# Access by key lookup
mask = encoding["attention_mask"]
# [1, 1, 1, 1, 1]

Tensor Conversion

# Convert lists to NumPy arrays
encoding.convert_to_tensors("np")
type(encoding.input_ids)
# <class 'numpy.ndarray'>

# Convert lists to PyTorch Tensors
encoding.convert_to_tensors("pt")
# Returns torch.Tensor

Serialization

plain_dict = encoding.to_dict()
# {"input_ids": [...],
#  "attention_mask": [...]}