Determines the optimal length for the next text chunk, ensuring it does not exceed the specified maximum length.
It attempts to split the text at the last occurrence of a separator within the max_length.
To avoid creating excessively small chunks, it only considers separators found in the latter half of the potential chunk (defined as max_length >> 1).
If no suitable separator is found, or if the text is shorter than max_length, it returns max_length (or the text length).
Parameters
text: string
The text to be split.
max_length: number
The hard limit for the chunk length. Must be a positive integer.
separators: string[] = ...
A list of characters or substrings to split by, in order of preference. Defaults to ['\n', ' ', '.'].
Determines the optimal length for the next text chunk, ensuring it does not exceed the specified maximum length.
It attempts to split the text at the last occurrence of a separator within the
max_length. To avoid creating excessively small chunks, it only considers separators found in the latter half of the potential chunk (defined asmax_length >> 1).If no suitable separator is found, or if the text is shorter than
max_length, it returnsmax_length(or the text length).