llama_core::embeddings

Function chunk_text

Source 
pub fn chunk_text(
    text: impl AsRef<str>,
    ty: impl AsRef<str>,
    chunk_capacity: usize,
) -> Result<Vec<String>, LlamaCoreError>
Expand description

Generate a list of chunks from a given text. Each chunk will be up to the chunk_capacity.

§Arguments

  • text - A reference to a text.

  • ty - Type of the text, txt for text content or md for markdown content.

  • chunk_capacity - The max tokens each chunk contains.

§Returns

A vector of strings.

§Errors

Returns an error if the operation fails.