API Reference

Call functions with arguments generated by a local large language model

class local_llm_function_calling.CompletionModelPrompter

Basic text prompter

property call_header: str

The header for the function call

Returns:

The header for the function call

Return type:

str

function_call(function_to_call: str | None = None) → str

Create a function call prompt

Parameters:

function_to_call (str | None) – The function to call.

Returns:

The function call prompt

Return type:

str

property head: str

The head of the prompt

Returns:

The head of the prompt

Return type:

str

prompt(prompt: str, functions: list[local_llm_function_calling.prompter.FunctionType], function_to_call: str | None = None) → str

Create a function call prompt

Parameters:

• prompt (str) – The natural language part of the prompt

• functions (list[FunctionType]) – The functions to choose from

• function_to_call (str | None) – The function to call.

Returns:

The function call prompt

Return type:

str

prompt_for_function(function: FunctionType) → str

Generate the prompt section for a function

Parameters:

function (FunctionType) – The function to generate the prompt for

Returns:

The prompt section for the function

Return type:

str

prompt_for_functions(functions: list[local_llm_function_calling.prompter.FunctionType]) → str

Generate the prompt section for a list of functions

Parameters:

functions (list[FunctionType]) – The functions to generate the prompt for

Returns:

The prompt section for the functions

Return type:

str

class local_llm_function_calling.Constrainer(model: Model[PrefixType])

Generate text with an LLM in a constrained way

advance_generation(generation: Generation, constraint: Callable[[str], tuple[bool, bool]], max_len: int | None = None) → bool

Advance the generation by one token

Parameters:

• generation (Generation) – The generation to use

• constraint (Callable[[str], tuple[bool, bool]]) – A function that takes a string and returns a tuple of (is_valid, is_complete)

• max_len (int | None) – The maximum length of the generated string

Returns:

Whether the generation is complete

Return type:

bool

gen_next_token(generation: Generation, constraint: Callable[[str], tuple[bool, bool]]) → tuple[bool, int]

Generate the next token and register it

Parameters:

• generation (Generation) – The generation to use

• constraint (Callable[[str], tuple[bool, bool]]) – A function that takes a string and returns a tuple of (is_valid, is_complete)

Raises:

NoValidTokensError – There are no valid tokens to generate

Returns:

A tuple, the first element is whether the

generation is complete, the second is the number of characters generated so far (or 0 if the generation is complete)

Return type:

tuple[bool, int]

generate(prefix: PrefixType, constraint: Callable[[str], tuple[bool, bool]], max_len: int | None = None, max_new_tokens: int | None = None) → str

Generate a string with the LLM

Parameters:

• prefix – The prefix to use; the type depends on the model

• constraint (Callable[[str], tuple[bool, bool]]) – A function that takes a string and returns a tuple of (is_valid, is_complete)

• max_len (int | None) – The maximum length of the generated string

• max_new_tokens (int | None) – The maximum number of tokens to generate

Raises:

NoValidTokensError – There are no valid tokens to generate

Returns:

The generated value

Return type:

str

exception local_llm_function_calling.ConstrainerError

An error in the constrainer

exception local_llm_function_calling.FunctionCallingError

An error in the function calling library

class local_llm_function_calling.FunctionType

Function type

class local_llm_function_calling.Generator(functions: list[local_llm_function_calling.prompter.FunctionType], model: Model[PrefixType], prompter: TextPrompter[PrefixType, PromptType] | None = None)

Generate the function call based on the schema

choose_function(prompt: PromptType, function_call: str | None = None, suffix: str = '') → str

Choose a function to call

Parameters:

• prompt (PromptType) – The prompt to use

• function_call (str | None) – The function to call Will be generated if not provided.

• suffix (str) – The suffix to terminate, in order to begin prefix clashes

Returns:

The function to call

Return type:

str

generate(prompt: PromptType, function_call: str | None = None, max_length: int | None = None, max_new_tokens: int | None = None, suffix: str = '') → FunctionCall

Generate the function call

Parameters:

• prompt (PromptType) – The prompt to use

• function_call (str | None) – The function call to use. Will be generated if not provided.

• max_length (int | None) – The maximum length of the generated sequence

• max_new_tokens (int | None) – The maximum number of tokens to generate

• suffix (str) – The suffix to terminate, in order to begin prefix clashes

Returns:

The generated function call

Return type:

FunctionCall

generate_arguments(prompt: PromptType, function_call: str, max_length: int | None = None, max_new_tokens: int | None = None) → str

Generate the arguments for the function

Parameters:

• prompt (PromptType) – The prompt to use

• function_call (str) – The function to call

• max_length (int | None) – The maximum length of the generated sequence

• max_new_tokens (int | None) – The maximum number of tokens to generate

Returns:

The arguments for the function, as a JSON string

(may not be complete)

Return type:

str

classmethod hf(functions: list[FunctionType], model: AutoModelForCausalLM | str, tokenizer: AutoTokenizer | str | None = None, prompter: TextPrompter[str, PromptType] | None = None) → Generator[str, PromptType]

Create a generator for the responses to a function call, using a Huggingface model

Parameters:

• functions (list[FunctionType]) – The functions to use.

• model (AutoTokenizer | str) – The model to use.

• tokenizer (AutoTokenizer | str | None) – The tokenizer to use. Defaults to the model’s tokenizer if not provided.

• prompter (TextPrompter) – The prompter to use. Will use the model’s default prompter if not provided.

Returns:

The generator, using a Huggingface model

natural_language(prompt: PromptType, max_new_tokens: int | None = None) → str

Generate a natural language response

Parameters:

• prompt (PromptType) – The prompt to use

• max_new_tokens (int | None) – The maximum number of tokens to generate

Returns:

The natural language response

Return type:

str

respond(prompt: PromptType, max_new_tokens: int | None = None) → str | FunctionCall

Generate a response

Parameters:

prompt (PromptType) – The prompt to use

Returns:

The response

Return type:

str | FunctionCall

should_call(prompt: PromptType) → bool

Determine if the function should be called

Parameters:

prompt (PromptType) – The prompt to use

Returns:

Whether the function should be called

Return type:

bool

class local_llm_function_calling.InstructModelPrompter

Basic prompter for instruct models

property head: str

The head of the prompt

Returns:

The head of the prompt

Return type:

str

prompt(prompt: str, functions: list[local_llm_function_calling.prompter.FunctionType], function_to_call: str | None = None) → str

Create a function call prompt

Parameters:

• prompt (str) – The natural language part of the prompt

• functions (list[FunctionType]) – The functions to choose from

• function_to_call (str | None) – The function to call.

Returns:

The function call prompt

Return type:

str

class local_llm_function_calling.JsonSchemaConstraint(schema: dict, style: StyleConfig | None = None)

A JSON schema constraint

validate(text: str) → ValidationResult

Validate the text against the schema

Parameters:

text (str) – The text to validate

Returns:

The validation result

Return type:

json_schema_enforcer.schema.ValidationResult

exception local_llm_function_calling.NoValidTokensError

There are no valid tokens to generate

exception local_llm_function_calling.SequenceTooLongError

The sequence is too long to generate

class local_llm_function_calling.TextPrompter(*args, **kwargs)

Prompter protocol for function calling with open source models

prompt(prompt: PromptType_contra, functions: list[local_llm_function_calling.prompter.FunctionType], function_to_call: str | None = None) → PrefixType_co

Prompt the user for input

If function_to_call is None, then the prompt’s aim should be to select the correct function to call. If function_to_call is not None, then the prompt’s aim should be to generate the correct arguments for the function.

Parameters:

• prompt – The natural language part of the prompt

• functions (list[FunctionType]) – The functions to choose from

• function_to_call (str | None) – The function to call. When None, the prompt should be to select the function to call.

A container for huggingface models specifically

class local_llm_function_calling.model.huggingface.HuggingfaceGeneration(model: AutoModelForCausalLM, tokenizer: AutoTokenizer, prefix: str | list[str | int])

A single generation sequence with a huggingface model

get_generated(candidate: int | None = None) → str

Get the generated sequence

Parameters:

candidate (int | None) – The token to add to the sequence

Returns:

The generated sequence

Return type:

str

get_sorted_tokens() → Iterator[int]

Get the tokens sorted by probability

Raises:

SequenceTooLongError – If the sequence is too long to generate

Yields:

The next of the most likely tokens

register_token(token: int) → None

Select the token for this generation step

Parameters:

token (int) – The token to select

class local_llm_function_calling.model.huggingface.HuggingfaceModel(model: AutoModelForCausalLM | str, tokenizer: AutoTokenizer | str | None = None)

A container for a huggingface model

default_prompter() → TextPrompter[str, str]

Get the default prompter for this model

Returns:

A generic CompletionModelPrompter

start_generation(prefix: PrefixType_contra) → HuggingfaceGeneration

Start a new generation sequence

Parameters:

prefix – The generation prefix, either as a string or as a list of strings/integers; token IDs and string to be automatically tokenized

Returns:

The generation sequence initialized with the

prefix

Return type:

HuggingfaceGeneration

This collects all of the prompters into a single module

class local_llm_function_calling.prompters.CodeLlamaFunctionCallingPrompter

A prompter for code llama function calling models

natural_language_prompt(prompt: str, functions: list[FunctionType]) → list[bytes | int]

Prompt the model to generate a natural language response

Parameters:

• prompt (str) – The natural language part of the prompt

• functions (list[FunctionType]) – The functions to choose from

Returns:

The natural language prompt

Return type:

list[bytes | int]

prompt(prompt: str | list[ChatMessage], functions: list[FunctionType], function_to_call: str | None = None) → list[bytes | int]

Generate the llama prompt

Parameters:

• prompt (str | list[ChatMessage]) – The prompt to generate the response from

• functions (list[FunctionType]) – The functions to generate the response from

• function_to_call (str | None) – The function to call. Defaults to None.

Returns:

The llama prompt, a function selection prompt if no

function is specified, or a function argument prompt if a function is specified

Return type:

list[bytes | int]

should_call_prompt(prompt: str | list[ChatMessage], functions: list[FunctionType]) → tuple[list[bytes | int], ShouldCallResponse]

Check if a function should be called

Parameters:

• prompt (str | list[ChatMessage]) – The prompt to generate the response from

• functions (list[FunctionType]) – The functions to choose from

Returns:

The function to call and the response

Return type:

tuple[str, ShouldCallResponse]