Guided Prompts

The Guider in OnPrem.LLM, a simple interface to the Guidance package, can be used to guide the output of an LLM based on conditions and constraints that you supply.

Let’s begin by creating an onprem.LLM instance.

from onprem import LLM
llm = LLM(n_gpu_layers=35) # set based on your system

Next, let’s create a Guider instance.

from onprem.guider import Guider
guider = Guider(llm)

The guider.prompt method accepts Guidance prompts as input. (You can refer to the Guidance documentation for information on how to construct such prompts.)

Here, we’ll show some examples (mostly taken from the Guidance documentation) and begin with importing some Guidance functions.

The `select` function

The select function allows you to guide the LLM to generate output from only a finite set of alternatives. The Guider.prompt method returns a dictionary with the answer associated with the key you supply in the prompt.

from guidance import select

guidance_program = f'Do you want a joke or a poem? A ' + select(['joke', 'poem'], name='answer') # example from Guidance documentation
guider.prompt(guidance_program)

Do you want a joke or a poem? A joke

{'answer': 'joke'}

The `gen` function

The gen function allows you to place conditions and constraints on the generated output.

from guidance import gen

guider.prompt(f'The capital of France is {gen("answer", max_tokens=1, stop=".")}')

The capital of France is Paris

{'answer': 'Paris'}

You can also use regular expressions to guide the output.

prompt = f"""Question: Luke has ten balls. He gives three to his brother. How many balls does he have left?
Answer: """ + gen('answer', regex='\d+')

guider.prompt(prompt)

Question: Luke has ten balls. He gives three to his brother. How many balls does he have left?
Answer: 7

{'answer': '7'}

prompt = '19, 18,' + gen('answer', max_tokens=50, stop_regex='[^\d]7[^\d]')
guider.prompt(prompt)

19, 18, 17, 16, 15, 14, 13, 12, 11, 10, 9, 8,

{'answer': ' 17, 16, 15, 14, 13, 12, 11, 10, 9, 8,'}

Structured Outputs

Using select and gen, you can guide the LLM to produce outputs conforming to the structure that you want (e.g., JSON).

Let’s create a prompt for generating fictional D&D-type characters.

sample_weapons = ["sword", "axe", "mace", "spear", "bow", "crossbow"]
sample_armour = ["leather", "chainmail", "plate"]

def generate_character_prompt(
    character_one_liner,
    weapons: list[str] = sample_weapons,
    armour: list[str] = sample_armour,
    n_items: int = 3
):

    prompt = ''
    prompt += "{"
    prompt += f'"description" : "{character_one_liner}",'
    prompt += '"name" : "' + gen(name="character_name", stop='"') + '",'
    # With guidance, we can call a GPU rather than merely random.randint()
    prompt += '"age" : ' + gen(name="age", regex="[0-9]+") + ','
    prompt += '"armour" : "' + select(armour, name="armour") + '",'
    prompt += '"weapon" : "' + select(weapons, name="weapon") + '",'
    prompt += '"class" : "' + gen(name="character_class", stop='"') + '",'
    prompt += '"mantra" : "' + gen(name="mantra", stop='"') + '",'
    # Again, we can avoid calling random.randint() like a pleb
    prompt += '"strength" : ' + gen(name="age", regex="[0-9]+") + ','
    prompt += '"quest_items" : [ '
    for i in range(n_items):
        prompt += '"' + gen(name="items", list_append=True, stop='"') + '"'  
        # We now pause a moment to express our thoughts on the JSON
        # specification's dislike of trailing commas
        if i < n_items - 1:
            prompt += ','
    prompt += "]"
    prompt += "}"
    return prompt

d = guider.prompt(generate_character_prompt("A quick and nimble fighter"))

{"description" : "A quick and nimble fighter","name" : "Katana","age" : 25,"armour" : "leather","weapon" : "sword","class" : "fighter","mantra" : "I am the blade of justice.","strength" : 10,"quest_items" : [ "Quest Item 1","Quest Item 2","Quest Item 3"]}

The Generated Dictionary:

{'items': ['Quest Item 3', 'Quest Item 2', 'Quest Item 1'],
 'age': '10',
 'mantra': 'I am the blade of justice.',
 'character_class': 'fighter',
 'weapon': 'sword',
 'armour': 'leather',
 'character_name': 'Katana'}

Convert to JSON

import json
print(json.dumps(d, indent=4))

{
    "items": [
        "Quest Item 3",
        "Quest Item 2",
        "Quest Item 1"
    ],
    "age": "10",
    "mantra": "I am the blade of justice.",
    "character_class": "fighter",
    "weapon": "sword",
    "armour": "leather",
    "character_name": "Katana"
}

The select function

The gen function

Structured Outputs

The Generated Dictionary:

Convert to JSON

The `select` function

The `gen` function