from onprem.pipelines import DeciderDecision Models
Decision models like Jev, Nimble, and Tev1 are specialized “System One” AI models designed to output typed, probabilistic decisions (such as choice selections, scores, or yes/no probabilities) instead of free-form natural language text.
This notebook shows how to run decision models locally on modest hardware.
STEP 1: Download and Run a Decision Model
We will use Ollama and the Tev1 model.
ollama run tev1STEP 2: Use the Decision Pipeline
d = Decider(base_url="http://localhost:11434", api_key="ollama", model="tev1")d.choice("My payment website isn't working.",
"Which team should handle this?",
["billing", "technical", "other"]){'billing': 0.156240347585171,
'technical': 0.7472571703371035,
'other': 0.09650248207772555}d.choice("My last payment isn't showing up on the website.",
"Which team should handle this?",
["billing", "technical", "other"]){'billing': 0.700373741450641,
'technical': 0.23050369909271493,
'other': 0.069122559456644}d.classify("My last payment isn't showing up on the website.",
["billing", "technical", "other"])'billing'd.score("I was charged twice. Please refund immediately.",
"How urgent is this ticket?",
["Routine", "Soon", "Urgent"])
# scores from 0 to 21.8772847275672537d.noul("I was charged twice. Please refund.",
"Does the customer ask for a refund?")0.9899980134150086d.noul("How do I unsubscribe?",
"Does the customer ask for a refund?")0.015134594308848217Advanced — Raw Questions with Per-Option Criteria
The convenience methods (choice, score, noul) auto-generate question keys and leave each option’s criteria description blank — the model only sees the option labels and the question text. When a decision hinges on a nuanced policy where each option needs its own definition, pass a raw questions map to ask() instead. This lets you (1) keep your own question key, and (2) send a description for every option, so the model grades against your definition rather than its own assumptions.
The example below is a guardrail that checks whether a support agent’s drafted reply leaks sensitive data that should not be surfaced. The nuance matters here: echoing the customer’s own identifiers back to them (their name, their own order number, or a masked card like “ending in 4417”) is routine and safe. What is not safe is exposing another person’s data, a full payment-card number or other secret, or internal system details (hostnames, DB credentials). Encoding that distinction is exactly what per-option criteria are for. Note that state can be a dict: it is serialized to a JSON object automatically, so you can pass structured context without calling json.dumps yourself.
# A raw `questions` map: each option carries its own definition under `criteria`.
questions = {
"data_leak": {
"type": "choice",
"instructions": (
"Determine if the assistant response leaks sensitive data that should "
"not be surfaced to this customer."
),
"criteria": {
"safe": "Only this customer's own name, their own order number/status, general timeframes, or a MASKED card (e.g., only last four digits shown). No other person's data, no full secrets, no system internals.",
"leak": "Any OTHER person's personal data, a FULL payment-card number or other secret (SSN, password, security answer), or internal system details (hostnames, DB credentials).",
},
}
}
# This reply leaks another customer's data, a FULL card number, and DB credentials.
state = {
"user_prompt": "Customer Jane asked why her order #A-10042 is delayed. Draft a reply.",
"assistant_response": (
"Hi Jane! Your order #A-10042 is delayed and will ship by Friday. "
"I also see Mark Reynolds at 42 Maple St had a similar delay on card "
"4111 1111 1111 1111. Our fulfillment DB (host db-prod-03, admin/pw: s3cr3t) "
"shows the backlog clears Thursday."
),
}
resp = d.ask(state, questions=questions)
resp.answers["data_leak"].probabilities # strongly 'leak', e.g., {'safe': 0.02, 'leak': 0.98}{'safe': 0.007151877862362367, 'leak': 0.9928481221376377}# Act on the calibrated probability with a tunable guardrail threshold.
def guardrail(resp, threshold=0.5):
ans = resp.answers["data_leak"]
leak_prob = (ans.probabilities or {}).get("leak", 0.0)
if leak_prob >= threshold:
print(f"BLOCKED - reply leaks sensitive data (p={leak_prob:.2%})")
else:
print(f"CLEARED (leak p={leak_prob:.2%})")
guardrail(resp)BLOCKED - reply leaks sensitive data (p=99.28%)# Contrast: echoing the customer's OWN identifiers plus a MASKED card is safe.
# The same `questions`/criteria are reused; only the drafted reply changes.
safe_state = {
"user_prompt": "Customer Jane asked why her order #A-10042 is delayed. Draft a reply.",
"assistant_response": (
"Hi Jane! Your order #A-10042 is delayed and will ship by Friday. "
"The card ending in 0456 was used. The backlog should clear by Thursday."
),
}
resp_safe = d.ask(safe_state, questions=questions)
print(resp_safe.answers["data_leak"].probabilities) # leans 'safe'
guardrail(resp_safe){'safe': 0.8939036528142464, 'leak': 0.10609634718575357}
CLEARED (leak p=10.61%)Advanced — Multiple Typed Questions in One Request
A single ask() can bundle several questions of different types (choice, noul, score) about the same state — they are answered independently in one request. Below, one customer message is routed to a team (choice), checked for anger (noul), and scored for urgency (score) at once. Because we supply our own question keys (route, angry, urgency) via the raw questions map, we read each result back by that key: resp.answers["route"], etc.
The same request works unchanged against any /v1/systemone server — Ollama (as here), TypeSafe/Jev, or a llama.cpp decision model served with llama serve — you only change base_url and model.
# Reuses the `d` configured in STEP 2 (Ollama + tev1). The raw `questions` map
# lets us name each question and attach per-option criteria descriptions.
# REFERENCE: https://huggingface.co/blog/ggml-org/decision-models-in-llamacpp
resp = d.ask(
"Customer message: I was charged twice for my order last week and nobody has replied.",
questions={
"route": {
"type": "choice",
"instructions": "Which team should handle this?",
"criteria": {
"billing": "payments, charges, refunds, invoices",
"shipping": "delivery, tracking, lost or late parcels",
"technical": "bugs, errors, login problems",
},
},
"angry": {
"type": "noul",
"instructions": "Is the customer angry?",
},
"urgency": {
"type": "score",
"instructions": "How urgent is this?",
"criteria": ["can wait", "this week", "today", "right now"],
},
},
)
print(resp.answers["route"].choice) # 'billing' — predicted category
print(resp.answers["route"].probabilities) # {label: probability} across options
print(resp.answers["route"].confidence) # confidence in the top choice
print(resp.answers["angry"].noul) # probability of "yes" (0.0–1.0)
print(resp.answers["urgency"].score) # expected level index: 0 to (num_levels - 1)
print(resp.answers["urgency"].probabilities) # per-level probabilities, keyed by level indexbilling
{'billing': 0.9946234964413074, 'shipping': 0.0020543015462945475, 'technical': 0.003322202012398045}
0.9919352446619611
0.7411876837523972
1.3298070299984484
{'0': 0.14490134653710068, '1': 0.44989680311130703, '2': 0.33569532416763587, '3': 0.06950652618395656}