llm backends

Support classes for different LLM backends (e.g., LlamaCpp, HuggingFace Transformers, AWS GovCloud LLMs)

ChatGovCloudBedrock

def ChatGovCloudBedrock(
    model_id:str, region_name:str='us-gov-east-1', endpoint_url:Optional[str]=None, profile_name:Optional[str]=None,
    aws_access_key_id:Optional[str]=None, aws_secret_access_key:Optional[str]=None, max_tokens:int=512,
    temperature:Optional[float]=None, streaming:bool=False, include_reasoning:bool=False, enable_thinking:bool=False,
    thinking_effort:Optional[str]=None, callbacks:Optional[List]=None, **kwargs
):

Custom LangChain Chat model for AWS GovCloud Bedrock.

This class provides integration with Amazon Bedrock running in AWS GovCloud regions, supporting custom VPC endpoints and GovCloud-specific configurations.

source

LlamaCpp

def LlamaCpp(
    model_path:str, max_tokens:Optional[int]=None, temperature:Optional[float]=0.8, top_p:Optional[float]=0.95,
    top_k:Optional[int]=40, repeat_penalty:Optional[float]=1.1, stop:Optional[List[str]]=None, n_ctx:int=512,
    n_batch:Optional[int]=8, n_gpu_layers:Optional[int]=None, verbose:bool=False, streaming:bool=True,
    callbacks:Optional[List[Any]]=None, grammar_path:Optional[str]=None, grammar:Optional[Any]=None,
    model_kwargs:Optional[Dict[str, Any]]=None, **kwargs:Any
):

A lightweight, LangChain-free wrapper around llama_cpp.Llama.

This class replaces langchain_community.llms.LlamaCpp (which is deprecated, as langchain-community has been sunset). It exposes:

client: the underlying llama_cpp.Llama instance (used directly by OnPrem for create_chat_completion and for prompt truncation/tokenization).
invoke(prompt, stop=..., **kwargs): returns the generated text as a string. When streaming is enabled, tokens are forwarded to any supplied callbacks (e.g., a streaming stdout handler) as they are generated.
get_num_tokens(text): convenience token counter.

Named arguments are the common generation/model parameters. Any additional keyword arguments (and the contents of model_kwargs) are forwarded directly to the llama_cpp.Llama constructor.

source

HFPipeline

def HFPipeline(
    model_id:str, max_tokens:Optional[int]=None, mute_stream:bool=False, tokenizer:Any=None, **kwargs:Any
):

A lightweight, LangChain-free wrapper around a Hugging Face transformers text-generation pipeline.

This replaces langchain_huggingface’s ChatHuggingFace/HuggingFacePipeline for the local-transformers generation path. It exposes:

pipeline: the underlying transformers pipeline (used for tokenizer access, generation_config, and prompt truncation).
invoke(prompt, stop=..., **kwargs): returns generated text as a string. Applies the tokenizer’s chat template (as a single user message) when available, and forwards stop via the pipeline’s stop_strings argument.
get_num_tokens(text): convenience token counter.

AWS GovCloud Examples

This example shows how to use OnPrem.LLM with cloud LLMs served from AWS GovCloud.

Recommended: AWS SSO/ADFS profile. If your organization uses SSO/ADFS, first create a profile with aws configure sso (e.g., named Bedrock) and authenticate with aws sso login --profile Bedrock. Then pass the profile name via profile_name. boto3 handles the short-lived SSO token (re-run aws sso login when it expires):

from onprem import LLM

inference_arn = "YOUR INFERENCE ARN"
endpoint_url = "YOUR ENDPOINT URL"
region_name = "us-gov-east-1" # replace as necessary

# set up LLM connection to Bedrock on AWS GovCloud using an SSO/ADFS profile
llm = LLM(
  f"govcloud-bedrock://{inference_arn}",
  region_name=region_name,
  endpoint_url=endpoint_url,
  profile_name="Bedrock",  # name of the profile created via `aws configure sso`
)

# send prompt to LLM
response = llm.prompt("Write a haiku about the moon.")

Still supported: static access keys via environment variables. The AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables remain fully supported. If you set them (or pass aws_access_key_id/aws_secret_access_key directly) and omit profile_name, those credentials are used as before — boto3 picks up the environment variables automatically:

from onprem import LLM

inference_arn = "YOUR INFERENCE ARN"
endpoint_url = "YOUR ENDPOINT URL"
region_name = "us-gov-east-1" # replace as necessary

# set up LLM connection to Bedrock on AWS GovCloud
llm = LLM(
  f"govcloud-bedrock://{inference_arn}",
  region_name=region_name,
  endpoint_url=endpoint_url,
)

# send prompt to LLM
response = llm.prompt("Write a haiku about the moon.")

Structured Outputs with AWS GovCloud Bedrock

AWS GovCloud Bedrock natively supports structured outputs.

from onprem import LLM
from pydantic import BaseModel, Field

inference_arn = "YOUR INFERENCE ARN"
endpoint = "YOUR ENDPOINT URL"
region = "us-gov-east-1" # replace as necessary

# setup LLM
llm = LLM(
  f"govcloud-bedrock://{inference_arn}",
  region_name=region,
  endpoint_url=endpoint,
)

# Define a Pydantic model for structured output
class PersonInfo(BaseModel):
    name: str = Field(description="name of person")
    age: int = Field(description="age of person")
    city:str = Field(description="city in which the person currently lives")
    occupation:str = Field(description="occupation of person")

# sent structured output prompt to LLM
prompt = """
  Extract the following information from this text:
  "Hi, I'm Sarah Johnson, I'm 28 years old, live in Seattle, and work as a software engineer."
"""
result = llm.prompt(prompt, response_format=PersonInfo)

# Print the structured result
print(f"Name: {result.name}")
print(f"Age: {result.age}")
print(f"City: {result.city}")
print(f"Occupation: {result.occupation}")
print(f"Type: {type(result)}")

Name: Sarah Johnson
Age: 28
City: Seattle
Occupation: software engineer
Type: <class '__main__.PersonInfo'>