llm backends

Support classes for different LLM backends (e.g., AWS GovCloud LLMs)

---

source

ChatGovCloudBedrock

def ChatGovCloudBedrock(
    model_id:str, region_name:str='us-gov-east-1', endpoint_url:Optional=None, aws_access_key_id:Optional=None,
    aws_secret_access_key:Optional=None, max_tokens:int=512, temperature:Optional=None, streaming:bool=False,
    include_reasoning:bool=False, enable_thinking:bool=False, thinking_effort:Optional=None, callbacks:Optional=None,
    **kwargs
):

Custom LangChain Chat model for AWS GovCloud Bedrock.

This class provides integration with Amazon Bedrock running in AWS GovCloud regions, supporting custom VPC endpoints and GovCloud-specific configurations.

Examples

This example shows how to use OnPrem.LLM with cloud LLMs served from AWS GovCloud.

The example below assumes you have set both AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY as environment variables. You can adjust the inference_arn, endpoint_url, and region_name based on your application scenario.

from onprem import LLM

inference_arn = "YOUR INFERENCE ARN"
endpoint_url = "YOUR ENDPOINT URL"
region_name = "us-gov-east-1" # replace as necessary

# set up LLM connection to Bedrock on AWS GovCloud
llm = LLM(
  f"govcloud-bedrock://{inference_arn}",
  region_name=region_name,
  endpoint_url=endpoint_url,
)

# send prompt to LLM
response = llm.prompt("Write a haiku about the moon.")

Structured Outputs with AWS GovCloud Bedrock

AWS GovCloud Bedrock natively supports structured outputs.

from onprem import LLM
from pydantic import BaseModel, Field

inference_arn = "YOUR INFERENCE ARN"
endpoint = "YOUR ENDPOINT URL"
region = "us-gov-east-1" # replace as necessary

# setup LLM
llm = LLM(
  f"govcloud-bedrock://{inference_arn}",
  region_name=region,
  endpoint_url=endpoint,
)

# Define a Pydantic model for structured output
class PersonInfo(BaseModel):
    name: str = Field(description="name of person")
    age: int = Field(description="age of person")
    city:str = Field(description="city in which the person currently lives")
    occupation:str = Field(description="occupation of person")

# sent structured output prompt to LLM
prompt = """
  Extract the following information from this text:
  "Hi, I'm Sarah Johnson, I'm 28 years old, live in Seattle, and work as a software engineer."
"""
result = llm.prompt(prompt, response_format=PersonInfo)

# Print the structured result
print(f"Name: {result.name}")
print(f"Age: {result.age}")
print(f"City: {result.city}")
print(f"Occupation: {result.occupation}")
print(f"Type: {type(result)}")

Name: Sarah Johnson
Age: 28
City: Seattle
Occupation: software engineer
Type: <class '__main__.PersonInfo'>