llm backends

Support classes for different LLM backends (e.g., AWS GovCloud LLMs)

---

source

ChatGovCloudBedrock

 ChatGovCloudBedrock (model_id:str, region_name:str='us-gov-east-1',
                      endpoint_url:Optional[str]=None,
                      aws_access_key_id:Optional[str]=None,
                      aws_secret_access_key:Optional[str]=None,
                      max_tokens:int=512, temperature:float=0.7,
                      streaming:bool=False, callbacks:Optional[List]=None,
                      name:Optional[str]=None, cache:Union[langchain_core.
                      caches.BaseCache,bool,NoneType]=None,
                      verbose:bool=<factory>,
                      tags:Optional[list[str]]=None,
                      metadata:Optional[dict[str,Any]]=None, custom_get_to
                      ken_ids:Optional[Callable[[str],list[int]]]=None, ca
                      llback_manager:Optional[langchain_core.callbacks.bas
                      e.BaseCallbackManager]=None, rate_limiter:Optional[l
                      angchain_core.rate_limiters.BaseRateLimiter]=None, d
                      isable_streaming:Union[bool,Literal['tool_calling']]
                      =False, client:Any=None)

*Custom LangChain Chat model for AWS GovCloud Bedrock.

This class provides integration with Amazon Bedrock running in AWS GovCloud regions, supporting custom VPC endpoints and GovCloud-specific configurations.*

Examples

This example shows how to use OnPrem.LLM with cloud LLMs served from AWS GovCloud.

The example below assumes you have set both AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY as environment variables. You can adjust the inference_arn, endpoint_url, and region_name based on your application scenario.

from onprem import LLM

inference_arn = "YOUR INFERENCE ARN"
endpoint_url = "YOUR ENDPOINT URL"
region_name = "us-gov-east-1" # replace as necessary

# set up LLM connection to Bedrock on AWS GovCloud
llm = LLM(
  f"govcloud-bedrock://{inference_arn}",
  region_name=region_name,
  endpoint_url=endpoint_url,
)

# send prompt to LLM
response = llm.prompt("Write a haiku about the moon.")