LLM Constructor. Extra kwargs (e.g., temperature) are fed directly the LangChain LLM or Transformer Pipeline.
Args:
model_url: URL to .GGUF model (or the filename if already been downloaded to model_download_path). To use an OpenAI-compatible REST API (e.g., vLLM, OpenLLM, Ollama), supply the URL (e.g., http://localhost:8080/v1). To use a cloud-based provider, replace URL with provider/model: openai/<name_of_model> (e.g., openai/gpt-3.5-turbo). Any LiteLLM-supported provider is supported. To use Ollama, use ollama/<model_name (e.g., ollama/llama3.2). If None, use the model indicated by default_model.
model_id: Name of or path to Hugging Face model (e.g., in SafeTensor format). Hugging Face Transformers is used for LLM generation instead of llama-cpp-python. Mutually-exclusive with model_url and default_model. The n_gpu_layers and model_download_path parameters are ignored if model_id is supplied.
default_model: One of {‘mistral’, ‘zephyr’, ‘llama’}, where mistral is Mistral-Instruct-7B-v0.2, zephyr is Zephyr-7B-beta, and llama is Llama-3.1-8B.
default_engine: The engine used to run the default_model. One of {‘llama.cpp’, ‘transformers’}.
prompt_template: Optional prompt template (must have a variable named “prompt”). Prompt templates are not typically needed when using the model_id parameter, as transformers sets it automatically.
model_download_path: Path to download model. Default is onprem_data in user’s home directory.
vectordb_path: Path to vector database (created if it doesn’t exist). Default is onprem_data/vectordb in user’s home directory.
store_type: One of dense for the default dense vector database (i.e., chroma) or sparse for the sparse vector store (i.e., a keyword search engine).
(Documents stored in sparse vector databases are converted to dense vectors at inference time when used with LLM.ask.)
vectorstore: an onprem.ingest.stores.base.VectorStore instance.
max_tokens: The maximum number of tokens to generate.
n_gpu_layers: Number of layers to be loaded into gpu memory. Default is None. Only used for llama-cpp backend.
n_ctx: Token context window. Only used for llama-cpp backend. For Ollama backend, explicitly supply num_ctx instead which is passed to LiteLLM. Hugging Face Transformers backend (i.e., when using the model_id parameter) sets context window automatically.
n_batch: Number of tokens to process in parallel. Only used for llama-cpp backend.
stop: a list of strings to stop generation when encountered (applied to all calls to LLM.prompt)
mute_stream: Mute ChatGPT-like token stream output during generation
callbacks: Callbacks to supply model
embedding_model_name: name of sentence-transformers model. Used for LLM.ingest and LLM.ask.
embedding_model_kwargs: arguments to embedding model (e.g., {device':'cpu'}). If None, uses GPU if available.
embedding_encode_kwargs: arguments to encode method of embedding model (e.g., {'normalize_embeddings': False}).
rag_num_source_docs: The maximum number of documents retrieved and fed to LLM.ask and LLM.chat to generate answers.
rag_score_threshold: Minimum similarity score for source to be considered by LLM.ask and LLM.chat.
confirm: whether or not to confirm with user before downloading a model
Get VectorStore instance. Use the vectorstore’s methods directly instead of accessing the underlying database. Supply custom_vectorstore to use your own VectorStore instance (i.e., subclass DenseStore or SparseStore). Supply reset=True to reload the default vectorstore.
Send prompt to LLM to generate a response. Extra keyword arguments are sent directly to the model invocation.
Args:
prompt: The prompt to supply to the model. Either a string or OpenAI-style list of dictionaries representing messages (e.g., “human”, “system”).
image_path_or_url: Path or URL to an image file
prompt_template: Optional prompt template (must have a variable named “prompt”). This value will override any prompt_template value supplied to LLM constructor.
stop: a list of strings to stop generation when encountered. This value will override the stop parameter supplied to LLM constructor.
truncate_prompt: Truncate long string prompts. Only applies to llama-cpp-python and transformers LLMs.
truncate_strategy: Either ‘first’ (keep latest) or ’last(keep earliest). Ignored iftruncate_prompt=False`.
response_format: A Pydantic model class for structured output. When provided, the LLM will return a Pydantic object instead of a string. Uses native structured output (OpenAI, Azure, etc.) when available, otherwise falls back to the pydantic_prompt method which uses prompt-based parsing. Invoke pydantic_prompt directly for more control over prompt-based parsing.
method: Method for structured output when using response_format. Options: ‘function_calling’ (default), ‘json_schema’, ‘json_mode’. Only applies to models that support these methods (e.g., ChatOpenAI). ‘json_schema’ provides guaranteed schema adherence for OpenAI models (gpt-4o-mini, gpt-4o-2024-08-06+).
strict: Enable strict mode for schema validation when using response_format. Only applies when method=‘json_schema’ or method=‘function_calling’ for OpenAI models.
include_raw: If True, returns dict with ‘raw’, ‘parsed’, and ‘parsing_error’ keys when using response_format. If False (default), returns only the parsed output.
Accept a prompt as string and Pydantic model describing the desired output. Output will be a Pydantic object in the requested format.
Args:
prompt: The prompt to supply to the model. Either a string or OpenAI-style list of dictionaries representing messages (e.g., “human”, “system”).
pydantic_model: A Pydanatic model (sublass of pydantic.BaseModel that describes the desired output format. Output will be a desired Pydantic object. If put_format=None, then output is a string.
attempt_fix: Use an LLM call in attempt to correct malformed or incomplete outputs
fix_llm: LLM to use for fixing (e.g., langchain_openai.ChatOpenAI()). If None, then existing LLM.llm used.
stop: a list of strings to stop generation when encountered. This value will override the stop parameter supplied to LLM constructor.
Async version of prompt method. For cloud/API models, uses native async. For local models (llama.cpp/transformers), runs in thread pool to avoid blocking.
Args:
prompt: The prompt to supply to the model. Either a string or OpenAI-style list of dictionaries representing messages (e.g., “human”, “system”).
image_path_or_url: Path or URL to an image file
prompt_template: Optional prompt template (must have a variable named “prompt”). This value will override any prompt_template value supplied to LLM constructor.
stop: a list of strings to stop generation when encountered. This value will override the stop parameter supplied to LLM constructor.
truncate_prompt: Truncate long string prompts. Only applies to llama-cpp-python and transformers LLMs.
truncate_strategy: Either ‘first’ (keep latest) or ’last(keep earliest). Ignored iftruncate_prompt=False`.
response_format: A Pydantic model class for structured output. When provided, the LLM will return a Pydantic object instead of a string. Uses native structured output (OpenAI, Azure, etc.) when available, otherwise falls back to the pydantic_prompt method which uses prompt-based parsing.
method: Method for structured output when using response_format. Options: ‘function_calling’ (default), ‘json_schema’, ‘json_mode’. Only applies to models that support these methods (e.g., ChatOpenAI). ‘json_schema’ provides guaranteed schema adherence for OpenAI models (gpt-4o-mini, gpt-4o-2024-08-06+).
strict: Enable strict mode for schema validation when using response_format. Only applies when method=‘json_schema’ or method=‘function_calling’ for OpenAI models.
include_raw: If True, returns dict with ‘raw’, ‘parsed’, and ‘parsing_error’ keys when using response_format. If False (default), returns only the parsed output.
def ingest( source_directory:str, # path to folder containing documents chunk_size:Optional[int]=None, # text is split to this many characters by `langchain.text_splitter.RecursiveCharacterTextSplitter`. If None, uses onprem.ingest.base.DEFAULT_CHUNK_SIZE. chunk_overlap:Optional[int]=None, # character overlap between chunks in `langchain.text_splitter.RecursiveCharacterTextSplitter`. If None, uses onprem.ingest.base.DEFAULT_CHUNK_OVERLAP. ignore_fn:Optional[Callable]=None, # callable that accepts the file path and returns True for ignored files batch_size:int=1000, # batch size used when processing documents(e.g, creating embeddings).**kwargs):
Answer a question based on source documents fed to the LLM.ingest method. This method delegates to RAGPipeline. See RAGPipeline.ask for parameter details. Extra keyword arguments are sent directly to LLM.prompt. Returns a dictionary with keys: answer, source_documents, question
We’ll use a small 3B-parameter model here for testing purposes. The vector database is stored under ~/onprem_data by default. In this example, we will store the vector store in temporary folders.
1. Luna - this name means "moon" in Latin and is perfect for a cat with soft, moon-like fur or bright green eyes that seem to glow like the full moon.
2. Willow - named after the delicate branches of a willow tree, this name would suit a sweet, gentle kitty who loves to snuggle and purr contentedly in your lap.
3. Marshmallow - if you have a fluffy cat with a round tummy and a plump body, why not call her Marshmallow? This adorable name is sure to melt your heart as soon as you see her cute little face.
Appending to existing vectorstore at /home/amaiya/onprem_data/vectordb
Loading documents from ./sample_data/1/
Loading new documents: 100%|██████████████████████| 1/1 [00:00<00:00, 3.52it/s]
Loaded 6 new documents from ./sample_data/1/
Split into 41 chunks of text (max. 500 chars each for text; max. 2000 chars for tables)
Creating embeddings. May take some minutes...