Summarization

The pipelines modules in OnPrem.LLM includes the Summarizer to summarize one or more documents with an LLM. This notebook shows a couple of examples.

Document Summarization

The Summarizer.summarize method runs multiple intermediate prompts and inferences, so we will set verbose-False and mute_stream=True. We will also set temperature=0 for more consistency in outputs. Finally, we will use the Zephyr-7B-beta and use the appropriate prompt template obtained from here. You can experiment with different, newer models to improve results.

from onprem import LLM
from onprem.pipelines import Summarizer
llm = LLM(model_url='https://huggingface.co/TheBloke/zephyr-7B-beta-GGUF/resolve/main/zephyr-7b-beta.Q4_K_M.gguf', 
          prompt_template= "<|system|>\n</s>\n<|user|>\n{prompt}</s>\n<|assistant|>",
          n_gpu_layers=-1, verbose=False, mute_stream=True, temperature=0) # set based on your system
summarizer = Summarizer(llm)

Next, let’s download the ktrain paper and summarize it.

!wget --user-agent="Mozilla" https://arxiv.org/pdf/2004.10703.pdf -O /tmp/ktrain.pdf -q

text = summarizer.summarize('/tmp/ktrain.pdf', max_chunks_to_use=5)
print(text['output_text'])

/home/amaiya/projects/ghub/onprem/onprem/pipelines/summarizer.py:141: LangChainDeprecationWarning: The class `LLMChain` was deprecated in LangChain 0.1.17 and will be removed in 1.0. Use :meth:`~RunnableSequence, e.g., `prompt | llm`` instead.
  map_chain = LLMChain(llm=langchain_llm, prompt=map_prompt)
/home/amaiya/projects/ghub/onprem/onprem/pipelines/summarizer.py:157: LangChainDeprecationWarning: This class is deprecated. Use the `create_stuff_documents_chain` constructor instead. See migration guide here: https://python.langchain.com/v0.2/docs/versions/migrating_chains/stuff_docs_chain/
  combine_documents_chain = StuffDocumentsChain(
/home/amaiya/projects/ghub/onprem/onprem/pipelines/summarizer.py:162: LangChainDeprecationWarning: This class is deprecated. Please see the migration guide here for a recommended replacement: https://python.langchain.com/docs/versions/migrating_chains/map_reduce_chain/
  reduce_documents_chain = ReduceDocumentsChain(
/home/amaiya/projects/ghub/onprem/onprem/pipelines/summarizer.py:171: LangChainDeprecationWarning: This class is deprecated. Please see the migration guide here for a recommended replacement: https://python.langchain.com/docs/versions/migrating_chains/map_reduce_chain/
  map_reduce_chain = MapReduceDocumentsChain(


Ktrain is a low-code Python library that simplifies the machine learning process by providing a unified interface for building, training, inspecting, and applying models using various types of data such as text, vision, and tabular. It automates where possible but also allows users to make choices based on their unique requirements. Ktrain supports TensorFlow Keras models and includes out-of-the-box support for tasks like text classification, sequence tagging, image classification, node classification, and link prediction. It offers state-of-the-art models like BERT and fastText, learning rate finders, optimization techniques, explainable AI tools, and a simple prediction API for deployment. Ktrain is an open-source machine learning platform that automates various tasks beyond just model selection and architecture search in AutoML approaches. It provides text classification, regression, sequence tagging, topic modeling, document similarity, recommendation, summarization, and question answering for text data, as well as node classification and link prediction for graph data. Overall, ktrain aims to augment human engineers' strengths rather than replace them in the ML process.

Tips: For faster summarizations, we set max_chunks_to_use=5, so that only the first five chunks of 1000 characters are considered (where chunk_size=1000 is set as the default). You can set max_chunks_to_use to None (or omit the parameter) to consider the entire document when generating the summarization.

Concept-Focused Summarization

Concept-focused summarization allows you to summarize a long document with respect to a concept of interest. This can be accomplished by invoking the summarizer.summarize_by_concept method and supplying a concept_description.

Let’s summarize the ktrain paper we just downloaded by different concepts of interest.

Summarize with respect to the concept of “question-answering”

summary, sources = summarizer.summarize_by_concept('/tmp/ktrain.pdf', concept_description="question answering")
print(summary)


The context provides an example of using the ktrain library to build an open-domain question-answering system. The system involves indexing documents from the 20 Newsgroups dataset using scikit-learn, locating documents containing words in a question using the search index, extracting paragraphs as contexts, and using a pretrained BERT model on the SQuAD dataset to parse out candidate answers. The system sorts and prunes candidate answers by confidence scores and returns results. Overall, ktrain is presented as a low-code Python library that simplifies building, training, inspecting, and applying machine learning models for various types of data, including text, vision, graph, and tabular data.

Summarize with respect to the concept of “computer vision”

summary, sources = summarizer.summarize_by_concept('/tmp/ktrain.pdf', concept_description="computer vision")
print(summary)


The context mentions "computer vision" in relation to an example of building an image classifier using the Dogs vs. Cats dataset with a pretrained ResNet50 model from ImageNet. This indicates that ktrain, the low-code Python library discussed in the text, supports tasks related to computer vision, specifically image classification. Other examples mentioned in the context include text classification, sequence tagging, and document similarity with one-class learning for text data; node classification and link prediction for graph data; and classification and regression for tabular data. Overall, ktrain provides a unified interface for solving a wide range of machine learning tasks using different types of data.

Summarize with respect to the concept of “agents”

summary, sources = summarizer.summarize_by_concept('/tmp/ktrain.pdf', concept_description="agents")
print(summary)

No text relevant to "agents" in document.

Summarize a blog post on LLMs with respect to `prompting`

from langchain.document_loaders import WebBaseLoader

loader = WebBaseLoader("https://lilianweng.github.io/posts/2023-06-23-agent/")
docs = loader.load()
with open('/tmp/blog.txt', 'w') as f:
    f.write(docs[0].page_content)

summary, sources = summarizer.summarize_by_concept('/tmp/blog.txt', concept_description="prompting")
print(summary)

WARNING:langchain_community.utils.user_agent:USER_AGENT environment variable not set, consider setting it to identify your requests.

The context mentions "prompting" as a technique used to enhance the performance of LLMs (Large Language Models) on complex tasks. Specifically, prompting involves instructing the model to think step by step through techniques like Chain of Thought (CoT), Tree of Thoughts, and ReAct. CoT and Tree of Thoughts decompose big tasks into smaller manageable steps, while ReAct integrates reasoning and acting within LLM by extending the action space to include both task-specific discrete actions and natural language prompting for generating reasoning traces. These techniques allow LLMs to better understand and solve complex problems by breaking them down into simpler subtasks and providing explicit steps for the model to follow.

More Options

If there is a need, you can experiment with different parameters, as described in our documentation.