Enhancing PDF extraction: multi-column layout and OCR

[JiaZhang42]

Hi Eduard,

Thank you for creating such a powerful package!

I wonder if you plan to extend the PDF extraction functionality in llm_message() to automatically detect whether the PDF is multi-column or requires OCR and then apply the appropriate extraction method. From my experience, pdftools::pdf_text() does not currently handle these scenarios effectively.

Additionally, I noticed that pdf_page_batch() prepares both the text and image of each PDF page as a list of LLM messages. I’m new to this multimodal functionality and wanted to ask: Is the inclusion of both text and images primarily to account for the layout or structure of the PDF?

[edubruell]

Thanks so much for the kind words! You're absolutely right—there’s plenty of room for improvement when working with PDFs. Ideally, something akin to pdfminer or marker (which excel in layout-aware extraction) would be a great addition to tidyllm. However, both are Python-based, and as of now, I’m not aware of equivalent R solutions. So, it might take some time to develop a robust fix for this within the R ecosystem.

Workaround Using pdf_page_batch()

In the meantime, pdf_page_batch() can serve as a useful workaround for handling PDFs with complex layouts, such as those with figures or multi-column formatting. Here’s an example I tried with a dual-column paper containing a few figures. While it’s far from perfect, it can at leats handle such cases partially:

markdown <- here::here("local_wip","example_aktivrente.pdf") |>
  pdf_page_batch("Transcribe this page to markdown.
  For figures put an expressive description of each figure into <figure> </figure> tags") |>
  send_batch(claude)

markdown |> 
  fetch_batch(claude) |>
  map(get_reply)

This approach mostly works, but I haven’t spent much time documenting it since it’s still a bit rough around the edges and needs a lot of work on writing good prompt and manual corrections, when it does not work.

Using the Google Gemini Upload Feature for Multimodal PDF Handling

Another (albeit API-specific) alternative is the file-upload feature in the Google Gemini API. This functionality, available in the development version of tidyllm and coming to CRAN with version 0.3.0, excels at multimodal PDF tasks. It can handle image-based pages, works with tidyllm_schema() and provides detailed descriptions of figures and tables.

Here’s an example workflow:

#Define a reply schema for figures
figure_schema <- tidyllm_schema(name="FigureSchema",
               FigureDescription = "character[]")    

#Use gemini_upload_file to upload PDF
example_upload <- gemini_upload_file(here::here("local_wip","example_aktivrente.pdf"))

#Get an assistant reply with the file in the model context and specifying a reply schema  
conversation <- llm_message("Describe the figures in this document in detail. 
                            Use at least a hundred words per figure") |>
  chat(gemini(.fileid = example_upload$name),.json_schema = figure_schema)

conversation |> get_reply_data()

For the same dual-column paper, this approach generated an excellent list of all figures and tables, complete with detailed descriptions—far superior to plain text extraction from PDFs.

I plan to write an article on this Gemini-specific feature in the next 2–3 weeks, covering its support for audio and video uploads (but perhaps mentioning that its really good for pdfs too). Perhaps I will add a section to the pdf-article on Gemini. Until then, you can check out the documentation for gemini_chat(), gemini_upload_file(), and gemini_delete_file().

If you come across any packages or methods that could improve PDF content extraction in llm_message(), I’d love to hear about them.

[JiaZhang42]

Thank you so much for the reply! I'm looking forward to the Gemini article.

Previously I had to answer structured questions from a large number of PDFs (research papers). My way was to preprocess the text and feed it into the LLM APIs. The tables & figures were less important in that task, but I had to get the text in the right sequence so that LLM could understand them. My imperfect workaround was to

1. determine if the PDF needs OCR: If the total number of words extracted from the first 5 pages is less than a threshold, or if the extraction contains garbled text (the proportion of common or stop words is less than a threshold), use pdftools::pdf_ocr_text() to extract.
2. If it doesn't need OCR, use tabulapdf::extract_text() to extract. tabulapdf handles multi-column PDFs well in general. Though it's possible to insert a footnote within a paragraph of the main body, it doesn't affect the general understanding much. However, the current version of it has a bug that results in a fatal error when the PDF contains special characters like $\bar{x}$ or $\hat{x}$. I have to wrap the extraction in a separate R session so that it won't kill the main session.

I'll check pdfminer or marker out to see if they provide more flexible extraction than tabulapdf (e.g., can detect the footnotes or headers) and consider integrating them into the workflow through reticulate.

[edubruell]

Hey, I found something that might be interesting for pdf-use-cases. This is olmo on Huggingface. The comparisons in their white-paper to other pdf extraction methods look really good. Once they have a GGUF version on huggingface, it should be possible to install this model with just ollama_download_model() and test it with the pdf_page_batch().

[JiaZhang42]

I'll check it out. Thanks a lot!