Search engines are essential to research across all fields.
With the evolving landscape of Artificial Intelligence, search engines are also evolving at a rapid pace. Just to name a few innovations from the past decade, we have witnessed the shift from keyword-matching to semantic matching using language models, we are seeing chatbots (with retrieval augmented generation) replacing the traditional search engine completely, and we are also seeing the first few examples of deep research agents.
In this rapidly evolving landscape, professionals increasingly find it difficult to keep up with the latest advancements.
In addition, many AI based search technologies are designed primarily around the needs and workflows of computer science and engineering, and they provide more immediate benefits in these disciplines than in fields such as the humanities, where research practices, data organization, and information access follow vastly different conventions.
In discussing the search experience with researchers in the History Department at CMU, we came across an interesting challenge.
The CMU historians are currently studying the life of Dr. Herbert Simon. Simon (June 15, 1916 – February 9, 2001) was a renowned CMU computer scientist with dual contributions to the fields of Computer Science and Economics. His primary research interest was decision-making within organizations and he is best known for the theories of “bounded rationality” and “satisficing”. He is the only person in the world who has won both the ACM Turing Award and the Nobel Prize in Economics. The researchers in the History Department at CMU were interested in understanding what reasoning process led to the discoveries that Simon made.
When Simon passed away in 2001, his family generously donated much of the papers he left behind to the CMU library. These papers include handwritten notes and diaries, school records and course materials, typewritten lectures and reports, personal correspondence, and more. The library was able to digitize over 37,000 of these documents. Within the physical archive, documents are grouped into folders and boxes of folders. Following standard archival practices, the digital database largely retains the structure of the physical storeroom. While a search engine is available, it only is capable of basic string matching using text extracted with an outdated Optical Character Recognition (OCR) library, Tesseract, which was never designed to handle complex handwritten documents containing math or diagrams.
The challenge posed to us by our historian collaborators was to build a less cumbersome search experience, one that could handle high-level semantic queries framed in natural language such as:
- what was herb simon reading around the time he wrote models of discovery?
- Find me handwritten documents that contain diagrams drawn by Simon. What themes are there across these diagrams?
- in what ways is simon’s 1998 article on learning not supported by the other documents in this collection?
With queries like these, the existing search engine returns no results at all.
Therefore, we set on a path to (1) run the corpus through a modern AI-based digitization pipeline and (2) build a new search engine using more recent Search Engine techniques. The rest of this post details how we accomplished these goals and our learnings in the hopes that it can be useful for someone doing something similar in the future.
OCR with Google’s Gemini
The first task for us was a better OCR. We decided to use Gemini because Gemini models are some of the strongest multimodal models available, and also because Google was kind enough to sponsor our group’s research with Gemini API credits. We also played around a bit with Claude and found it works about equally well.
Setting up the task
We used the following prompt for the Gemini model:
You are an OCR system that extracts text from single page image files. You will be given a page, and you need to extract the text from it. If there is an image in the page, please provide a description of the image if possible. The image description should be wrapped in <image> tags like this: <image>description of the image</image>. If there is a diagram in the document, please provide a description of the diagram if possible. The diagram description should be wrapped in <diagram> tags like this: <diagram>description of the diagram</diagram>. Math equations should be wrapped in <math> tags like this: <math>math equations</math>. Please provide accurate and complete text extraction. My career depends on your performance, so please do your best.
Notice that we asked the model to describe the images and diagrams to provide better searchability. We also asked the model to extract mathematical equations because after talking to the researchers, we thought this might be useful to have in the future. The Gemini API supports requesting structured output, and we used the following Pydantic model for the outputs:
class ImageText(BaseModel):
content: str = Field(description="The extracted text from the image.")
date: Optional[datetime] = Field(description="The date on which the page was created if mentioned in the page.")
has_picture: bool = Field(description="Whether the page contains a picture or not.")
has_diagram: bool = Field(description="Whether the page contains a diagram or not.")
has_math: bool = Field(description="Whether the page contains math equations or not.")
Each page of a document was provided to us as a tiff image file. We decided to do OCR page-by-page. Across all the documents, there were 136,807 pages, each of which we made 1 Gemini API call for.
Picking a model and determining cost
Choice of model is pretty important—many libraries and archives operate on shoestring budgets, so we didn’t want to just thoughtlessly go with the biggest model available. We ended up going with gemini-3-flash-preview, which in our initial experiments seemed to offer a good compromise between quality and cost. However, we think even a cheaper model like gemini-3.1-flash-lite could do pretty well. In the table below, we show a few examples of pages from the Simon archives, and their digitization from a few different models.
In total, we spent about $350 dollars. The cost for gemini-3-flash-preview is $0.50 per million input tokens and $3.00 per million output tokens.
As a note, at the time, Google’s Vertex AI platform also provided a Document Intelligence service that was capable of OCR, but we did not find this OCR to be as good as that of the Gemini API. We were impressed that the Gemini API was easily able to read handwritten notes that were hardly readable by us.
Some challenges
While running the OCR script, we found a few more challenges. The first was that sometimes the Gemini API returned invalid text or no response. This might have been because of network errors or API overload; however, this was easily resolved using the retry logic from the tenacity python package. The second problem we encountered was that the Gemini API refused to perform OCR for certain documents which it thought would infringe copyright. This, although generally a positive feature, hindered us because we had permission to use the documents (but no way to tell the API that). Because of this, we were unable to perform OCR for ~100 documents.
We also ran into challenges setting up the Batch API, which would have halved the cost of our total Gemini API bill. We ran into a specific bug when using optional fields with Pydantic. Not all documents have a date in them, but for the ones that did, it was important for our collaborators that we could extract these data. This meant our optional date field was necessarily optional and we could not change this. We hope that Google will soon fix the bugs in the Batch API so that those wanting to reproduce our pipeline can take advantage of discounted batch prices.
Examples
Below you can see what the OCR looks like for three example documents, using three different Gemini models at varying pricepoints.
|
|
$$\int_{0}^{t}m\frac{dT}{dt}dt=-\int_{0}^{t}P\frac{dV}{dt}dt+\int_{0}^{t}PT_{0}dt-\int_{0}^{t}PTdt$$
m$$\frac{dT}{dt}=-P\frac{dV}{dt}+PT_{0}-PT$$ (1) $$m\frac{dT}{dt}+PT=-P\frac{dV}{dt}+PT_{0}$$ PVT, $$T_{0}$$
(2) $$PV=mT$$ $$P\frac{dV}{dt}+V\frac{dP}{dt}=m\frac{dT}{dt}$$ $$PVT_{0}$$ (3)
$$P\frac{dV}{dt}+V\frac{dP}{dt}+\frac{\rho}{m}PV=-P\frac{dV}{dt}+PT_{0}$$
$$2P\frac{dV}{dt}+V\frac{dP}{dt}+\frac{\rho}{m}PV=\rho T_{0}$$ $$m[T(1)-T(0)]+\rho
T(0)=-P(0)[V(1)-V(0)]+\rho T_{0}$$ $$P(0)V(0)=mT(0)$$ P(1) T(1) V(1) P(0) T(0) V(0) P T V X X X X X X X
|
Final processing and Vertex AI Search
Once we had the documents’ OCR text, we merged the data with metadata that we were provided. This metadata had document details such as the box number, folder number and the type of the document. We created a single txt file for each document page and moved the txt files into a Google Cloud Storage bucket. Then we created a Search Data Store by importing the data from the Cloud Storage bucket. We chose to maintain granularity of the Datastore at page level to be able to identify relevant pages in very long documents. Finally, we created a search engine using Google’s Vertex AI Search (which in typical Google style, has since been rebranded to “Agent Search”).
Once created, we deployed the Search Engine using Firebase. The cool thing about these tools is that they allow you to deploy a Google-like search interface with basically no programming (aside from munging data into the expected format).
Vertex AI Search supports some useful features such as AI-generated summaries of the search results and the ability to ask follow-up questions of the summaries (this is enabled by turning on “search with follow-ups” in the UI). One caveat is that these RAG features still seem a bit experimental, and sometimes the app refuses to produce a summary even when there are search results. We’re not sure yet if our historian collaborators will find the AI summaries helpful, but we appreciated having them during our own testing.
Vertex AI Search also supports passing in an instruction to tailor the style of the generated summaries. We played around with this quite a bit. Here is the instruction we ended up using:
Given the conversation between a user and a helpful assistant and some search results, create a final answer for the assistant. The answer should use all relevant information from the search results, not introduce any additional information, and use exactly the same words as the search results when possible. The assistant’s answer should be no more than 20 sentences. Your job is to provide well-cited answers to questions about Herbert Simon. You should communicate like an academic. You should not make statements that can’t be verified by the retrieved items.
Remember: the data store you are retrieving from is the Herb Simon archive, located at Carnegie Mellon University. The collection includes scientific papers by Simon and others; project reports and research proposals; lecture materials, book and paper drafts, publications and journal article reprints; personal papers and awards; external correspondence, CMU interoffice Memoranda, e-mail, student papers, and research materials.
While the bulk of the collection has been digitized, some material, such as incoming correspondence and computer program print outs, are not available online due to copyright or privacy concerns. In some cases, as noted below, entire series have not been digitized. In other cases, individual documents or sets of documents may not be available in the datastore.
Beyond setting the style, our goal with this was to make sure the AI understood that all documents in the dataset were connected to Herb Simon, even if they never mentioned his name. Overall though, we don’t think this prompt changed the generations much, so this was a bit of a failed effort.
Finally, we also integrated 8 metadata filters for researchers to have more control over their search queries. These filters allow the user to quickly narrow down the search results.
| Filter Name |
Source |
| Box Number |
Document Metadata |
| Genre |
Document Metadata |
| Has Diagram? |
Gemini OCR |
| Has Picture? |
Gemini OCR |
| Series Title |
Document Metadata |
| Year |
Gemini OCR |
| Year Lower |
Document Metadata |
| Year Upper |
Document Metadata |
Our observations
You can access our demo here. Here are some of the things we learned by building out this demo.
Queries that impressed us
The search engine does well for queries that are well-supported by Simon’s personal writings. For example, the AI was able to generate coherent and well-cited answers to How did his experience of WW2 affect Simon's thinking later in life? and Did Simon's views on the impact of AI chess players change over the course of his life?. We were also especially impressed when the search engine was able to retrieve documents with little word overlap with our query. For example, in response to How did school and departmental politics at CMU get on Simon's nerves?, the search engine retrieved a magazine clipping critiquing university cost cutting measures, an email in which Simon complains about too-high teaching loads and the challenges facing a highly interdisplinary departnment, and another email where he deplores university customs gradually morphing into inflexible policies. It seems university sore points have not changed much over the decades.
Where did it fail?
This solution isn’t perfect. We humans have knowledge about the data that Google’s search engine doesn’t have. For example, we know that if some book exists in the dataset, then it was physically in Simon’s possession before he passed away. Even if the book never mentions Simon anywhere, it is connected to him. Looking back at the query, what was herb simon reading around the time he wrote models of discovery?, the AI retrieves correspondence where Simon wrote to
David Klahr (a fellow CMU faculty member), “I have now had a chance to read your book on discovery, and I like it very much.” This seems like a promising lead, until one notices that this letter is dated to May 1998, decades after Simon’s own book was published. The search results seem to entirely miss books and documents that are dated to around Simon’s book’s date. Even when we revise the query to be , what was herb simon reading around the time he wrote models of discovery (published in 1977)?, the search engine mostly returns texts Simon wrote, not texts he was reading.
The search engine and RAG summarization layer also very often fail to understand what parts of the query are important to historians. For example, a faculty member in our department, Bob Frederking, asked us if he was in the dataset. We queried What was the relationship between Herb Simon and Robert Frederking? The RAG summary unhelpfully responds:
The provided sources do not contain any information about a relationship between Herbert Simon and Robert Frederking. However, the sources do indicate that Herbert Simon corresponded with several individuals named Robert, including Robert Lindsay, Robert S. Sullivan, and Robert Mehrabian.
This response is bad, both because a historian interested in Robert Frederking is unlikely to be interested in other individuals with first name Robert, and also because Robert Frederking does actually appear in at least two documents (documents which are successfully retrieved when we query with just Robert Frederking). The promise of AI is that it might obviate the need for painstaking query construction, but the technology is still far from achieving this goal.
The Search Engine API has a feature to finetune the base model on “company-specific queries,” and we are curious whether this feature could be used in the future to improve how the search engine handles these sorts of queries.
The challenge, of course, as with any finetuning, is to collect enough examples of what realistic queries look like.
Libraries need systems they can rely on
Finally, we would like to point out how our reliance on Google services, especially for the search engine deployment is problematic. Many libraries and archives have an aversion to software they have limited control over. Google especially has something of a reputation for killing popular products and APIs. Our colleagues have asked us if there is a good open-source alternative to Vertex AI Search, and this is something we’d like to explore further. Alternatives include building a search engine using ElasticSearch or using an off-the-shelf embedding model with a react frontend.
Even in the few months we have had our Simon Search Engine up running, we have noticed that behaviour can be unpredictable. Queries that return many results on one day return no results a month later. Our assumption is Google is changing the underlying models under the hood. This sort of instability decreases user trust in applications. Just as in STEM fields, historians need their research process to be reproducible; another historian ought to be able to replicate the research carried out. If an AI agent becomes part of the research process, it needs to give the same response each time it is queried.
Finally, cost is also a significant concern for most libraries. In-house search solutions are attractive because they have predictable costs. Low and predictable costs are crucial for broad deployment.
Acknowledgements
We would like to thank Professor Christopher Phillips at CMU for his involvement at every stage in this project and his feedback on this blog post. We would also like to thank Vincent Sha, Associate Dean of IT and Operations at CMU for his feedback on the search engine and suggestions for search queries to try out.