Tukuk-OSBefore I touched Meilisearch, I built my own inverted index. I wanted to understand what happens inside the black box before buying it. That small project — one JavaScript file, a few hundred lines — became the biggest stepping stone in building Tukuk-OS.
What Is an Inverted Index?
Naive search means reading every document and looking for the word you want. With a hundred documents it might work. With a hundred thousand it would take an unreasonable amount of time.
An inverted index reverses the work: from documents to words. Every unique word maps to a list of document IDs where it appears. Search for "meilisearch" and you instantly get the correct list of documents without reading a single document in full.
My First Design
My first design was simple: one JavaScript object. Keys are words (normalised to lowercase and stripped of punctuation), values are lists of document IDs. When I add a document I split its text into words, filter out stop words (and, the, is, at), and add the ID to each list.
The first test felt extraordinary. Across a few hundred articles my search returned in a blink. I wrapped console.time around it and saw 0.2ms for a typical query. I finally understood why search engines feel fast — not because their servers are powerful, but because their data structures are right.
The Real Challenge: Being Smarter Than Keywords
A basic index only supports "find documents containing all these words". But real search engines must handle partial matches, prefer matches that appear in titles, and compute relevance scores.
I added three layers:
- Frequency scoring: a word that appears often in a short document matters more than in a long one.
- Position: a match in the title earns a higher score.
- Proximity: when two words appear near each other in the same sentence, that is a strong signal.
The third layer took longest but produced the biggest quality jump. It also taught me why you must stop effort at some point: computing proximity for every word pair in every document is too expensive, so I only do it for queries with two or three words.
How I Measured Index Quality
One problem I did not think about while building: how do I know it works well? Fast scores mean nothing if the returned results are irrelevant.
I wrote a set of test queries — ten questions with answers I knew should appear at the top. Every time I changed the scoring algorithm I ran that set and compared results. Several times my "improvements" made common results worse while improving rare ones — and the test set caught it before the change stuck.
This also taught me that search quality is subjective for some queries. "javascript" could mean tutorials, jobs, or news. For such queries I learned not to over-measure — sometimes a range of reasonable results is better than one forced "correct" answer.
The practice I eventually kept: every search system needs a hand-written set of reference queries. It is small (ten to twenty queries), cheap to run, and it is the only way to know whether your change improves the system or breaks it. I use the same set today when tuning Meilisearch settings for Tukuk-OS.
Where My Design Failed
My index failed in two ways. First, size: the JavaScript object held everything in memory, and as I added more documents memory usage grew unchecked. Second, features: no faceted search, no fuzzy matching, no autocomplete, and no index management.
I realised I was rebuilding an open source search engine that had taken teams years. That is when I decided to use Meilisearch — but with a deep understanding of what it does under the hood. When I tune filterableAttributes and searchableAttributes in Tukuk-OS, I know exactly why each field is placed there.
Lessons for Search Engine Builders
If you want to understand search engines, do not start with a library. Build the simplest one first: one object, one search function, one test. Then add simple ranking, then scoring, then filters. You will discover for yourself why each feature you create matters — and you will appreciate other people's search engines in a way that is impossible if you have never struggled to build your own.
The inverted index I wrote by hand still exists in my archive. Tukuk-OS does not use it, but it is the most valuable learning document I have ever produced.