How we built AirCase

AirCase is a free casebook of US aviation accident investigations by the NTSB (National Transportation Safety Board), from 2008 to September 2026. It has keyword search with filters, case pages, a dashboard, a compare view and an AI search.

I built it for three reasons: to learn how to build retrieval and LLM apps properly, to learn Langfuse on a real app, and to see how far AI coding agents can go when I act only as the product owner.

"We" in the title means me and two agents: Claude Code from Anthropic as the lead agent, and Codex from OpenAI for the first design, the reviews and later some of the code. We built it within two weeks.

The data #

Each month the NTSB publishes its aviation data from 2008 onward as one file, avall.zip, with a Microsoft Access database inside. We read it with mdbtools. The latest file has 31,453 events, and 25,510 of them are US records that you can browse in AirCase. The AI search uses 24,414 of them: the ones that our rules treat as completed investigations with a probable cause.

The cases live in PostgreSQL 16. Each import is a new "generation", and going live is one atomic database switch. Elasticsearch 9 holds two indexes: one for the cases, and one for the narrative passages with their text and vectors. Everything is TypeScript on Node 24: a Fastify 5 API, a React 19 web app and Zod schemas shared by both. It runs in Docker Compose on my own computer. Visitors reach it only through a Cloudflare Tunnel, so the app needs no public port. Dev and live have separate databases, index names and Langfuse projects.

Search: keywords and meaning #

Keyword search uses BM25, the classic keyword ranking: a case scores higher when it contains your words, and a rare word counts more than a common one. We added aviation synonyms, so "carb ice" finds "carburetor icing".

Meaning search uses embeddings. An embedding model turns a text into a long list of numbers, a vector. Texts with a similar meaning get similar vectors, so we can find passages about the same thing even when the words differ. We cut 69,023 narrative sections into 91,898 passages and embedded them with OpenAI's text-embedding-3-small. The full pass was 28.5 million input tokens and cost about $0.57.

The AI search runs three channels on your question: keywords over the passages, vector similarity over the passages (kNN, the passages whose vectors are nearest to the question's vector), and keywords over the case fields such as the probable cause. Each channel returns its top 200 hits, collapsed to cases. The lists are then merged with Reciprocal Rank Fusion (RRF). Each list gives a case 1 divided by (60 + rank) points, so a case near the top of several lists collects the most. We wrote the fusion in TypeScript, because Elasticsearch's built-in RRF needs a paid license.

AI search shows evidence, not answers #

The original plan was the usual one: Claude writes a short summary with citations, and a second "checker" step checks every claim against the sources before anything is shown. Real tests killed it.

• First paid summaries. Both were withheld. One failed a citation check. In the other, the checker flagged all 35 claims, and about 80% of the flags were wrong.
• After calibration, the best checker still withheld the summary in 17 of 21 test runs, all on summaries that we had labelled correct. One doubtful claim out of 20 to 30 blocks the whole text.
• Showing only the checked sentences let 2 of 171 planted errors through: quotes whose meaning changes without the sentence before them.
• Stricter rules let none of 158 planted errors through, but at first kept only 24% of the correct statements. Later fixes and a new check of our labels changed these numbers, but they also found new errors.

So I decided to launch the AI search as an evidence view: each result is a real investigation with its probable cause and whole-paragraph excerpts from the NTSB text, ranked and highlighted. The AI suggests search settings and orders the results, but it writes no summary and no answer. What you read is NTSB text. The order and the highlights don't prove that every part of your question is true for a case, and the page says so. Showing the evidence is honest, cheap and fast. Summaries stay off. Maybe later.

When you ask a question, Claude Haiku 5.5 reads it once. It suggests filters (a US state, years, an aircraft category) and search terms. Filters are the biggest win in the AI search. In our tests, applying the suggested filters more than doubled how many of the expected cases appear in the first 20 results, from 30% to 71%. So when your question names a US state for the accidents, AirCase now applies it on its own. That is harder than it sounds. In "aircraft maintained within Alaska", Alaska is not where the accident happened, and the hand-written rules we tried first kept missing wordings like that. Now two locks must agree. Simple rules accept only a state that the question spells out, and they step aside for anything tricky, such as two separate requests or "anywhere but Texas". Then a second, small Claude Haiku call says what the words describe, the accident's location or something else, like where the aircraft was maintained, and it must quote them. The state shows on the page as "from your question", and taking it out is free. Years and aircraft types stay suggestions for now, like everything else the AI suggests.

After the three channels and the merge, Claude Haiku reorders the leading 50 cases. It reads short excerpts of each case and grades them. Only the order inside those 50 can change. If it fails, the merged order stays.

My first live searches took 5 to 10 seconds and cost about $0.003 each. Daily and monthly budgets cap the spending. When a question is outside the archive or asks for flying advice, you still get the normal results, with a fixed note above them. On one test set the notes caught 70 of 77 such questions and appeared once by mistake. A missing note is no guarantee. Every result carries a "research only" reminder, and visitors can mark a search Useful or Not useful.

Prompt injection and abuse #

Anyone can type anything into the AI search, and the NTSB text that Claude reads could hide instructions too. So we treat both as untrusted data and give the model as little power as possible.

• The model can't do much. Claude Haiku gets no tools, no browsing and no database access. Each step must answer in a strict format that the server checks. Reading the question returns a question type, filter values from fixed lists and search terms of at most 300 characters. The reranker returns only a grade from 0 to 3 for each case. Anything else counts as a failure, and the search goes on without that step.
• Questions and case text are data. They go to the model inside tags, look-alike tags are defused, and the prompts tell the model to ignore any instructions inside.
• The worst case is small. The search runs on your question as typed. The only filter AirCase adds by itself is a US state that your question spells out and the second check ties to the accident, and the model can only drop it, never invent one. Other suggestions apply only when you click them, and the only model-written text on the page is the suggested search terms. A planted instruction in a narrative could at most change the order inside the leading 50, because narratives never reach the steps that choose filters.
• Limits. Without an account you get 10 AI searches a day, and each one must pass Cloudflare Turnstile, a check against bots. There are also limits per minute, per network and for the whole site, and a question can have at most 1,500 characters.
• Budgets. Each AI search reserves its worst-case cost before any model call, and the real cost goes into a spend ledger. The monthly budget is paced into a daily allowance of at most $3, and searches without an account get at most a quarter of it. When the money runs out, the AI search pauses and keyword search keeps working.
• One way in. The app has no public port and refuses any connection that doesn't come through the Cloudflare Tunnel. It sends strict security headers, and the admin page answers only to my own accounts.

Tests replay injection attempts and check that even a hostile answer from the model leaves the results unchanged. No test can prove that a model always ignores a planted instruction, so the design limits what one could change.

Testing search without an aviation expert #

I am not an aviation expert, and I had nobody to label results for me. So we built two references. Both are imperfect.

• Track C has 43 questions and uses the facts that NTSB investigators coded for each case. A question about carburetor icing, for example, expects the cases coded with carburetor icing. A case without the code can still fit the question, so this is a reference, not the full truth.
• Track N uses two AI judges from different providers, Anthropic and OpenAI. They rate how well a case's narrative fits a question, and they must quote evidence. A third step settles some of their disagreements. Together they got 102 of 103 test items right, then judged 3,893 case-question pairs, with 447 left unresolved. Track N cost $11.94 in total.

Every score is labelled "coded-reference" or "model-judged", never "expert".

Our main measure is narrow recall@20. For each question with 1 to 20 coded cases, it is the share of those cases found in the first 20 results, averaged over the questions.

Every test question names its limits in words: a US state, a range of years or an aircraft category. Words alone can't make the search keep only "Hawaii" or "2023 to 2026", but filters can. The AI suggests those filters, and one click on "Apply these filters" runs the search again with them. So we measured each step in one run on the current data, on the 12 narrow development questions, three times each:

Elasticsearch alone means keywords and embeddings on the question as typed. The AI reordering is Claude Haiku grading the leading 50 cases.

One click on the AI's filters roughly doubles what the search finds. On 24 fresh narrow questions that we never tuned anything on, the same three steps scored 22%, 24% and 45%. Across both sets, the AI suggested filters in 58 of the 60 searches on narrow questions, and 52 times they were exactly the right ones. In the other eight it added an extra filter, missed the state or suggested none.

On the development questions, every question improved. The reordering also helps more once the filters are in: without it, the filtered search scored 51%. Broad questions gain too: on the development set, the coded precision in the first 10 results went from 16% to 27% with the reordering, and to 44% with the filters.

The fresh questions are harder: most have only one or two coded cases, so a single case decides a question's score. We chose the reordering on the development questions, and on the fresh ones it added only 2 points by itself.

Filters without the click #

The click works, but most people never click. So we tested whether AirCase can apply filters by itself without applying wrong ones. Six AI writers wrote 1,644 fresh questions, and a second AI labeler checked every expected answer on its own. About 60% are traps that name a state, years or an aircraft type that must not become a filter, like "accidents involving pilots trained in Arizona" or "crashes of aircraft built in 1975". Codex from OpenAI wrote 150 more, aimed at our rules and our prompt. We fixed problems only on a separate set of practice questions. The 1,644 fresh questions decided, by a rule we wrote down before the first run.

The one wrong year was "nothing after 2010", read as 2011 and later. That is exactly the kind of mistake we don't want to make silently, so years wait. In a new run, the state alone, applied by itself, raised recall@20 from 30% to 60% on the development questions and from 19% to 48% on the fresh ones. One click on all the suggested filters reached 70% and 51% in the same run. The price is caution: the checks apply a state in about half of the questions that name one, and leave the rest as a suggestion. The simple rules are not perfect either. Codex from OpenAI still finds rare wordings that get past them, like "accidents in Boston and also in Ohio": the rules let Ohio through alone, and only the AI check can stop it. If a wrong state ever gets through, it shows on the page, and taking it out is free. Because of wordings like this, Codex's last review did not clear the filter, so turning it on was my decision.

Earlier, the same test with each question's correct filters helped us choose the second retrieval version. In that setting it raised narrow recall@20 from 62% to 72%, and the AI judges also preferred its first six results.

A fresh test of a top-100 reranking gained 8.1 points, but it did not pass our confirmation rule. We could not build enough fresh questions to confirm the top-50 version, so turning reranking on was my product decision, not a proven win. I made it because the reordered results are nicer to use: the cases that best fit your question come first, so you read less before you find what you need.

The target is 90%, so even with the filters we are not there yet. The evaluations cost about $28 in total.

Langfuse on a real app #

When tracing works, each AI search becomes one trace in Langfuse: the record of one request with all its steps. Ours has a request span and three steps: scope (Claude Haiku reads the question), retrieve (Elasticsearch, with the OpenAI embed call inside) and rerank (Claude Haiku). When the question names a state, a fourth step, attribution, checks it. Each model call records the model name, tokens, cost and timing. The retrieval step records its timing, version and counts. If tracing fails, the search still works.

• Prompts. The two prompts are versioned in Langfuse and linked to the steps that used them.
• Scores. Each Useful or Not useful rating is a score, visitor_feedback, on its trace.
• Datasets and experiments. The ranking evaluation lives in Langfuse as datasets, with four experiment runs side by side. They cost $0.21.

Live traces hold no question text and no results, only allowlisted numbers and names: timings, counts, models and costs. That is for privacy, and it is enough to debug speed and cost.

Fresh data every month #

Once a day the importer looks at the NTSB download page. When a new avall.zip appears, dev goes first. The importer:

1. Downloads it, with size and time limits.
2. Extracts it in a sandbox with no network.
3. Compares it with the serving data. The counts of new, changed and removed cases are only for information. The one guard is size: a file with less than 90% of the cases we serve is treated as damaged and is not imported.
4. Loads it and builds new search indexes, embedding only new or changed text, with a cap of $2 per file in each environment.
5. Probes the new indexes.
6. Publishes them with one atomic database switch.
7. Checks the running app. It must show the new data within 5 minutes and then pass two checks 10 minutes apart.

Live runs the same steps only after dev has passed, with the same file, the same code and the same app build. If a run fails before the switch, the old data keeps serving. If a check fails after the switch, the importer switches back to the previous data by itself and checks the site again.

Nobody approves anything. A temporary problem, like the NTSB site being down, is simply retried the next day. The admin page shows an alert only when something is wrong: "Delayed" after temporary failures on 3 days, and "Needs a fix" when the importer cannot go on by itself, after 7 days of failures, or when no import check has run for 36 hours.

The first night showed why the checks matter. A missing await made the dev run fail before its switch. Live was skipped and left untouched. The bug was fixed, with tests, the same night.

A private admin page shows usage, costs against the budgets, search ratings and the import status.

How we worked #

I was the product owner: I made the decisions and approved changes. I proposed the idea and the main tech stack, such as Elasticsearch, OpenAI embeddings and Langfuse, and then we discussed together what the best approach would be. The agents wrote the code.

Claude Code from Anthropic was the lead agent. It ran Claude Opus 5.5, first at extra high effort and then at max effort for most of the work. It wrote the interface design and the plan with me, then split the work into tasks. Each task was done by a fresh sub-agent and checked by a fresh reviewer agent until the review was clean. The sub-agents were mostly Opus 5.5 and Sonnet 5.5, at their default effort.

Codex from OpenAI ran at max effort. It wrote the first design documents with me and reviewed every step that touches data, from the extraction to the embeddings, retrieval and the AI prompts. It also reviewed security before releases. Later it wrote server code too: non-UI code went to Codex, the UI to Claude. So models from two AI companies reviewed each other's work.

Much of the work ran while I slept, under three rules: Codex clears each data step before it runs on real data, paid AI calls go only through budget gates and a spend ledger, and risky features stay off until I decide.

Every release runs the full test suite: about 5,400 unit and web tests, 831 integration tests against real local PostgreSQL and Elasticsearch with separate test data, and 110 browser tests, 82 against a mocked API and 28 against a real server. The importer has its own drills in throwaway containers.

What I learned #

• Show evidence, don't generate. For safety material, a wrong sentence is worse than no sentence.
• Suggest first, act only with two locks. The model reads a question well, but not well enough to change the search on its own. Hand-written rules alone kept leaking, and the AI alone was too eager. Rules that accept only what the question spells out, plus a separate AI check that only says what the words describe, made no wrong state filter on 348 fresh trap questions.
• Know what a number measures. Our 30% measured the search before you click the suggested filters. After that one click it was 71%. Both are test results on development questions, not measures of real searches.
• Models from two companies catch more. Codex's security reviews kept the first automatic state filter off, then pushed the new checks through round after round of stricter rules, and reviews caught misleading wording on the admin page before release. Codex also checked this post and found 21 places where a number or a claim needed more context.
• Agents forget. When a long session is compacted, the agent loses its memory. The lead agent keeps a progress ledger file and trusts that file and git more than its own memory.
• Small things bite. Google Search Console couldn't fetch one sitemap, because sitemaps shared the API's rate limits. They got their own.
• Trust the source, alert when it breaks. Review gates on good data would have cost me time every month. Now the importer only guards against a broken file and shows an alert when something is wrong.

Try it #

AirCase is at aircase.app. Search for something you know, ask the AI search a plain question, and click the rating under the result. Feedback is welcome on LinkedIn.