This page is the dashboard and blog for the whole document-extraction project — not just the real-estate data below. The general pipeline (Gemini + Claude, see the homepage) is still in development, so there's no versioned accuracy history to show yet. What exists right now is one complete, verified case study, and this page is both the proof of that work and the methodology behind it.
If you read one page on this site, read this one — it explains where every number below comes from, how it's computed, and, deliberately, what it refuses to calculate.
There's no versioned accuracy benchmark yet — the general extraction pipeline hasn't been built (Phase 1 on the roadmap). Instead of describing that as though it existed, here is what has actually been done and measured, live from the database behind this site, not a snapshot:
35
projects covered
4,343
verified transactions
96.8%
field accuracy (last measured)
79.18%
extraction completeness
Project/transaction counts computed live from the database at request time. Accuracy and completeness are measured by hand (see below), not recomputed on every page load — see the "Extraction completeness" section further down for how and when.
Blog
Published 2026-09-10
Before writing a general-purpose extraction pipeline, this project tested the underlying approach on something concrete and unglamorous: 83 real PDF and Excel files published by the Hanoi Department of Construction (Sở Xây dựng Hà Nội), listing foreign-buyer apartment purchases. Government registries, no consistent format, real defects — corrupted ZIP containers inside .xlsx files, shifted and merged spreadsheet cells, mixed date formats, and in a few PDFs, two overlapping text layers.
.xlsx archives. An .xlsx file is a ZIP container; some had one damaged internal entry out of six, and standard libraries refused to open the whole file over a single bad entry even though the other sheets were physically intact. Fixed by reading the ZIP entries directly and salvaging what was readable, logging honestly what wasn't."hàn" (South Korea) inside "thành" (city), "áo" (Austria) inside "báo cáo" (report). The same class of bug showed up three times in different files before it was fixed properly: a closed dictionary with word-boundary checks, plus a dedicated trap fixture that must match zero times on any future change.This is a live, working system, not a static write-up — the same data is browsable on a live aggregated dashboard →, and the day-by-day engineering log is at /changelog. The rest of this page is the detailed methodology behind these numbers.
Every transaction record on this site originates from a foreign-buyer purchase list published by the Hanoi Department of Construction (Sở Xây dựng Hà Nội, soxaydung.hanoi.gov.vn). These lists are official public disclosures: under Vietnamese housing regulations, developers report foreign purchases to the provincial construction authority, and the authority periodically publishes the consolidated lists as part of its foreign ownership quota (30% of units per building) oversight duty. Publishing an aggregated, source-linked version of already-public government records is the basis on which this project exists.
Coverage today: Hanoi only. The codebase contains scaffolding referencing Ho Chi Minh City and Da Nang (seepipeline/spiders/), but no data from those cities has ever been loaded — they are unused code, not a live source. Claims elsewhere on earlier versions of this site implying multi-city or real-time coverage were incorrect and have been removed.
The dataset behind this site was built from a batch of PDF and Excel files already published on the Hanoi Department of Construction site, processed bypipeline/etl/hanoi_bulk_processor.py. This is a manual, on-demand pipeline run — not a scheduled scraper. The server cron only runs a weekly database backup; there is no automated crawl.
A Scrapy-based spider (pipeline/spiders/hanoi.py) exists in the repository but has never been run against production (no run logs, no proxy credentials configured). Its own code comments note that Hanoi's portal blocks non-Vietnamese IPs and would require residential proxies to reach — which would mean bypassing an access restriction. That conflicts with this project's rule to use only open sources without circumventing protections, so that spider is documented here as unused, unverified, and not endorsed for activation without a deliberate decision to either respect the IP block (and find another lawful access path) or not collect from that endpoint at all.
Any future automated collection from this or any source is expected to keep reasonable delays between requests, identify itself, and respect the source's published access rules — the same standard this document holds the current batch process to, even though today there is no live crawler to apply it to.
We measure our own extraction losses and publish them here — that is a stronger claim than staying quiet about them, not a weaker one.
How it's measured. Vietnamese foreign-buyer publications number their rows in order in an "STT" (số thứ tự) column — the document itself states how many rows it contains, via its highest row number. For every source file (or, for spreadsheets with multiple independent lists, every sheet) we compare the row numbers we actually extracted against that maximum and record exactly which numbers are missing. This is a measurement, not an estimate: it tells us precisely how many rows and which ones, not a guess.
79.18%
Overall extraction completeness across reconcilable source files, measured 24 Aug 2026. Calculated as (rows extracted ÷ highest source row number), summed across all reconcilable files.
Completeness is not correctness. The number above measures how many rows we extracted out of how many the source document says exist — it says nothing about whether the values inside those rows (date, contract number, nationality, project) are right. We measure that separately, by hand: a random sample of extracted rows checked one field at a time against the original document image, before looking at what the database says for that row.
96.8%
Field-level accuracy on a random 50-record sample (seed 42), measured 24 Aug 2026. Progression on the same sample as parser fixes landed: 84.8% → 92.4% → 96.8%. Not recomputed automatically — a fresh figure needs a fresh random sample and a fresh manual read, not a formula. See how this is measured.
Not every file can be reconciled this way. A file (or spreadsheet sheet) counts as reconcilable only when we can confidently identify an STT column at all — a small number of files use a layout where that isn't structurally certain, and those are excluded from the percentage above rather than assumed complete or incomplete.
Known classes of loss (full breakdown with row counts per class, as last measured, in the engineering log): rows dropped because a required field — usually the address — couldn't be located on that row; rows on pages where the source table structure wasn't recognized; rows correctly excluded because the underlying table concerned a resale to a third party (personal data of a non-buyer, out of scope for this dataset by design) rather than a purchase; and a structural finding specific to this measurement — some files carry a quota-summary row (total apartment count for the building) that briefly got mistaken for the row-count ceiling itself, inflating the apparent loss for a handful of files, corrected once found.
What's being done about it. Fixes are applied one loss class at a time, largest first, each verified against a fixed reference set of files before and after the change so a fix can't silently drop rows elsewhere while gaining them here. Scanned pages without any extractable text layer are a separate, harder case — today's source files all have a text layer, so that ceiling is currently zero rows for this dataset, but it won't stay that way as new sources are added.
All figures on /insights and the public API (GET /api/insights, full spec at /openapi.yaml) are computed live from the transactions and projects tables at request time — nothing is a stale precomputed snapshot.
contract_date. Rows with no parseable date are excluded, not assigned to a month.buyer_nationality, overall and broken out by year of contract_date.district, extracted from the transaction's address text where a Vietnamese administrative district name was recognizable.In the data
Not in the data
contract_date values exist inside later filings, but pre-2024 coverage itself is not complete)Vietnamese housing law caps foreign ownership at 30% of units per building. It would be natural to want a "X% of quota used" figure for each project. This site deliberately does not compute one.
The reason is simple: that calculation needs the total unit count per building as its denominator, and there is no reliable public source for that number across our project list. A small number of individual filings happen to mention a building's total unit count, but most don't, and treating a partial, inconsistent field as if it were a complete dataset would produce numbers that look authoritative while being wrong for most projects. A percentage computed from an unreliable denominator is worse than no percentage — it invites a decision on data we don't actually have.
If a trustworthy, complete source for total-unit counts becomes available, this is the page that would be updated to say so — and only then would a quota-fill figure be added.