What happens to this data and this system, dated, stated as numbers.
Every entry here is, in some sense, an admission of a mistake found in the data or the code that produces it. That's deliberate — a page that only ever says "improved things" wouldn't be believable. A system that finds and names its own errors, with a number attached, is. See /methodology for how those numbers are computed.
Applied the expanded personal-data filter (passport and ID-card terms) to the live database: 216 already-stored records from the 49 affected source files were marked no longer current, on top of the 71 tables already excluded going forward.
Extraction completeness measured against the current parser: 83.74% → 79.18% (see /methodology). A deliberate trade of coverage for third-party data protection, not a regression.
Found a parsing defect: on multi-column PDF page headers, text was read across columns instead of down them, and the resulting unreadable fragment was written into the address field in place of the confirmed project address carried from earlier pages.
Records no longer linked to a project: 1,054 → 163 (-891, about 80% of all unlinked records). Confirmed independent of the personal-data hole below — different file, different code path, zero overlap.
Found a hole in the personal-data check: a buyer surname with no digits and no ID-related label passed all three existing checks, because none of them ever asked whether a value sitting in the address field looked like an address at all.
99 live records had a buyer name in the address field — cleared (the name is not retained in any form) and marked no longer current. The check now includes a name-vs-address heuristic, with a regression test proving it catches this exact case if it recurs.
Measured extraction accuracy for the first time, not just completeness: 50 records picked at random, values read from the source document image before looking at what the database stored for that row, compared field by field.
84.8% of fields correct (212/250) before fixes. Full breakdown and methodology on this page above and in the linked report.
Fixed the two largest accuracy defect classes found by that measurement: one source template wrote contract dates with a 2-digit year our parser did not recognize at all, and six source files wrote the contract number with the project name embedded in a way that broke the existing pattern match.
Contract dates recovered: 927 → 64 missing. Contract numbers recovered: 346 → 82 missing. Re-measured accuracy on the same 50 records: 92.4% (231/250).
Replaced the record-versioning key: a content hash was being used both to identify a record and to detect when it changed, so fixing any field the hash was built from (like the date fixes above) silently orphaned the old row forever instead of superseding it — this had already happened three times.
New stable identifier (project + unit + contract number) decoupled from content-change detection, with a regression test for the exact failure mode. Found 57 groups (119 records) where more than one active record now shares an identifier — flagged, not auto-resolved.
Expanded the personal-data filter: it recognized the Vietnamese national ID card terms but not the passport term, so a table listing both parties’ passport numbers in a resale/gift transaction passed through uninspected.
71 tables across 49 source files now correctly excluded (previously 0). No buyer names or ID numbers were ever stored in the database from the missed tables — verified directly, not assumed.
Diagnosed (did not yet fix) why 28.46% of transactions aren’t linked to a project: three unrelated causes, the largest being a single source file where the project name is stated once on page 1 and never carried forward to the other 19 pages.
Breakdown by cause published, largest class alone accounts for 815 of 1111 unlinked records.
Fixed the city filter on the homepage search offering "Ho Chi Minh City" and "Da Nang" as if there were data for them — there never has been, coverage is Hanoi only. Replaced broken-looking emoji icons with plain inline icons that don’t depend on the viewer’s system having an emoji font installed.
City picker now shows Hanoi only, honestly. 5 icon instances replaced across the homepage and the paid-report unlock panel.
Restored the informal "Mỹ" (US) nationality label without reintroducing its known collision with the "Mỹ Đình" district name — read only from a recognized nationality column, exact match.
9 records corrected (American nationality).
Found that the extraction-completeness measurement itself was inflated: a quota-summary row (total apartment count per building) on page 1 of several files was being mistaken for the row-count ceiling of the buyer list.
Corrected: measured extraction completeness moved from 51.9% to 82.68% (see /methodology) — most of that gap was a measurement bug, not missing data.
Fixed the single largest real extraction-loss class: files dedicated to one project (identified reliably by filename) were dropping every row because the row carried only an apartment code, not address text.
723 additional transaction rows recovered across 2 source files, zero rows lost anywhere else (verified against a fixed reference set of 83 files).
Closed a class of bug where a parsing function could return zero rows silently, with no way to tell "genuinely empty" from "something went wrong" — this is the third time that exact failure mode had caused a real loss (an unactivated Python environment, a missing database filter, revoked file permissions).
Every zero-row parse result now logs an explicit reason. Added a regression test asserting a permission-denied file raises an error instead of returning nothing.
Introduced row-number (STT) reconciliation as the completeness measurement — Vietnamese source publications number their own rows, so extracted rows can be checked against the source's own count.
Measured extraction completeness: 51.9%.
Found and fixed a typo in a source spreadsheet: a contract date of 2004, a year legally impossible for this kind of foreign-buyer transaction.
2 records corrected to the neighboring cells' actual year.
Fixed two contract-date extraction bugs (a seller company's registration date being read instead of the sale date; the address cell's apartment number being read as the contract date/number).
108 records corrected; 106 now-superseded prior versions marked inactive rather than deleted (append-only history).
Fixed three nationality-matching bugs caused by substring collisions — short nationality keys ("hàn", "áo") matching inside unrelated Vietnamese words ("thành phố", "báo cáo") instead of only whole words.
185 nationality labels corrected.
Found that 12 source files had had their read permissions accidentally revoked, and the parser was silently treating "permission denied" the same as "this document has no transactions" — one of those 12 files alone held 853 real records.
Permissions restored; the parser now lets a file-open failure surface as an error rather than a silent zero. Recount: 2,141 → 3,120 unique transactions.
Found and fully excluded a table format used for third-party unit reassignments — these carry the personal data (ID numbers, names) of a person who is not the foreign buyer this dataset is about, and are out of scope for this project by design.
Reassignment tables are now detected structurally and dropped entirely, rather than partially parsed.