Skip to content

Issue/2183/5 perf: detect duplicate keys of stop_times and shapes without a per-row map - #2190

Open
CarloMendola wants to merge 1 commit into
MobilityData:masterfrom
CarloMendola:issue/2183/5-table-container-index
Open

Issue/2183/5 perf: detect duplicate keys of stop_times and shapes without a per-row map#2190
CarloMendola wants to merge 1 commit into
MobilityData:masterfrom
CarloMendola:issue/2183/5-table-container-index

Conversation

@CarloMendola

Copy link
Copy Markdown

Summary:

Fifth of the six PRs #2183 is split into, in the grouping requested here. On its own, as you asked, because it changes generated code for every table container and turns byTranslationKey into a lazily built, synchronized fallback used from a multi-threaded loader. Based directly on master and independent of the other PRs in the series.

For every table with a composite primary key, the generated setupIndices builds a HashMap<CompositeKey, Entity> with one entry per row — used only to detect duplicate keys, and to serve byTranslationKey lookups when translations.txt refers to the table by its primary key. One entry costs about 60 bytes, so on a large feed that map alone holds hundreds of megabytes for stop_times.txt and shapes.txt, whether or not any translation ever needs it.

Both of those tables have a two-column primary key whose first column is indexed and whose second column is the sequence that index is sorted by (trip_id + stop_sequence, shape_id + shape_pt_sequence). For each entity, the generated code now binary-searches its own group of that sorted index for the leftmost entity with the same sequence: that entity is the first one holding the primary key, so any other entity with the same key is a duplicate of it. No map, no allocation. Entities are visited in load order, so DuplicateKeyNotices are produced exactly as the map produced them — same pairs, same order.

The map is kept only for the translation lookups that need it, built on first use behind a synchronized accessor, so a feed that does not translate stop_times.txt or shapes.txt never pays for it. Every other table with a composite primary key keeps the map-based detection unchanged: those tables are small, and the map-based code is simpler to read.

Implementation report for the whole series: https://github.com/CarloMendola/gtfs-validator/blob/9f409204bcefff7387f05a3f70118fb03134443b/prompts/memory-optimization/MEMORY_OPTIMIZATION_REPORT.md

Expected behavior:

Same notices, same order, lower memory. Verified on the CTA Chicago feed (95 MB zip, 413 MB of CSV, 1439 shapes, 100 287 trips): the notices object of report.json is identical to master's and report.html differs only in the generation timestamp. Nothing to screenshot for that reason.

The generated code is the interesting part of this diff. IdAndSequencePrimaryKeySchemaTest in processor/tests covers the generator, and I read the generated setupIndices of the two optimized containers and of two control containers that must keep the previous behaviour. GtfsStopTimeDuplicateKeyTest and GtfsShapeDuplicateKeyTest are new end-to-end tests over the real containers: duplicates at the start, in the middle, at the end, several duplicates of one key, and the no-duplicate case, asserted against the notices the previous implementation emitted.

Nothing under docs/ describes the generated indices, so no documentation change is needed.

Please make sure these boxes are checked before submitting your pull request - thanks!

  • Run the unit tests with gradle test to make sure you didn't break anything
  • Add or update any needed documentation to the repo
  • Format the title like "feat: [new feature short description]". Title must follow the Conventional Commit Specification(https://www.conventionalcommits.org/en/v1.0.0/).
  • Linked all relevant issues
  • Include screenshot(s) showing how this pull request works and fixes the issue(s) — no visible change: the notices are identical to master's

…w map

For every table with a composite primary key the generated `setupIndices` built
a `HashMap<CompositeKey, Entity>` with one entry per row, only to find duplicate
keys and to serve translation lookups. One entry costs about 60 bytes, so on a
large feed that map alone holds hundreds of megabytes for stop_times.txt and
shapes.txt.

Both tables have a two-column primary key whose first column is indexed and
whose second column is the sequence that index is sorted by. Each entity is now
looked up in its own group of that sorted index by a binary search for the
leftmost entity with the same sequence: that entity is the first one holding the
primary key, so any other entity is a duplicate of it. Entities are visited in
the order they were loaded, so the notices are reported exactly as the map
reported them, at no memory cost.

The map itself is kept only for the translation lookups that need it, and built
on first use, so a feed that does not translate stop_times.txt or shapes.txt
never pays for it. Every other table with a composite primary key is unchanged:
they are small, and the map-based detection is simpler.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@CLAassistant

CLAassistant commented Sep 3, 2026

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants