Issue/2183/5 perf: detect duplicate keys of stop_times and shapes without a per-row map - #2190
Open
CarloMendola wants to merge 1 commit into
Open
Conversation
…w map For every table with a composite primary key the generated `setupIndices` built a `HashMap<CompositeKey, Entity>` with one entry per row, only to find duplicate keys and to serve translation lookups. One entry costs about 60 bytes, so on a large feed that map alone holds hundreds of megabytes for stop_times.txt and shapes.txt. Both tables have a two-column primary key whose first column is indexed and whose second column is the sequence that index is sorted by. Each entity is now looked up in its own group of that sorted index by a binary search for the leftmost entity with the same sequence: that entity is the first one holding the primary key, so any other entity is a duplicate of it. Entities are visited in the order they were loaded, so the notices are reported exactly as the map reported them, at no memory cost. The map itself is kept only for the translation lookups that need it, and built on first use, so a feed that does not translate stop_times.txt or shapes.txt never pays for it. Every other table with a composite primary key is unchanged: they are small, and the map-based detection is simpler. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary:
Fifth of the six PRs #2183 is split into, in the grouping requested here. On its own, as you asked, because it changes generated code for every table container and turns
byTranslationKeyinto a lazily built, synchronized fallback used from a multi-threaded loader. Based directly onmasterand independent of the other PRs in the series.For every table with a composite primary key, the generated
setupIndicesbuilds aHashMap<CompositeKey, Entity>with one entry per row — used only to detect duplicate keys, and to servebyTranslationKeylookups when translations.txt refers to the table by its primary key. One entry costs about 60 bytes, so on a large feed that map alone holds hundreds of megabytes for stop_times.txt and shapes.txt, whether or not any translation ever needs it.Both of those tables have a two-column primary key whose first column is indexed and whose second column is the sequence that index is sorted by (
trip_id+stop_sequence,shape_id+shape_pt_sequence). For each entity, the generated code now binary-searches its own group of that sorted index for the leftmost entity with the same sequence: that entity is the first one holding the primary key, so any other entity with the same key is a duplicate of it. No map, no allocation. Entities are visited in load order, soDuplicateKeyNotices are produced exactly as the map produced them — same pairs, same order.The map is kept only for the translation lookups that need it, built on first use behind a synchronized accessor, so a feed that does not translate stop_times.txt or shapes.txt never pays for it. Every other table with a composite primary key keeps the map-based detection unchanged: those tables are small, and the map-based code is simpler to read.
Implementation report for the whole series: https://github.com/CarloMendola/gtfs-validator/blob/9f409204bcefff7387f05a3f70118fb03134443b/prompts/memory-optimization/MEMORY_OPTIMIZATION_REPORT.md
Expected behavior:
Same notices, same order, lower memory. Verified on the CTA Chicago feed (95 MB zip, 413 MB of CSV, 1439 shapes, 100 287 trips): the
noticesobject ofreport.jsonis identical to master's andreport.htmldiffers only in the generation timestamp. Nothing to screenshot for that reason.The generated code is the interesting part of this diff.
IdAndSequencePrimaryKeySchemaTestinprocessor/testscovers the generator, and I read the generatedsetupIndicesof the two optimized containers and of two control containers that must keep the previous behaviour.GtfsStopTimeDuplicateKeyTestandGtfsShapeDuplicateKeyTestare new end-to-end tests over the real containers: duplicates at the start, in the middle, at the end, several duplicates of one key, and the no-duplicate case, asserted against the notices the previous implementation emitted.Nothing under
docs/describes the generated indices, so no documentation change is needed.Please make sure these boxes are checked before submitting your pull request - thanks!
gradle testto make sure you didn't break anything