v3.0.1

Main update

The main difference between the v3.0.0 and v3.0.1 is the modification of ShapeIncreasingDistanceValidator. After a discussion in #1070, we realized that we had a problem with the notice DecreasingOrEqualShapeDistanceNotice (previously an ERROR): because of the way scheduling software produce data, we can have records in shapes.txt that have equal coordinates, and equal values for shape_dist_traveled. This can happen when two shape points are really close to one another. It creates a somewhat duplicative record, and this should not be an ERROR. A true ERROR happens if values of shape_distance_traveled are equal for two shape points that have different coordinates.

We decided to replace this notice with three new ones (in PR #1083):
1- DecreasingShapeDistanceNotice as an ERROR: if two consecutive shape points have decreasing values for shape_dist_traveled.
2- EqualShapeDistanceDiffCoordinatesNotice as an ERROR: if two consecutive points have equal shape_dist_traveled and different lat/lon coordinates in shapes.txt.
3- EqualShapeDistanceSameCoordinatesNotice as a WARNING: if two record have the same values for shape_dist_traveled and lat/lon coordinates in shapes.txt.

List of merged PRs

• fix: ShapeIncreasingDistanceValidator by @lionel-nj in #1083
• build: Maven central by @ed-g in #1090
• build: Jitpack by @lionel-nj in #1099
• fix: Upgrade to Guava 31 and use ImmutableMap.Builder.buildOrThrow() instead of build() by @aababilov in #1103
• fix: Updates scripts to modify mobilitydatabase.org to old.mobilitydatabase.org by @maximearmstrong in #1108
• docs: Add README badge for acceptance tests by @barbeau in #1104

New Contributors

• @ed-g made their first contribution in #1090

Full Changelog: v3.0.0...v3.0.1

v3.0.0

We're excited to bring you the 3.0.0 release, it's a big one!

We three had main goals for this release:

➡️ Solve discrepancies with the official GTFS specification.
26 new rules have been added in total:

• New validation rules were added for the adopted GTFS extensions pathways.txt, levels.txt, translations.txt
• New custom rules were added for parts of the spec that were missing from this validator
• Some existing rules were improved to better reflect the specification
• The severity of some rules was updated to reflect the specification after the update to RFC 2119 has been done

Please refer to the V2.0.0 to V3.0.0 rule mapping available in NOTICE_MIGRATION.md for more details on the rule updates.

➡️ Implement automated tests that will evaluate if a new rule results in datasets from the mobility database getting additional errors. If so, the Pull Request is blocked and a report will be generated for further investigation. This ensures the stability and reliability of this validator, while still being updated with updates and extensions in the specification.

➡️ Make sure MobilityData's work has related public communication to facilitate community contribution (issues opened, public sprint board and bug tracking board, roadmap, ways for users to voice what they'd like to see next and for contributors to see how to get involved).

There are also new command line parameters that can be found in USAGE.md.

There is a lot more work included in this release, thanks to our amazing contributors. See details below!

Work ⚙️

Solve discrepancies with the spec

• chore: downgrade LeadingOrTrailingWhitespacesNotice to WARNING in #929
• chore: upgrade EmptyColumnNameNotice to ERROR in #1019
• chore: new rule - additional requirements urls (stops.txt, routes.txt, agency.txt) in #932

routes.txt

• feat: Reduce severity of SameNameAndDescriptionForRouteNotice to WARNING in #917

stops.txt

• chore: check stop name and desc in #937
• chore: check stops.zone_id presence in #936

stop_times.txt

• chore: new rule - additional check on stops.location_type in #938
• chore: upgrade LocationWithUnexpectedStopTimeNotice to ERROR in #1021

pathways.txt and levels.txt

• feat: Validate how locations are referenced from stop_times.stop_id in #960
• feat: Validate that pathway endpoints have correct types in #961
• feat: verify conditional requirement of levels.txt presence in #966
• feat: additional checks on pathways.txt and stops.txt in #967
• feat: Validate that locations are reachable if a station has pathways in #997
• feat: Validate that a pathway is not a loop in #999
• feat: Validate field presence and key integrity for translations.txt in #968
• chore: upgrade PathwayUnreachableLocationNotice to ERROR in #1028

translations.txt

• chore: upgrade TranslationForeignKeyViolationNotice to ERROR in #1027

Automated test to see if a PR results in additional errors

• implementation of acceptance test (#1037, #1038, #926)

Other work included

• We ran some analytics on all datasets from the mobility database (#1041)
  Better rules
• improve rule logic (#839, #944, #946, #956, #957, #958, #978, #972)

Better notice schema export

• implement RTTI machine for notice schema (#922, #924, #925, #951, #923)

Better documentation

• documentation improvements (#842, #836, #837, #940, #902, #905, #1024, #1009, #988, #1057)

Bugs

• fix: Prevent NullPointerException when Throwable.getMessage returns null in #871
• fix: Catch any runtime exception when parsing a GTFS field in #1012
• fix: Swap arguments of isAssignableFrom in #872
• fix: bump debian version used in docker image in #891
• fix: Protect against start_date and end_date in wrong order in #897
• fix: Raise IllegalArgumentException for invalid date in #977
• fix: change limit on notice count during validation to 100_000 in #1018
• fix: validate route, stops and agency consistency urls in #1003
• feat: Prevent OOM in NoticeContainer and speed up hasValidationErrors in #895
• fix: Skip phone number validation when country code is unknown in #1062
• fix: False positives for StopTimeTimepointWithoutTimesNotice in #1044

Other good stuff

• ci: check code formatting on pull requests in #953
• ci: publish a new docker image on every push on master in #913
• ci: verify gradle wrapper in #964
• ci: bug triage in #982
• feat: Provide enum setters and clearFIELD methods in #843
• feat: make report names user configurable via CLI args in #852
• feat!: deprecate -f in #851
• feat: Pass severityLevel to constructor of some notices in #863
• feat: Propagate InterruptedException during feed loading in #869
• feat: Catch CSV parse errors and add validation notices in #874
• feat: print help menu in #888
• feat: Export infinity and NaN values in JSON in #889
• feat: Treat empty files as parsed unsuccessfully in #898
• feat: introduce -p CLI parameter to output beautified reports in #901
• feat: Add byPrimaryKey and getKeyColumnNames methods to GtfsTableContainer in #1000
• chore: upgrade to gradle wrapper to 7.2 in #965
• chore: Upgrade to Java 11 in #1014

Contributors 🙏

Massive shoutout to:
🎉 @aababilov
🎉 @asvechnikov2
🎉 @barbeau
🎉 @lionel-nj
🎉 @maximearmstrong
🎉 @isabelle-dr

v2.0.0

What's new 🆕

We're excited to bring you this v2 release of the gtfs-validator! A lot has changed since v1, and we think you'll appreciate the new features.

• Faster 🚀 and stronger 💪 thanks to a new multi-threaded architecture (thank you @aababilov!) (PR #486)
  • Supports the largest known real-world GTFS datasets. For example, the Netherlands GTFS previously failed with an OutOfMemory error, and now it can be validated in 2 minutes and 12 seconds on an ordinary laptop!
  • Validates datasets much faster. For example, MBTA used to take 38 seconds to validate, and now it only takes 14 seconds - less than half the time!
• ⏬ Reduces the amount of code needed for adding new validation rules and support for new GTFS files and fields by leveraging Java annotations and AutoValue
  • A new rule can now be added in a single .java file. See the "Adding new rules" documentation for examples.
  • Basic data type, range, and primary key validation for new GTFS files and fields can be added in a few lines of code. See the "Adding new files and fields" documentation for examples.
• 📝 Notice definitions for validation rules have been updated
  • Notice severities now align with the canonical GTFS spec (e.g., issues #472, #417). ERRORS are for items that the spec explictly requires or prohibits, and WARNINGS are for items that will impact the quality of GTFS datasets but aren't strictly required (suggestions or best practices).
  • Notices have ported to the new architecture (e.g., PR #592, PR #710). See the "Rules" documentation for the new v2 notice definitions and the "v1 to v2 Migration Guide" for a table mapping v1 notices to v2 notices.
• 🛠 Better error handling and reporting
  • In case of problems with the validator software itself, an error report is now written to a file system_errors.json to help with troubleshooting (PR #596). Normal validation results can be found in report.json.
• 🗺 New project roadmap
  • Check out the project wiki. Please feel free to add things you are interested in!

v1.4.0

What's new 🆕

• We now generate a validation report even if the software unexpectedly crashed: users will be informed that it happened and are invited to report the error.
• The validation report name has been reworked to include information regarding the feed's feed_info.publisher_name or agency.agency_id.
• Various bug fixes.
• Our documentation for releases has been updated to match the latest development.

NPEs ❌

#469

Fixes 🔧

#462
#465

Changes ⚠️

#479

#481
#488
#487

Doc 📖

#463
#478
#490

v1.3.1 -- Deep Dive Oct. 28th

Fixes

#459
#452

Doc

#453
#456

v1.3.0

What's new

You can now run the validator directly on GitHub, see the relevant readme section: https://github.com/MobilityData/gtfs-validator/blob/master/README.md#via-github-actions---run-the-validator-on-any-gtfs-archive-available-on-a-public-url

We now have a local web UI

We now provide Docker images, see https://github.com/orgs/MobilityData/packages/container/gtfs-validator/192493
Release process completely automated, release cycle should be on a fixed schedule now

Our README has been updated.

NPEs

#387
#423

Fixes

#394
#429
#433

Changes

#395
#438
#439
#443

v1.2.2

Description

Extend the number of supported rules to achieve feature parity with the python Google validator for what is strictly in spec.

New rules

• conditional presence of calendar.txt or calendar_dates.txt - #355

stops

• stops too far from trip shape - #340

agency

• agency.agency_lang match feed_info.feed_lang - #352

shapes

• increasing shape_dist_traveled along a shape - #363

csv

• no duplicate column in CSV - #326
• no empty csv file - #317

routes

• names unicity - #351

stop_times

• stop times departure time after arrival time - #198
• stop times travel backward in time - #338
• fast travel between stops - #332
• increasing distance stop times - #350

stop_times and trips

• verify no overlap in trip block #346

trips

• trip start and end time verification - #327
• additional verifications #344

frequencies

• frequencies start time before end time #335
• verify no overlap - #349

Behavior changes

• abort on error - #339

v1.2.1

Release notes

• Complete implementation of CSV validation

  • agency.txt, stops.txt, routes.txt, trips.txt, stop_times.txt, levels.txt, transfers.txt, calendar.txt, pathways.txt, calendar_dates.txt, shapes.txt, feed_info.txt

• Agency related validation

• Validate route id reference in trips

• Cross validation between agency.txt and routes.txt

• Fix for issues #288 and #322

• Validate trips.service_id

• Validate shape_id references

• Feed related validation

Additional info

• Starting with this release we provide our tests results as well as javadoc
• You can find the complete set of implemented rules in RULES.md
• You can find snapshot versions of the latest merge on master via our GitHub action page

v1.2.0

• Notice format has been reworked

  • level and code have been separated
  • extra data is serialized in a noticeSpecific object in the JSON output

• Add feature to run the validator with a json configuration file in place of command line arguments

• Add a feature to run the validation only on a subset of files

• Add GTFS semantics validation

• Add documentation detailing implemented rules

• Add documentation on how to extend the validation rules

• Various bug fixes

v1.1.0

First stable release

• add supports for all files that can be contained in a GTFS archive
• add support of all GTFS types for CSV validation and base GTFS validation (numeric ranges and string formats)
• add JSON export of validation results
• add protobuf export of validation results

See the v1.1.0 README for usage information.

Note that command-line parameters in the master branch have changed since this release.

Note: README.md link to GTFS specification schema should point here