[BUG] Breaking Changes compares individual files using case-sensitive matching

[josefree]

API Spec link

#35645

API Spec version

2025-01-01-preview

Describe the bug

In 2025-06-01 and 2025-09-01 of Microsoft.SecurityInsight, there is a model named “PremiumMdtiDataConnectorDataTypes”. However, In 2025-07-01-preview, there isn’t. In the pull request Dev sentinel securityinsights microsoft.security insights 2025 07 01 preview #35645. It's clear this breaking change was not captured.

Expected behavior

According to Azure Service Versioning Guidelines - any public or private preview service version must be a superset of that last GA service version (unless a preview version introduces a breaking change). Therefore, the expectation behavior for Breaking Change Detection tool is it should compare the new preview with the latest stable to identify the stable APIs missing from the former.

Actual behavior

In the pull request Dev sentinel securityinsights microsoft.security insights 2025 07 01 preview #35645. It's clear no breaking change was captured.

Reproduction Steps

1. create a pull request of a new preview version
2. manually a model already published in previous stable version
3. validate if the breaking change detection is triggered

Environment

GitHub

[mikeharder]

This is, primarily, another instance of #38245 (PR checks should treat files as case-insensitive). The spec uses different casing for some files across stable (DataConnectors.json) and preview (dataConnectors.json). For example:

https://github.com/Azure/azure-rest-api-specs/blob/14df0d25acfedb5aaa9927c8fe3495db04d1a4ba/specification/securityinsights/resource-manager/Microsoft.SecurityInsights/stable/2025-09-01/DataConnectors.json

https://github.com/Azure/azure-rest-api-specs/blob/14df0d25acfedb5aaa9927c8fe3495db04d1a4ba/specification/securityinsights/resource-manager/Microsoft.SecurityInsights/preview/2025-07-01-preview/dataConnectors.json

Since BreakingChanges compares each swagger file individually, and matches files using a case-sensitive comparison, it never compares stable/DataConnectors.json to preview/dataConnectors.json.

One fix would be for BreakingChanges to use a case-insensitive comparison.

Another fix, though probably too large and risky to be worth it now, would be for BreakingChanges to work on the spec level rather than the file level. Rather than comparing each JSON file, we'd effectively merge all the JSON files in a spec folder into a single file, and compare that across versions. This way, if you refactored a spec to say merge a.json and b.json into c.json, you could still pass BreakingChanges, as long as the overall spec was compatible.

@raych1: Do you know why BreakingChanges analyzes individual files rather than spec folders as a whole?

[josefree]

Could there be an enhancement to detect abnormal inconsistency in spec files and service spec folder structure. I mean under a service folder, adding a new spec or configuration file should be fine, but removing any spec file or configuration file already published, there should be an alarm. And this check should be case sensitive, because GitHub URI is case sensitive.

[mikeharder]

Could there be an enhancement to detect abnormal inconsistency in spec files and service spec folder structure. I mean under a service folder, adding a new spec or configuration file should be fine, but removing any spec file or configuration file already published, there should be an alarm.

@josefree: How is this different than our current "Breaking Changes" check? Can you create a draft PR, showing an example change, that is currently allowed, but you think should be blocked?

And this check should be case sensitive, because GitHub URI is case sensitive.

I'm not sure what you mean. Can you create another example, showing what should and shouldn't be allowed in terms of case sensitivity?