Infiniti-loop (?) in analyzer / reporter on huge Gradle project #9763
Description
Describe the bug
We found an Infiniti-loop in the analyzer.
The project we want to analyse is https://github.com/hashgraph/hedera-services and the .ort.yml can be found here: https://github.com/hendrikebbers/ort-scan-hedera/blob/main/ort-config/hedera-services.yml
When we initially executed the analyzer it always crashed with an OutOfMemoryError in the Gradle Daemon. Even if we set the memory to 30 GB. We figured out, that the problem is located to OrtModelBuilder.tk. We did a local fix for that class and introduced a cache of type mutableMapOf<ComponentIdentifier, OrtDependency>() that is used in the toOrtDependencies method. Instead of creating new OrtDependencyImpl instance all the time in the toOrtDependencies method we check if the cache already contains an instance with the given ID and than return that instance. I already discussed that change with @sschuberth and since some changes are needed I have not created a PR so far.
Our concrete changes as a git patch:
Subject: [PATCH] Adding cache to gradle analyzer
---
Index: plugins/package-managers/gradle-plugin/src/main/kotlin/OrtModelBuilder.kt
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
diff --git a/plugins/package-managers/gradle-plugin/src/main/kotlin/OrtModelBuilder.kt b/plugins/package-managers/gradle-plugin/src/main/kotlin/OrtModelBuilder.kt
--- a/plugins/package-managers/gradle-plugin/src/main/kotlin/OrtModelBuilder.kt (revision 75cf08a046a115a216dc8610b041f99492bf14f0)
+++ b/plugins/package-managers/gradle-plugin/src/main/kotlin/OrtModelBuilder.kt (revision de517cf972efdcea271805d06e70b1012a802c1c)
@@ -56,6 +56,8 @@
private val warnings = mutableListOf<String>()
private val globalDependencySubtrees = mutableMapOf<String, List<OrtDependency>>()
+ private val cache = mutableMapOf<ComponentIdentifier, OrtDependency>()
+
override fun canBuild(modelName: String): Boolean = modelName == OrtDependencyTreeModel::class.java.name
override fun buildAll(modelName: String, project: Project): OrtDependencyTreeModel {
@@ -141,6 +143,7 @@
}.filterIsInstance<ResolvedArtifactResult>()
}
+
private fun Collection<DependencyResult>.toOrtDependencies(
poms: Map<String, ModelBuildingResult>,
visited: Set<ComponentIdentifier>
@@ -165,6 +168,10 @@
// Cut the graph on cyclic dependencies.
if (id in visited) return@mapNotNull null
+ if(cache.containsKey(id)) {
+ return@mapNotNull cache[id]
+ }
+
when (id) {
is ModuleComponentIdentifier -> {
val pomFile = if (selectedComponent is ResolvedComponentResultInternal) {
@@ -211,7 +218,7 @@
selectedComponent.dependencies.toOrtDependencies(poms, visited + id)
}
- OrtDependencyImpl(
+ val dependency = OrtDependencyImpl(
groupId = id.group,
artifactId = id.module,
version = id.version,
@@ -232,13 +239,15 @@
},
localPath = null
)
+ cache[id] = dependency
+ dependency
}
is ProjectComponentIdentifier -> {
val moduleId = selectedComponent.moduleVersion ?: return@mapNotNull null
val dependencies = selectedComponent.dependencies.toOrtDependencies(poms, visited + id)
- OrtDependencyImpl(
+ val dependency = OrtDependencyImpl(
groupId = moduleId.group,
artifactId = moduleId.name,
version = moduleId.version.takeUnless { it == "unspecified" }.orEmpty(),
@@ -251,6 +260,8 @@
mavenModel = null,
localPath = id.projectPath
)
+ cache[id] = dependency
+ dependency
}
else -> {
With that change the Gradle deamon does not dies anymore and the analyse results are send back sucessfully to the ORT process. Sadly that runs endlessly. I killed the process after 16 hours but attached an VisualVm agend in between to check if it is still running or just blocked. The process was running all the time in the same code areas as you can see on the attached screenshots.
To Reproduce
- Checkout https://github.com/hashgraph/hedera-services
- Create a
.ort.ymlfile with the content of https://github.com/hendrikebbers/ort-scan-hedera/blob/main/ort-config/hedera-services.yml - Checkout https://github.com/oss-review-toolkit/ort/
- Change
OrtModelBuilder.tkbased on the given git patch - Run ORT with those parameters:
--debug -P ort.analyzer.allowDynamicVersions=true analyze -i PATH/TO/REPO -o PATH/TO/REPO
Expected behavior
The process should finish after some time and the analyser result should be stores in a file.