feat(amazonq): Adding UTG build and execute logic back for internal Amazon customers. #5396

Randall-Jiang · 2025-02-20T02:27:27Z

Types of changes

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)

Description

This PR introduces a build-exec loop for internal Amazon users, allowing them to run a customized build command after Amazon Q generates unit tests. The goal is to ensure that the newly generated unit tests do not break the initial build.

Solution

Adding back build and execute logic for internal amazon users to fix the errors when generating unit tests. Allow users to run 3 iterations to fix the codes if there are any issues cause build failure.

TO-DO

Add telemetry for build & execute loop
Fixing the UI base on the PM's requirement

Checklist

My code follows the code style of this project
I have added tests to cover my changes
A short description of the change has been added to the CHANGELOG if the change is customer-facing in the IDE.
I have added metrics for my changes (if required)

License

I confirm that my contribution is made under the terms of the Apache 2.0 license.

.../src/software/aws/toolkits/jetbrains/services/amazonqCodeTest/CodeWhispererUTGChatManager.kt

.../software/aws/toolkits/jetbrains/services/amazonqCodeTest/model/BuildAndExecuteStatusIcon.kt

@@ -25,15 +28,19 @@
        BuildAndExecuteStatusIcon.WAIT.icon
    } else if (progressStatus == BuildAndExecuteProgressStatus.RUN_EXECUTION_TESTS) {
        BuildAndExecuteStatusIcon.CURRENT.icon
+    } else if (progressStatus == BuildAndExecuteProgressStatus.BUILD_FAILED || progressStatus == BuildAndExecuteProgressStatus.FIXING_TEST_CASES) {


laileni-aws · 2025-02-20T18:12:16Z

...ws/toolkits/jetbrains/services/codewhisperer/codetest/sessionconfig/CodeTestSessionConfig.kt

+            val buildLogFilePath = Path.of(utgDir, buildAndExecuteLogDir, "buildAndExecuteLog")
+                .toString().replace("\\", "/") // Ensures consistent path format
+
+            LOG.debug { "Adding build logs to ZIP at: $buildLogFilePath" }


Do we need this log?

I added this log to make sure the build log is added to the currect folder.

Yeah but we should not print this out for the customers. So, we can remove this.

laileni-aws · 2025-02-20T18:14:33Z

...oftware/aws/toolkits/jetbrains/services/amazonqCodeTest/controller/CodeTestChatController.kt

+
+                    return
+                }
+//                taskContext.progressStatus = BuildAndExecuteProgressStatus.RUN_EXECUTION_TESTS


Do we need this commented code? as this is being added as part of this PR

I think this commented code is for execution loop.

seems like a bad practice to leave commented out code as future TODO. Can you put it in a different branch and PR when ready? lets not add clutter

I think we can remove this code, anyway we will have git history to track.

laileni-aws · 2025-02-20T18:15:35Z

...oftware/aws/toolkits/jetbrains/services/amazonqCodeTest/controller/CodeTestChatController.kt

@@ -821,13 +828,12 @@ class CodeTestChatController(
                    "Successfully sent test generation telemetry. RequestId: ${
                        testGenerationEventResponse.responseMetadata().requestId()}"
                }
-                sessionCleanUp(session.tabId)
+                // sessionCleanUp(session.tabId)


Remove commented code.

laileni-aws · 2025-02-20T18:17:07Z

...oftware/aws/toolkits/jetbrains/services/amazonqCodeTest/controller/CodeTestChatController.kt

@@ -691,24 +695,27 @@ class CodeTestChatController(
                        canBeVoted = false
                    )
                )
-                sessionCleanUp(session.tabId)
+                if (!isInternalUser) {


Did you test this by login with Builder Id vs internal user?

.../src/software/aws/toolkits/jetbrains/services/amazonqCodeTest/CodeWhispererUTGChatManager.kt

laileni-aws · 2025-02-20T18:20:14Z

.../src/software/aws/toolkits/jetbrains/services/amazonqCodeTest/CodeWhispererUTGChatManager.kt

@@ -175,7 +179,7 @@ class CodeWhispererUTGChatManager(val project: Project, private val cs: Coroutin
        var shortAnswer = ShortAnswer()
        LOG.debug {
            "Q TestGen session: ${codeTestChatHelper.getActiveCodeTestTabId()}: " +
-                "polling result for id: ${job.testGenerationJobId()}, group name: ${job.testGenerationJobGroupName()}, " +
+                "polling result for id: ${job.testGenerationJobId()}, group name: ${session.testGenerationJobGroupName}, " +


nit: not required as both job.testGenerationJobGroupName() and session.testGenerationJobGroupName will be same

laileni-aws · 2025-02-20T18:21:29Z

.../src/software/aws/toolkits/jetbrains/services/amazonqCodeTest/CodeWhispererUTGChatManager.kt

@@ -250,10 +255,9 @@ class CodeWhispererUTGChatManager(val project: Project, private val cs: Coroutin
                    if (shortAnswer.stopIteration == "true") {
                        throw CodeTestException("TestGenFailedError: ${shortAnswer.planSummary}", "TestGenFailedError", shortAnswer.planSummary)
                    }
-                    val fileName = shortAnswer.sourceFilePath?.let { Path.of(it).fileName.toString() } ?: path.fileName.toString()


Why did we remove this check? Any reason?

laileni-aws · 2025-02-20T18:22:38Z

.../src/software/aws/toolkits/jetbrains/services/amazonqCodeTest/CodeWhispererUTGChatManager.kt

                    codeTestChatHelper.updateAnswer(
                        CodeTestChatMessageContent(
-                            message = generateSummaryMessage(fileName) + shortAnswer.planSummary,
+                            message = generateSummaryMessage(path.fileName.toString()) + shortAnswer.planSummary,


What if user opens test file and run /test. This says Generating unit test for Test file but this should be Source file.

when user opens test file and run /test there, if this says Generating unit test for Test file, that's expected right? Because utg is generating unit test for opening files?

I don't think this code change is from me, I rebased from main and the code change is there. Maybe @ashishrp-aws or @chungjac has more info about this.

Can you add this back?
Looks like it was removed as part of this PR

laileni-aws · 2025-02-20T18:24:35Z

.../src/software/aws/toolkits/jetbrains/services/amazonqCodeTest/CodeWhispererUTGChatManager.kt

@@ -297,7 +301,7 @@ class CodeWhispererUTGChatManager(val project: Project, private val cs: Coroutin
            // TODO: Modify text according to FnF
            codeTestChatHelper.addAnswer(
                CodeTestChatMessageContent(
-                    message = message("testgen.error.generic_technical_error_message"),
+                    message = message("testgen.message.failed"),


Was this necessary?
We should show users
I am experiencing technical difficulties at the moment. Please try again … instead of Test generation failed

laileni-aws · 2025-02-20T18:25:58Z

.../src/software/aws/toolkits/jetbrains/services/amazonqCodeTest/CodeWhispererUTGChatManager.kt

-            previousIterationContext?.buildLogFile,
-            previousIterationContext?.testLogFile
-        )
+        val combinedBuildAndExecuteLogFile =


variable declaration is not needed here, if required can we change variable name to buildLogFile etc..

I keeped this in case we need excute in the future, will do it in the next pr.

...-community/src/software/aws/toolkits/jetbrains/services/amazonqCodeTest/utils/UTGChatUtil.kt

...oftware/aws/toolkits/jetbrains/services/amazonqCodeTest/controller/CodeTestChatController.kt

-                taskContext.progressStatus = BuildAndExecuteProgressStatus.FIXING_TEST_CASES
-                val buildAndExecuteMessageId = updateBuildAndExecuteProgressCard(taskContext.progressStatus, messageId, session.iteration)
+                // TODO: only go to future iterations when buildExitCode or testExitCode > 0, right now iterate regardless
+                if (taskContext.buildExitCode > 0) {


…& add telemetry for build&execute

.../src/software/aws/toolkits/jetbrains/services/amazonqCodeTest/CodeWhispererUTGChatManager.kt

...oftware/aws/toolkits/jetbrains/services/amazonqCodeTest/controller/CodeTestChatController.kt

-                    requestId = session.startTestGenerationRequestId
-                )
+                if (session.iteration == 1) {
+                    AmazonqTelemetry.utgGenerateTests(


...-community/src/software/aws/toolkits/jetbrains/services/amazonqCodeTest/utils/UTGChatUtil.kt

+
+    // Find the nearest Gradle root directory
+    var packageRoot: File? = testFileAbsolutePath.parentFile
+    while (packageRoot != null && packageRoot != projectRoot) {


laileni-aws · 2025-02-20T18:48:05Z

.../src/software/aws/toolkits/jetbrains/services/amazonqCodeTest/CodeWhispererUTGChatManager.kt

@@ -469,10 +473,8 @@ class CodeWhispererUTGChatManager(val project: Project, private val cs: Coroutin
                previousIterationContext.selectedFile
            }

-        val combinedBuildAndExecuteLogFile = combineBuildAndExecuteLogFiles(


Can we remove combineBuildAndExecuteLogFiles function as this is not being used

laileni-aws · 2025-02-22T02:32:29Z

...ws/toolkits/jetbrains/services/codewhisperer/codetest/sessionconfig/CodeTestSessionConfig.kt

+            val buildLogFilePath = Path.of(utgDir, buildAndExecuteLogDir, "buildAndExecuteLog")
+                .toString().replace("\\", "/") // Ensures consistent path format
+
+            LOG.debug { "Adding build logs to ZIP at: $buildLogFilePath" }


Yeah but we should not print this out for the customers. So, we can remove this.

laileni-aws · 2025-02-22T02:33:18Z

...oftware/aws/toolkits/jetbrains/services/amazonqCodeTest/controller/CodeTestChatController.kt

+
+                    return
+                }
+//                taskContext.progressStatus = BuildAndExecuteProgressStatus.RUN_EXECUTION_TESTS


I think we can remove this code, anyway we will have git history to track.

.../src/software/aws/toolkits/jetbrains/services/amazonqCodeTest/CodeWhispererUTGChatManager.kt

laileni-aws · 2025-02-25T20:09:25Z

.../src/software/aws/toolkits/jetbrains/services/amazonqCodeTest/CodeWhispererUTGChatManager.kt

                    codeTestChatHelper.updateAnswer(
                        CodeTestChatMessageContent(
-                            message = generateSummaryMessage(fileName) + shortAnswer.planSummary,
+                            message = generateSummaryMessage(path.fileName.toString()) + shortAnswer.planSummary,


Can you add this back?
Looks like it was removed as part of this PR

laileni-aws · 2025-02-25T20:11:14Z

.../src/software/aws/toolkits/jetbrains/services/amazonqCodeTest/CodeWhispererUTGChatManager.kt

+                        promptInputDisabledState = true,
+                        promptInputProgress = buildAndExecuteProgrogressField,
+                    )
+                }


You can optimize this!

codeTestChatHelper.updateUI( promptInputDisabledState = true, promptInputProgress = if (session.iteration == 1) { testGenProgressField(0) } else { buildAndExecuteProgrogressField } )

...-community/src/software/aws/toolkits/jetbrains/services/amazonqCodeTest/utils/UTGChatUtil.kt

...oftware/aws/toolkits/jetbrains/services/amazonqCodeTest/controller/CodeTestChatController.kt

.../src/software/aws/toolkits/jetbrains/services/amazonqCodeTest/CodeWhispererUTGChatManager.kt

rli · 2025-02-25T21:44:04Z

...-community/src/software/aws/toolkits/jetbrains/services/amazonqCodeTest/utils/UTGChatUtil.kt

+    // Find the nearest Gradle root directory
+    var packageRoot: File? = testFileAbsolutePath.parentFile
+    while (packageRoot != null && packageRoot != projectRoot) {
+        if (File(packageRoot, "settings.gradle.kts").exists() || File(packageRoot, "build.gradle.kts").exists()) {


what if you're using the groovy version of the gradle build script?

...-community/src/software/aws/toolkits/jetbrains/services/amazonqCodeTest/utils/UTGChatUtil.kt

rli · 2025-02-25T21:45:16Z

...-community/src/software/aws/toolkits/jetbrains/services/amazonqCodeTest/utils/UTGChatUtil.kt

+        packageRoot = packageRoot.parentFile
+    }
+
+    // If no valid Gradle directory is found, fallback to the project root


? does that actually make sense?

just in case no gradle files are found.

rli · 2025-02-25T21:45:33Z

...-community/src/software/aws/toolkits/jetbrains/services/amazonqCodeTest/utils/UTGChatUtil.kt

@@ -109,10 +130,14 @@ fun runBuildOrTestCommand(
    }

    val processBuilder = ProcessBuilder()
-        .command(listOf(command) + args)
-        .directory(File(repositoryPath))
+        .command(listOf("zsh", "-c", "source ~/.zshrc && $command ${args.joinToString(" ")}"))


what if they are on windows? or don't use zsh?

...-community/src/software/aws/toolkits/jetbrains/services/amazonqCodeTest/utils/UTGChatUtil.kt

rli · 2025-02-28T17:02:04Z

...-community/src/software/aws/toolkits/jetbrains/services/amazonqCodeTest/utils/UTGChatUtil.kt

 ) {
+    val brazilPath = "${System.getProperty("user.home")}/.toolbox/bin:/usr/local/bin:/usr/bin:/bin:/sbin"


why are we hardcoding internal user paths

@rli
I think this is to get the path to access the bb, else it always throws an error saying bb command not found while running build and execute commands.

.../src/software/aws/toolkits/jetbrains/services/amazonqCodeTest/CodeWhispererUTGChatManager.kt

+                        credentialStartUrl = getStartUrl(project),
+                        jobGroup = session.testGenerationJobGroupName,
+                        jobId = session.testGenerationJob,
+                        result = if (e.message == message("testgen.message.cancelled")) MetricResult.Cancelled else MetricResult.Failed,


.../src/software/aws/toolkits/jetbrains/services/amazonqCodeTest/CodeWhispererUTGChatManager.kt

+                        jobId = session.testGenerationJob,
+                        result = if (e.message == message("testgen.message.cancelled")) MetricResult.Cancelled else MetricResult.Failed,
+                        reason = (e as CodeTestException).code ?: "DefaultError",
+                        reasonDesc = if (e.message == message("testgen.message.cancelled")) "${e.code}: ${e.message}" else e.message,


...oftware/aws/toolkits/jetbrains/services/amazonqCodeTest/controller/CodeTestChatController.kt

@@ -114,6 +116,7 @@
    private val authController: AuthController = AuthController(),
    private val cs: CoroutineScope,
 ) : InboundAppMessagesHandler {
+    var buildResult = false


...oftware/aws/toolkits/jetbrains/services/amazonqCodeTest/controller/CodeTestChatController.kt

+            "utg_skip_and_finish" -> {
+                codeTestChatHelper.addAnswer(
+                    CodeTestChatMessageContent(
+                        message = message("testgen.message.success"),


…s not found

laileni-aws · 2025-03-04T22:04:40Z

...-community/src/software/aws/toolkits/jetbrains/services/amazonqCodeTest/utils/UTGChatUtil.kt

    // If no valid Gradle directory is found, fallback to the project root
-    val gradleWrapper = File(packageRoot, "gradlew")
-    val workingDir = if (gradleWrapper.exists()) packageRoot else projectRoot
+    val gradleWrapper = File(packageRoot ?: projectRoot, "gradlew")


where we are using this gradleWrapper?

Randall-Jiang requested a review from a team as a code owner February 20, 2025 02:27

github-advanced-security bot found potential problems Feb 20, 2025

View reviewed changes

laileni-aws reviewed Feb 20, 2025

View reviewed changes

Randall-Jiang requested a review from a team as a code owner February 20, 2025 18:12

laileni-aws reviewed Feb 20, 2025

View reviewed changes

.../src/software/aws/toolkits/jetbrains/services/amazonqCodeTest/CodeWhispererUTGChatManager.kt Show resolved Hide resolved

laileni-aws reviewed Feb 20, 2025

View reviewed changes

...-community/src/software/aws/toolkits/jetbrains/services/amazonqCodeTest/utils/UTGChatUtil.kt Show resolved Hide resolved

add build-execute code

24a0832

Randall-Jiang force-pushed the build-execute branch 2 times, most recently from bfac8e6 to 24a0832 Compare February 21, 2025 21:50

address comment

4168e78

github-advanced-security bot found potential problems Feb 21, 2025

View reviewed changes

create progress bar for fixing build failure

b5de261

Randall-Jiang force-pushed the build-execute branch from 83095f6 to b5de261 Compare February 21, 2025 22:35

Randall-Jiang added 3 commits February 24, 2025 15:28

fix the bug which might cause user can not execute the build command …

9cf1786

…& add telemetry for build&execute

fix format error

ca2c128

correct the logic to send the telemetry

9151dce

github-advanced-security bot found potential problems Feb 25, 2025

View reviewed changes

Randall-Jiang force-pushed the build-execute branch from c14981b to 84062da Compare February 25, 2025 20:37

addressed comment

b5792bc

Randall-Jiang force-pushed the build-execute branch from 84062da to b5792bc Compare February 25, 2025 20:39

nit

5231789

laileni-aws reviewed Feb 25, 2025

View reviewed changes

rli reviewed Feb 25, 2025

View reviewed changes

addressing comments

9e323fe

Randall-Jiang force-pushed the build-execute branch from 78d4edf to 9e323fe Compare February 25, 2025 22:36

add PATH

3601d49

Randall-Jiang force-pushed the build-execute branch from 5ddf101 to 3601d49 Compare February 28, 2025 00:42

rli reviewed Feb 28, 2025

View reviewed changes

Randall-Jiang added 2 commits February 28, 2025 11:43

add gradle file for groovy version of the gradle build script

fc8d07b

fix UI error

d195b1a

Randall-Jiang requested review from rli and laileni-aws March 3, 2025 05:50

github-advanced-security bot found potential problems Mar 3, 2025

View reviewed changes

fix the issue that workspace has the same name with prject, the dir i…

b5f6ce1

…s not found

Randall-Jiang force-pushed the build-execute branch from 2c3ddba to b5f6ce1 Compare March 4, 2025 21:53

laileni-aws reviewed Mar 4, 2025

View reviewed changes

laileni-aws mentioned this pull request Mar 7, 2025

feat(amazonq): UTG build and execute for internal users. #5436

Open

6 tasks

laileni-aws closed this Mar 7, 2025

		) {
		val brazilPath = "${System.getProperty("user.home")}/.toolbox/bin:/usr/local/bin:/usr/bin:/bin:/sbin"

feat(amazonq): Adding UTG build and execute logic back for internal Amazon customers. #5396

feat(amazonq): Adding UTG build and execute logic back for internal Amazon customers. #5396

Uh oh!

Conversation

Randall-Jiang commented Feb 20, 2025 • edited by laileni-aws Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Types of changes

Description

Solution

TO-DO

Checklist

License

Uh oh!

Uh oh!

Check warning

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

laileni-aws Feb 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Check warning

Uh oh!

Uh oh!

Check warning

Check warning

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Randall-Jiang commented Feb 20, 2025 •

edited by laileni-aws

Loading

laileni-aws Feb 20, 2025 •

edited

Loading