fix(jira): update epic collector to use new API endpoint and include

[GoSimplicity]

Fixed the Epic collector to use the new Jira API endpoint and ensured all required fields are included when fetching Epic data. This resolves the synchronization failures caused by the deprecated endpoint (fix #8544 ).

[klesh]

hanks for your contribution!
This should resolve the JIRA Cloud data collection issue.
However, I’m concerned it may not work with older JIRA Server versions.
I think we should detect the JIRA version and call the appropriate APIs accordingly.

For reference, here’s some related code:

incubator-devlake/backend/plugins/jira/tasks/epic_collector.go

Line 54 in ac3b1d9

if data.JiraServerInfo.DeploymentType == models.DeploymentServer && len(data.JiraServerInfo.VersionNumbers) == 3 && data.JiraServerInfo.VersionNumbers[0] <= 8 {

[GoSimplicity]

hanks for your contribution! This should resolve the JIRA Cloud data collection issue. However, I’m concerned it may not work with older JIRA Server versions. I think we should detect the JIRA version and call the appropriate APIs accordingly.

For reference, here’s some related code:

incubator-devlake/backend/plugins/jira/tasks/epic_collector.go

Line 54 in ac3b1d9

if data.JiraServerInfo.DeploymentType == models.DeploymentServer && len(data.JiraServerInfo.VersionNumbers) == 3 && data.JiraServerInfo.VersionNumbers[0] <= 8 {

Based on your suggestion, I have made the new and old version endpoints compatible.

[klesh]

That was fast — great job!

However, I made a mistake in pointing to an inappropriate line of code for this case.
The version check on line 54 was originally added because the performance of Jira versions ≤ 8 was quite poor, so we reduced the batch size to avoid overloading or crashing the Jira server.

In the current scenario, checking only the server type should be sufficient.
I'm not sure whether Jira Server 9 supports the api/v3 endpoint or not.

[narrowizard]

Based on the information provided in the Atlassian developer changelog, the new api/3/search/jql endpoint has changed the way data is paginated. Specifically, random page access has been replaced with a continuation token API. This means that you won't be able to fetch multiple pages simultaneously using parallel threads. The startAt parameter has been replaced with nextPageToken.

Given these changes, simply updating the API endpoint without adjusting the pagination logic will result in incomplete data retrieval. The existing code that relies on the startAt parameter for pagination will no longer work as expected. It's crucial to implement the new nextPageToken mechanism to handle pagination properly and ensure that all data can be fetched correctly.

We’ll replace random page access with a continuation token API. This means you won’t be able to get multiple pages at the same time with parallel threads. startAt parameter will be replaced with nextPageToken.

[GoSimplicity]

Excellent catch on the pagination changes! You're absolutely correct that api/3/search/jql uses nextPageToken instead of startAt. I'll update the implementation to:
1.Use different pagination strategies based on the API version
2.Set concurrency to 1 for api/3 (sequential pagination only)
3.Handle the nextPageToken properly in the response parser
I'll revise the PR to address these issues. Would you prefer separate collector implementations for each API version, or a single collector with conditional logic?

[klesh]

There’s already a mechanism for handling this kind of API. Here’s how you can enable it:

1. Specify GetNextPageCustomData in ApiCollectorArgs to extract the nextPageToken.

   • This token will be stored in the RequestData.CustomData field.
   • Note: this also forces the collector to run in sequential mode (no parallel fetching).

2. In the Query function, read the CustomData and plug it into the query string.

Here’s an example:

incubator-devlake/backend/plugins/bitbucket_server/tasks/pr_collector.go

Lines 47 to 48 in 52433a8

	GetNextPageCustomData: GetNextPageCustomData,
	Query: GetQueryForNextPage,

[GoSimplicity]

That was fast — great job!

However, I made a mistake in pointing to an inappropriate line of code for this case. The version check on line 54 was originally added because the performance of Jira versions ≤ 8 was quite poor, so we reduced the batch size to avoid overloading or crashing the Jira server.

In the current scenario, checking only the server type should be sufficient. I'm not sure whether Jira Server 9 supports the api/v3 endpoint or not.

You're right about the performance considerations for JIRA Server ≤ 8. I'll keep the reduced batch size for those versions. Regarding JIRA Server 9's support for api/v3, I'll need to investigate further and potentially add version-specific handling.

[klesh]

I see that there are new commits, and the code looks good to me. Just to confirm, is it ready for merging? Have you tested it against Jira Cloud, and does the pagination work as expected?

[GoSimplicity]

I see that there are new commits, and the code looks good to me. Just to confirm, is it ready for merging? Have you tested it against Jira Cloud, and does the pagination work as expected?

Yes, the code is ready for merging. I have done some testing and it works properly.

[klesh]

LGTM
Thanks for your contribution