Fix : Unable to poll results in async API #515
Conversation
jprakash-db
requested review from
andrefurlan-db,
benc-db,
jackyhu-db,
rcypher-databricks and
yunbodeng-db
as code owners
src/databricks/sql/thrift_backend.py
Outdated
| # For async operation we don't want the direct results | ||
| getDirectResults=ttypes.TSparkGetDirectResults( | ||
| maxRows=max_rows, maxBytes=max_bytes | ||
| maxRows=0 if async_op else max_rows, |
There was a problem hiding this comment.
Instead of setting maxRows as 0, I would recommend not specifying the direct result object in the request altogether. This is to make the intention clear of not expecting the direct results.
There was a problem hiding this comment.
Have made the object None
jprakash-db
had a problem deploying
to
azure-prod
GitHub Actions
Failure
| getDirectResults=None | ||
| if async_op | ||
| else ttypes.TSparkGetDirectResults( | ||
| maxRows=max_rows, | ||
| maxBytes=max_bytes, | ||
| ), |
There was a problem hiding this comment.
do we have a linter for python driver yet? if not, could you please open an issue to community or internally?
There was a problem hiding this comment.
@jayantsing-db I can raise a new PR for that
jprakash-db
had a problem deploying
to
azure-prod
GitHub Actions
Failure
jprakash-db
had a problem deploying
to
azure-prod
GitHub Actions
Failure
jprakash-db
temporarily deployed
to
azure-prod
GitHub Actions
Inactive
| with self.cursor() as cursor: | ||
| while duration < min_duration: | ||
| assert scale_factor < 512, "Detected infinite loop" | ||
| assert scale_factor < 1024, "Detected infinite loop" |
There was a problem hiding this comment.
Could you explain this change?
There was a problem hiding this comment.
For long running queries the previous upper limit of scale factor was not sufficient to run queries for atleast 5 minutes, hence increased it
Description
Polling using the async API returned the
RESOURCE_NOT_FOUNDerror for async executed queriesFix
Force retention of data on the server side by not returning the results in the
direct resultswithin theexecute_queryresponseTesting
Added e2e tests