Exportify has encountered an unexpected error (5XX)

[danseare]

Consistently getting this error for several days. When I go to the Spotify Web API Status page it reports "All Systems Operational". I suspect the most likely culprit is the very large number of liked songs I have in my account (28,375). I haven't yet tried "unliking" a portion of them but will try when time permits.

[danseare]

On 2023.04.15 I was able to do a full export when I had 28,276 liked songs.
Shortly after that with 28,455, Exortify was timing out at log in.
On 4/24, I decreased the number of likes to 19,999 and was finally able to log on. Increasing to 20,000 reintroduced the same login time out I experienced with 28,455.
Although I was able to login with 19,999 liked songs, when I initiated an export I saw that same message about 5xx error on Spotify’s side.
To get exporting to work I had to decrease the number of liked songs to 10,999. At 11,000, initiating an export reintroduced the message about 5xx error on Spotify’s side.
One more data point: I am seeing a very similar issue on https://www.nativenoise.co.za/spotify/follow-all-artists.

[flawedgun]

Oof. Another data point: I've got over 24k songs liked and now I'm getting the error. I used to use Exportify w/ no problem with ~22k-23k songs liked. The other version of Exportify is still working for me, but that one doesn't allow you to export your liked song or to do a search for playlists. Would love to see this fixed.

[watsonbox]

Thanks for the feedback. I checked on the volumes of these issues and indeed there do seem to be more of them since around April 3rd. The response from the Spotify API is:

{
  "error": {
    "status": 502,
    "message": "Bad gateway."
  }
}

As the Exportify error message explains, 5XX errors are server-side issues which can't be resolved by the client. In this case the 502 error is a generic type of error indicating that the API gateway encountered some unknown error from an upstream service.

The Spotify API does have rate limiting policies which become important when exporting large amounts of data, and returns a 429 error when the client must wait before proceeding. 4XX responses are caused by the client and are usually actionable. Exportify honors these responses, leaving it entirely up to Spotify to indicate how many requests are allowed to be made in any given time window.

That said, it's not impossible that Spotify have begun to respond to large numbers of requests with a 502 response instead of a 429 response, but that's a server-side bug and doesn't suggest that anything can be done on the client to resolve it. I haven't made any code changes to Exportify since last year, well before the increase in this type of error.

The other version of Exportify is still working for me, but that one doesn't allow you to export your liked song or to do a search for playlists.

As this issue seems to be related to the amount of data being exported, it's not surprising to me that a version which exports less data still works. I suppose one possibility might be to add a setting to "Skip large playlists", say 20k or larger. They would then have to be downloaded separately or not at all.

You might also like to try this approach to see if using a separate API application works better.

To summarize: unfortunately, any attempt to resolve a 5XX error is going to be guesswork at best, because the API isn't telling us what went wrong 😞.

[watsonbox]

I happened to see this one first, but I note that it's actually a duplicate of #140, where as @erricrice suggests, there's no harm in increasing the wait time a little to see if it helps... I'll pick that up over there.