First pass at fixing tests from Py2->3 upgrade

[fungjj92]

Overview

This PR is a first go at fixing the failing tests from the Py2->3 upgrade.
The tests should be fixed for the exporter app 🎉 One down.

I touched tests / code across the django apps api, exporter, importer, treemap. Except for exporter, they continue to have errors. I'm not sure how the test runner selects the order of tests to run because it skipped around these 4 apps when running ./scripts/manage.py test --failfast

Most of the changes thus far have been about unicode/byte - string translation.
The celery and Django-tinsel upgrades were to fix test-related errors, see commit df782b1

Notes

In retrospect, I would have worked on one app's tests at a time. My advice to parallelize this work moving forward is just that:
./scripts/manage.py test <app_name> --failfast

It's impossible to linearly compare progress for various reasons. But anyway, we're making progress!

Before:

Ran 1115 tests in 275.613s

FAILED (failures=19, errors=305, skipped=23)

And this is what remains after related-PRs 😳

Ran 1115 tests in 331.760s

FAILED (failures=26, errors=125, skipped=23)

Testing

In the app VM, pip install the new python dependencies:
vagrant ssh app -c 'cd /opt/app/core && envdir /etc/otm.d/env pip3 install -r ./requirements.txt'

Checkout the otm-addons companion branch https://github.com/OpenTreeMap/otm-addons/pull/1566

I suppose you'll want to test each of the apps that was touched and check if any still failling tests are a result of lines of codes touched in this PR.
./scripts/manage.py test <app_name>

Also ensure that the changes don't chip away important context just to get tests to work.

[jwalgran]

The majority of these changes are clear and straightforward. I made a few comments and suggestions.

[fungjj92]

Standby. I am in the middle of fixing auth test issues in this branch, I realized there were some errors I was making in encoding/decoding.

[fungjj92]

Alright this is ready to be looked at.

Ran 1115 tests in 297.171s

FAILED (failures=25, errors=125, skipped=23)

I had been clobbering the basic auth tests with unwanted layers of encoding & decoding. I'm not sure how these failures didn't arise earlier, but alas. I was tipped off by the lack of output at the regex parsers; no surprise but regex is great at preserving and communicating its desired data format, which was helpful in figuring out how the auth data should look.

[jwalgran]

I got 264 errors instead of 125

Ran 1115 tests in 322.333s

FAILED (failures=25, errors=264, skipped=23)
Destroying test database for alias 'default'...

I verified that I had the correct branches and SHAs checked out.

~/Projects/otm-cloud (python3)
> kj git cb
===== otm-cloud =====
python3
===== otm-core =====
jf/fix-tests-2to3
===== otm-addons =====
jf/fix-tests-2to3
===== otm-tiler =====
develop
===== otm-ecoservice =====
develop

~/Projects/otm-cloud (python3)
> kj git status
===== otm-cloud =====
On branch python3
nothing to commit, working tree clean
===== otm-core =====
On branch jf/fix-tests-2to3
Your branch is up to date with 'origin/jf/fix-tests-2to3'.

nothing to commit, working tree clean
===== otm-addons =====
On branch jf/fix-tests-2to3
Your branch is up to date with 'origin/jf/fix-tests-2to3'.

nothing to commit, working tree clean
===== otm-tiler =====
On branch develop
Your branch is up to date with 'origin/develop'.

nothing to commit, working tree clean
===== otm-ecoservice =====
On branch develop
Your branch is up to date with 'origin/develop'.

nothing to commit, working tree clean

~/Projects/otm-cloud (python3)
> kj git rev-parse HEAD
===== otm-cloud =====
25eda864651968a4f67298b5c1acada7d437d3a8
===== otm-core =====
82c79a2991aa2bd407706880ae734d859ff773c9
===== otm-addons =====
4b94756207566b68736e9e6fe3fe089cb743d11a
===== otm-tiler =====
65328ada3d6303b59b60d31d16dbc2909e3c2816
===== otm-ecoservice =====
8aeb53b2d858a3227e443c7f912d13bca77bacb0

I will reprovision my VM from scratch to make sure something strange isn't going on.

I also commented on a concern I have about removing base64 encoding from the auth process.

[jwalgran]

Consider updating the commit message to explain why we are removing this base 64 encoding. It is my understanding that basic auth requires base 64.

In basic HTTP authentication, a request contains a header field in the form of Authorization: Basic <credentials>, where credentials is the base64 encoding of id and password joined by a single colon :.

from https://en.wikipedia.org/wiki/Basic_access_authentication

[fungjj92]

Thanks so much for this detail. I took another dive in with this better baseline understanding, and get what is going on 100% now. In py3, there are ripple effect from having to have the initial credentials start as bytes. This complication appears on these lines: https://github.com/OpenTreeMap/otm-core/pull/3288/files#diff-c7aac248bb6ed915db70251295a116a3R96-R103. I pushed up changes to this effect.

[jwalgran]

Consider adding a comment explaining why it is necessary to convert the reader to a list. It looks like the only use of reader_rows in in the list comprehension below, which I would expect to be able to iterate over reader as-is.

[jwalgran]

I did provision a fresh app VM and got the same results

FAILED (failures=25, errors=264, skipped=23)

[jwalgran]

Thank you for your diligent work on this PR. I compared the summary output of ./scripts/manage.sh test.

On the python3 branch

FAILED (failures=19, errors=392, skipped=23)

On this branch

FAILED (failures=25, errors=264, skipped=23)

👍

I also made a few nonessential suggestions.

[fungjj92]

Thank you for the well-considered rounds of reviews. Onward! Merging.