FIX: Numpy pre-release accommodations

[effigies]

This has turned into something of an omnibus PR, which does the following things:

• Updates the setup script to use setuptools and take advantage of conditional dependencies
• Require bz2file for Python 2.7 only
• Add Opener.readinto method (via RF: Circumvents a deprecation warning from np.fromstring #702)
• Use bytearray and filelike.readinto in numerous places to preserve zero-copy construction of numpy arrays
• Remove deprecated numpy.fromstring (via RF: Circumvents a deprecation warning from np.fromstring #702)

Closes #702.

---

Original

It looks like we've been taking advantage of numpy letting us declare a memory block writeable even when the underlying object is ostensibly immutable. This has allowed us to avoid copies for objects that we knew wouldn't be accessed through their original interfaces, but they decided this was a bug and closed this loophole in numpy/numpy#11739.

Don't know what else to do but make the copies they think we should be making, but perhaps others have more numpy arcana to share.

Fixes #697.

[matthew-brett]

For volumeutils, I think this only represents a change in behavior for bz2 files on Python 2.7, because:

• we first check if the file-like object has a readinto method, and use it if so, to read into a bytearray. This is mutable. GzipFile, io.BytesIO and standard file objects have readinto in Python 2.7 and >=3.5. BZ2File also had this method for Python >= 3.5, but not for Python 2.7. So bz2 files fall through to the next check, on Python 2.7.
• Then we check if the file is in the SAFE_STRINGERS tuple; this only includes GzipFile and BZ2File;
• If so, then we attempt to avoid the copy. Here's where the BZ2File objects will cause a new double copy, because we cannot set the read contents as writeable.

The neater fix, for volumeutils, would be to drop the SAFE_STRINGERS check. This will make the code simpler at the expense of making BZFile objects copy for numpies, where we can set the writeable flag.

[matthew-brett]

For the trackvis reader, we could probably avoid nearly all the memory inflation by using the readinto trick that volumeutils is already using.

[matthew-brett]

This suggestion would help - we could depend on bz2file on Python 2.7, and then drop the SAFE_STRINGERS logic, with no loss of memory performance on any Numpy version.

[matthew-brett]

We still need to use the bz2file.BZ2File object instead of the the bz2.BZ2File object, in openers, for Python 2. Maybe:

import sys
if sys.version_info[0] < 3:
    from bz2file import BZ2File
else:
    from bz2 import BZ2File

and so on, maybe importing from here for other uses of BZ2File. I can't think immediately of a test for saving memory here, can you?

[effigies]

As in to avoid an early import? pkgutil.find_loader() is a Python 2/3-safe way to see if a module can be imported without actually importing.

[matthew-brett]

No, sorry, the bz2file package provides an alternative BZ2File implementation, it does not replace the implementation in the bz2 package. If you check, I think you'll find that bz2.BZ2File still doesn't have readinto after you've installed bz2file, but bz2file.BZ2File does have readinto (on Python 2.7).

[effigies]

Ah, reparsed. Yeah, I still need to make that change. On the whole, this PR just isn't ready. To get tests working again, I still need to figure out why we can't use version selectors in install_requires. I suspect it's a distutils or setuptools problem.

[coveralls]

Coverage decreased (-0.004%) to 91.846% when pulling ea1b0cc on effigies:fix/writable_bug into d9c864b on nipy:master.

[codecov-io]

Codecov Report

Merging #700 into master will decrease coverage by 0.01%.
The diff coverage is 92.3%.

@@            Coverage Diff             @@
##           master     #700      +/-   ##
==========================================
- Coverage   88.88%   88.86%   -0.02%     
==========================================
  Files          93       93              
  Lines       11449    11455       +6     
  Branches     1892     1894       +2     
==========================================
+ Hits        10176    10180       +4     
- Misses        933      934       +1     
- Partials      340      341       +1

Impacted Files	Coverage Δ
nibabel/volumeutils.py	92.75% <100%> (-0.03%)	⬇️
nibabel/streamlines/trk.py	94.31% <100%> (+0.01%)	⬆️
nibabel/gifti/parse_gifti_fast.py	84.51% <100%> (ø)	⬆️
nibabel/streamlines/tck.py	98.9% <100%> (+0.01%)	⬆️
nibabel/info.py	100% <100%> (ø)	⬆️
nibabel/nifti1.py	91.22% <100%> (ø)	⬆️
nibabel/openers.py	75.8% <71.42%> (-0.67%)	⬇️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d9c864b...ea1b0cc. Read the comment docs.

[effigies]

@arokem I ended up needing your Opener.readinto to resolve the streamlines complaints, so I just merged your PR into this one.

(Moved here from the edited top post, since I'm not sure if adding a handle in an edit notifies people.)

[effigies]

@matthew-brett This one is ready for review.

[matthew-brett]

Great - thanks - just a couple of comments.

[matthew-brett]

Might be worth a comment explaining use of readinto.

[matthew-brett]

I guess it's just possible someone else is using nisext. Any way to avoid the API change?

[effigies]

@matthew-brett Any more comments?

[effigies]

@matthew-brett Going to merge this and start the 2.3.2 process. If you have further concerns, can you open a PR with a suggested fix?

[matthew-brett]

Good call - looks fine to me.