Improve package's doctests

[bonjourmauko]

Hi there!

I really enjoy OpenFisca, but I recently encountered an issue.

Here is what I did:

pytest openfisca_core openfisca_web_api --maxfail 0 --continue-on-collection-errors --quiet

Here is what I expected to happen:

XXX passed, X xfailed, XXX warnings in X.XXs

Here is what actually happened:

========================================================================================= short test summary info ==========================================================================================
FAILED openfisca_core/commons/formulas.py::openfisca_core.commons.formulas.apply_thresholds
FAILED openfisca_core/commons/formulas.py::openfisca_core.commons.formulas.switch
FAILED openfisca_core/holders/holder.py::openfisca_core.holders.holder.Holder.get_memory_usage
...
ERROR openfisca_core/scripts/measure_numpy_condition_notations.py::test_switch_select

Context:

Python provides doctests to help keep a testable documentation.

They provide some benefits:

• They help maintainers and contributors to spot bugs and improvement opportunities.
• They helps reusers to understand the code quickly
• They help separating contexts (unit, functional, ...)
• They help produce the official doc

Complications:

Today the code coverage by doctests is mild, they are not run, and most of them are broken.

Some observed consequences:

• The code is in overall hard to grasp without a consequent time investment, thus hard to contribute to
• The code is hard to test unitarily due to its current design
• There are then very few unit tests
• The documentation is was broken

Proposal:

Fix, complete, and run doctests systematically!

TODO

• Fix commons doctests #1250
• Fix data_storage doctests #1249
• Fix entities doctests #1254
• Fix holders doctests #1266
• Fix enums doctests #1267
• Fix parameters doctests #1268
• Fix periods doctests #1269
• Fix populations doctests #1276
• 9. Improve openfisca_core.projectors doctests
• 10. Improve openfisca_core.reforms doctests
• 11. Improve openfisca_core.simulation_builder doctests
• 12. Improve openfisca_core.simulations doctests
• 13. Improve openfisca_core.taxbenefitsystems doctests
• 14. Improve openfisca_core.taxscales doctests
• 15. Improve openfisca_core.tracers doctests
• 16. Improve openfisca_core.variables doctests
• 17. Improve openfisca_web_api.loader doctests

[MattiSG]

Thanks @maukoquiroga for providing a bit more background to this impressive series of PRs 🙂

Could you explain what “to work” and “it didn't” means for pytest? Did you not get failures when you expected to get them? Did the command simply not run?