-
Notifications
You must be signed in to change notification settings - Fork 32
Open
Description
Either I have totally misunderstood the documentation or printing of Fixed
numbers is wrong.
julia> Fixed{Int8,2}(0.25)
0.2Q5f2
julia> Fixed{Int8,3}(0.125)
0.12Q4f3
Looking into FixedPointNumbers.jl/show
leads to
digits=ceil(Int, f * log10_2)
which is wrong, as for 2^-f
the number of decimal fraction digits scales linearly and not logarithmic with f
's number of binary fraction digits, as you can see here
julia> 2.0.^(-1:-1:-10)
10-element Vector{Float64}:
0.5
0.25
0.125
0.0625
0.03125
0.015625
0.0078125
0.00390625
0.001953125
0.0009765625
Consequently this can be fixed by using digits=f
.
Metadata
Metadata
Assignees
Labels
No labels