Allow user to disable unnecessary model evaluations after #2202

[torfjelde]

After #2202 we will perform one additional evaluation per MCMC iteration than before.

In most use-cases we and users encounter in practice, this is negligible compared to the inference itself, e.g. when using NUTS, but in some cases it can be a significant increase in computational cost.

Unfortunately this is necessary to support "all" possible models and stealthy bugs causing incorrect results in some cases, e.g. see referenced issues in #2202. But IMO we should also provide the user a way to turn this off, as it is not required for most models encountered in practice.

One simple way of doing this is to let the user tell Turing.jl that "hey, this model is 'static' (in some sense), so perform optimizations where you can". At the moment, this would have to be done by the user manually, e.g. by setting a variable attached to the Model itself telling us that it has static support, etc., which is not ideal, but I don't really see a way around it in the near future (long-term we can definitely do better, e.g. analyze the IR in the model).

I'm of the opinion that adding something like this now is worth it, because I know I would personally be annoyed if I'm forced to perform redundant operations when my model does not require it, without an easy way to turn it off. But I know there are other opinions floating around regarding this.

Thoughts? @devmotion @yebai @sunxd3

[yebai]

Can we simply pass an additional argument to the sample function, e.g. sample(;is_static=true)?

[yebai]

For HMC, this is per MCMC proposal instead of per leapfrog iteration, right?

[torfjelde]

Can we simply pass an additional argument to the sample function, e.g. sample(;is_static=true)?

Technically possible, but this will be very invasive as it requires propagating kwargs through sooo much of the Turing.jl code base. I started out doing this in an attempt to add this and then just gave up because it got so unwieldy.

[sunxd3]

I am checking out the has_static_support stuff, but it's removed. Tor are you suggesting some similar mechanism?

[torfjelde]

Exactly:)

[penelopeysm]

There are now multiple reasons why the model is re-evaluated, e.g. tracking the correct (unlinked) log-densities (previously the samplers would just return :lp in whatever space they were operating in). The values in unlinked space (values_as_in_model) are now calculated in the same pass. So I don't think it's easy to cut out the extra evaluation any more.

On the bright side, one could sort of consider this fixed, in that we needed to add an extra evaluation to calculate logp anyway (there's no real way around that apart from maybe tearing up LogDensityFunctions). So by lumping this into it, we kind of have prevented the extra evaluation (i.e. we evaluate once instead of twice).