Replace `Metadata.flags` with `Metadata.trans` #1060

mhauru · 2025-09-29T17:03:41Z

Now that the "del" flag is gone (#1058), the only flag that is ever used is "trans". Hence, no need to bother with having the Dict{String, BitVector} for Metadata.flags, and can instead have a single BitVector for Metadata.trans. EDIT: Renamed to Metadata.is_transformed.

You may wonder, given that Metadata is presumably on its way out, why bother? Two reasons:

I tried running the benchmark suite locally with VectorVarInfo, and there were some horrendous performance regressions there compared to using Metadata. Hence, we might not be about to switch over the VarNamedVector imminently.
The above experience made me wonder why there was such a performance difference, and whether the Metadata.flags field might actually be a significant cost compared to a BitVector.

My local benchmarking suggests that indeed, this makes a difference:

Before

┌───────────────────────┬───────┬─────────────┬───────────────────┬────────┬────────────────┬─────────────────┐
│                 Model │   Dim │  AD Backend │           VarInfo │ Linked │ t(eval)/t(ref) │ t(grad)/t(eval) │
├───────────────────────┼───────┼─────────────┼───────────────────┼────────┼────────────────┼─────────────────┤
│ Simple assume observe │     1 │ forwarddiff │             typed │  false │           16.0 │             1.7 │
│           Smorgasbord │   201 │ forwarddiff │             typed │  false │          790.6 │            46.1 │
│           Smorgasbord │   201 │ forwarddiff │ simple_namedtuple │   true │          382.0 │            84.3 │
│           Smorgasbord │   201 │ forwarddiff │           untyped │   true │         1431.7 │            36.0 │
│           Smorgasbord │   201 │ forwarddiff │       simple_dict │   true │        10511.1 │            21.6 │
│           Smorgasbord │   201 │ reversediff │             typed │   true │         1495.9 │            42.4 │
│           Smorgasbord │   201 │    mooncake │             typed │   true │         1637.4 │             3.4 │
│    Loop univariate 1k │  1000 │    mooncake │             typed │   true │         8635.9 │             3.2 │
│       Multivariate 1k │  1000 │    mooncake │             typed │   true │         1266.1 │             8.5 │
│   Loop univariate 10k │ 10000 │    mooncake │             typed │   true │        90116.3 │             3.2 │
│      Multivariate 10k │ 10000 │    mooncake │             typed │   true │        10364.2 │             9.7 │
│               Dynamic │    10 │    mooncake │             typed │   true │          235.0 │             5.7 │
│              Submodel │     1 │    mooncake │             typed │   true │           24.0 │             4.2 │
│                   LDA │    12 │ reversediff │             typed │   true │         1391.7 │             2.0 │
└───────────────────────┴───────┴─────────────┴───────────────────┴────────┴────────────────┴─────────────────┘

After

┌───────────────────────┬───────┬─────────────┬───────────────────┬────────┬────────────────┬─────────────────┐
│                 Model │   Dim │  AD Backend │           VarInfo │ Linked │ t(eval)/t(ref) │ t(grad)/t(eval) │
├───────────────────────┼───────┼─────────────┼───────────────────┼────────┼────────────────┼─────────────────┤
│ Simple assume observe │     1 │ forwarddiff │             typed │  false │           10.8 │             2.5 │
│           Smorgasbord │   201 │ forwarddiff │             typed │  false │          695.1 │            53.0 │
│           Smorgasbord │   201 │ forwarddiff │ simple_namedtuple │   true │          319.1 │           104.9 │
│           Smorgasbord │   201 │ forwarddiff │           untyped │   true │         1114.3 │            45.0 │
│           Smorgasbord │   201 │ forwarddiff │       simple_dict │   true │        10323.5 │            22.3 │
│           Smorgasbord │   201 │ reversediff │             typed │   true │         1190.0 │            52.4 │
│           Smorgasbord │   201 │    mooncake │             typed │   true │         1263.0 │             3.8 │
│    Loop univariate 1k │  1000 │    mooncake │             typed │   true │         5606.7 │             4.4 │
│       Multivariate 1k │  1000 │    mooncake │             typed │   true │         1236.0 │             8.7 │
│   Loop univariate 10k │ 10000 │    mooncake │             typed │   true │        63260.7 │             4.2 │
│      Multivariate 10k │ 10000 │    mooncake │             typed │   true │        11029.4 │             9.4 │
│               Dynamic │    10 │    mooncake │             typed │   true │          216.4 │             6.4 │
│              Submodel │     1 │    mooncake │             typed │   true │           19.0 │             4.6 │
│                   LDA │    12 │ reversediff │             typed │   true │         1341.4 │             2.0 │
└───────────────────────┴───────┴─────────────┴───────────────────┴────────┴────────────────┴─────────────────┘

Curious to see whether GHA benchmarks come out looking similar.

github-actions · 2025-09-29T17:05:24Z

Benchmark Report for Commit `a011dd6`

Computer Information

Julia Version 1.11.7
Commit f2b3dbda30a (2025-09-08 12:10 UTC)
Build Info:
  Official https://julialang.org/ release
Platform Info:
  OS: Linux (x86_64-linux-gnu)
  CPU: 4 × AMD EPYC 7763 64-Core Processor
  WORD_SIZE: 64
  LLVM: libLLVM-16.0.6 (ORCJIT, znver3)
Threads: 1 default, 0 interactive, 1 GC (on 4 virtual cores)

Benchmark Results

┌───────────────────────┬───────┬─────────────┬───────────────────┬────────┬────────────────┬─────────────────┐
│                 Model │   Dim │  AD Backend │           VarInfo │ Linked │ t(eval)/t(ref) │ t(grad)/t(eval) │
├───────────────────────┼───────┼─────────────┼───────────────────┼────────┼────────────────┼─────────────────┤
│ Simple assume observe │     1 │ forwarddiff │             typed │  false │            7.4 │             1.6 │
│           Smorgasbord │   201 │ forwarddiff │             typed │  false │          598.5 │            49.3 │
│           Smorgasbord │   201 │ forwarddiff │ simple_namedtuple │   true │          423.2 │            57.6 │
│           Smorgasbord │   201 │ forwarddiff │           untyped │   true │         1063.4 │            32.1 │
│           Smorgasbord │   201 │ forwarddiff │       simple_dict │   true │         6740.1 │            29.0 │
│           Smorgasbord │   201 │ reversediff │             typed │   true │          914.1 │            46.1 │
│           Smorgasbord │   201 │    mooncake │             typed │   true │          875.0 │             5.8 │
│    Loop univariate 1k │  1000 │    mooncake │             typed │   true │         4455.6 │             5.6 │
│       Multivariate 1k │  1000 │    mooncake │             typed │   true │         1020.9 │             9.3 │
│   Loop univariate 10k │ 10000 │    mooncake │             typed │   true │        51734.7 │             4.9 │
│      Multivariate 10k │ 10000 │    mooncake │             typed │   true │         8677.5 │            10.3 │
│               Dynamic │    10 │    mooncake │             typed │   true │          132.2 │            10.9 │
│              Submodel │     1 │    mooncake │             typed │   true │           10.4 │             5.6 │
│                   LDA │    12 │ reversediff │             typed │   true │          992.7 │             2.1 │
└───────────────────────┴───────┴─────────────┴───────────────────┴────────┴────────────────┴─────────────────┘

codecov · 2025-09-29T17:21:01Z

Codecov Report

❌ Patch coverage is 88.50575% with 10 lines in your changes missing coverage. Please review.
✅ Project coverage is 82.36%. Comparing base (ec65b4f) to head (db64645).
⚠️ Report is 2 commits behind head on breaking.

Files with missing lines	Patch %	Lines
src/abstract_varinfo.jl	66.66%	3 Missing ⚠️
src/simple_varinfo.jl	78.57%	3 Missing ⚠️
src/threadsafe.jl	75.00%	2 Missing ⚠️
ext/DynamicPPLEnzymeCoreExt.jl	0.00%	1 Missing ⚠️
src/varinfo.jl	97.67%	1 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##           breaking    #1060      +/-   ##
============================================
- Coverage     82.38%   82.36%   -0.03%     
============================================
  Files            42       42              
  Lines          3820     3787      -33     
============================================
- Hits           3147     3119      -28     
+ Misses          673      668       -5

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

github-actions · 2025-09-30T08:56:45Z

DynamicPPL.jl documentation for PR #1060 is available at:
https://TuringLang.github.io/DynamicPPL.jl/previews/PR1060/

mhauru · 2025-09-30T08:58:52Z

CI benchmarks. Target branch:

┌───────────────────────┬───────┬─────────────┬───────────────────┬────────┬────────────────┬─────────────────┐
│                 Model │   Dim │  AD Backend │           VarInfo │ Linked │ t(eval)/t(ref) │ t(grad)/t(eval) │
├───────────────────────┼───────┼─────────────┼───────────────────┼────────┼────────────────┼─────────────────┤
│ Simple assume observe │     1 │ forwarddiff │             typed │  false │            8.5 │             1.6 │
│           Smorgasbord │   201 │ forwarddiff │             typed │  false │          635.2 │            43.6 │
│           Smorgasbord │   201 │ forwarddiff │ simple_namedtuple │   true │          411.8 │            52.7 │
│           Smorgasbord │   201 │ forwarddiff │           untyped │   true │         1163.6 │            29.7 │
│           Smorgasbord │   201 │ forwarddiff │       simple_dict │   true │         6444.2 │            28.6 │
│           Smorgasbord │   201 │ reversediff │             typed │   true │         1022.9 │            40.9 │
│           Smorgasbord │   201 │    mooncake │             typed │   true │          980.1 │             4.5 │
│    Loop univariate 1k │  1000 │    mooncake │             typed │   true │         5750.3 │             4.3 │
│       Multivariate 1k │  1000 │    mooncake │             typed │   true │          964.6 │             9.1 │
│   Loop univariate 10k │ 10000 │    mooncake │             typed │   true │        64679.1 │             3.9 │
│      Multivariate 10k │ 10000 │    mooncake │             typed │   true │         8179.8 │            10.3 │
│               Dynamic │    10 │    mooncake │             typed │   true │          129.7 │            11.3 │
│              Submodel │     1 │    mooncake │             typed │   true │           12.2 │             5.1 │
│                   LDA │    12 │ reversediff │             typed │   true │         1006.2 │             2.0 │
└───────────────────────┴───────┴─────────────┴───────────────────┴────────┴────────────────┴─────────────────┘

This branch:

┌───────────────────────┬───────┬─────────────┬───────────────────┬────────┬────────────────┬─────────────────┐
│                 Model │   Dim │  AD Backend │           VarInfo │ Linked │ t(eval)/t(ref) │ t(grad)/t(eval) │
├───────────────────────┼───────┼─────────────┼───────────────────┼────────┼────────────────┼─────────────────┤
│ Simple assume observe │     1 │ forwarddiff │             typed │  false │            7.4 │             1.7 │
│           Smorgasbord │   201 │ forwarddiff │             typed │  false │          597.3 │            49.0 │
│           Smorgasbord │   201 │ forwarddiff │ simple_namedtuple │   true │          422.1 │            57.4 │
│           Smorgasbord │   201 │ forwarddiff │           untyped │   true │          969.2 │            35.2 │
│           Smorgasbord │   201 │ forwarddiff │       simple_dict │   true │         6575.6 │            31.0 │
│           Smorgasbord │   201 │ reversediff │             typed │   true │          883.4 │            47.6 │
│           Smorgasbord │   201 │    mooncake │             typed │   true │          854.6 │             5.1 │
│    Loop univariate 1k │  1000 │    mooncake │             typed │   true │         4305.0 │             5.6 │
│       Multivariate 1k │  1000 │    mooncake │             typed │   true │          991.4 │             9.5 │
│   Loop univariate 10k │ 10000 │    mooncake │             typed │   true │        50138.4 │             5.1 │
│      Multivariate 10k │ 10000 │    mooncake │             typed │   true │         9003.3 │            10.1 │
│               Dynamic │    10 │    mooncake │             typed │   true │          128.2 │            11.4 │
│              Submodel │     1 │    mooncake │             typed │   true │            9.9 │             5.9 │
│                   LDA │    12 │ reversediff │             typed │   true │          989.8 │             2.1 │
└───────────────────────┴───────┴─────────────┴───────────────────┴────────┴────────────────┴─────────────────┘

Roughly in line with what I saw locally. Seems worth it to me, especially if you look at the Loop univariate 1k and 10k models.

yebai · 2025-09-30T11:50:53Z

I suggest we take this chance to rename Metadata.trans to a more readable term, e.g., Metadata.is_unconstrained / Metadata.is_transformed.

mhauru · 2025-09-30T14:17:01Z

Good idea, done.

src/varnamedvector.jl

penelopeysm · 2025-09-30T16:29:34Z

src/simple_varinfo.jl

-islinked(vi::SimpleVarInfo) = istrans(vi)
+islinked(vi::SimpleVarInfo) = is_transformed(vi)


like this line is just the same function but duplicated. so it feels like to me we could just pick one and roll with it!

penelopeysm

Happy to merge after this! Very pleased with the improved performance

HISTORY.md

penelopeysm · 2025-10-09T17:11:10Z

@mhauru I'll leave you to click the green button :)

* Bump minor version * bump benchmarks compat * add a skeletal changelog * `InitContext`, part 3 - Introduce `InitContext` (#981) * Implement InitContext * Fix loading order of modules; move `prefix(::Model)` to model.jl * Add tests for InitContext behaviour * inline `rand(::Distributions.Uniform)` Note that, apart from being simpler code, Distributions.Uniform also doesn't allow the lower and upper bounds to be exactly equal (but we might like to keep that option open in DynamicPPL, e.g. if the user wants to initialise all values to the same value in linked space). * Document * Add a test to check that `init!!` doesn't change linking * Fix `push!` for VarNamedVector This should have been changed in #940, but slipped through as the file wasn't listed as one of the changed files. * Add some line breaks Co-authored-by: Markus Hauru <[email protected]> * Add the option of no fallback for ParamsInit * Improve docstrings * typo * `p.default` -> `p.fallback` * Rename `{Prior,Uniform,Params}Init` -> `InitFrom{Prior,Uniform,Params}` --------- Co-authored-by: Markus Hauru <[email protected]> * use `varname_leaves` from AbstractPPL instead (#1030) * use `varname_leaves` from AbstractPPL instead * add changelog entry * fix import * tidy occurrences of varname_leaves as well (#1031) * `InitContext`, part 4 - Use `init!!` to replace `evaluate_and_sample!!`, `predict`, `returned`, and `initialize_values` (#984) * Replace `evaluate_and_sample!!` -> `init!!` * Use `ParamsInit` for `predict`; remove `setval_and_resample!` and friends * Use `init!!` for initialisation * Paper over the `Sampling->Init` context stack (pending removal of SamplingContext) * Remove SamplingContext from JETExt to avoid triggering `Sampling->Init` pathway * Remove `predict` on vector of VarInfo * Fix some tests * Remove duplicated test * Simplify context testing * Rename FooInit -> InitFromFoo * Fix JETExt * Fix JETExt properly * Fix tests * Improve comments * Remove duplicated tests * Docstring improvements Co-authored-by: Markus Hauru <[email protected]> * Concretise `chain_sample_to_varname_dict` using chain value type * Clarify testset name * Re-add comment that shouldn't have vanished * Fix stale Requires dep * Fix default_varinfo/initialisation for odd models * Add comment to src/sampler.jl Co-authored-by: Markus Hauru <[email protected]> --------- Co-authored-by: Markus Hauru <[email protected]> * `InitContext`, part 5 - Remove `SamplingContext`, `SampleFrom{Prior,Uniform}`, `{tilde_,}assume` (#985) * Remove `SamplingContext` for good * Remove `tilde_assume` as well * Split up tilde_observe!! for Distribution / Submodel * Tidy up tilde-pipeline methods and docstrings * Fix tests * fix ambiguity * Add changelog * Update HISTORY.md Co-authored-by: Markus Hauru <[email protected]> --------- Co-authored-by: Markus Hauru <[email protected]> * fix missing import * Shuffle context code around and remove dead code (#1050) * Delete the `"del"` flag (#1058) * Delete del * Fix a typo * Add HISTORY entry about del * Fixes for Turing 0.41 (#1057) * setleafcontext(model, ctx) and various other fixes * fix a bug * Add warning for `initial_parameters=...` * Remove `resume_from` and `default_chain_type` (#1061) * Remove resume_from * Format * Fix test * remove initial_params warning * Allow more flexible `initial_params` (#1064) * Enable NamedTuple/Dict initialisation * Add more tests * fix include_all kwarg for predict, improve perf (#1068) * Fix `include_all` for predict * Fix include_all for predict, some perf improvements * Replace `Metadata.flags` with `Metadata.trans` (#1060) * Replace Medata.flags with Metadata.trans * Fix a bug * Fix a typo * Fix two bugs * Rename trans to is_transformed * Rename islinked to is_transformed, remove duplication * Change pointwise_logdensities default key type to VarName (#1071) * Change pointwise_logdensities default key type to VarName * Fix a doctest * Fix DynamicPPL / MCMCChains methods (#1076) * Reimplement pointwise_logdensities (almost completely) * Move logjoint, logprior, ... as well * Fix imports, etc * Remove tests that are failing (yes I learnt this from Claude) * Changelog * logpdf * fix docstrings * allow dict output * changelog * fix some comments * fix tests * Fix more imports * Remove stray n Co-authored-by: Markus Hauru <[email protected]> * Expand `logprior`, `loglikelihood`, and `logjoint` docstrings --------- Co-authored-by: Markus Hauru <[email protected]> * Remove `Sampler` and its interface (#1037) * Remove `Sampler` and `initialstep` * Actually just remove the entire file * forgot one function * Move sampling test utils to Turing as well * Update changelog to correctly reflect changes * [skip ci] Make changelog headings more consistent --------- Co-authored-by: Markus Hauru <[email protected]>

Replace Medata.flags with Metadata.trans

756cc25

github-actions bot assigned mhauru Sep 29, 2025

Fix a bug

1091986

mhauru added 2 commits September 30, 2025 12:21

Fix a typo

ece8fb5

Fix two bugs

a011dd6

Rename trans to is_transformed

0f8c9b1

Merge remote-tracking branch 'origin/breaking' into mhauru/delete-flags

4f85f2b

mhauru requested a review from penelopeysm September 30, 2025 16:19

penelopeysm reviewed Sep 30, 2025

View reviewed changes

src/varnamedvector.jl Show resolved Hide resolved

penelopeysm reviewed Sep 30, 2025

View reviewed changes

mhauru added 2 commits October 9, 2025 13:53

Rename islinked to is_transformed, remove duplication

0103a97

Merge remote-tracking branch 'origin/breaking' into mhauru/delete-flags

db64645

mhauru requested a review from penelopeysm October 9, 2025 15:18

penelopeysm requested changes Oct 9, 2025

View reviewed changes

HISTORY.md Show resolved Hide resolved

mhauru merged commit 01bf0bc into breaking Oct 10, 2025
18 of 19 checks passed

mhauru deleted the mhauru/delete-flags branch October 10, 2025 08:55

penelopeysm mentioned this pull request Oct 15, 2025

del "del": remove del flag #982

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Replace `Metadata.flags` with `Metadata.trans` #1060

Replace `Metadata.flags` with `Metadata.trans` #1060

Uh oh!

mhauru commented Sep 29, 2025 •

edited

Loading

Uh oh!

github-actions bot commented Sep 29, 2025 •

edited

Loading

Uh oh!

codecov bot commented Sep 29, 2025 •

edited

Loading

Uh oh!

github-actions bot commented Sep 30, 2025

Uh oh!

mhauru commented Sep 30, 2025

Uh oh!

yebai commented Sep 30, 2025 •

edited

Loading

Uh oh!

mhauru commented Sep 30, 2025

Uh oh!

Uh oh!

penelopeysm Sep 30, 2025 •

edited

Loading

Uh oh!

penelopeysm left a comment

Uh oh!

Uh oh!

penelopeysm commented Oct 9, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

		islinked(vi::SimpleVarInfo) = istrans(vi)
		islinked(vi::SimpleVarInfo) = is_transformed(vi)

Replace Metadata.flags with Metadata.trans #1060

Replace Metadata.flags with Metadata.trans #1060

Uh oh!

Conversation

mhauru commented Sep 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Before

After

Uh oh!

github-actions bot commented Sep 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmark Report for Commit a011dd6

Computer Information

Benchmark Results

Uh oh!

codecov bot commented Sep 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

github-actions bot commented Sep 30, 2025

Uh oh!

mhauru commented Sep 30, 2025

Uh oh!

yebai commented Sep 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mhauru commented Sep 30, 2025

Uh oh!

Uh oh!

penelopeysm Sep 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

penelopeysm left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

penelopeysm commented Oct 9, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Replace `Metadata.flags` with `Metadata.trans` #1060

Replace `Metadata.flags` with `Metadata.trans` #1060

mhauru commented Sep 29, 2025 •

edited

Loading

github-actions bot commented Sep 29, 2025 •

edited

Loading

Benchmark Report for Commit `a011dd6`

codecov bot commented Sep 29, 2025 •

edited

Loading

yebai commented Sep 30, 2025 •

edited

Loading

penelopeysm Sep 30, 2025 •

edited

Loading