Handle ROOT's memberwise splitting

[jpivarski]

Consider the weird serialization in scikit-hep/uproot3#373, scikit-hep/uproot3#374, scikit-hep/uproot3#403, scikit-hep/uproot3#475, and scikit-hep/uproot3#495. It's field-at-a-time inside of each entry. I had thought it was Boost-inside-ROOT, but not for most of the above. It may be some ROOT serialization mode that I'm unaware of.

[jpivarski]

Add to that one more: I think scikit-hep/uproot3#510 is another example of that. Look at the second file, uproot-issue510b.root:

>>> import uproot4, skhep_testdata
>>> t = uproot4.open(skhep_testdata.data_path("uproot-issue510.root"))["EDepSimEvents"]
>>> b = t["Event"]["Trajectories.Points"]
>>> b.debug(0)

--+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+-
 64   0 101 102  64   9   0   1   0   0   0   2   0   1   0   0   0   0   2   0
  @ ---   e   f   @ --- --- --- --- --- --- --- --- --- --- --- --- --- --- ---
--+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+-
  0   0   0   1   0   0   0   0   2   0   0   0  64   0   0  60   0   4   0   1
--- --- --- --- --- --- --- --- --- --- --- ---   @ --- ---   < --- --- --- ---
--+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+-
  0   0   0   0   2   0   0   0  64   0   0  36   0   3   0   1   0   0   0   0
--- --- --- --- --- --- --- ---   @ --- ---   $ --- --- --- --- --- --- --- ---
--+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+-
  2   0   0   0  64 103 237  14  20  44 204 192  64  99 170 169 116  55  10  48
--- --- --- ---   @   g --- --- ---   , --- ---   @   c --- ---   t   7 ---   0
--+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+-
 64 192  63 130 249   6 230 103  63 240   0   0   0   0   0   0  64   0   0  60
  @ ---   ? --- --- --- ---   g   ? --- --- --- --- --- --- ---   @ --- ---   <
--+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+-
  0   4   0   1   0   0   0   0   2   0   0   0  64   0   0  36   0   3   0   1
--- --- --- --- --- --- --- --- --- --- --- ---   @ --- ---   $ --- --- --- ---
--+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+-
  0   0   0   0   2   0   0   0  64 104 149  31 100 192  97 100  64  98 140 241
--- --- --- --- --- --- --- ---   @   h --- ---   d ---   a   d   @   b --- ---
--+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+-
140  93 110   7  64 192  67  18 202 151 200 123  63 240 171 196  70 133 147  27
---   ]   n ---   @ ---   C --- --- --- ---   {   ? --- --- ---   F --- --- ---
--+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+-
 64   0   0  36   0   3   0   1   0   0   0   0   2   0   0   0  64   4 167 135
  @ --- ---   $ --- --- --- --- --- --- --- --- --- --- --- ---   @ --- --- ---
--+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+-
146 180 211 210 192  17 142 116  33 185 246 118  64  12   3 158  90 174 184  82
--- --- --- --- --- --- ---   t   ! --- ---   v   @ --- --- ---   Z --- ---   R
--+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+-
 64   0   0  36   0   3   0   1   0   0   0   0   2   0   0   0   0   0   0   0
  @ --- ---   $ --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- ---
--+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+-
  0   0   0   0 128   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0
--- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- ---
--+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+-
  0   0   0   0   0   0   0   2
--- --- --- --- --- --- --- ---
--+---+---+---+---+---+---+---+

This is a collection of std::vector<TG4TrajectoryPoint>, where TG4TrajectoryPoint is

>>> tree.file.streamer_named("TG4TrajectoryPoint").show()
TG4TrajectoryPoint (v1): TObject (v1)
    Position: TLorentzVector (TStreamerObject)
    Momentum: TVector3 (TStreamerObject)
    Process: int (TStreamerBasicType)
    Subprocess: int (TStreamerBasicType)

The first 6 bytes is header as usual: 64 0 101 102 64 9. (That's the right num_bytes for the entry.)

Next, we're looking at a split std::vector header:

| 0   1 | 0   0   0   2 | 0   1 | 0   0   0   0 | 2   0   0   0 | 0   1 | 0   0   0   0 | 2   0   0   0 |
|       |  two objects  |            bits for #1                |            bits for #2                |

Then follow two TLorentzVectors:

[191.4079686045643, 157.33318529844246, 8319.023224699498, 1.0]
[196.6600822217398, 148.4044858765412, 8326.14680764474, 1.0419352297551032]

and two TVector3:

[2.5818015538629675, -4.389114882445588, 3.5017668804712594]
[0.0, -0.0, 0.0]

and two integers, 0 and 2.

Following that is a header for 0 objects and then a header for 32 objects:

--+---+---+---+---+---+---+---+---+---+---+---+-
  0   0   0   0   0   0   0  14   0   0   0  32
--- --- --- --- --- --- --- --- --- --- ---    
--+---+---+---+---+---+---+---+---+---+---+---+-

and, indeed, there are 32 ten-byte std::vector headers:

--+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+-
  0   1   0   0   0   0   2   0   0   0   0   1   0   0   0   0   2   0   0   0
--- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- ---
--+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+-
  0   1   0   0   0   0   2   0   0   0   0   1   0   0   0   0   2   0   0   0
--- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- ---
--+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+-
  0   1   0   0   0   0   2   0   0   0   0   1   0   0   0   0   2   0   0   0
--- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- ---
--+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+-
  0   1   0   0   0   0   2   0   0   0   0   1   0   0   0   0   2   0   0   0
--- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- ---
--+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+-
  0   1   0   0   0   0   2   0   0   0   0   1   0   0   0   0   2   0   0   0
--- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- ---
--+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+-
  0   1   0   0   0   0   2   0   0   0   0   1   0   0   0   0   2   0   0   0
--- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- ---
--+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+-
  0   1   0   0   0   0   2   0   0   0   0   1   0   0   0   0   2   0   0   0
--- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- ---
--+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+-
  0   1   0   0   0   0   2   0   0   0   0   1   0   0   0   0   2   0   0   0
--- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- ---
--+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+-
  0   1   0   0   0   0   2   0   0   0   0   1   0   0   0   0   2   0   0   0
--- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- ---
--+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+-
  0   1   0   0   0   0   2   0   0   0   0   1   0   0   0   0   2   0   0   0
--- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- ---
--+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+-
  0   1   0   0   0   0   2   0   0   0   0   1   0   0   0   0   2   0   0   0
--- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- ---
--+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+-
  0   1   0   0   0   0   2   0   0   0   0   1   0   0   0   0   2   0   0   0
--- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- ---
--+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+-
  0   1   0   0   0   0   2   0   0   0   0   1   0   0   0   0   2   0   0   0
--- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- ---
--+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+-
  0   1   0   0   0   0   2   0   0   0   0   1   0   0   0   0   2   0   0   0
--- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- ---
--+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+-
  0   1   0   0   0   0   2   0   0   0   0   1   0   0   0   0   2   0   0   0
--- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- ---
--+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+-
  0   1   0   0   0   0   2   0   0   0   0   1   0   0   0   0   2   0   0   0
--- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- ---
--+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+-

Right after that, the TLorentzVectors start up again:

--+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+-
 64   0   0  60   0   4   0   1   0   0   0   0   2   0   0   0  64   0   0  36
  @ --- ---   < --- --- --- --- --- --- --- --- --- --- --- ---   @ --- ---   $
--+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+-
  0   3   0   1   0   0   0   0   2   0   0   0  64 103 237  14  20  44 204 192
--- --- --- --- --- --- --- --- --- --- --- ---   @   g --- --- ---   , --- ---
--+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+-
 64  99 170 169 116  55  10  48  64 192  63 130 249   6 230 103  63 240   0   0
  @   c --- ---   t   7 ---   0   @ ---   ? --- --- --- ---   g   ? --- --- ---
--+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+-
  0   0   0   0
--- --- --- ---
--+---+---+---+

This one is

[191.4079686045643, 157.33318529844246, 8319.023224699498, 1.0]

Similarly, there's also a "name" field that claims to have type std::string:

>>> t["Event"]["Trajectories.Name"].streamer
<TStreamerSTLstring at 0x7f33475eaf10>

but it's clearly a collection of strings (53 of them):

>>> t["Event"]["Trajectories.Name"].debug(0)

--+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+-
 64   0   0 214   0   9   5 103  97 109 109  97   3 109 117  45   6 112 114 111
  @ --- --- --- --- --- ---   g   a   m   m   a ---   m   u   - ---   p   r   o
--+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+-
116 111 110   6 112 114 111 116 111 110   6 112 114 111 116 111 110   6 112 114
  t   o   n ---   p   r   o   t   o   n ---   p   r   o   t   o   n ---   p   r
--+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+-
111 116 111 110   6 112 114 111 116 111 110   6 112 114 111 116 111 110   7 110
  o   t   o   n ---   p   r   o   t   o   n ---   p   r   o   t   o   n ---   n
--+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+-
101 117 116 114 111 110   7 110 101 117 116 114 111 110   7 110 101 117 116 114
  e   u   t   r   o   n ---   n   e   u   t   r   o   n ---   n   e   u   t   r
--+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+-
111 110   7 110 101 117 116 114 111 110   2 101  45   2 101  45   2 101  45   2
  o   n ---   n   e   u   t   r   o   n ---   e   - ---   e   - ---   e   - ---
--+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+-
101  45   2 101  45   2 101  45   2 101  45   2 101  45   2 101  45   2 101  45
  e   - ---   e   - ---   e   - ---   e   - ---   e   - ---   e   - ---   e   -
--+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+-
  2 101  45   2 101  45   2 101  45   2 101  45   2 101  45   2 101  45   2 101
---   e   - ---   e   - ---   e   - ---   e   - ---   e   - ---   e   - ---   e
--+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+-
 45   2 101  45   2 101  45   2 101  45   2 101  45   2 101  45   2 101  45   2
  - ---   e   - ---   e   - ---   e   - ---   e   - ---   e   - ---   e   - ---
--+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+-
101  45   2 101  45   2 101  45   2 101  45   2 101  45   2 101  45   2 101  45
  e   - ---   e   - ---   e   - ---   e   - ---   e   - ---   e   - ---   e   -
--+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+-
  2 101  45   2 101  45   2 101  45   2 101  45   2 101  43   7 110 101 117 116
---   e   - ---   e   - ---   e   - ---   e   - ---   e   + ---   n   e   u   t
--+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+-
114 111 110   2 101  45   2 101  45   2 101  45   2 101  45   2 101  45
  r   o   n ---   e   - ---   e   - ---   e   - ---   e   - ---   e   -

The key: I think these are both TClonesArrays! They have a non-empty fTClonesName member.

[jpivarski]

@tamasgal, I know that you're busy with Unroot.jl, but if you're ever interested in solving a mystery, we now have 6 issues that are due to cases in which ROOT writes the subentries 1, 2, 3 of structs with fields a b as

a1 a2 a3 b1 b2 b3

instead of

a1 b1 a2 b2 a3 b3

It's a lot like branch-splitting, but this happens inside of each entry. I've been able to reverse engineer the fact that this is happening, but not why it's happening: what information in the TBranch(Element), its parent branches, maybe the TTree itself, and associated streamers might tell us that we should deserialize it this way instead of the normal way. If you find anything that might shed some light on it (or even what this mode is named!), I'd be grateful.

The cases in this function:

https://github.com/scikit-hep/uproot4/blob/ccf790f9bb4c25363fdc9fabe455dd99c85d69d8/uproot4/interpretation/identify.py#L1180-L1186

are guesses, based on a few examples, so don't copy them without care!

Also, if you know of anyone else who's inclined to dig into these details, let me know. I'm looking for help!

[tamasgal]

Oh, that seems to be a tough one. I bookmarked it and try to see if I find something new! Currently I am more busy with my PhD than anything else though, but I am certainly interested 😅

I think I might have seen something similar in KM3NeT data, I have to dig in my notes, I hope I find it since for that I could also provide the source code...

[jpivarski]

I just learned from Philippe that it's called "memberwise streaming" (as opposed to "objectwise streaming") and the ROOT code for deserializing these objects is here:

https://github.com/root-project/root/blob/e87a6311278f859ca749b491af4e9a2caed39161/io/io/src/TStreamerInfoReadBuffer.cxx#L1220-L1277

The way to identify that a particular object is serialized this way is by checking for bit 14 (2**14 == 16384) in the instance version, TBufferFile::kStreamedMemberWise. Also, that bit has to be removed from the instance version before comparing it with the class version (second line of the quoted code above).

Indeed, in the uproot-issue510b.root file I investigated above, the version number does have bit 14 set:

>>> import uproot4, skhep_testdata
>>> t = uproot4.open(skhep_testdata.data_path("uproot-issue510b.root"))["EDepSimEvents"]
>>> b = t["Event"]["Trajectories.Points"]
>>> b.debug(0, limit_bytes=6)

--+---+---+---+---+---+
 64   0 101 102  64   9
  @ ---   e   f   @ ---
--+---+---+---+---+---+

The first four bytes is the size of this entry (with the kByteCountMask bit removed),

>>> numpy.array([64, 0, 101, 102], "u1").view(">u4") & ~(2**30)
array([25958])

and the next two bytes is the version number with a kStreamedMemberWise bit set,

>>> numpy.array([64, 9], "u1").view(">u2") & ~(2**14)
array([9], dtype=int32)

So these things are identified one object at a time (even though a branch is likely to consist entirely of one type of serialization or the other).

For making tests, I think the way a class can be put into this mode is by calling TClass::SetCanSplit(true) on its TClass object (TClass::GetClass("class name")). I'm not 100% certain whether this controls memberwise/objectwise splitting, ordinary branch splitting, or both. But it would be nice to see the same class written as memberwise and as objectwise, for confidence that we're doing it right.

root/test/bench.cxx might make examples with and without memberwise splitting, but this is part of ROOT's benchmark tests and relies on other code that I haven't followed to its definitions. It might be possible to simply run this benchmark to generate files with memberwise and objectwise serialization.

TVirtualStreamerInfo has a SetStreamMemberWise(bool) method, but I don't know if that means we can directly use it to make tests.

I'm just writing these things here as notes, so that this information does not get lost.

[jpivarski]

This should be considered a bug at least until we have a "not implemented" error message for this case, but fully implementing it is a feature. I think I'll put in one PR to add the "not implemented" message and then remove the "bug" label from this issue.

[tamasgal]

Oh wow, I didn't even have a chance 😅

Nice to hear that the mystery is mostly solved.

You most certainly also found this thread (also from Philippe) https://root-forum.cern.ch/t/splitability-of-classes-with-custom-streamer/32974 for me especially this statement from Philippe was quite new:

If a class has a custom Streamer we have to assume that it is for a good reason :). When splitting is used, the custom Streamer is not used at all and thus we are (silently) not doing what the user (likely) intended often leading to corrupted results.

Btw. I searched our codebase for SetCanSplit but have not found any use of that. I also have not found my notes which were about a strange split structure just like you described, only some sketches of the split-branch strategy which is well-known.

[jpivarski]

I just asked him about it at the ROOT I/O meeting, which is every Friday:

https://indico.cern.ch/category/526/

[tamasgal]

Oh nice, it seems to be open for externals, at least I was able to join the video room ;)