You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: README.md
+46-39Lines changed: 46 additions & 39 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -36,55 +36,70 @@ Also see the [IETF specification](https://www.ietf.org/archive/id/draft-devault-
36
36
## Project Goals
37
37
38
38
**Goals:**
39
+
39
40
-**Fast** — Self-contained binary encoding, similar to a tuple structure
40
41
-**Simple** — Can be reimplemented in under an hour
41
42
-**Portable** — Cross-language support with well-defined standardization
42
43
43
44
**Non-goals:**
45
+
44
46
-**Data compactness** — That's what gzip is for
45
-
-**Provide an RPC layer** — This is trivial to implement yourself based on your specific requirements
47
+
-**RPC layer** — This is trivial to implement yourself based on your specific requirements
46
48
47
49
## Use Cases
48
50
49
-
- Defining network protocols
50
-
- Storing data at rest that needs to be upgradeable:
51
-
- Binary data in databases
52
-
- File formats
51
+
-**Network protocols** — Allow the server to cleanly evolve the protocol version without breaking old clients
52
+
-**Data at rest** — Upgrade your file format without breaking old files
53
+
54
+
## Overview
55
+
56
+
VBARE works by declaring a schema file for every possible version of the schema, then writing conversion functions between each version of the schema.
57
+
58
+
**Versions**
53
59
54
-
## At a Glance
60
+
Each message has an associated version, an unsigned 16 bit integer. Each version of the protocol increases monotonically from 1 (e.g. `v1`, `v2`, `v3`). This version is specified in the file name (e.g. `my-schema/v1.bare`).
55
61
56
-
- Every message has a version associated with it, either:
57
-
- Pre-negotiated (via mechanisms like HTTP request query parameters or handshakes)
58
-
- Embedded in the message itself
59
-
- Applications provide functions to upgrade between protocol versions
60
-
- There are no evolution semantics in the schema itself — simply copy and paste the schema to write a new version
62
+
**Schema Evolution & Converters**
61
63
62
-
## Evolution Philosophy
64
+
On your server, you manually define code that will convert between versions for both directions (upgrade for deserialization, downgrade for serialization).
63
65
64
-
- Declare discrete versions with predefined version indexes
65
-
- Manual evolutions simplify application logic by putting complex defaults in your application code
66
-
- Stop making big breaking v1 to v2 changes — instead, make much smaller changes with more flexibility
67
-
- Reshaping structures is important, not just changing types and names
66
+
There are no evolution semantics in the schema itself — simply copy and paste from `v1` to `v2` the schema to write a new version.
68
67
69
-
## Specification
68
+
**Servers vs Clients**
70
69
71
-
### Versions
70
+
Servers need to include converters between all versions.
72
71
73
-
Each schema version is a monotonically incrementing integer. _[TODO: Specify exact integer type]_
72
+
Clients only need to inlucde a single version of the schema since the server is responsible for version conversion no matter what version you connect with.
74
73
75
-
### Embedded Version
74
+
**Embedded vs Negotiated Versions**
76
75
77
-
Embedded version works by inserting an integer at the beginning of the buffer. This integer is used to define which version of the schema is being used. _[TODO: Specify exact integer type]_
76
+
Every message has a version associated with it. This version is either:
77
+
78
+
- Pre-negotiated (via mechanisms like HTTP request query parameters or handshakes)
79
+
- For example, you can extract the version from a request like `POST /v3/users`
80
+
- Embedded in the message itself in the first 2 bytes of the message (see below)
81
+
82
+
**Embedded Binary Format**
83
+
84
+
Embedded version works by inserting an unsigned 16 bit integer at the beginning of the buffer. This integer is used to define which version of the schema is being used.
78
85
79
86
The layout looks like this:
80
87
81
88
```
82
-
[TODO: Add layout diagram]
89
+
+-------------------+-------------------+
90
+
| Schema Version | BARE Payload |
91
+
| (uint16, 2B) | (variable N B) |
92
+
+-------------------+-------------------+
83
93
```
84
94
85
-
### Pre-negotiated Version
95
+
##Philosophy
86
96
87
-
Often, you specify the protocol version outside of the message itself. For example, when making an HTTP request with the version in the path like `POST /v3/users`, we can extract version 3 from the path. In this case, VBARE does not insert a version into the buffer. For this use case, VBARE simply acts as a step function for upgrading or downgrading version data structures.
97
+
The core of why VBARE was designed this way is:
98
+
99
+
- Manual evolutions simplify application logic by putting all complex evolutions & defaults in a conversion code instead of inside your core applciation logic
100
+
- Manual evolution forces you to handle edge cases of migrations & breaking changes at the cost of more verbose migration code
101
+
- Stop making big breaking v1 to v2 changes — instead, make much smaller schema changes with more flexibility
102
+
- Schema evolution frequently requires more than just renaming properties (like Protobuf, Flatbuffers, Cap'n'proto) — more complicated reshaping & fetching data from remote sources is commonly needed
88
103
89
104
## Implementations
90
105
@@ -108,25 +123,10 @@ _Adding an implementation takes less than an hour — it's really that simple._
-[File system driver](https://github.com/rivet-dev/rivetkit/tree/b81d9536ba7ccad4449639dd83a770eb7c353617/packages/rivetkit/schemas/file-system-driver)
110
125
111
-
## Embedded vs Negotiated Version
112
-
113
-
_[TODO: Add detailed comparison]_
114
-
115
126
## Comparison with Other Formats
116
127
117
128
[Read more](./docs/COMPARISON.md)
118
129
119
-
## Clients vs Servers
120
-
121
-
- Only servers need to have the evolution steps
122
-
- Clients just send their version
123
-
124
-
## Downsides
125
-
126
-
- Extensive migration code required
127
-
- The older the version, the more migration steps needed (though these migration steps should be effectively free)
128
-
- Migration steps are not portable across languages, but only the server needs the migration steps, so this is usually only implemented once
129
-
130
130
## FAQ
131
131
132
132
### Why is copying the entire schema for every version better than using decorators for gradual migrations?
@@ -153,6 +153,13 @@ Yes, but after enough pain and suffering from running production APIs, this is w
153
153
154
154
Most of the time, structures will match exactly, and most languages can provide a 1:1 migration. The most complicated migration steps will be for deeply nested structures that changed, but even that is relatively straightforward.
155
155
156
+
### What are the downsides?
157
+
158
+
- More verbose migration code — but this is usually because VBARE forces you to handle all edge cases
159
+
- The older the version, the more migration steps need that need to run to bring it to the latest version — though migration steps are usually negligible in cost
160
+
- Migration steps are not portable across languages, but only the server needs the migration steps, so this is usually only implemented once
0 commit comments