Unclear whether control characters are allowed in comments

It is not clear from the specification whether control characters (U+0000 .. U+001F and U+007F) are allowed in comments.

The section Comment only states that "A hash symbol marks the rest of the line as a comment.".
This implies that any non-newline characters are allowed after the hash symbol.

Discussion about control characters only appears in the section String, which should have no implication on comments.

The ABNF definition forbids control characters (except tab) in comments, but the ABNF is not authoritative.

I think the section Comment should explicitly state which characters are allowed in comments.
Note that it would be unlogical to allow e.g. form-feed inside comments, because form-feed is traditionally a stronger separator than newline.

[pradyunsg]

ABNF is not authoritative.

It is.

I think the section Comment should explicitly state which characters are allowed in comments.

If there's more people who think this would add value, we might do this. I don't want to complicate an otherwise straightforward explanation.

ABNF is not authoritative.

It is.

The general theme in issues #566, #567, #568, #569 is that I believe the specification to be ambiguous. I assumed the text document is the complete, stand-alone, authoritative specification, while the ABNF has only experimental status.

If indeed the ABNF is authoritative, perhaps these issues are resolved.
However I think in that case the specification text should explicitly declare the ABNF grammer to be authoritative and defer to it for details. Otherwise how will anybody know where to look for a precise definition of TOML ?

[rmunn]

From the point of view of a parser, it's probably easier to allow control characters (except for newline) in comments, because then the rule is "Once you see a comment marker, ignore everything until the next newline". This is simpler (at least for some parser libraries) to implement than "Once you see the comment marker, ignore all valid characters but throw an error for 0x00-0x1f and 0x7f, and switch modes at the newline".

So I'd personally prefer for the rule to be "control characters are legal (and ignored) in comments; after a comment marker is seen, only the next newline matters."

[ChristianSi]

@rmunn On the other hand, if control chars are prohibited in comments too, they would be prohibited anywhere in a TOML document since they are also prohibited in strings of all kinds (and they are certainly not allowed outside of strings or comments). So prohibiting them everywhere might actually be quite simple for many parsers.

[pradyunsg]

Currently, I am ambivalent on this issue tbh. I don't see any major gains either way so I'm inclined to say status quo would win here.

I can be convinced either way though.

[ChristianSi]

@pradyunsg I assume if you say "status quo" you mean the status quo as defined by the ABNF (which prohibits control chars in comments)?

The original poster's point was exactly that, in the human-readable spec, the status quo is not defined. I too would advice clarifying this in the written spec in addition to the ABNF, for people who can read English better than Backus–Naur.

[pradyunsg]

Thanks everyone for the patience here -- I took a long time to come back around to this.

Yes, I think forbidding control characters in comments makes sense. If there's a significant use case for control characters in comments, do holler.

Meanwhile, I think what we need here is a PR clarifying in text what's already clear in the ABNF -- that control characters are not allowed in comments.