Honor incoming Content-Type encoding

[fbiville]

The encoding is currently ignored and we assume UTF-8 on the way in and out.

[scothis]

The linked comment is from when the invoker directly handled remote http request that could come in with any encoding. Now the http adapter handles all requests and converts them to the streaming protocol which the invoker speaks.

What if we defined text in the streaming protocol to always be utf-8 and converted any other encoding to utf-8 in the http adapter?

[Kehrlann]

@scothis This sounds reasonable ; and then individual invokers do not need to implement transcoding themselves. It should be explicitly laid out in the invoker spec I imagine.

If a user function expected a different charset for their payload, they will have to transcode it from utf-8 specifically, but I guess this doesn't happen very often ?

On a side note, I took a look at the http-adapter, and it does not support the HTTP accept-charset header: streaming-http-adapter code / accept-charset header ; so we might want to open an issue for that.

[Kehrlann]

Here are some notes from my research on encoding in node, in case we still want to do it at the invoker level:

1. Node only supports UTF-8 and UTF-16 by default (https://nodejs.org/api/util.html#util_whatwg_supported_encodings)
2. Node can support more encodings, but it needs to be built with ICU data. https://nodejs.org/api/intl.html
3. This ICU data can be quite big, as it contains lots of stuff related to i18n, not just charsets (e.g. datetime locales)
4. The current riff images for Node do not contain extra charsets, this was run on a riff image:

> let dec = new TextDecoder('latin1');
Thrown:
RangeError [ERR_ENCODING_NOT_SUPPORTED]: The "latin1" encoding is not supported
    at new TextDecoder (internal/encoding.js:372:15)

5. The other widespread way to do transcoding in node is to use iconv-lite . It's a small-ish library, about 300Kb of charsets.
6. Fun fact: node's TextDecoder can decode all available charsets, but the TextEncoder only supports UTF-8, because it is considered Best Practice™

[ericbottard]

I'd rather have support for any encoding in the streaming case as well

[fbiville]

Now the http adapter handles all requests and converts them to the streaming protocol which the invoker speaks.

The http adapter is not involved with streaming functions. The payloads' encoding will still be user-defined.

[scothis]

I'm advocating that the streaming protocol itself define utf-8 as the encoding for text based content. Of course, binary data is still binary data and a text content could be encoded in any character set and treated as binary data. But, if we set utf-8 as the standard, then it alleviates the invokers from needing to support all encodings. Supporting an additional encoding is the same as supporting an additional content-type (an opt-in behavior that may limit compatibility).

[ericbottard]

Maybe we're saying the same thing: utf-8 should be a mandatory supported encoding for everything text based (where it makes sense), which doesn't preclude authors from supporting other encoding if they want. In the same way that text/plain, application/json and octet-stream are the 3 mandatory content-types today. In which case I +1

[Kehrlann]

I'm not sure I understand what the different arguments are here. Suppose I make an HTTP request with the following headers:

Content-Type: application/json;charset=iso-8859-1
Accept: application/json
Accept-Charset: iso-8859-1

Whose responsibility is it to decode the payload and encode the response ? To me there are 4 possibilities, here's my current understanding. I think noone advocates for Option 4, but I'm not sure about the others. I hope this clarifies the discussion, at least for me 🙂

Option 1: The streaming http adapter

• The streaming http adapter takes the incoming requests, decodes the payload from iso-8859-1 to an utf-8 string, and then makes a riff call specifying the payload is applicaton/json encoded in utf8
• The invoker receives the payload, reads it as an utf-8 string, makes an object by parsing the json, and then forwards the object to the user function (with keys and string values as utf-8)
• The user function reads the payload, performs computation and returns an object with keys and string values in utf-8
• The invoker marshalls the content as JSON, without changing any encoding, and writes it to the output stream
• It receives the response, as application/json and utf8, then encodes the response to iso-8859-1

In this case this requires changes to the stream-http-adapter to support encoding. The streaming input/output processors should also have encoding/decoding capabilities.
However, the invokers all speak utf-8 only, and there's no need to update this particular invoker.

Option 2: The node invoker

• The streaming http adapter receives the request and forwards it to the invoker as is
• The invoker receives the requests, the decodes the payload from iso-8859-1 into a utf-8 string, then makes the json object, etc.
• [rest is left as an exercise for the reader]

This means virtually no changes to the streaming-http-adapter or streaming in/outs.
However this particular invoker (and others !) should support all possible encodings for text/plain and application/json content-types/accept-charset ; and honor the user-defined charsets.
(Small caveat, the streaming-http-adapter probably needs to capture the "Accept-Charset" http header and pass it in the Expected content types of the start frame)

Option 3: A user-provided un-marshaller

• The streaming http adapter receives the request and forwards it to the invoker as is
• The invoker detects that despite the request having an application/json content-type, the charset is not utf-8 ; so it looks up user-provided unmarshallers for an application/json+iso-8859-1 unmarshaller
• If it finds one, it uses that to make the object passed to the function
• [rest is left as an exercise for the reader]

Same as option 2, little changes are required for the streaming-http-adapter and streaming endpoints.
This particular invoker needs to detect that it receives a textual payload in an unsupported encoding ; and look up an (un)marshaller for that, instead of trying to make an object from a string (which it now supposes is utf-8). If I understand correctly, custom (un)marshallers are not implemented yet but will be, see #87

Option 4: Riff only supports utf-8 throughout the entire process

• The streaming http adapter receives the request and forwards it to the invoker as is
• The invoker detects an unsupported encoding, and calls the function with a byte array and potentially params specifying the encodings; it's up to the function to decode the payload and make whatever it wants with it.

Same as option 2, little changes are required for the streaming-http-adapter and streaming endpoints.
This particular invoker needs to detect unsupported encoding and avoid all decoding, passing pure binary data in that case. And it needs to signal to the function which encoding the end-user/caller specified.

[scothis]

@Kehrlann thanks for laying out the options so clearly.

I'm advocating to explore Option 1 as it centralizes where character encodings need to be dealt with.

Options 2 and 3 are effectively the same from the backplane's perspective. It would be up to each invoker to choose if/how to expose supporting additional content-types (and likewise for charsets). The difference is in who's is responsible for providing the codec once the message is passed to the invoker.

[ericbottard]

I think we're indeed all saying the same thing: the adapter converts to utf-8, which is a required encoding for invokers to support, yet not forcing them to support only that encoding.