Tokenizer not respecting line continuation in no parens call with signed argument

[lukaszsamson]

Elixir and Erlang/OTP versions

Erlang/OTP 27 [erts-15.2.5] [source] [64-bit] [smp:12:12] [ds:12:12:10] [async-threads:1] [jit]

Elixir 1.18.4 (compiled with Erlang/OTP 27)

Operating system

any

Current behavior

When a function is called with an argument both single line and line continued forms produce equivalent AST

asd 1

produces

iex(15)> Code.string_to_quoted!("asd 1")
{:asd, [line: 1], [1]}

and

asd\
1

produces

iex(16)> Code.string_to_quoted!("asd\\\n1")
{:asd, [line: 1], [1]}

Whereas for signed arguments it's not consistent

asd -1

produces

iex(17)> Code.string_to_quoted!("asd -1")
{:asd, [ambiguous_op: nil, line: 1], [{:-, [line: 1], [1]}]}

but

asd\
-1

produces

iex(18)> Code.string_to_quoted!("asd\\\n-1")
{:-, [line: 2], [{:asd, [line: 1], nil}, 1]}

The bug is in tokenizer, handle_space_sensitive_tokens which is responsible for special https://github.com/elixir-lang/elixir/blob/d21a13b9152a487e82c0e41ebb20ecb0d00e521d/lib/elixir/src/elixir_tokenizer.erl#L989C1-L989C30 treatment of identifier followed by dual_op token is only called after tokenizer consumes a horizontal space in

elixir/lib/elixir/src/elixir_tokenizer.erl

Line 602 in d21a13b

handle_space_sensitive_tokens(Remaining, Line, Column + 1 + Stripped, Scope, Tokens);

. When an escaped newline is processed in

elixir/lib/elixir/src/elixir_tokenizer.erl

Line 621 in d21a13b

tokenize("\\\n" ++ Rest, Line, _Column, Scope, Tokens) ->

, tokenizer advances line and skips spaces in

elixir/lib/elixir/src/elixir_tokenizer.erl

Line 723 in d21a13b

tokenize_eol(Rest, Line, Scope, Tokens) ->

without invoking the special identifier followed by dual_op code.

Expected behavior

Line continuation should produce equivalent AST