Huge input causes OOM.

[kenmasumitsu]

With the changes in Correctly split text into sentences #204, SudachiTokenizer analyzed all characters (was only first 4096).

The change is fine. But due to the change, I see OOM issue.
SudachiTokenizer.reset() analyzes all text and store the result in ArrayList<MorphemeList>. It causes OOM due to large list size.

I think it would be better to change the analyzing to be done gradually with the SudachiTokenizer.incrementToken() function, instead of all at once with the SudachiTokenizer.reset() function. as well as StandartTokenizer.java

[kenmasumitsu]

Further investigation,

JapaneseTokenizer.tokenizeSentences() in the Sudachi library is analyzing all text at once and storing it in local list, which seems to be the cause of the OOM.

Therefore there are two approach to resolve this.

Option 1. Fix Sudachi Plugin (this repo). SudachiTokenizer pass small chunk of text to JapaneseTokenizer.
Option 2. Fix Sudachi. JapaneseTokenizer.tokenizeSentences() does not analyze all text all at once.

[kenmasumitsu]

Could you please close this issue, because #132 is merged.

Option 1. Fix Sudachi Plugin (this repo). SudachiTokenizer pass small chunk of text to JapaneseTokenizer.
Option 2. Fix Sudachi. JapaneseTokenizer.tokenizeSentences() does not analyze all text all at once.

It applied Option 1.