NLP2023 PDF Morphological analysis would improve performance.

  • I kind of thought it might be, but I had no proof.
  • Well tested and verified, thank goodness.

relevance - GPT3 reverses the information density. - Is a Japanese language model necessary?

https://twitter.com/hpp_ricecake/status/1636951339459829761?s=46&t=gkSZtjGEtUZPO0JCzBxCBw


This page is auto-translated from /nishio/日本語Tokenizerの違いは下流タスク性能に影響を与えるか? using DeepL. If you looks something interesting but the auto-translated English is not good enough to understand it, feel free to let me know at @nishio_en. I’m very happy to spread my thought to non-Japanese readers.