nishio I’m wondering if someone could do a solid comparison experiment, but I have better skin in the game when interacting with ChatGPT in English than in Japanese. [nishio If this gap does not narrow, I think the dynamics of language will change dramatically.
-
The reason most Japanese people speak Japanese in their daily lives is that their conversation partners around them speak Japanese, so they are only stuck in the local minimum.
lempiji In the MMLU task, it is best to listen in English, and then there are official indications that listening in Japanese on GPT-4 is better performance than listening in English on GPT-3.5, etc… There is certainly a language power balance…
- MMLU: Massive Multitask Language Understanding(MMLU)
- GPT-4
-
Many existing ML benchmarks are written in English. To get an initial sense of capability in other languages, we translated the MMLU benchmark—a suite of 14,000 multiple-choice problems spanning 57 subjects—into a variety of languages using Azure Translate (see Appendix).
This page is auto-translated from /nishio/言語の力学 using DeepL. If you looks something interesting but the auto-translated English is not good enough to understand it, feel free to let me know at @nishio_en. I’m very happy to spread my thought to non-Japanese readers.