Thoughts on the Japanese Language Model

2023-02-21

• I was asked the question verbally a few days ago and answered it verbally, but I drew the diagram again because I couldn’t convey the image in my brain in words.
• (I later realized that I had unexpectedly done “What the language model cannot do”).

• For Thumbnail

Write down your thoughts on the question, “Can creating a language model specifically for the Japanese language give it an advantage over existing language models due to circumstances unique to the Japanese language?

• Q: Isn’t the Japanese language more expressive than English?

• A: I don’t think so.

  • Natural language has developed for the purpose of communication between homo sapiens
    • So “getting the message across” is most important.
    • The average listener’s ability to understand the language does not increase due to bottlenecks in the ability to express the language.
    • If the “average listener’s ability to understand” does not differ significantly by ethnicity or race, then the natural language used by that ethnic group will also have the same ability to express itself.
    • 
      • (In science fiction terms, people who have lived in a harsh colony for generations might have a higher level of comprehension than the average homo sapiens, but, well, it’s not much different on Earth today.)
    • For example, in mathematics, artificial vocabulary and syntax are used to distinguish “concepts that are used vaguely and indistinguishably in natural language.
      • This artificial language need not be understood by the average homo sapiens.
      • So it can have higher expressive power than natural language.
      • 
      • However, the ability to express yourself in areas you are not interested in is sometimes discarded.
    • Both English and Japanese are natural languages, which caps the ability to express oneself.
      • Easy compared to the ability to understand “the type of language that can only be understood by trained experts,” such as technical discussions among mathematicians or the behavior of complex algorithms
  • Side-by-side view
    • 

• There is a difference or there isn’t.

  • 

  • For example, in Japanese, the subject is naturally omitted, making it easy to discard the subject.

    • This, in turn, means that it is easy to overlook the subject distinction.

    • This means that there is a cultural difference in what is chopped up into small pieces and what is chopped up into large pieces in articulating the world.

    • It is not that one language is superior to the other.

    • 

      • A person of one culture looks at the language of another culture and thinks, “I’m an idiot for calling B without distinguishing between B and C,” but the other culture thinks likewise, “I’m an idiot for calling D without distinguishing between C and D.”
      • There won’t be much difference in the performance of the human brain, so there won’t be much difference in the number of pieces inscribed in the world as a whole.
      • concrete example
        • 
        • 
          • I told the AI that it was an A. He said no, it’s a B.

  • Using multiple languages, with the most finely divided language for each domain, provides a better understanding of the world than any single language.

    • 
      • Related: Cognitive Resolution.
    • To use a high school math analogy, if you know both vectors and complex numbers, when you want to rotate a vector, you can convert it to a complex number, rotate it, and then rotate it back.Vector and complex number metaphor
      • By mapping the system to the one that is easiest to operate, operating it there, and then returning to it, it is easy to do things that were difficult to do in the original system.
    • Better to have several different systems.
      • Ancient India, which developed before European ideas flowed in and was not given back to Europe much due to high translation costs.
      • Muslim community that had an incentive not to share knowledge due to religious opposition to Christianity.
      • It would be interesting to see better language models and low-cost references to literature from these cultures.
      • It is also worth noting that these areas are still likely to see population growth
    • A Japanese person asks a question in Japanese, and the AI thinks once in Arabic, then thinks in Pali based on the result, and finally answers by translating the conclusion into Japanese, saying, “This guy only understands Japanese, so it’s no use.

Q: Is there value in a better language model for Japanese?

• A: Yes.
• This does not mean that “Japanese-only models have value.”
• The importance of a thicker pipeline to the “AI that thinks across languages” that will grow more and more in the future.
  • 
  • ‘AI that thinks across languages’ is like a newly discovered oil field, value bubbling up
  • Users of languages with narrower pipes do not enjoy much of the value that springs from this.

relevance - Work to open up the forest of what has yet to be written. - In the process of trying to summarize the contents of this page (Worldview after LLM), the language

---

This page is auto-translated from /nishio/日本語言語モデルについて考えたこと using DeepL. If you looks something interesting but the auto-translated English is not good enough to understand it, feel free to let me know at @nishio_en. I’m very happy to spread my thought to non-Japanese readers.