Intellectual Production Techniques for Engineers Using LLM (Presentation Material)

2024-06-06 NISHIO Hirokazu @nishio - Senior Researcher, Cybozu Labs, Inc. - General Incorporated Association Unexplored Director - Founder of unexplored junior and mentor

The impetus for this year’s event

• More reprints of The Intellectual Production of Engineers. The Intellectual Production of Engineers, published in 2018, was reprinted more than five years after publication.
• Becoming a long seller = continuing to sell over the long term (p.15)
• Books become less successful as time passes after publication.
  • Some books, however, continue to sell. That’s because people who read them recommend them to others (a social proof of value).
  • When I ego search on X/Twitter from time to time, I see posts like “I read this because someone I know recommended it” (p. 141).
• What it means that the book is still selling five years after publication.
  • Withstood 5 years of [Test of Time
  • It contains content of long-term value that will not obsolescence in the last five years.
  • Half of the cover’s “still useful 10 years later” was demonstrated.
    • 

Is the content worthwhile in the future?

• What about the next five years?
• Development of LLM represented by [ChatGPT]
  • Totally unanticipated at the time of writing (2017)
  • How does this technological change affect the methods of intellectual production described in this book?
• I want a point of view other than my own, so I asked LLM (experiment).
• Prompt: “What is the

• This is the preface and table of contents to my 2018 book “The Intellectual Production of Engineers” Please look back and comment from the perspective of the present day 2024, when LLM technology has developed and the understanding and generation of text is powerfully accelerated by machines.

• Claude 3 Opus:

• From a 2024 perspective, the content of “The Intellectual Production of Engineers” is becoming increasingly important.

• While the development of LLM (Large Language Models) has accelerated information gathering and knowledge acquisition, the importance of chewing over the information obtained and constructing one’s own model remains unchanged. The “Learning Cycle” proposed in this book is considered to be an effective framework even today.

• 
  • This is exactly what we are doing right now.
    • Gather information by researching and experiencing LLM firsthand
      • A prediction model is created and a hypothesis is generated that says, “If we use it this way, will it yield useful results?” and a hypothesis is generated.
    • Practice and test that hypothesis, if it works, it’s useful, if it doesn’t Learning Opportunities.
• Continued output is also presented (gray is AI output)

• The chapter on “motivation” suggests that human creativity and will will continue to be important even as AI advances in automation.

  • Yes, Enthusiasm is a scarce resource.

• “Memory training” techniques are still important skills for humans today, even as AI has dramatically improved our ability to remember and retrieve information. It is not just about memorizing information, but structuring knowledge and the ability to apply it.

  • The storage of information was now delegated to machines. The skills of structuring and applying knowledge to use those tools efficiently became important.

• “Efficient reading” skills are even more important now that the flood of information has further accelerated. The ability to quickly grasp the essence and select the necessary information is required.

  • Recently, I have been reading papers to Claude 3 Opus. This chapter, which reconsiders the “act of reading” and the “purpose of reading” in order to read efficiently, will serve as a foundation for considering changes in “how to read” as tools change.

• “Putting thoughts together” is still an important inherent human ability, even with the development of language models such as ChatGPT. We are expected to critically examine machine-generated information and form our own ideas.

  • You are absolutely right, LLM has allowed us to generate a lot of material for brainstorming ideas, etc. We need guidelines on how to “put it all together”. We need guidelines on how to “put that material together”.

• The process of “coming up with an idea” is an area where human creativity can be demonstrated, even today with the development of AI. Even though machines can assist with ideation, it is still the job of humans to ultimately create valuable ideas.

  • It is important to consider what constitutes a “worthwhile idea.” An idea that is not implemented has no value, nor does an idea that is not feasible have value. The process of nurturing the seeds of ideas that will lead to future value will become increasingly important.

• “Deciding what to learn” is becoming even more important with the development of AI. Not only are they required to efficiently complete the tasks given to them, but also to the ability to determine what to learn on their own.

  • When major changes occur in the world, we cannot be sure that what was once considered “good to learn this” will continue to be useful in the future. Since no one knows what the future holds, we cannot leave it to others; we must choose for ourselves.

• Overall, the ideas and skills presented in “The Engineer’s Art of Intellectual Production” can be evaluated as a good book that provides important suggestions for surviving the AI age. It is considered to be very useful today as a guideline for harnessing the power of machines and demonstrating creativity as a human being at the same time.

• The time has come when such texts can be produced at low cost by using LLM.
  • If a person reading this sentence thinks it is interesting, then the value of “interesting” has been produced.
  • If people who read this text find it compelling and buy the book, it means that cash value has been produced by shining a light on the intellectual output of the past.
• I find the output itself interesting, as well as the process of “I let LLM read the book and this output came out”.
  • This is a real example of “Intellectual Production with LLM
  • The “Engineer’s Art of Intellectual Production” was born out of observations of the intellectual production process that led to the writing of a book, “The Art of Supporting Coding.”
  • New intellectual products are now emerging from observations of the process of “intellectual production using LLM

Changes after writing

• First to talk about the evolution of the methodology of intellectual production and the changes from 2018 to now
  • Otherwise, we lack the prerequisite knowledge to talk about the present and the future.
• There are three major categories.
  • From Blog to Scrapbox
  • From Paper KJ Method to Kozaneba
  • Development of LLM

From Blog to Scrapbox

• I’ve been blogging since the early 2000s.
  • In 2000, Google Search started its service in Japanese. If you write notes of what you learned on the Internet, you can find them with a Google search when you think, “That thing I did before, what was it?
  • I started out with a personal note as a motivation, but many people have thanked me for it, and with the development of social networking sites, it has become easier to notice mentions.

  • If you publish your materials on the Internet, they will be mentioned on social networking sites from time to time, triggering a reminder. The higher the social demand, the more frequently they are mentioned, and thus the more frequently they are reviewed and improved. (p.141)

    • (I personally call them “Social Triggers,” although the book doesn’t mention it.)
• 3 Advantages
  • Advantages of writing: it won’t disappear (p. 149)
  • Advantages of writing digitally: searchable (p. 20)
  • Benefits of writing in a public forum: social value generation (p. 141)
• Scrapbox (Note: Scrapbox changed its name to Cosense in May 2024, but will be referred to as Scrapbox in this talk)
  • Started trying Scrapbox in 2017 while writing “The Intellectual Production of Engineers” and really liked it by the time of publication in 2018
  • but due to the structure, it was not possible to add a chapter on Scrapbox to the book (just barely mentioned in a footnote on p. 175).
  • As of 2024, approximately 20,000 articles, or 60 books worth of text, have been written in Scrapbox
• Scrapbox Features
  • Link-specific notation: simply enclose a link in [] to make it a link. Can express relationships between information at low cost.
  • Links to non-existent pages: Links can be created even if the page to which they refer does not yet exist. This facilitates knowledge expansion.
  • Automatic fuzzy search: automatically searches when creating links and suggests pages that may be relevant. Helpful for finding knowledge relevance.
• These features accelerate Scrapbox’s representation of knowledge as a network structure.
  • The more accumulation accumulates, the more benefits.
  • In the early stages when there is little accumulation, it is difficult to realize the benefits because the network is not fully formed
• The advantages of Scrapbox
  • Low-cost stocking of associations that come to mind for you.
    • These stocks of associations will grow Knowledge Network.
    • Like a plant, it branches out and grows lushly.
  • Follow the links and re-discover what you thought in the past. - Scrapbox reminds me of what I’ve forgotten.
    • The link that spread Expand the pertinence of useful concepts.
  • Encountering descriptions of the past that are relevant to what you are thinking about now will give you new insights.
    • Revisit past experiences from your current point of view
    • Comparing past and present experiences as “similarities” leads to deeper understanding and abstraction
  • The more important things are mentioned repeatedly, the fuller the page.
    • Social triggers were more about luck than others.
    • Scrapbox causes interaction with “Past self”.

From Paper KJ Method to Kozaneba

• I wrote about the electronic version of the KJ method on p. 176, I actually implemented it after I finished writing
• I made digital stationery Kozaneba and I’m happy enough with it, so I don’t do KJ method on paper anymore.
• Advantages of the Electronic KJ Method
  • It doesn’t occupy physical space and can be stored without contamination in the long term.
  • Can copy-paste
    • If you are worried about breaking something, you can try a new structure with confidence by copying it and backing it up.
  • Line Drawing Function
    • Draw lines to explicitly show relationships between information
    • Can be moved with the relationship explicitly stated.
    • Physical paper makes it cumbersome to move the position of the dipstick after drawing a line, and drawing a line prevents you from moving it
  • Zoom function
    • An infinitely wide canvas is difficult to prepare physically, but easy with digital
    • The zoom function allows you to change the display range at will, whether you want to get a bird’s eye view of the whole picture or focus on a specific part. This makes it easier to explore the relevance of ideas.
    • Important labels can be enlarged for easy viewing.
      • This is also a hassle with paper, but easy with digital
• I personally use Kozaneba, but KJ Method Study Session @ Loftwork I used Miro.
  • It is better to use what you are familiar with. Miro is not specifically designed for the KJ method, but the overall functionality is well done.
  • Especially when several people are doing the KJ method on one screen at the same time, being able to see what the person next to you is doing has the effect of “understanding is improved by comparing with similar things”.
  • (To be honest, I did not write enough about the KJ method. I added a lot to it in the material of the above-mentioned workshop, so if you are interested, please read it).

Development of LLM (coupled with personal knowledge base)

• November 2022 OpenAI’s ChatGPT appears.
  • I had tried the GPT3 API over the summer, but didn’t understand its power until I saw ChatGPT
• March 2023 - I connected my Scrapbox to ChatGPT. ([Details https://scrapbox.io/nishio/%E8%87%AA%E5%88%86%E3%81%AEScrapbox%E3%82%92ChatGPT%E3%81%AB%E3%81% A4%E3%81%AA%E3%81%84%E3%81%A0%E8%A9%B1%E5%8B%89%E5%BC%B7%E4%BC%9A]) - Attempts to connect personal knowledge base and LLM
  • LLMs began reading and talking about the 20,000 articles and 60 books worth of content in their Scrapboxes.
    • Can use my past knowledge and ideas in my interactions with LLMs
  • Now called RAG (Retrieval-Augmented Generation, retrieval-enhanced generation)
    • RAG is a technique for generating more specific and contextualized responses by retrieving and leveraging information from a specific database in addition to the general knowledge possessed by the LLM.
    • Allows them to have knowledge that generic LLMs do not have
    • If you put the same prompt in the same LLM as someone else has, similar content will be generated, this will not lead to differentiation (commoditization)
    • Can be personalized to your liking by combining with personalized data
• Released as a custom GPT [NISHIO Hirokazu’s Assistant] for ChatGPT using GPT Builder
  • The contents of this Scrapbox and the contents of “The Intellectual Production of Engineers” and “The Technology Behind Coding” are included.
• It’s in digital data, so you can put it in.
  • Seven years of writing in digital format in Scrapbox is being utilized.
  • Writing makes it indelible, accumulative, and searchable.
  • That “search” has advanced with LLM technology.
• Search for the keyword “search” in the Engineer’s Guide to Intellectual Productivity PDF
  • 
  • LLMs do the work of looking through the hits and putting them together many times faster than humans.
  • Briefly explain in writing the effect of search on intellectual production.
  • The effect of search on intellectual production is the rapid acquisition of information and [* efficient organization of knowledge. By utilizing search, you can quickly locate the information you need and develop a detailed understanding of the whole picture. This gives you a sense of approaching problem solving and improves the quality and speed of your intellectual production.
  • Interesting angle on “efficient organization of knowledge.”
    • Gather where certain keywords appear and read them.
    • By “comparing” similar “sentences in which the key word is used,” “understanding is advanced.”
    • p.39-41

Development of LLM (competition to expand context window)

• The context window is the maximum amount of text that the LLM can process at one time. The larger this context window, the better the LLM can understand longer sentences and conversational flow and generate consistent responses.
• March 2024, Claude 3 Opus at Anthropic, 200K tokens with context window
  • Roughly speaking, one paperback book is about 100K.
  • I experimented with putting 50 pages in a technical book with more words than a paperback book and having it summarized, but it works fine.
  • I use it daily to paste entire English papers and have them explained in Japanese.
• The Intellectual Production of Engineers, p. 122.
• 

• If you read too fast and do not understand enough, you can read again, but if you read too slow and waste time, there is no way to regain it, so if you do not know the proper balance, you should knock it on the fast side.

• Claude 3 Opus reads a 50 page book and outputs a summary in tens of seconds, so less than 1 second per page
  • I read faster than a native Japanese speaker can read a book in Japanese.
  • We can run that in English or Chinese.
  • Ask a question and they will answer it = Dig deeper based on interests = Personalized summaries tailored to individual interests

Context Window Expansion → Maintain Consistent Context

• Larger context windows make it easier to maintain consistent context
  • For example, a conversation over a number of days about a specific project
  • Human memory is volatile after a period of time, and it is expensive to read back past conversation logs.
  • If the LLM can have a conversation with me with the logs in context, I’ll act like someone who remembers the “project memory”.
• In fact, the “Intellectual Production Techniques for Engineers Using LLM” is part of an experiment to let LLM manage projects.
  • It began as an experiment to have LLM manage a “book writing project.”
  • In the middle of this, I was approached to give a lecture, so I’m stopping by to prepare materials for the lecture.
  • Practice will reveal what can be made useful with LLM and what cannot yet be done with LLM.
• The “Intellectual Production of an Engineer Using LLM” project is being carried out while publishing intellectual products and intellectual production processes on Scrapbox. - LENCHI_Preface - LENCHI_Preface Generation Process
• LENCHI_FOREWORD
  • We are in the midst of a dramatic evolution of Large Language Models (LLMs), AI that understands and generates language, and LLMs such as ChatGPT have the potential to go beyond mere conversation and writing tools to transform intellectual production itself.
  • For many years, I have been working to improve the intellectual productivity of individuals and teams; in my 2013 book, “The Art Behind Coding,” I rethought the intellectual production of programming from the aspect of language, and in my 2018 book, “The Intellectual Production of Engineers,” I tried to present a universal technique that is not limited to engineering, but is applicable to all intellectual In 2018, I attempted to present a universal technique that is applicable to all intellectual production, not just engineering.
  • As an extension of this awareness, LLMs have suddenly appeared before my eyes as an aid, and sometimes even a substitute, for human intellectual production. This should be an opportunity to rethink the nature of intellectual production itself. As someone who has been looking at the relationship between language and intellectual production and exploring its universal techniques, I cannot help but see the advent of LLM as an excellent opportunity to advance my own research in a big way.
  • In this book, we will explore the possibilities of practically using LLM in our own intellectual production. My skills as a programmer, my curiosity as a researcher, and my sense of language as an author. I, a man of many faces, will share with you the process of exploring a new way of intellectual production through dialogue and collaboration with LLM.
  • This book is neither a book of know-how filled with examples nor a book of grandiose theories. Rather, it is a record of one professional’s struggle to evolve his own intellectual production by “mastering” LLM, and should be called a trail of an experiment to reexamine the essence of intellectual production and create a new form of it.
  • Now, let’s take LLM with us and break new ground in intellectual production.
• LENCHI_FOREWORD GENERATION PROCESS excerpt

  • You are a talented editor. I am the author of the book “The Intellectual Production of Engineers” I am planning a book on intellectual production using LLM. Assist me.

    • Roles and objectives are clearly stated.

  • It’s cliche. The reason why it is cliche is because there is no necessity that is unique to the author writing this book. More likely, you don’t seem to know the author’s information, so I will give it to you. PASTED The Intellectual Production of Engineers Author’s official page.

    • Commodity stale content proposed because it doesn’t give specific information, specific knowledge.

  • The author’s name is Yasukazu Nishio, although I think the policy is fine with that. Self-introduction data is attached. PASTED self-introduction.

    • Giving knowledge.

  • In this case, I don’t think the “let’s make chapter headings first” approach that is common in traditional book planning is not a good idea. Instead, I think it is better to start with a small start and sprint, and use LLM as an example by using it to its fullest in the process.

    • major policy statement.

  • An emergent approach to verbalizing our know-how as “craftsmen of intellectual production who master LLM”, that’s a good phrase. In addition, this chat itself will be used as an example. I think an appropriate start for the book would be a short preface that can be read by people without context, followed by a brief introduction and examples of real-world examples.

    • major policy statement.

  • I think the way you’re going in is emo, but don’t think you should connect with the author’s past actions early on to show that something unique to this author will be written.

    • major policy statement.

Current Issues and Future

• Repeated searches or putting in too large content eats up context and causes amnesia.
  • This is a problem with the current chat UI design
    • Most people only perceive it as a chat and do not want to control the content of the context
    • So it’s not designed to do that.
    • Actually, ChatGPT allows tree exploration but it’s hard to use & the playground allows you to delete some history but not in the chat UI.
  • Chat has been a tool for “human-to-human communication” and not for editing the other person’s memory, etc.
    • In human conversation, it’s similar to discussing while writing on a whiteboard, or discussing while collaboratively editing the minutes of a meeting.
    • A UI will be needed where humans and LLMs can work together to gather and edit information to create a “good context”.
    • It’s probably closer to Scrapbox than chat
  • (Or the context window will be 10100 times larger and 10100 times faster, eliminating the need for organization, or the LLM will organize better than humans without human help.)
• Convergence assistance needed.
  • Accelerated brainstorming with LLM is useful for divergent thinking
    • One person can have multiple points of view.
    • Related: [Not subjective or objective, but from the subjectivity of one to the subjectivity of many.
  • On the other hand, how to support convergent thinking in LLM is a challenge
  • Context broad LLM can help with purpose and contextual convergence (= variants of summarization)
  • There could be a combination of KJ legal methods and LLM to support convergent thinking.

Summary Development

• Just put an English paper on Claude and say “Summarize” and he will summarize it for you.

• Same for GPT4, but much different in output.

• The word “summary” is too vague a term to specify what a human wants an LLM to do, and there is a wide range of interpretation among LLMs

• The same LLM can vary a lot depending on how the prompt is written.

  • Give Claude an audio transcription of the videoconference to “summarize” and “take minutes.”
  • Hand them a paper and say, “Summarize it,” “What’s great about this paper?” “How is it different from X?”
  • The “instructional verbalization skills” that allow you to output the information you want to get will make a big difference in effectiveness.

• Fractal Reader

  • A service that presents a summary with different levels of detail for each input (created by @blu3mo, a student doing research and development on computer interaction).
    • This lecture material as input: You can see the real thing here:
  • This UI design is interesting p.20 Tools for a “rough first” approach
  • Humans cannot specify the appropriate level of detail before looking at a summary subject (whether you want to read in detail or just skim depends on the content).
  • Users can obtain the information they need in a flexible manner by first looking at a rough summary and then looking at more detailed summaries only where they want to see more detail.

• broad listening

• Developments in summarization technology are not just helping people read books; they are changing the way people communicate

• 

• Advances in information replication technology have led to “broadcasts” that can be heard by large numbers of people.

  • Broadcasting refers to the one-way dissemination of information, as in broadcasting.
  • But this was a one-way communication from the sender of information to the receiver.

• Developments in summarization technology make it easier to hear the opinions of many people.

  • This generates a new form of communication: “broad listening.

• The book “Plurality: Collaborative Technology and the Future of Democracy” by Audrey Tang and Glen Weyl has more details.

  • 
  • In fact, I am the Contributor & Editor of this book and my name is listed in the credits.
  • Japanese version will be translated by Hiroo Yamagata and published by Cybozu Shiki Books
  • I have created a forum in Scrapbox to have conversations using the machine-translated content of books as a means of experimenting with collaboration in assembling understanding from reading books.
  • What I wrote here is Embedded daily in Github Actions to create a custom GPT [⿻Plurality Assistant https://scrapbox.io/plurality-japanese/%E2%BF% BBPlurality_Assistant] data.
    • (= AI assistant that reads all conversations in the community every day)
  • If you are interested, please join us here: [/plurality-japanese/read first](https://scrapbox.io/plurality-japanese/read first).

Changes in the act of “reading a book

• Traditional “read a book” activities
  • Mainstream linear reading experience of picking up a paper book and reading through it word by word
  • Readers were getting information in the order the author intended, making it difficult to tailor the reading to their own pace and interests
• LLM brings about a change in the act of “reading a book”.
  • LLM enabled me to summarize the content of the book and extract the parts of the book that are of interest to me
  • Readers will be able to explore the book’s content in a non-linear fashion according to their own interests.
  • With LLM, you can quickly find the information you need to know without having to read through the entire book. The experience is like acquiring knowledge through interacting with a book.
  • Plurality is a CC0 license for manuscripts, so you can experiment without worrying about book copyrights.
  • Fractal Reader summarizes and presents the content of a book in different levels of detail. You can read a broad summary first, then read a more detailed summary of the parts that interest you, and so on, understanding the book at your own pace.
    • Summary experiment for one book: [/plurality-japanese/fractal-summary page with all table of contents](https://scrapbox.io/plurality-japanese/fractal-summary page with all table of contents).
  • Read while multiple people write and discuss the same book through Scrapbox
    • Reading” is changed from a work in which each individual builds up understanding “inside himself/herself” in solitude to a work in which everyone cooperates to create comprehension support contents as “digital public goods” “outside himself/herself”.
  • An AI assistant will look at all the activity done by the community and answer questions.
    • In the future they will support better cooperation
• The traditional “book” format is not the best means of knowledge transfer

summary

• The development of LLM is bringing about significant changes in intellectual production
• The ideas advocated by “The Engineer’s Art of Intellectual Production” continue to have value today
• It is important to utilize LLM while taking into account the evolution of tools and human creativity
• The way of learning is changing, and the ability to identify what to learn on one’s own is becoming increasingly important.
• Important to continue to explore the transformative potential of intellectual production brought about by LLM and the methodologies to take advantage of it

---

This page is auto-translated from /nishio/LLMを使いこなすエンジニアの知的生産術(講演資料) using DeepL. If you looks something interesting but the auto-translated English is not good enough to understand it, feel free to let me know at @nishio_en. I’m very happy to spread my thought to non-Japanese readers.