LLM下一个发展未来是什么:
- LLM的未来可分为现状修复和开拓增量两类
- 现状修复方向包括居高不下的推理成本、垂直适配难题和上下文中注意力失控
- 对推理成本的解决将有助于推动LLM的广泛应用
- 垂直适配难题需要解决头部大模型价格过高和微调任务能力有限的问题
- 上下文中注意力失控也是需要解决的问题
2. 现状修复:
- 居高不下的推理成本：目前推理成本非常昂贵，只有少数业务可以承担，需要通过开源平替小模型、上下文压缩和LLM的优化来降低成本
- 垂直适配难题：微调垂直领域的LLM面临价格高昂和任务能力有限的问题，需要开放微调模型价格和提高微调任务能力
- 上下文中注意力失控：需要探索如何处理更长的上下文和更好地管理注意力的问题
3. 开拓增量:
- 开源平替小模型：通过开源平替小模型实现针对性微调，在更小参数量级上对齐千亿参数的效果
- 上下文压缩：通过文件分块后语义检索或长文分段摘要来缩减上下文长度
- LLM本身的优化：包括在芯片层的工作以及算法、模型结构和任务调度方面的优化
- 这些方法可以进一步提升LLM的性能和应用领域
4. LLM’s Context and Attention Mechanism:
- LLM requires a comprehensive understanding of the input context, utilizing attention mechanism to calculate the relationship between every word.
- Attention weights determine which words should be used to generate the answer.
5. Controlling Attention in LLM:
- Customizable and controllable attention mechanism is a crucial optimization point.
- OpenAI’s System field allows developers to define key attention points, but has limitations in meeting diverse demands.
6. Security Measures in LLM:
- OpenAI established safety standards for LLM, including avoiding answering unknown topics and promoting positive content.
- GPT-4, released in March 2023, addresses safety concerns.
- Anthropic, a company formed by former OpenAI members, focuses on LLM safety with RLAIF (Reinforcement Learning AI Feedback).
7. Interpretability in LLM:
- Interpretability is crucial in decision-making AI systems.
- High-cost actions require explanations and information backing.
- Addressing interpretability challenges in LLM is difficult, but can be partially alleviated through context, product design, and source positioning.
8. Essence Learning in LLM:
- LLM should focus on learning the essence of tasks and collecting effective data through lossless compression.
- Essence learning enables a deeper understanding beyond surface-level knowledge.
9. 先验胜于后验:
- OpenAI emphasizes the importance of understanding concepts and frameworks before gathering information.
- They believe in using external tools like calculators, wikis, and search engines to enhance their capabilities.
10. 业务安全:
- OpenAI recognizes the data security issues many enterprises face.
- They predict that the next focus will be on cloud privatization with Azure, which smaller model manufacturers can take advantage of.
11. 降低门槛:
- OpenAI sees cost and development frameworks as two factors contributing to barriers of entry.
- They also mention the possibility of low-code development and AI-art workflows.
- OpenAI believes that empowering everyone to build their own apps has significant value.
12. 理解（多模态）:
- OpenAI envisions LLM understanding not only images, audio, and video but also other forms of quantitative data like gyroscopes, touch, temperature, infrared, etc.
- They believe that applications that utilize this data structure will be easier to develop and integrate.
13. 干预:
- OpenAI’s current interventions include the ChatGPT plugin, which has limitations for safety reasons.
- They have also invested in robot companies and have technological expertise in robotic arms.
- However, there are safety concerns with LLM directly influencing the world.
14. 多端融合:
- Microsoft’s current efforts are focused on PC applications, while other devices like smartphones, VR, headphones, etc., are in a slow adaptation phase.
- OpenAI suggests paying attention to different information inputs, device characteristics, low-cost solutions, and new interaction paradigms.
15. Introduction:
- The ChatGPT plugin has been released, but it is not as impressive as expected.
- It is more similar to Alexa Echo’s Skill Store than Apple’s App Store.
- The limited features and input/output constraints of Skill Store pose challenges for further development.
16. Future Development:
- GPT-4’s image modality needs to be opened up.
- ChatGPT should be usable on more application platforms to obtain richer information input and multiple output modes.
- The key is to focus on what future plugins can do.
17. Promising Directions:
- Understanding the fundamental differences brought by the technology.
- Analyzing if the direction is for an existing market or a new market.
- Considering if the direction will be disrupted by tech giants.
- Examining the technical feasibility and business potential.
18. Technological Advancements:
- More mature CUI capabilities allow for a wide range of tasks.
- Improved understanding abilities include text, images, audio, video, and 3D.
- Enhanced generation abilities extend beyond content to code and data.
- Some reasoning abilities enable basic logic reasoning.
- Building trust through improved human-AI interactions.
19. Market Analysis:
- Distinguishing between existing and new markets.
- Existing markets may have high entry barriers and limited AI breakthroughs.
- Expanding existing markets depends on whether improved experiences will drive incremental growth.
- New markets have blue ocean opportunities with equal competition.
20. Disruption by Tech Giants:
- Considering whether the direction aligns with tech giants’ strategies.
- Being cautious of OpenAI’s future development, frameworks, and integration workflows.
21. Iteration of GPT Models:
- Speculating the iterative cycle of GPT-X models.
- Collecting data on fine-tuning and prompts within the capabilities of a certain version.
- Releasing new versions based on the collected data.
- Endlessly repeating the process to improve LLM’s thinking framework.
22. Technological Prepositions and Commercial Potential:
- Evaluating technical understanding and business judgment.
- Considering if the technology has a solid foundation and if there is sufficient commercial potential.
23. Challenges in emotional direction:
- Retention: Long-term user retention is low in emotion-based applications. This can be addressed through capturing user needs sensitively.
- Trust: Users are skeptical of AI in initial interactions. Waiting for new technologies and mindset shifts is necessary.
- Duration and energy: Emotional investment consumes mental energy, leading to user fatigue and attrition. Building flow is a solution, but challenging.
24. Emerging paradigms with new technologies:
- New technologies bring new paradigms (e.g., Notion AI) that aim to unify various formats and enhance creativity.
- Exploring new file formats and paradigms is essential for seizing opportunities in creative, productive, and lifestyle domains.
- Innovation in multiple dimensions is required to establish new standards in the age of emerging technologies.
25. Opportunities in the AI wave:
- Helping others adapt to AI advancements creates opportunities in 2C and 2B markets.
- Earning through tools combined with AI, such as ChatPDF or integrating AI into SaaS, drives business prosperity.
- Monetizing self-trained models for individuals or enterprises offers a potential revenue stream.
26. Opportunities in labor migration:
- AI-driven labor surplus prompts migration to AI-related services, content industries, or possibly returning to traditional sectors.
- Analyzing the US labor market can provide insights into the direction of labor migration.
- Labor migration presents opportunities, although specific prospects remain uncertain.
27. Cultural shifts with changing productivity:
- Productivity changes influence cultural shifts. For instance, industrial revolution fostered gender equality and pushed for individuality.
- LLM technology may enhance trust in AI, potentially reshaping human-AI dynamics.
- The impact of LLM technology on culture is uncertain, ranging from increased trust to concerns over AI crises.
28. AI技术带来的焦虑和影响:
- AI技术的发展引发了人们对未来更深远的影响的担忧，如创作者两级分化和信任转向真实KOL
- 人文思潮的变化主要影响内容行业和创作者领域
29. 准备思考框架:
- 需要判断技术浪潮带来的改变，并准备相应的生产力飞跃
- 生产力飞跃会带来认知错乱、观念建立的打破和产生的阵痛
30. 拥抱UGA元年:
- LLM的辐射范围不仅包括内容创作领域，还包括应用创作领域
- 需要使用各种工具努力将创意变成实际应用
31. 找本质，先验胜于后验:
- 在LLM时代，表层知识价值降低，需要更本质、先验的思考框架
- 目标是始终找到更优雅的理解世界的方式
32. 保持好奇心，但做好信息管理:
- 选择信源时要随着自我迭代而变化，移除不再有帮助的作者
- 进行高效飞速阅读，并做有输入的阅读
33. 拥抱不可控性:
- AI产品经理需要掌握对不可控性的控制能力
- 这也是时代风暴下所有人需要掌握的能力
34. Contact Information:
- If your company is working on LLM and has HC products, feel free to call me.
- If you are involved in entrepreneurship, investment, or any other field, you are welcome to reach out and exchange knowledge. Please introduce yourself briefly when adding me.
