90大语言模型能力来源与边界张奇复旦大学
大语言模型基础理论突破，发表大模型相关论文80+篇仅修改130亿参数中的1个就会使模型完全混乱输入:FudanUniversityislocatedin,LLaMA2-138 (PPL5.877): Fudan University is located in Shanghai, China. Itis locally known as复旦大学. The university was established in 1905. It isaccredited by Ministry of Education of the People's Republic of China. There are over 40,000 students studying in various courses offered by FudanUniversity. The language of instruction is Chinese.AMA2-138改语言核心区1维扩大10倍(PPL376079936):Fudan University is located ina>axeeeggeeggegeesxeeeee@Nie:ksYoueTheres>Thats>ksThise>s>lsxpeseThiscs><><><Thsspep<pts<s>Thece/e>s>Whats>a>peepepssssee:esssssssseessLLaMA2-138-修改语言非核心区LayerNorm其他维度扩大10倍(PPL5.914): Fudan University is located in Sthanghai,Cthina,the largest city with themost econom and cul ural a tiv ties n Ch ina. With the mos ad anced infastructure and the best Ilving conditon,it has become the internationaleducation center with the largest oversea students. It consists oflinan. Kanaaiao and Fenglin campus, which boasts the best resourcesfrom botheducation andresearch.FudanUniversityhasbeen a famous and attractive university forinternationalstudents,especiallyin the pastonedecade from国际上首次提出的大语言模型语言核心区和维度依赖理论,可以有效指导大语言模型训练过程UnveilingLinguisticRegions in Large LanguageModels, ACL2024
1.大语言模型语言核心区与维度依赖破坏'Arabic/Vietnamese区域CaseStudy:ChineseBase-7812,38544一年有366天，一年有months12个月w/o Arabic regionThere are 365 days in a year and 121216365一年有365天。一年有months in a year12个月w/o VietnamesereglonThere are365daysin ayearand1212.365-年有365天。一份有months in ayear12个月for generation.ModelMMLUArabic-MMLU(filtered)Base-7B42.4625.6W/o Arabicregion39.27W/oVietnameseregion39.2826.7*Arabic.MMLU(filteredimeansremovingquestionswhosecorrectansweris WiArabic MMLU:Assessing Massive Multitask L anguage Understanding in Arabic (K oto etal, arXiv 2024)UnveilingLinguisticRegions inLargeLanquaaeModes.ACL2024
1.大模型能力边界在哪里?
知识利用层次图大模型目前在哪个层级?未来可以到哪个层级?KnowledgeNewCreationComprehensionand Application自RecaMemorizationDark KnowedgeWangetal.Knowledge Mechanismsin Large Language Models:ASurveyandPerspective,EMNLP2024
知识利用层次图大模型目前在哪个层级?未来可以到哪个层级?KnowledgeNewCreation目前?Comprehensionngand ApplicationRecaMemorizationDark KnowedgeWangetal.Knowledge Mechanismsin Large Language Models:ASurveyandPerspective,EMNLP2024
知识利用层次图KnowledgeNev我认为:目前仍然是记忆Creation长上下文建模Comprehensionand Application多任务学习自Recall一Memorization跨语言迁移性文本生成能力Dark KnowedgeWangetal. Knowledge Mechanismsin Large Language Models:ASurveyandPerspective,EMNLP2024
知识利用层次图AGI系统才能“理解"KnowledgeNew理解物理世界Creation拥有长久准确记忆(O.可以推理Comprehensionand Application自Reca可以分层次规划MemorizationDarkKnowedgeWangetal.Knowledge Mechanismsin Large Language Models:ASurveyandPerspective,EMNLP2024
大模型能力边界的买践计究
当前大模型依然无法完成直工的“理解”与“推理"
大模型“参加”2024高考数学情况Table 1:202...
