Al Agents Beyond ChatGPTLLMLLMLLMZhou(Jo)YuColumbia UniversityArklex AlWho supports AI Agents?Bill GatesAgents are bringing about the biggestCurrent agents are just thinrevolution in computing since we went fromwrappers around LLMs.typing commands to tapping on icons.Autoregressive LLMs canAndrew Ngnever reason or plan.I think Al agentic workflows will drivemassive AI progress this year.Auto-GPT's limitations in...revealSam Altmanthat it is far from being a practicalsolution.2025 is when agents will work.Slides adapted from Yu SuWhat are AI Agents?Perception:Multimodal inputs including,text,image,audio,video,touch,etc.AgentSensorsPerceptsPlanning(Inner Monologue):ReasoningChain-of-Thought reasoning over tokensEnvironmentthat powered by LLMsInnerMonologueReflection:meta-reasoning in every stopActions:function/tool calling,embodiedActionsactions.Adapted from Russell Norvig (2020)AI Agent Deployment ConsiderationPHASE 1RESEARCHPHASE 2SCALINGPHASE 3INNOVATINGLevel 1Level 2Level 3Level 4Level 5"Your Work Assistant'Agent-as-a-Servico"Autonomous Agents"TASK 1A2LLMPROTO AGIA completion to theAn LLM-centric softwareA service-centric system withAn autonomous system withhuman prompt wit世tssystem for assisting real-LLMs as core components forLLMs as core components forvehicle,an L5 agent isworld tasks.completing various tasksSlide:Alex Wang @Scale Al18Overview3.AI agent self-improvement via tree search (vuet al.ICLR 2025)Background:In-Context Self-ImprovementInput:Q:Calculate (4 *1)-(2 3)=?运营动脉运营动脉运营动Xiao Yu,Baolin Peng,Michel Galley,Jianfeng Gao,Zhou Yu,Teaching Language Models to Self-Improve through2Background:In-Context Self-ImprovementInput:Q:Calculate (4 *1)-(2 3)=?Q:Calculate 1+2=?Ans:3nu3noun-jo-ureyQ:Calculate (4 *-1)+(2 *3)=?Q:Calculate…oys-MajLet's think step by step:Q:Calculate (4 1)-(2 3)=?Step1(4*1)-(2*3)=4-6.Ans:-2Step2:4-6=-2Ans:-23Background:In-Context Self-ImprovementInput:Q:Calculate (4 *1)-(2 3)=?Self-Improvement Prompting(Madaan,et al,2023)Step1:(4*1)-(2*3)=4-6Step2:4-6=-3Ans:-3Madaan,A.et al.(2023)'Self-Refine:Iterative Refinement with Self-Feedback'4Background:In-Context Self-ImprovementInput:Q:Calculate (4 *1)-(2 3)=?Self-Improvement Prompting(Madaan,et al,2023)Step1:(4*1)-(2*3)=4-6Step2:4-6=-3promptAns:-3feedbackIn step 2 the part "4-6=-3"isincorrect.This is because...promptupdateStep1:(4*1)-(2*3)=4-6Step2:4-6=-2Ans:-2Madaan,A.et al.(2023)'Self-Refine:Iterative Refinement with Self-Feedback'5Background:In-Context Self-ImprovementInput:Q:Calculate (4 *1)-(2 3)=?Self-Improvement Prompting(Madaan,et al,2023)Step1:(4*1)-(2*3)=4-6Step2:4-6=-3promptAns:-3feedbackIn step 2 the part "4-6=-3"isincorrect.This is because...promptpromptfeedbackupdateStep1:(4*1)-(2*3)=4-6Step2:4-6=-2Ans:-2Madaan,A.et al.(2023)'Self-Refine:Iterative Refinement with Self-Feedback'6Background:In-Context Self-ImprovementMultistep ArithmeticCodex (175B)5LLaMa (7B)+2.0Problem 1:small LM cannot self-improve via prompting!-5.2-5.1Logical DeductionCodex (175B)5+4.4LLaMa (7B)4.1+CoT prompt+SI.prompt+ft (finetune)+ft Sl.demoBackgroundMotivationApproachExperiments7Background:In-Context Self-ImprovementMultistep Arithmetic■Codex(175B)5LLaMa (7B)+2.0Problem 1:small LM cannot self-improve via prompting!-5.2-5.1Logical Deduction0Step1:(4*1)-(2*3)=4-6tep2:4-6=-3Codex (175B)5+4.4LLaMa (7B)Ans:-3In step1 the part“2*3=6”is4.1incorrect.This is because...+CoT prompt+SI.prompt+ft (finetune)+ft Sl.demo..error propagates!BackgroundMotivationApproachExperiments8Background:In-Context Self-ImprovementMultistep ArithmeticCodex (175B)5LLaMa (7B)Problem 2:small LM cannot leam+2.0"self-improvement"from LLM demonstrations!Logical DeductionCodex (175B)5+4.4LLaMa (7B)054.1+CoT prompt+SI.prompt+SI.prompt+ft(finetune)+ft Sl.demoBackgroundMotivationApproachExperiments9Background:In-Context Self-ImprovementMultistep ArithmeticCodex (175B)5LLaMa(7B)Problem 2:small LM cannot lear+2.0"self-improvement"from LLM demonstrations!5.25.1Q:Calculate 4-0 *-1*8+6=?Logical DeductionCodex(175B)5+4.4LLaMa (7B)=4-(0*-1*-8)+600=4-(0*-1*-8)+60=4-(0)+6=4-(0+8)+6=4-(0+6)=4-8+65-4.1=4-6=-2+6=-2=4+CoT prompt+ft (finetune)+ft Sl.demofeedback:..irrelevant demonstrations!10MotivationPrior work shows that self-improvement(S.I.)is useful for task performance/generalization (Madaan,et al,2023)We find prompt-based S.I./simple distillation methods fails with small LMMadaan.A.et al.(2023)'Self-Refine:Iterative Refinement with Self-Feedback'BackgroundMotivationApproachExperiments