Back to U-A-Goose

Winter 2027 · University of Alberta

CMPUT 663: AI for Software Engineering

Term
Winter 2027
Instructor
Prof. Zhou Yang
Department
Computing Science

Course overview

This graduate course explores how AI is built and used for software engineering, with a focus on large language models of code and coding agents. It connects these systems to software engineering tasks and examines how to evaluate their usefulness, reliability, and limitations.

The course brings together software engineering and machine learning, including the security, privacy, and human factors that matter when AI is used in real software development.

Interested in research? For University of Alberta students interested in doing research with us, we strongly recommend taking this course.

Course timeline

15 teaching weeks · 2 classes per week · Reading Week excluded

  1. Week 01

    Introduction to Software Engineering

    This week introduces the goals, processes, tasks, and challenges of software engineering, covering requirements engineering, design, implementation, testing, deployment, and maintenance. Students will become familiar with software representations, formal specifications, testing, fuzzing, program analysis, agile development, and modern development tools, providing a foundation for understanding the engineering problems that AI can help address.

  2. Week 02

    Research Methods in Software Engineering

    This week introduces the research process in software engineering, including research questions and hypotheses, literature search and review, research design and evaluation, academic writing, and peer review. Students will learn to critically read existing work, assess research claims and supporting evidence, identify open problems, and develop an initial proposal for their group project.

  3. Week 03

    Mining Software Repositories and Software Analytics

    This week explores how source code, version histories, issue reports, code reviews, and execution logs can provide insights into software development and support task automation. Topics include data collection, feature and label design, classification, regression, clustering, and deep learning. Students will learn to construct research datasets and examine how data quality, sampling bias, and data leakage affect their findings.

  4. Week 04

    Code Representation Learning and Pretraining

    This week examines code representation learning and language model pretraining, covering tokenization, model architectures, training data, and pretraining objectives. Examples include code completion, masked language modeling, and fill-in-the-middle training. Students will explore how data and training objectives shape model capabilities, and learn to prepare training examples and conduct small-scale training experiments.

    Check Reading List:19 papers
    1. code2vec: Learning Distributed Representations of CodeUri Alon, Meital Zilberstein, Omer Levy, et al., POPL, 2019.
    2. CodeBERT: A Pre-Trained Model for Programming and Natural LanguagesZhangyin Feng, Daya Guo, Duyu Tang, et al., Findings of EMNLP, 2020.
    3. CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and GenerationShuai Lu, Daya Guo, Shuo Ren, et al., NeurIPS Datasets and Benchmarks Track, 2021.
    4. Evaluating Large Language Models Trained on CodeMark Chen, Jerry Tworek, Heewoo Jun, et al., OpenAI Technical Report, 2021.
    5. Efficient Training of Language Models to Fill in the MiddleMohammad Bavarian, Heewoo Jun, Nikolas Tezak, et al., OpenAI Technical Report, 2022.
    6. CodeGen: An Open Large Language Model for Code with Multi-Turn Program SynthesisErik Nijkamp, Bo Pang, Hiroaki Hayashi, et al., ICLR, 2023.
    7. CodeGen2: Lessons for Training LLMs on Programming and Natural LanguagesErik Nijkamp, Hiroaki Hayashi, Caiming Xiong, et al., Salesforce Technical Report, 2023.
    8. Textbooks Are All You NeedSuriya Gunasekar, Yi Zhang, Jyoti Aneja, et al., Microsoft Research Technical Report, 2023.
    9. Code Llama: Open Foundation Models for CodeBaptiste Rozière, Jonas Gehring, Fabian Gloeckle, et al., Meta AI Technical Report, 2023.
    10. DeepSeek-Coder: When the Large Language Model Meets Programming—The Rise of Code IntelligenceDaya Guo, Qihao Zhu, Dejian Yang, et al., DeepSeek Technical Report, 2024.
    11. StarCoder 2 and The Stack v2: The Next GenerationAnton Lozhkov, Raymond Li, Loubna Ben Allal, et al., arXiv, 2024.
    12. Granite Code Models: A Family of Open Foundation Models for Code IntelligenceMayank Mishra, Matt Stallone, Gaoyuan Zhang, et al., IBM Technical Report, 2024.
    13. CodeGemma: Open Code Models Based on GemmaCodeGemma Team, Heri Zhao, Jeffrey Hui, et al., Google Technical Report, 2024.
    14. DeepSeek-Coder-V2: Breaking the Barrier of Closed-Source Models in Code IntelligenceDeepSeek-AI, Qihao Zhu, Daya Guo, et al., DeepSeek Technical Report, 2024.
    15. Qwen2.5-Coder Technical ReportBinyuan Hui, Jian Yang, Zeyu Cui, et al., Alibaba Technical Report, 2024.
    16. Seed-Coder: Let the Code Model Curate Data for ItselfByteDance Seed, Yuyu Zhang, Jing Su, et al., ByteDance Technical Report, 2025.
    17. Mellum: Production-Grade in-IDE Contextual Code Completion with Multi-File Project UnderstandingNikita Pavlichenko, Iurii Nazarov, Ivan Dolgov, et al., JetBrains Technical Report, 2025.
    18. Qwen3-Coder-Next Technical ReportRuisheng Cao, Mouxiang Chen, Jiawei Chen, et al., Alibaba Technical Report, 2026.
    19. Mellum2 Technical ReportMarko Kojic, Ivan Bondyrev, Aral de Moor, et al., JetBrains Technical Report, 2026.
  5. Week 05

    Post-Training Code Models

    This week examines how pretrained code models are adapted to follow instructions, perform software engineering tasks, and learn from feedback. Topics include supervised fine-tuning and instruction tuning, preference optimization, knowledge distillation, and reinforcement learning with methods such as GRPO. Students will learn to design training examples and objectives, compare post-training methods, and investigate how learning signals affect model behavior and performance.

    Check Reading List:19 papers
    1. Training Language Models to Follow Instructions with Human FeedbackLong Ouyang, Jeffrey Wu, Xu Jiang, et al., NeurIPS, 2022.
    2. Self-Instruct: Aligning Language Models with Self-Generated InstructionsYizhong Wang, Yeganeh Kordi, Swaroop Mishra, et al., ACL, 2023.
    3. Scaling Instruction-Finetuned Language ModelsHyung Won Chung, Le Hou, Shayne Longpre, et al., Journal of Machine Learning Research, 2024.
    4. LoRA: Low-Rank Adaptation of Large Language ModelsEdward J. Hu, Yelong Shen, Phillip Wallis, et al., ICLR, 2022.
    5. Direct Preference Optimization: Your Language Model is Secretly a Reward ModelRafael Rafailov, Archit Sharma, Eric Mitchell, et al., NeurIPS, 2023.
    6. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language ModelsZhihong Shao, Peiyi Wang, Qihao Zhu, et al., DeepSeek Technical Report, 2024.
    7. DeepSeek-R1 Incentivizes Reasoning in LLMs through Reinforcement LearningDaya Guo, Dejian Yang, Haowei Zhang, et al., Nature, 2025.
    8. CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement LearningHung Le, Yue Wang, Akhilesh Deepak Gotmare, et al., NeurIPS, 2022.
    9. WizardCoder: Empowering Code Large Language Models with Evol-InstructZiyang Luo, Can Xu, Pu Zhao, et al., ICLR, 2024.
    10. Magicoder: Empowering Code Generation with OSS-InstructYuxiang Wei, Zhe Wang, Jiawei Liu, et al., ICML, 2024.
    11. OpenCodeInterpreter: Integrating Code Generation with Execution and RefinementTianyu Zheng, Ge Zhang, Tianhao Shen, et al., Findings of ACL, 2024.
    12. Training Language Models to Self-Correct via Reinforcement LearningAviral Kumar, Vincent Zhuang, Rishabh Agarwal, et al., ICLR, 2025.
    13. Training Software Engineering Agents and Verifiers with SWE-GymJiayi Pan, Xingyao Wang, Graham Neubig, et al., ICML, 2025.
    14. SWE-RL: Advancing LLM Reasoning via Reinforcement Learning on Open Software EvolutionYuxiang Wei, Olivier Duchenne, Jade Copet, et al., NeurIPS, 2025.
    15. SWE-smith: Scaling Data for Software Engineering AgentsJohn Yang, Kilian Lieret, Carlos E. Jimenez, et al., NeurIPS Datasets and Benchmarks Track, 2025.
    16. SERA: Soft-Verified Efficient Repository AgentsEthan Shen, Daniel Tormoen, Saurabh Shah, et al., ICML, 2026.
    17. Toward Training Superintelligent Software Agents through Self-Play SWE-RLYuxiang Wei, Zhiqing Sun, Emily Mcmilin, et al., ICML, 2026.
    18. LEGO-RL: Harness-Native Reinforcement Learning for Coding AgentsYiming Du, Yuxin Jiang, Tao Yuan, et al., arXiv, 2026.
    19. Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?Yang Yue, Zhiqi Chen, Rui Lu, et al., NeurIPS, 2025.
  6. Week 06

    Inference-Time Scaling

    This week explores how additional computation at inference time can improve performance on software engineering tasks. Topics include sampling multiple candidates, reasoning, search, iterative refinement, execution feedback, and verification. Students will learn to distinguish candidate generation from candidate selection, compare strategies under matched computational budgets, and analyze trade-offs among task success, latency, and computational cost.

    Check Reading List:20 papers
    1. S*: Test Time Scaling for Code GenerationDacheng Li, Shiyi Cao, Chengkun Cao, et al., Findings of EMNLP, 2025.
    2. CodeMonkeys: Scaling Test-Time Compute for Software EngineeringRyan Ehrlich, Bradley Brown, Jordan Juravsky, et al., arXiv, 2025.
    3. DARS: Dynamic Action Re-Sampling to Enhance Coding Agent Performance by Adaptive Tree TraversalVaibhav Aggarwal, Ojasv Kamal, Abhinav Japesh, et al., ACL, 2025.
    4. Wider or Deeper? Scaling LLM Inference-Time Compute with Adaptive Branching Tree SearchYuichi Inoue, Kou Misaki, Yuki Imajuku, et al., NeurIPS, 2025.
    5. Satori-SWE: Evolutionary Test-Time Scaling for Sample-Efficient Software EngineeringGuangtao Zeng, Maohao Shen, Delin Chen, et al., arXiv, 2025.
    6. Dynamic Scaling of Unit Tests for Code Reward ModelingZeyao Ma, Xiaokang Zhang, Jing Zhang, et al., ACL, 2025.
    7. Learning to Generate Unit Tests for Automated DebuggingArchiki Prasad, Elias Stengel-Eskin, Justin Chih-Yao Chen, et al., COLM, 2025.
    8. R2E-Gym: Procedural Environment Generation and Hybrid Verifiers for Scaling Open-Weights SWE AgentsNaman Jain, Jaskirat Singh, Manish Shetty, et al., COLM, 2025.
    9. CodeChemist: Test-Time Scaling for Low-Resource Code Generation via Functional Knowledge TransferKaixin Wang, Tianlin Li, Xiaoyu Zhang, et al., ICML, 2026.
    10. Code Generation by Differential Test Time ScalingYifeng He, Ethan Wang, Jicheng Wang, et al., arXiv, 2026.
    11. GraphAHA: Graph-Based Adaptive Search with Heterogeneous Actions for Test-Time Code GenerationXitao Li, Haijun Wang, Gege Yuan, et al., arXiv, 2026.
    12. TipCoder: Reinforcement Learning Boosted Test-time Instruction Proposer for Code GenerationMinyu Chen, Sihao Wu, Ling-I Wu, et al., Findings of EMNLP, 2026.
    13. Verification Trap: Understanding Test-Time Selection Failures under False Premises in Code GenerationFeng He, Hejia Wang, Linghao Meng, et al., EMNLP, 2026.
    14. Adaptive Generate-Rank-Verify: Inference-Time Search with Costly VerificationShaddin Dughmi, Mahdi Haghifam, Yusuf Hakan Kalayci, NeurIPS (accepted), 2026.
    15. Verifiers in the Loop: Decoding Time Verification for Code TranslationTianyang Zhou, Somesh Jha, Mihai Christodorescu, et al., NeurIPS (accepted), 2026.
    16. An Iterative Test-and-Repair Framework for Competitive Code GenerationLingxiao Tang, Muyang Ye, Zhaoyang Chu, et al., arXiv, 2026.
    17. Towards Functional Correctness of Code Models with Selective GenerationJaewoo Jeong, Taesoo Kim, Sangdon Park, ICML, 2026.
    18. A Theoretical Analysis of Test-Driven Code GenerationNicolas Menet, Michael Hersche, Andreas Krause, et al., NeurIPS (accepted), 2026.
    19. ExPairT-LLM: Exact Learning for LLM Code Selection by Pairwise QueriesTom Yuviler, Dana Drachsler-Cohen, AAAI, 2026.
    20. Reason-Code: Reliable Code Generation via Test-Driven Monte Carlo Tree SearchZixu Li, Zhiqi Peng, ACL Industry Track, 2026.
  7. Week 07

    Coding Agents

    This week introduces agents that combine language models, tool use, and environmental feedback to perform software engineering tasks. Topics include task decomposition, repository exploration, planning, memory, and multi-agent collaboration. Through examples of fault localization, code modification, testing, and repair, students will learn to build a basic agent loop, analyze execution traces, and compare different workflows.

    Check Reading List:20 papers
    1. LocAgent: Graph-Guided LLM Agents for Code LocalizationZhaoling Chen, Xiangru Tang, Gangda Deng, et al., ACL, 2025.
    2. RepoMaster: Autonomous Exploration and Understanding of GitHub Repositories for Complex Task SolvingHuacan Wang, Ziyi Ni, Shuo Zhang, et al., NeurIPS, 2025.
    3. Multi-CoLoR: Context-Aware Localization and Reasoning across Multi-Language CodebasesIndira Vats, Sanjukta De, Subhayan Roy, et al., SANER, 2026.
    4. PatchPilot: A Cost-Efficient Software Engineering Agent with Early Attempts on Formal VerificationHongwei Li, Yuheng Tang, Shiqi Wang, et al., ICML, 2025.
    5. Unified Software Engineering Agent as AI Software EngineerLeonhard Applis, Yuntong Zhang, Shanchao Liang, et al., ICSE, 2026.
    6. Trae Agent: An LLM-based Agent for Software Engineering with Test-time ScalingTrae Research Team, Pengfei Gao, Zhao Tian, et al., ByteDance Technical Report, 2025.
    7. debug-gym: A Text-Based Environment for Interactive DebuggingXingdi Yuan, Morgane M Moss, Charbel El Feghali, et al., Microsoft Research Technical Report, 2025.
    8. RepoAudit: An Autonomous LLM-Agent for Repository-Level Code AuditingJinyao Guo, Chengpeng Wang, Xiangzhe Xu, et al., ICML, 2025.
    9. RefAgent: A Multi-agent LLM-based Framework for Automatic Software RefactoringKhouloud Oueslati, Maxime Lamothe, Foutse Khomh, ICSE, 2026.
    10. Multi-Agent Coordinated Rename RefactoringAbhiram Bellur, Mohammed Raihan Ullah, Dima Golubenko, et al., arXiv, 2026.
    11. RefactorAssist: Agentic Refinement for Reliable Code RefactoringJonathan Cordeiro, Shayan Noei, Ying Zou, arXiv, 2026.
    12. Multi-Agent LLM Collaboration for Unit Test Generation via Human-Testing-Inspired WorkflowsQuanjun Zhang, Ye Shang, Siqi Gu, et al., arXiv, 2026.
    13. Archer: Towards Agentic Review for Compiler OptimizationsYunbo Ni, Shaohua Li, arXiv, 2026.
    14. Adversarial Review: Structured Disagreement for Grounded Agentic Code ReviewEric S. Qiu, Joyce Gill, ICML DL4C Workshop, 2026.
    15. SWE-MeM: Learning Adaptive Memory Management for Long-Horizon Coding AgentsShuzheng Gao, Wenhao Zeng, Zhaojian Yu, et al., arXiv, 2026.
    16. MemTrace: State-Consistent Memory for Long-Horizon Coding AgentsHongming Xu, Le Zhou, ZhongHe Jin, et al., arXiv, 2026.
    17. SE-Agent: Self-Evolution Trajectory Optimization in Multi-Step Reasoning with LLM-Based AgentsYifu Guo, Jiaye Lin, Huacan Wang, et al., NeurIPS, 2025.
    18. Scaling Inference-Time Compute with Teams of Coding AgentsKeenan Samway, Daniel D'souza, Omar Darwiche Domingues, et al., arXiv, 2026.
    19. ChatDev 2.0: A No-Code Multi-Agent Platform for Developing EverythingYufan Dang, Shu Yao, Bowen Lai, et al., EMNLP System Demonstrations, 2026.
    20. DeepCode: Open Agentic CodingZongwei Li, Zhonghang Li, Zirui Guo, et al., arXiv, 2025.
  8. Week 08

    Agent Harnesses

    This week examines agent harnesses, the software infrastructure that supports and controls agent execution. Topics include tool interfaces, execution environments, context management, state persistence, resource budgets, and failure handling and recovery. Students will learn how these choices affect the information, actions, and feedback available to an agent, and gain experience modifying harness components and evaluating their interaction with the model.

    Check Reading List:20 papers
    1. The OpenHands Software Agent SDK: A Composable and Extensible Foundation for Production AgentsXingyao Wang, Simon Rosenberg, Juan Michelini, et al., MLSys, 2026.
    2. AgentScope 1.0: A Developer-Centric Framework for Building Agentic ApplicationsDawei Gao, Zitao Li, Yuexiang Xie, et al., arXiv, 2025.
    3. Configuring Agentic AI Coding Tools: An Exploratory StudyMatthias Galster, Seyedmoein Mohsenimofidi, Jai Lal Lulla, et al., AIware, 2026.
    4. Harness Engineering: Anatomy, Architecture, and Evolution of Coding Agents -- A Source-Code Study of Eleven SystemsPaul Barbaste, Tristan Darrigol, Germain Vu, et al., arXiv, 2026.
    5. Code as Agent HarnessXuying Ning, Katherine Tieu, Dongqi Fu, et al., arXiv, 2026.
    6. LLM-as-Code: Agentic Programming for Agent HarnessJunjia Qi, Zichuan Fu, Jingtong Gao, et al., KDD AgenticSE Workshop, 2026.
    7. The Complexity Trap: Simple Observation Masking Is as Efficient as LLM Summarization for Agent Context ManagementTobias Lindenbauer, Igor Slinko, Ludwig Felder, et al., NeurIPS DL4C Workshop, 2025.
    8. An Empirical Study of Harness Design for Coding AgentsRun-Ze Fan, Zihao Zhang, Simin Ma, et al., arXiv, 2026.
    9. Beyond the Model: Demystifying Harness Effects in Software Engineering AgentsHaichuan Hu, Quanjun Zhang, Shengcheng Yu, et al., arXiv, 2026.
    10. CTXAgent - Self-Controlled Context Management of LLM AgentsMatthias Busch, Marius Tacke, Roland C. Aydin, et al., arXiv, 2026.
    11. ACM: Agentic Context Management for Long Horizon TasksXiaochuan Li, Ryan Ming, Meng Chu, et al., arXiv, 2026.
    12. Code Isn't Memory: A Structural Codebase Index Inside a Coding AgentIshaan Bhola, Adithyan Krishnan, Sravanth Kurmala, et al., arXiv, 2026.
    13. RefactorPlatform: An Open-Source Harness for Controlled Evaluation of Repository-Scale Refactoring AgentsAziz Ben Amor, Drish Mali, Mann Acharya, et al., arXiv, 2026.
    14. Agentic Harness Engineering: Observability-Driven Automatic Evolution of Coding-Agent HarnessesJiahang Lin, Shichun Liu, Chengjun Pan, et al., arXiv, 2026.
    15. Meta-Harness: End-to-End Optimization of Model HarnessesYoonho Lee, Roshen Nair, Qizheng Zhang, et al., arXiv, 2026.
    16. LEGO-RL: Harness-Native Reinforcement Learning for Coding AgentsYiming Du, Yuxin Jiang, Tao Yuan, et al., arXiv, 2026.
    17. Effective Context Engineering for AI AgentsPrithvi Rajasekaran, Ethan Dixon, Carly Ryan, et al., Anthropic Engineering Report, 2025.
    18. Effective Harnesses for Long-Running AgentsJustin Young, Anthropic Engineering Report, 2025.
    19. Harness Design for Long-Running Application DevelopmentPrithvi Rajasekaran, Anthropic Engineering Report, 2026.
    20. Harness Engineering: Leveraging Codex in an Agent-First WorldRyan Lopopolo, OpenAI Engineering Report, 2026.
  9. Week 09

    AI for Software Engineering in Industry

    Industry practitioners will share their experience developing and deploying AI tools for software engineering. Discussions will examine how models and agents fit into development workflows, including code review and team collaboration, and how quality, cost, and operational constraints influence adoption. Students will use concrete cases to assess benefits and limitations, understand engineering bottlenecks, and identify research opportunities.

    Check Reading List:22 papers
    1. ★ "My productivity is boosted, but ..." Demystifying Users' Perception on AI Coding AssistantsYunbo Lyu, Zhou Yang, Jieke Shi, et al., ASE, 2025.
    2. The SPACE of AI: Real-World Lessons on AI’s Impact on DevelopersBrian Houck, Travis Lowdermilk, Cody Beyer, et al., Microsoft Research Report, 2025.
    3. DORA 2025 State of AI-assisted Software Development ReportDerek DeBellis, Kevin Storer, Nathen Harvey, et al., DORA / Google Research Report, 2025.
    4. On the Use of Agentic Coding: An Empirical Study of Pull Requests on GitHubMiku Watanabe, Hao Li, Yutaro Kashiwa, et al., ACM Transactions on Software Engineering and Methodology, 2026.
    5. Programming by Chat: A Large-Scale Behavioral Analysis of 11,579 Real-World AI-Assisted IDE SessionsNingzhi Tang, Chaoran Chen, Zihan Fang, et al., ASE, 2026.
    6. From Delegation to Action Approval: An Exploratory Study of Developers' Practices in Collaborating with AI Coding AgentsYinan Wu, Dominik Wermke, Ze Shi Li, et al., arXiv, 2026.
    7. Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer ProductivityJoel Becker, Nate Rush, Elizabeth Barnes, et al., METR Technical Report, 2025.
    8. We Are Changing Our Developer Productivity Experiment DesignJoel Becker, Nate Rush, Tom Cunningham, et al., METR Research Update, 2026.
    9. Measuring the Self-Reported Impact of Early-2026 AI on Technical Worker ProductivityJoel Becker, METR Research Report, 2026.
    10. Writing Code vs. Shipping Code: Productivity Effects Across Generations of AI Coding ToolsMert Demirer, Leon Musolff, Liyuan Yang, NBER Working Paper 35275, 2026.
    11. AI Writes Faster Than Humans Can Review: A Longitudinal Study of an Enterprise 2x MandateHao He, Shyam Agarwal, Yegor Denisov-Blanch, et al., arXiv, 2026.
    12. Agentic Coding and Persistent Returns to ExpertiseZoe Hitzig, Maxim Massenkoff, Eva Lyubich, et al., Anthropic Research Report, 2026.
    13. How AI Impacts Skill FormationJudy Hanwen Shen, Alex Tamkin, arXiv, 2026.
    14. The Impact of AI Coding Assistants on Software Engineering: A Longitudinal StudyAnnie Vella, Kelly Blincoe, ACM Transactions on Software Engineering and Methodology, 2026.
    15. Does Working with AI Agents Change How Developers Code, Test, and Review?Muhammad Hamza, Dominik Siemon, Wardah Naeem Awan, ESEM, 2026.
    16. Testing with AI Agents: An Empirical Study of Test Generation Frequency, Quality, and CoverageSuzuka Yoshimoto, Shun Fujita, Kosei Horikawa, et al., MSR Mining Challenge, 2026.
    17. Debt Behind the AI Boom: A Large-Scale Empirical Study of AI-Generated Code in the WildYue Liu, Ratnadira Widyasari, Yanjie Zhao, et al., arXiv, 2026.
    18. Engineering Pitfalls in AI Coding Tools: An Empirical Study of Bugs in Claude Code, Codex, and Gemini CLIRuixin Zhang, Wuyang Dai, Hung Viet Pham, et al., FSE Industry Track, 2026.
    19. How AI Is Transforming Work at AnthropicSaffron Huang, Bryan Seethor, Esin Durmus, et al., Anthropic Research Report, 2025.
    20. Harness Engineering: Leveraging Codex in an Agent-First WorldRyan Lopopolo, OpenAI Engineering Report, 2026.
    21. Scaling Long-Running Autonomous CodingWilson Lin, Cursor Engineering Report, 2026.
    22. Agent Swarms and the New Model EconomicsWilson Lin, Cursor Engineering Report, 2026.
  10. Week 10

    Benchmarking and Evaluation

    This week examines how software engineering benchmarks are constructed and evaluated, including task selection, data splits, execution environments, and evaluation metrics. We will consider functional correctness, the runtime performance of generated software, and agent efficiency, alongside data leakage, test adequacy, variability, and the reliability of human and model-based evaluation. Students will learn to design reproducible comparisons and assess which conclusions the results support.

    Check Reading List:22 papers
    1. Multi-SWE-bench: A Multilingual Benchmark for Issue ResolvingDaoguang Zan, Zhirong Huang, Wei Liu, et al., NeurIPS Datasets and Benchmarks Track, 2025.
    2. SWE-PolyBench: A multi-language benchmark for repository level evaluation of coding agentsMuhammad Shihab Rashid, Christian Bock, Yuan Zhuang, et al., arXiv, 2025.
    3. SWE-bench Goes Live!Linghao Zhang, Shilin He, Chaoyun Zhang, et al., NeurIPS Datasets and Benchmarks Track, 2025.
    4. SWE-rebench V2: Language-Agnostic SWE Task Collection at ScaleIbragim Badertdinov, Maksim Nekrashevich, Anton Shevtsov, et al., arXiv, 2026.
    5. SWE-Lancer: Can Frontier LLMs Earn $1 Million from Real-World Freelance Software Engineering?Samuel Miserendino, Michele Wang, Tejal Patwardhan, et al., ICML, 2025.
    6. FeatureBench: Benchmarking Agentic Coding for Complex Feature DevelopmentQixing Zhou, Jiacheng Zhang, Haiyang Wang, et al., arXiv, 2026.
    7. APEX-SWEAbhi Kottamasu, Chirag Mahapatra, Sam Lee, et al., arXiv, 2026.
    8. Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line InterfacesMike A. Merrill, Alexander G. Shaw, Nicholas Carlini, et al., arXiv, 2026.
    9. SWE-Serve: Benchmarking Agentic Engineering For Production Inference ServingJennifer Williams, Dave Farris, Jeff Farris, et al., arXiv, 2026.
    10. SlopCodeBench: Benchmarking How Coding Agents Degrade Over Long-Horizon Iterative TasksGabriel Orlanski, Devjeet Roy, Alexander Yun, et al., arXiv, 2026.
    11. Multi-step SWE: Can Coding Agents Keep Building on Their Own Code Over Long Horizons?Xiangkai Zeng, Zheng Liu, Zihao Lin, arXiv, 2026.
    12. SWE-Together: Evaluating Coding Agents in Interactive User SessionsYifan Wu, Zhuokai Zhao, Songlin Li, et al., arXiv, 2026.
    13. SWE-Gate: Passing Functional Tests Is Not Enough for Software Engineering AgentsXin He, Yanlin Wang, Mingwei Liu, et al., arXiv, 2026.
    14. Code Review Agent BenchmarkYuntong Zhang, Zhiyuan Pan, Imam Nur Bani Yusuf, et al., arXiv, 2026.
    15. SWE-fficiency: Can Language Models Optimize Real-World Repositories on Real Workloads?Jeffrey Jian Ma, Milad Hashemi, Amir Yazdanbakhsh, et al., ICML, 2026.
    16. Are “Solved Issues” in SWE-bench Really Solved Correctly? An Empirical StudyYou Wang, Michael Pradel, Zhongxin Liu, ICSE, 2026.
    17. SWE-Bench Pro V2: A Cleaner, Harder-to-Game Coding BenchmarkSoham Dan, Jeff Da, Sanyam Bhutani, et al., arXiv, 2026.
    18. Well-Specified and Well-Tested? Assessing SWE-bench-Style Datasets with a Self-Evolving AgentZhonghao Jiang, Xuyang Zheng, Yanfu Yan, et al., arXiv, 2026.
    19. ★ Beyond Text Matching: Towards Reference-Free Evaluation for Human-Oriented Binary Reverse EngineeringXiuwei Shang, Li Hu, Xiao Jiang, et al., ASE, 2026.
    20. Quantifying Memorization Advantage in Code LLMsAlberick Euraste Djire, Abdoul Kader Kaboré, Jordan Samhi, et al., ICSE, 2026.
    21. Reflections on the Reproducibility of Commercial LLM Performance in Empirical Software Engineering StudiesFlorian Angermeir, Maximilian Amougou, Mark Kreitz, et al., ICSE, 2026.
    22. Quantifying Infrastructure Noise in Agentic Coding EvalsGian Segato, Anthropic Engineering Report, 2026.
  11. Week 11

    Reliability, Security, and Privacy

    This week examines the reliability, security, and privacy of coding agents and generated software, as well as fairness requirements in relevant applications. Topics include secure code generation, prompt injection, unsafe tool use, access control, runtime verification, and human oversight. Students will learn to identify risks, evaluate mitigation strategies, and understand their guarantees, limitations, and effects on task completion.

    Check Reading List:26 papers
    1. Teaching Code Agents to Ask Without Leaking Private CodeWenjia Jiang, Jieke Shi, Zhensu Sun, et al., arXiv, 2026.
    2. PrivCode++: Latent-Conditioned Differentially Private Code Generation for Comprehensive GuaranteesZheng Liu, Chen Gong, Terry Yue Zhuo, et al., ICML, 2026.
    3. Scrub It Out! Erasing Sensitive Memorization in Code Language Models via Machine UnlearningZhaoyang Chu, Yao Wan, Zhikun Zhang, et al., ICSE, 2026.
    4. PrivCode: When Code Generation Meets Differential PrivacyZheng Liu, Chen Gong, Terry Yue Zhuo, et al., NDSS, 2026.
    5. Watermarking LLM Agent TrajectoriesWenlong Meng, Chen Gong, Terry Yue Zhuo, et al., ICML, 2026.
    6. Backdoors in Code Summarizers: How Bad Is It?Chenyu Wang, Zhou Yang, Yaniv Harel, et al., ASE, 2025.
    7. Insecure Coding Preferences in Long-Term Memory: Security Risks for LLM-based Code GenerationYuchen Chen, Wei Cheng, Yuan Xiao, et al., ISSTA, 2026.
    8. SkillJect: Effectively Automating Skill-Based Prompt Injection for Skill-Enabled AgentsXiaojun Jia, Jie Liao, Simeng Qin, et al., arXiv, 2026.
    9. MCPTox: A Benchmark for Tool Poisoning on Real-World MCP ServersZhiqiang Wang, Yichao Gao, Yanting Wang, et al., AAAI, 2026.
    10. BaxBench: Can LLMs Generate Correct and Secure Backends?Mark Vero, Niels Mündler, Victor Chibotaru, et al., ICML, 2025.
    11. Is Vibe Coding Safe? Benchmarking Vulnerability of Agent-Generated Code in Real-World TasksSongwen Zhao, Danqing Wang, Kexun Zhang, et al., ICML, 2026.
    12. SecureVibeBench: Benchmarking Secure Vibe Coding of AI Agents via Reconstructing Vulnerability-Introducing ScenariosJunkai Chen, Huihui Huang, Yunbo Lyu, et al., ACL, 2026.
    13. Benchmarking Correctness and Security in Multi-Turn Code GenerationRuchit Rawal, Jeffrey Yang Fan Chiang, Chihao Shen, et al., EMNLP, 2026.
    14. DeepGuard: Secure Code Generation via Multi-Layer Semantic AggregationLi Huang, Zhongxin Liu, Yifan Wu, et al., ACL, 2026.
    15. Learning from Failures: Error Notebook-guided Secure Code GenerationXinyu Zhong, Peng Lan, Zhifang Liao, Findings of ACL, 2026.
    16. AgentSpec: Customizable Runtime Enforcement for Safe and Reliable LLM AgentsHaoyu Wang, Christopher M. Poskitt, Jun Sun, ICSE, 2026.
    17. Defeating Prompt Injections by DesignEdoardo Debenedetti, Ilia Shumailov, Tianqi Fan, et al., arXiv, 2025.
    18. Safety Sidecar: Reflection-Driven Runtime Control for Safer AgentsBin Wang, Jiazheng Quan, Xingrui Yu, et al., Findings of ACL, 2026.
    19. HarnessSecurity-Bench: Do Security Mechanisms Really Protect Coding Agent Harnesses?Zhengyang Zhu, Liming Huang, Runmin Ji, et al., arXiv, 2026.
    20. SABER: Benchmarking Operational Safety of LLM Coding Agents in Stateful Project WorkspacesQi Hu, Yifeng Tang, Qinghua Wang, et al., EMNLP, 2026.
    21. AttriGuard: Defeating Indirect Prompt Injection in LLM Agents via Causal Attribution of Tool InvocationsYu He, Haozhe Zhu, Yiming Li, et al., USENIX Security, 2026.
    22. The Attacker Moves Second: Stronger Adaptive Attacks Bypass Defenses Against LLM Jailbreaks and Prompt InjectionsMilad Nasr, Nicholas Carlini, Chawin Sitawarin, et al., USENIX Security, 2026.
    23. How Reasoning Shapes Social Bias in LLM-Generated Code?Weifeng Sun, Jieke Shi, Zhou Yang, et al., ASE, 2026.
    24. FairCoder: Probing LLM Bias in High-Stakes Decision Making via Coding TasksYongkang Du, Jen-tse Huang, Jieyu Zhao, et al., arXiv, 2025.
    25. From If-Statements to ML Pipelines: Revisiting Bias in Code-GenerationMinh Duc Bui, Xenia Heilmann, Mattia Cerrato, et al., Findings of ACL, 2026.
    26. Hidden Licensing Risks in the PTMware EcosystemBo Wang, Yueyang Chen, Jieke Shi, et al., ISSTA, 2026.
  12. Week 12

    Applications in Hardware, Robotics, and Algorithm Optimization

    This week explores applications of code models and agents in hardware and integrated circuit design, robotics, and algorithm optimization. Through examples of domain adaptation, code generation, tool use, and verification, students will compare the domain knowledge, program representations, execution environments, and feedback required in each setting. They will examine where existing methods transfer and identify new engineering challenges.

    Check Reading List:30 papers
    1. Comprehensive Verilog Design Problems: A Next-Generation Benchmark Dataset for Evaluating Large Language Models and Agents on RTL Design and VerificationNathaniel Pinckney, Chenhui Deng, Chia-Tung Ho, et al., arXiv, 2025.
    2. FormalRTL: Verified RTL Synthesis at ScaleKezhi Li, Min Li, Xiangyu Wen, et al., arXiv, 2026.
    3. Veri-Sure: Multi-Agent RTL Code Generation with Temporal Tracing, Slicing and Formal VerificationJiale Liu, Taiyu Zhou, Tianqi Jiang, arXiv, 2026.
    4. SYMDIREC: A Neuro-Symbolic Divide-Retrieve-Conquer Framework for Enhanced RTL Synthesis and SummarizationPrashanth Vijayaraghavan, Apoorva Nitsure, Luyao Shi, et al., EACL Industry Track, 2026.
    5. Exploring the Agentic Frontier of Verilog Code GenerationPatrick Yubeaton, Siddharth Garg, Chinmay Hegde, arXiv, 2026.
    6. GoGoTB: Agentic RTL Verification with Specification-Grounded Coverage ClosureXin Xin, Jincheng Lou, Junhui Li, et al., arXiv, 2026.
    7. Closing the Loop on LLM-Generated RTL Assertions with Quality-Aware Formal VerificationRamesh Krishnamurthy, Danial Chitnis, Themis Prodromakis, arXiv, 2026.
    8. Will AI mess my RTL? A snapshot-based approach to verifying AI-written RTLBernat Homs, Oscar Palomar, Miquel Moreto, et al., arXiv, 2026.
    9. LongRTL: Graph-Similarity-Guided LLM-driven Long Context RTL OptimizationYuyang Ye, Che-Kuan Shen, Xiangfei Hu, et al., DATE, 2026.
    10. Toward Autonomous RTL and Synthesis Optimization: Benchmarking and Self-Improving AgentsWenji Fang, Yao Lu, Jinghua Wang, et al., arXiv, 2026.
    11. VPR-Evolve: Multi-Agent-Driven Algorithm Evolution for FPGA Place and RouteQihang Wu, Taizun Jafri, Aman Arora, et al., arXiv, 2026.
    12. AlphaEvolve: A Coding Agent for Scientific and Algorithmic DiscoveryAlexander Novikov, Ngân Vũ, Marvin Eisenberger, et al., Google DeepMind Technical Report, 2025.
    13. Kevin: Multi-Turn RL for Generating CUDA KernelsCarlo Baronio, Pietro Marsella, Ben Pan, et al., ICLR, 2026.
    14. CUDA-L1: Improving CUDA Optimization via Contrastive Reinforcement LearningXiaoya Li, Xiaofei Sun, Albert Wang, et al., ICLR, 2026.
    15. Geak: Introducing Triton Kernel AI Agent & Evaluation BenchmarksJianghui Wang, Vinay Joshi, Saptarshi Majumder, et al., arXiv, 2025.
    16. KernelFalcon: Autonomous GPU Kernel Generation via Deep AgentsLaura Wang, PyTorch Team at Meta, Meta/PyTorch Engineering Report, 2025.
    17. CUDA Agent: Large-Scale Agentic RL for High-Performance CUDA Kernel GenerationWeinan Dai, Hanlin Wu, Qiying Yu, et al., arXiv, 2026.
    18. KernelEvolve: Scaling Agentic Kernel Coding for Heterogeneous AI Accelerators at MetaGang Liao, Hongsen Qin, Ying Wang, et al., ISCA, 2026.
    19. KernelOPT: Dispatch-Aware Agentic Search for GPU Kernel OptimizationAheli Poddar, Sanskar Prasad, Arindam Samanta, et al., arXiv, 2026.
    20. Are LLM-Generated GPU Kernels Production-Ready? A Trace-Driven Benchmark and Optimization AgentLingyun Yang, Yuxiao Wang, Shenghao Liang, et al., arXiv, 2026.
    21. HIERA: Workload-Aware Planning Across Implementation Spaces for GPU Kernel OptimizationJinghao Wang, Qiqi Gu, Chenpeng Wu, et al., arXiv, 2026.
    22. AutoMisty: A Multi-Agent LLM Framework for Automated Code Generation in the Misty Social RobotXiao Wang, Lu Dong, Sahana Rangasrinivasan, et al., IROS, 2025.
    23. CaP-X: A Framework for Benchmarking and Improving Coding Agents for Robot ManipulationLetian Fu, Justin Yu, Karim El-Refai, et al., ICML, 2026.
    24. RHO: Your Coding Agent is Secretly a RoboticistKarim Elmaaroufi, Justin Svegliato, Sarunas Kalade, et al., arXiv, 2026.
    25. ENPIRE: Agentic Robot Policy Self-Improvement in the Real WorldWenli Xiao, Jia Xie, Tonghe Zhang, et al., CoRL, 2026.
    26. ASPIRE: Agentic /Skills Discovery for RoboticsRunyu Lu, Yubo Wu, Ethan Kou, et al., arXiv, 2026.
    27. SEAR: Self-Evolving Agentic RoboticsXinyi Wang, Bangzheng Li, Hanyu Wang, et al., arXiv, 2026.
    28. RoboAuto: Autonomous Robot Program Search through Multi-Level Physical UnderstandingBuzhao Liu, Jianing Ye, Xiu Yuan, et al., arXiv, 2026.
    29. ACE-CaP-0: Real-World Code-as-Policies in the Era of ExperienceACE-CaP Team, Changxu Wei, Yunpeng Qing, et al., arXiv, 2026.
    30. ★ ManimAgent: Self-Evolving Multimodal Agents for Visual EducationWenjia Jiang, Zongyuan Cai, Yuanhang Shao, et al., EMNLP, 2026.
  13. Week 13

    Research Project Presentations I

    The first set of groups will present their research projects and papers, covering the research questions, related work, methods, experimental results, and limitations. Through demonstrations, questions, and peer feedback, students will explain how the evidence supports their contributions, discuss reproducibility, and identify revisions needed to strengthen their work.

  14. Week 14

    Research Project Presentations II

    The remaining groups will present their research projects and papers, covering the research questions, related work, methods, experimental results, and limitations. Through demonstrations, questions, and peer feedback, students will explain how the evidence supports their contributions, discuss reproducibility, and identify revisions needed to strengthen their work.

  15. Week 15

    Project Revisions and Research Artifacts

    Groups will present their revisions and respond to feedback from earlier sessions. Discussions will focus on additional evidence, changes to methods or conclusions, remaining limitations, and reproducibility. Students will refine their papers and research artifacts, justify their final claims, and identify the work needed before submission to a research venue.

Course materials

The weekly topics and available reading lists are included in the timeline above. More readings and lecture materials will be added here as they become available.

Course timeline
View weekly topics
Reading lists
Week 04 · 19 papers
Week 05 · 19 papers
Week 06 · 20 papers
Week 07 · 20 papers
Week 08 · 20 papers
Week 09 · 22 papers
Week 10 · 22 papers
Week 11 · 26 papers
Week 12 · 30 papers
Lecture materials
To be added

Contact

For course enquiries, email zy25@ualberta.ca.