Research Problem
The explosion of Large Language Models (LLMs) like GPT-4, LLaMA, and Codex has revolutionized software engineering (SE) tasks, yet the literature remains highly fragmented. Researchers and practitioners lack a comprehensive taxonomy mapping which LLM architectures, training paradigms, optimization techniques, and evaluation metrics are best suited for different software engineering tasks (such as code generation, code comment generation, unit testing, and vulnerability detection).
Proposed Solution
The authors perform a systematic literature review (SLR) covering papers published up to mid-2023. They synthesize a taxonomy classifying LLM applications in SE, detailing: (1) what SE tasks are addressed, (2) which LLM architectures are utilized, (3) how models are adapted (pre-training, fine-tuning, prompting), and (4) how performance is evaluated.
Methodology & Evaluation
Followed the standard SLR guideline (Kitchenham & Charters). Screened 229 relevant papers from digital libraries and arXiv. Coded each paper based on research questions focusing on SE tasks, LLM characteristics, optimization methods, and evaluation datasets, providing a synthesis of state-of-the-art results.
Main Contributions
- Established a standard taxonomy classifying LLM-based software engineering research across 10 distinct task domains.
- Provided a detailed mapping of LLM architectures (Encoder-only, Decoder-only, and Encoder-Decoder) and their performance tradeoffs in SE.
- Identified key open challenges, including the 'evaluation crisis' (the inadequacy of BLEU/ROUGE for code correctness), prompt sensitivity, and hallucination containment in development settings.
Limitations & Weaknesses
- Given the extreme velocity of LLM research, some papers published in late 2023 and 2024 are omitted, meaning newer techniques like advanced retrieval-augmented generation (RAG) are not fully cataloged.
- The survey relies heavily on academic publications, with less coverage of proprietary engineering techniques used in tools like GitHub Copilot or Cursor.
Key Takeaways
“Decoder-only models are dominating code generation, but encoder-decoder architectures remain highly effective for code-to-code translation (refactoring).”
“Evaluating code via execution-based tests (like Pass@k) is far superior to textual overlap metrics (like BLEU).”
How This Relates to My Research
Strategic Alignment
Directly relates to my research interest in AI for Software Engineering. By presenting a systematic overview of the entire field, this paper points me directly to under-explored niches, such as applying LLMs to software architecture and technical debt detection, rather than simple code generation.
Future Research Ideas
- Build a benchmark focusing specifically on evaluating LLMs on complex, multi-file refactoring and architectural restructuring tasks.
- Design an evaluation suite that uses AST-level syntax and semantic checks rather than regex or string matches for code reviews.
References
Hou, X., Zhao, Y., Liu, Y., Yang, Z., Wang, K., Li, L., Bissyandé, T. F., Klein, J., & Lo, D. (2023). Large language models for software engineering: A systematic literature review. *ACM Transactions on Software Engineering and Methodology*, 33(3), 1-45.