Self-Programming AI: Code-Learning Agents for Autonomous Refactoring and Architectural Evolution — part 4/7
Cleaned-text code points [40889, 54812). Bibliography omitted here; source.md includes it.
maintainability, and adherence to best practices, all without direct human code writing – a significant advancement in autonomous software development. 4. Results We evaluate the effectiveness and robustness of our proposed Self-Programming AI (SPA) framework through a series of practical experiments. Specifically, we assess the SPA’s ability to autonomously generate correct, maintainable, and high-quality Python code across a diverse set of tasks. The evaluation metrics directly reflect the criteria encoded in our iterative refinement pipeline: functional correctness (all tests pass), test coverage (≥80%), cyclomatic complexity (≤10 per function), maintainability index (higher values indicate easier maintainability), and lint score (zero errors). All experiments utilize GPT-3.5-turbo for both initial code generation and iterative refinement, with each OpenAI API interaction robustly wrapped in exponential backoff retries to handle potential rate-limit issues. Experimental Setup We selected a set of 5 representative coding tasks spanning different complexity levels and application domains, including: 1. File System Management: Generate and manipulate sets of files (e.g., creating numbered files). 2. Robust Retry Mechanism: Implement a retry decorator with exponential backoff functionality. 3. Data Parsing and Cleaning: Parse and sanitize CSV data from diverse sources. 4. Mathematical Algorithm Implementation: Implement common algorithms (e.g., prime-number detection, Fibonacci calculations). 5. Web Scraping and Content Extraction: Scrape structured information from simple HTML pages. For each task, we provided the SPA with a brief natural-language specification. The SPA autonomously ran through the self-programming loop, logging metrics at each iteration via MLflow until meeting all success criteria or reaching the maximum iteration limit (set at 5 iterations per task). The experiments were conducted using Python 3.11 on standard virtual machines with 4 CPUs and 16 GB RAM. Quantitative Results Task Description Iterations to Success Tests Passed (%) Coverage (%) Complexity (max) Maintainability Index Lint Errors File System Management 2 100% 95% 3 84 0 Robust Retry Mechanism 3 100% 92% 6 81 0 Data Parsing and Cleaning 3 100% 90% 8 79 0 Mathematical Algorithm 1 100% 100% 4 88 0 Web Scraping & Content Extraction 4 100% 88% 9 78 0 As summarized in the table, SPA successfully converged to correct and well-structured implementations for all tested tasks. Functional correctness was achieved across the board, with every task ending at a 100% test pass rate. Test coverage consistently exceeded our target threshold of 80%, ranging between 88% to 100%. The final cyclomatic complexity values remained comfortably below our set threshold of 10, ensuring that generated solutions were structurally simple and maintainable. Maintainability index scores (ranging from 78 to 88) further confirmed the generated code was highly readable and clean. Importantly, lint errors were fully resolved in every final implementation, indicating strict adherence to best coding practices. Iterative Refinement Analysis We observed that simpler tasks (such as basic mathematical algorithms) typically reached the success criteria within a single iteration, highlighting the model’s capability to produce correct and clean implementations directly from the initial specification. More complex tasks (like retry mechanisms and web scraping) required multiple refinement iterations. Examination of the MLflow logs showed a consistent pattern: the first iteration typically resolved fundamental correctness issues, while subsequent iterations progressively increased coverage, reduced complexity, and eliminated lint errors. Notably, the AST-level patching guided by GPT-3.5-turbo's suggested modifications led to meaningful structural improvements. In the retry mechanism task, for instance, early iterations introduced basic exponential backoff logic, and later refinements simplified conditional logic to reduce complexity scores from initially higher values (around 12-15) down to acceptable levels (≤6). Robustness and Stability Throughout our experiments, the Retry and Rate Limit Wrapper consistently managed API usage constraints, successfully mitigating transient rate-limit errors by automatic retries with exponential backo ff. Analysis of system logs confirmed that no experimental run failed due to external API errors or interruptions, underscoring the robustness and operational stability of the SPA framework in real-world, quota-constrained API conditions. Qualitative Observations Qualitative inspection of the generated code revealed high readability and professional coding standards across final outputs. The automated AST-level patching via LibCST preserved code style, ensuring edits were minimally intrusive yet effective. Additionally, the self-refining loop's reliance on clearly structured feedback (coverage metrics, complexity reports, lint results) allowed GPT-3.5 to generate targeted improvements with minimal deviation from prior iterations, ensuring evolutionary, rather than revolutionary, code improvements. Summary of Results Overall, the experiments demonstrate that our Self-Programming AI architecture can autonomously and iteratively evolve initial LLM-generated code into robust, maintainable software implementations. The system consistently achieves comprehensive correctness (passing 100% of tests), excellent coverage (≥8 8%), controlled complexity, high maintainability, and flawless adherence to linting standards. These results affirm the potential of autonomous, self-refining code-generation agents to significantly enhance automated software dev elopment practices, minimizing manual developer oversight, and efficiently producing production-quality code. Figure: Iterations to Convergence The bar graph illustrates the number of iterations required for the self-progr amming agent to reach convergence across five distinct tasks. Simpler tasks such as mathematical algorithm implementation required only 1 iteration, while more complex tasks like web scraping and retry mechanism needed 3–4 iterations, demonstrating that convergence speed correlates with task complexity. Figure: Test Coverage Progression This line chart tracks the improvement in test coverage percentage over successive iterations for each task. All tasks show a consistent upward trend, with coverage increasing steadily as the agent refines its code and test suite. The graph highlights the agent’s ability to identify gaps and enhance test thoroughness autonomously, ultimately surpassing the 80% target coverage threshold in all cases. Figure: Cyclomatic Complexity Reduction This chart shows the progressive decline in cyclomatic complexity across iterations for each task. Complex implementations like Retry Mechanism and Data Parsing saw the sharpest reductions as the agen t refactored overly nested logic into simpler structures. Tasks like Mathematical Algorithms required minimal simplification, reflecting their inherently low logical branching. The graph confirms the agent’s capacity to autonomously restructure code for improved maintainability. Figure: API Retry Occurrences This bar chart illustrates the number of retry attempts triggered due to OpenAI API rate-limit responses across four tasks. Tasks with more iterations and longer prompts, such as Retry Mechanism and Data Parsing, experienced higher retry counts, while simpler tasks like Mathematical Algorithms had minimal API backoff events. This demonstrates the system's ability to gracefully recover from external service constraints without halting the refinement loop. 1. Iterative Refinement Metrics Across Tasks Task Iterations Initial Coverage Final Coverage Initial Complexity Final Complexity Final Lint Errors File System Management 2 61% 95% 7 3 0 Retry Mechanism 3 52% 92% 14 6 0 Data Parsing & Cleaning 3 66% 90% 12 8 0 Mathematical Algorithm 1 100% 100% 5 4 0 Web Scraping & Content Extraction 4 58% 88% 13 9 0 Purpose: Shows concrete improvements the agent made over time in coverage, complexity, and code quality. 2. Agent Actions and Behaviors per Iteration Iteration Common Agent Action Effect Observed Metric Improved 1 Initial code and test generation Baseline functionality (often failing tests) None / Low baseline 2 Added edge-case handling Test failures reduced ↑ Test pass rate 3 Refactored large functions Complexity lowered ↓ Cyclomatic complexity 4 Added new helper tests Coverage improved ↑ Test coverage 5 Applied naming/style corrections Pylint score reached zero 0 lint errors Purpose: Illustrates how the agent’s behavior evolves naturally across iterations. 3. Agent Autonomy Comparison by Task Type Task Category Human Intervention Needed Agent Confidence Success in ≤3 Iterations Comments Simple logic (Math) High Solved directly with tests + style fixes File operations Medium–High Minor path handling adjustments needed Retry / decorator logic Medium Needed more rounds to reduce complexity Data parsing Medium Refined progressively via feedback Web scraping Medium Final version stable at iteration 4 Purpose: Shows where the agent succeeded autonomously vs. needed more refinement effort. 5. Discussion During experimentation, our self-programming agent consistently exhibited patterns akin to experienced human developers. For example, it often simplified complex logic by extracting common sub-steps into helper functions. When it encountered repeated conditions or lengthy boolean expressions, the agent automatically refactored them into separate methods – effectively implementing “extract method” refactorings to reduce cyclomatic complexity. This emergent behavior reduced code duplication and improved readability. In parallel, the agent frequently decomposed large functions into modular components. We observed entire monolithic implementations being split into smaller functions or classes, each handling a well-defined subtask. This divide-and- conquer strategy mirrors classical software engineering advice: breaking a problem into subproblems greatly eases development. By introducing new helper routines and shifting logic into dedicated modules, the agent lowered overall coupling and made the codebase more maintainable. In summary, the agent’s coding habits – pulling out helper functions and rearranging code into smaller pieces – reflect common hu man practices (e.g. reducing code smells, simplifying control flow) that are known to improve software design. These emergent coding strategies often translated into robust solutions. For instance, by consolidating repetitive calculations into a single helper, the agent not only reduced duplication but also made test construction easier. In one case, repeatedly nested loops were replaced by a single helper that handled the looping logic; this both improved performance and made the behavior clearer to subsequent test passes. Crucially, these habits were consistent across different tasks and iterations: the agent reliably defaulted to cleaner, simpler logic whenever possible. This suggests that, through its training and iterative self-feedback, the model has implicitly learned f undamental design principles such as DRY (Don’t Repeat Yourself) and single-responsibility. Such behavior is noteworthy because it arose without any explicit instruction about refactoring or modularity, indicating true emergent structural reasoning in the agent’s code-generation process. Limitations Despite these promising tendencies, the agent still suffers from well-known limitations of cur rent LLM-based coding. One major issue is hallucination, i.e. confidently producing incorrect or nonsensical content. In our experiments, the agent sometimes generated test cases or code fragments that violated the project context. For example, it occasionally instantiated classes that did not exist or used function names with undefined parameters – effectively inventing API details. Such hallucinations frequently caused compilation failures in the test suite. In line with recent studies, we found that a large fraction of invalid autogenerated tests were due to such hallucinated identifiers. In concrete terms, over 30% of the agent’s invalid tests involved unresolved symbols or parameter mismatches. These errors not only broke the validation pipeline but also misled the agent: chasing feedback on these nonexistent symbols consumed iterations that could have been spent on genuine improvements. The model also occasionally misinterpreted problem specifications or metrics (for instance, confusing percentage and probability values), indicating semantic misalignment in understanding test prompts. Such errors highlight that the agent’s reasoning is only as reliable as the model’s grounding – a known risk in deploying GPT-based coders. Another critical limitation is the quality of the test suite driving the loop. Our agent operates by passing unit tests, so inadequate tests can easily give the wrong incentives. In practice we observed that when tests were shallow (e.g. covering only trivial cases) the agent converged on superficially correct yet flawed log ic. Put simply, it “got what it tested for.” If the provided tests did not catch an edge case or logical bug, the agent had no impetus to fix it and would stabilize the faulty solution. This dependence means that any blind spots or omissions in the test suite will be baked into the agent’s final code. This phenomenon aligns with standard software-engineering wisdom: correct specifications (here, tests) are essential for correct code. In fact, in several runs where we deliberately weakened the