Pseudocode Controlflow Project (Working Title)

# What

The Pseudocode Controlflow Project is a research project focused on developing and evaluating methods for handling complex tasks using Large Language Models (LLMs). The project aims to create a system where tasks are broken down into executable instructions, which can be translated into code or managed with LLM calls.

The process is structured in two main stages:

1. Generation: Task → Instructions: Transforming a general or specific task into a series of subtasks or instructions.
2. Compilation: Instructions → Execution: Converting these instructions into a form that can be executed, either by generating code for well-defined steps or by using LLMs to handle more ambiguous parts.

For example, a task like "grade this student's paper using a rubric" could be broken down into a sequence of smaller steps, such as checking criteria, scoring, and repeating until all criteria are evaluated.

Why

Project Goals:

• The overall purpose is to automate repetitive or detailed tasks.
  • To automate difficult tasks via breaking them down into simpler tasks.
  • To automate repetitive tasks efficiently and consistently

Meta Goals:

• Provide team members with hands-on experience in research and quick, iterative development with LLMs.
• Document the project comprehensively as a case study for the club's research efforts.

How

The project will explore different strategies for the two main stages:

1. Generation (Task → Instructions)

   • Possible Approaches:
     • Experimenting with various prompting techniques for LLMs, such as [few-shot learning](https://www.ibm.com/topics/few-shot-learning#:~:text=IBM-,What is few-shot learning%3F,suitable training data is scarce.).
     • Developing an algorithm-creation pipeline that breaks down a task into its core elements before generating a set of instructions.
   • Generates instructions in the form of pseudocode, a simplified representation that separates clear, codable parts (like loops, conditionals, and step orders) from ambiguous parts that require creative interpretation (e.g., "implement solution").
   • Should be a recursive algorithm which can be called on ambiguous lines, which allows for iterative specification of the instructions.
     • The more specified, the simpler the remaining ambiguous parts should be.
     • The number of iterations would depend on things like difficulty and estimated frequency of the Task.

   Example:

   Task: Debug code given a set of errors.

   Output Pseudocode:

   while errors not empty:
       for each error in errors:
           generate solutions for error
           if solution is small:
               implement solution
               run tests
           else:
               test simplified version of solution
               implement if tests pass
   

   • Example of recursive specification:

2. Compilation (Instructions → Execution)

   • Clearly codable parts will be directly converted into code.
     • For efficiency and reliability.
   • Remaining ambiguous parts will be handled via LLM approximation.

Who