Some prompt engineering techniques are simple and easy to implement (like "According to.." prompting). Some are extremely powerful, but require a more advanced setup with many more API calls, leading to increased costs and latency (like Tree of Thoughts).
Researchers at Virginia Tech and Microsoft have recently developed a new method that stands up against the Tree of Thoughts methhod with a remarkable reduction of 100 times fewer queries needed. Yes, you read that correctly — 100 times fewer queries. (link to the full paper here)
Enter, Algorithm of Thoughts (AoT).
What is Algorithm of Thoughts prompting?
Algorithm of Thoughts (AoT) is a prompting engineering method designed to mimic algorithmic thinking in large language models (LLMs). It breaks down tasks into defined sub-steps like defining the problem, gathering and analyzing information, formulating a hypothesis, testing it, and coming to a final conclusion.
Why Algorithm of Thoughts?
Large Language Models (LLMs) are extremely powerful, but they aren’t perfect. Prompt engineering exists because consistently getting high-quality outputs isn’t a walk in the park.
Many of the top-performing, and widely adopted methods are computationally heavy. For example, the Tree of Thoughts (ToT) method requires multiple rounds of querying as it traverses dozens of branches and nodes.
Designed to address these challenges, Algorithm of Thoughts presents a structured path of reasoning for LLMs. It's a solution that delivers on efficiency without compromising on the quality of outputs.
How Algorithm of Thought promptingworks
Algorithm of Thoughts is designed to mimic algorithmic thinking. It is broken down into a few steps:
- Define the Problem: Algorithm of Thoughts begins by clearly stating the problem behind the query or task.
- Gather Information: Before diving into solutions, Algorithm of Thoughts prompts the LLM to get the necessary context.
- Analyze the Information: Next, the LLM breaks down the gathered information, identifying patterns, relationships, or anomalies.
- Formulate a Hypothesis: Based on the analysis, the LLM puts together an initial solution.
- Test the Hypothesis: The LLM then thinks of ways to validate or refute its hypothesis, envisioning potential outcomes.
- Draw Conclusions: After testing, the LLM summarizes its findings, providing a refined solution to the initial problem.
- Reflect: Lastly, the LLM considers the broader implications of its conclusion, thinking of potential next steps or further questions.
Algorithm of Thoughts 'structured approach ensures the LLM isn’t left to wander aimlessly down many paths.
Algorithm of Thoughts prompt template
The Algorithm of Thoughts prompt technique isn’t the easiest to implement, but we’ve done our best to make it plug and play. Generally you want to follow this structure:
Here’s a practical example if you were using Algorithm of Thoughts to do research on the environmental implications of data centers.
- Problem Statement: What are the environmental implications of increased data center usage?
- Background Information: Data centers account for about 1% of global electricity use.
- Initial Hypothesis: Implementing renewable energy sources can mitigate the environmental impact.
- Reasoning: Evaluate the feasibility and impact of using renewable energy for data centers.
- Conclusion: Based on the analysis, is renewable energy a viable solution for data centers?
You could then restructure that into something a little more LLM-friendly.
"Given that data centers account for about 1% of global electricity use, what are the environmental implications of increased data center usage? I hypothesize that implementing renewable energy sources can mitigate the environmental impact. Can you evaluate the feasibility and impact of using renewable energy for data centers? Based on your analysis, is renewable energy a viable solution for data centers?”
We built a template in PromptHub that converts a normal prompt into a new prompt, following the Algorithm of Thoughts framework. You can access the template here and try it out right away.
If you don't have PromptHub access but want to try it out, reply to the email that gets sent when you join the waitlist and I'll share an access code with you.
Experiments: Game of 24
Setup
The researchers pulled 100 rounds of the Game of 24 from 4nums.com. For a game of 24 to be considered successful, it must use the four given numbers to get to a final answer of 24 using only basic arithmetic operations.
Baselines
Standard prompting and Chain of Thought (CoT) were evaluated under a 5-shot framework. Tree of Thoughts (ToT) was implement with a breadth of 5.
For Algorithm of Thoughts the same 5-shot framework was used as in standard prompting. These methods were sampled 100 times, with the average success rates being documented.
GPT-4 was the only model used, and the temperature was set to 0.
Algorithm of Thoughts prompt example
Here is the full version of the Algorithm of Thoughts prompt used in the game of 24 experiment.
Results
Algorithm of Thoughts destroys CoT and standard prompting. More importantly, Algorithm of Thoughts outperforms ToT with just a single LLM query, compared to ToT’s 109. Algorithm of Thoughts is working smarter, not harder.
Experiments: Mini crosswords
Setup
A collection of 20 games is drawn from goobix.com.
Baselines
The same baselines used in the previous experiment are used here. Algorithm of Thoughts is compared against standard prompting, ToT, and CoT.
Results
Once again, Algorithm of Thoughts outperforms standard prompting and CoT.
Algorithm of Thoughts slightly underperforms compared to ToT, but it uses significantly fewer queries, by a factor of 100. That’s a lot of additional API calls, tokens, and latency.
Other takeaways
I found one of the final takeaways to be one of the more interesting. The researchers tested to see if Algorithm of Thoughts could outperform the algorithm it is designed after (depth-first search, DFS). If it could, that would signal that LLMs have the potential to not just replicate, but surpass the algorithm's efficiency.
The results are in, and it’s looking pretty good for Algorithm of Thoughts.
This graph shows that Algorithm of Thoughts systematically visits fewer nodes compared to DFS. This implies that Algorithm of Thoughts is able to get to a correct final answer, in fewer steps (less node visits). Algorithm of Thoughts is able to integrate the uniform strategy of the depth-first algorithm, while also integrating intelligent recursive reasoning to limit the steps needed.
Wrapping up
We cover a lot of the latest research in prompt engineering and AI. Algorithm of Thoughts might be our favorite new method because of it’s ease of use and performance. It is able to compete right alongside ToT, while using significantly less resources.
Additionally, since Algorithm of Thoughts is only a single query, anyone can try it out. Unlike other methods that demand external resources, Algorithm of Thoughts requires nothing more than a correctly structured prompt.
