solatis/claude-config

Experiments on 30+ tasks across 13 task families and multiple data domains show that UDR significantly outperforms baselines and further analyses show the effectiveness of each proposed component and UDR's strong ability under various scena

Code: https://github.com/itayle/diverse-demonstrations

We use $k=16$ examples as demonstrations by default for all experiments in the paper, unless otherwise specified. Examples are sampled at uniform from the training data. We choose a set of $k$ training examples using 5 different random seed

For "demonstrations w/ $a$% accurate labels" ($0 \leq a \leq 100$), we use $k \times a/100$ correct pairs and $k \times (1-a/100)$ incorrect pairs in a random order, as described in Algorithm [\[alg:abl_accuracy\]](#alg:abl_accuracy). For "

**Code and data:** https://github.com/wzhouad/context-faithful-llm

Modern large language models (LLMs) exhibit a remarkable capacity for role-playing, enabling them to embody not only human characters but also non-human entities. This versatility allows them to simulate complex human-like interactions and 

- http://localhost:7770/electronics/headphones.html

- http://localhost:7770/electronics.html?cat=60

- A Reddit website (Postmill): {webarena_root}:9999 - A GitLab website: {webarena_root}:8023 - A map website (OpenStreetMap): http://ec2-3-131-244-37.us-east-2.compute.amazonaws.com:3000

- A GitLab website: {webarena_root}:8023 - A map website (OpenStreetMap): http://ec2-3-131-244-37.us-east-2.compute.amazonaws.com:3000 - A Wikipedia website: http://ec2-3-131-244-37.us-east-2.compute.amazonaws.com:8888/wikipedia_en_all_maxi

- A map website (OpenStreetMap): http://ec2-3-131-244-37.us-east-2.compute.amazonaws.com:3000

- A Wikipedia website: http://ec2-3-131-244-37.us-east-2.compute.amazonaws.com:8888/wikipedia_en_all_maxi_2022-05/A/User:The_other_Kiwix_guy/Landing

- A Reddit website (Postmill): {webarena_root}:9999 - A GitLab website: {webarena_root}:8023 - A map website (OpenStreetMap): http://ec2-3-131-244-37.us-east-2.compute.amazonaws.com:3000

- A GitLab website: {webarena_root}:8023 - A map website (OpenStreetMap): http://ec2-3-131-244-37.us-east-2.compute.amazonaws.com:3000 - A Wikipedia website: http://ec2-3-131-244-37.us-east-2.compute.amazonaws.com:8888/wikipedia_en_all_maxi

- A map website (OpenStreetMap): http://ec2-3-131-244-37.us-east-2.compute.amazonaws.com:3000

- A Wikipedia website: http://ec2-3-131-244-37.us-east-2.compute.amazonaws.com:8888/wikipedia_en_all_maxi_2022-05/A/User:The_other_Kiwix_guy/Landing

While Large language models (LLMs) have the capability to iteratively reflect on their own outputs, recent studies have observed their struggles with knowledge-rich problems without access to external resources. In addition to the inefficie

To analyse the effects of different LLMs quantitatively, we calculate the average pairwise semantic similarity between multiple directions for one question, then 1-similarity to obtain the diversity measurement shown below. The pretrained m

Prompting serves as the major way humans interact with Large Language Models (LLM). Commercial AI systems commonly define the role of the LLM in system prompts. For example, ChatGPT uses "You are a helpful assistant" as part of its default 

Incorporating personas into prompts has been an important approach for designing system prompts and enabling role-playing with LLMs. However, the effects of adding these personas on model performance were previously unclear. In this study, 

We utilize the Ollama framework (https://github.com/ollama/ollama-python) to conduct all evaluations described in the paper. Generally, we make use of the 4-bit quantized [10.1145/3623402] versions of the tested models to maintain consisten

| Model Name  | Hub Link                                        | Quantization Method | | ----------- | ----------------------------------------------- | ------------------- | | Gemma 2B    | https://www.ollama.com/library/gemma:2b         

| ----------- | ----------------------------------------------- | ------------------- | | Gemma 2B    | https://www.ollama.com/library/gemma:2b         | Q4                  | | Gemma 7B    | https://www.ollama.com/library/gemma:7b         

| Gemma 2B    | https://www.ollama.com/library/gemma:2b         | Q4                  |

| Gemma 7B    | https://www.ollama.com/library/gemma:7b         | Q4                  |

| Llama3 7B   | https://www.ollama.com/library/llama3:8b        | Q4                  |

| Llama3 80B  | https://www.ollama.com/library/llama3:70b       | Q4                  |

| Phi3 3.8B   | https://ollama.com/herald/phi3-128k:latest      | Q5                  |

| Mistral 7B  | https://www.ollama.com/library/mistral:7b       | Q4                  |

| GPT-4o-Mini | https://huggingface.co/                         | N/A                 |

| Qwen 2 1.5B | https://huggingface.co/Qwen/Qwen2-1.5B-Instruct | FP16                |

| Aya 35B     | https://ollama.com/library/aya:35b              | Q4                  |

Prompting methods play a crucial role in enhancing the capabilities of pre-trained large language models (LLMs). We explore how contrastive prompting (CP) significantly improves the ability of large language models to perform complex reason

All code is available at <https://github.com/yao8839836/cp>.

All prompts are available at <https://github.com/yao8839836/cp/blob/master/main.py>.

Our experimental logs are available at <https://github.com/yao8839836/cp/tree/master/log>.

The 240 examples, along with our annotations, can be accessed at the following link: <https://anonymous.4open.science/r/cp-712E/results/zero_shot_cp_gpt4_240_examples_labeled.txt>.

: The datasets are available at <https://github.com/kojima-takeshi188/zero_shot_cot/tree/main/dataset>.

: <https://docs.llama-api.com/quickstart>

: <https://prek-math-te.stanford.edu/operations/analyzing-thinking-underlying-wrong-answers>

: <https://mathmistakes.org/category/elementary-school/>

: <https://www.gutenberg.org/ebooks/38769>

: <https://www.proprofs.com/quiz-school/story.php?title=common-sense-quiz_1>

: <https://www.wikihow.com/Common-Sense-Quiz>

The answering quality of an aligned large language model (LLM) can be drastically improved if treated with proper crafting of prompts. In this paper, we propose ExpertPrompting to elicit the potential of LLMs to answer as distinguished expe

**5) Vicuna:** Trained from LLaMA 13B with user-shared conversations collected from ShareGPT (https://sharegpt.com/).

- **Authors**: Pedro Henrique Luz de Araujo, Paul Rottger, Dirk Hovy, Benjamin Roth - **Affiliations**: University of Vienna, Bocconi University - **Code**: https://github.com/peluz/principled-personas

- **Code**: https://github.com/peluz/principled-personas

- Prompt chaining is introduced at https://www.promptingguide.ai/techniques/prompt_chaining

- Stepwise prompt is similar to specifying the steps required to complete a task at https://platform.openai.com/docs/guides/prompt-engineering/tactic-specify-the-steps-required-to-complete-a-task

- Dataset source: https://www.bbc.com/news

- OpenAI models documentation: https://platform.openai.com/docs/models

- ACL Ethics Policy: https://www.aclweb.org/portal/content/acl-code-ethics

Code available: https://github.com/Xalp/ECHO

We also investigate whether rationale-augmented ensembling is robust to different templates or verbalizers, since previous work has shown that templates or verbalizers can have a significant effect on final performance [bach2022promptsource

4. Model checkpoints at https://github.com/google-research/google-research/tree/master/ul2.

5. Public API available at https://openai.com/api/.

10. https://beta.openai.com/docs/api-reference/making-requests

We use certain functions from the existing implementations of Kojima et al. [kojima2022large] (available at https://github.com/kojima-takeshi188/zero_shot_cot) and Wang et al. [wang2022self] (available via their ICLR supplement), but largel

[^1]: <https://github.com/oriyor/reasoning-on-cots>

[^4]: <https://serpapi.com/>

[^6]: <https://leaderboard.allenai.org/strategyqa>

[^7]: https://serpapi.com/organic-results

- **arXiv**: [2305.14628](https://arxiv.org/abs/2305.14628)

- **Affiliations**: University of Maryland, University of Washington, NYU Shanghai, Stanford University - **Venue**: EMNLP 2023 - **Code**: https://github.com/NoviScl/MoRE

- **Code**: https://github.com/NoviScl/MoRE

| ARCADE   | SC-Exec (strict match) | 29.8               | N/A                    |

| ARCADE   | SC-Exec (fuzzy match)  | **30.3**           | N/A                    |

Language models are increasingly being deployed for general problem solving across a wide range of tasks, but are still confined to token-level, left-to-right decision-making processes during inference. This means they can fall short in tas

**Input-output (IO) prompting** is the most common way to turn a problem input `latex $x$ ` into output `latex $y$ ` with LM: `latex $y \sim p_\theta(y | \texttt{prompt}_{{IO}}(x))$ `, where `latex $\texttt{prompt}_{IO}(x)$ ` wraps input `l

**Task Setup.** We scrape data from [4nums.com](https://www.4nums.com/game/difficulties/), which has 1,362 games that are sorted from easy to hard by human solving time, and use a subset of relatively hard games indexed 901-1,000 for testin

**Task setup.** We sample random sentences from [randomwordgenerator.com](https://randomwordgenerator.com/sentence.php) to form 100 inputs, and there is no groundtruth passage for each input constraint. As we find that GPT-4 can follow the 

**Task setup.** We scrape data from [GooBix](https://www.goobix.com/crosswords/0505/), which contains 156 games of 5x5 mini crosswords. As we observe adjacent games contain similar clues, we use 20 games with indices 1, 6, ..., 91, 96 for t

All code is available at <https://github.com/princeton-nlp/tree-of-thought-llm>.

All prompts are available at <https://github.com/princeton-nlp/tree-of-thought-llm/tree/master/src/tot/prompts>.

Trajectories are available at <https://github.com/princeton-nlp/tree-of-thought-llm/tree/master/logs>.

- In practice, we set c=100, b=5, n in {3, 5}, and |V| < 20. - For a given reasoning task, we perform this process once (for GPT-3.5 Turbo), and re-use our selection across all LLMs we evaluate. - Token pricing information: https://openai.c

- Token pricing information: https://openai.com/pricing

**Code:** https://github.com/skpig/MPSC

Q: Did Clark Gable marry more women once than Richard Burton? A: [Let's think step by step. Richard Burton was married to four different women. Clark Gable was married to five different women, one time each. Thus, Clark Gable married more w

A: [Let's think step by step. Richard Burton was married to four different women. Clark Gable was married to five different women, one time each. Thus, Clark Gable married more women than Richard Burton.]{.mark} The answer is yes. Q: Can a 

Q: Can a human heart last from NYC to Raleigh NC by Toyota Hiux?

A: [Let's think step by step. Human hearts can last up to 6 hours outside the body. The distance from NYC to Raleigh, NC is 505 miles. The top speed of a Toyota Hilux is 105 miles per hour. The time needed to travel from NYC to Raleigh, NC 

Q: Take the last letters of the words in "Gabe Ventura Woody" and concatenate them. A: [Let's think step by step. The last letter of "Gabe" is "e". The last letter of "Ventura" is "a". The last letter of "Woody" is "y". Concatenating them i

[^1]: Our code is available at <https://github.com/shizhediao/active-prompt>.

[^2]: <https://openai.com/api/>

[^3]: <https://openai.com/form/researcher-access-program>

[^4]: <https://openai.com/api/>

**Code and Data:** https://github.com/xiye17/TextualExplInContext

We have publicly released the source code of an intelligent conversational programme on GitHub, implemented based on the reponse refinement paradigm discussed in this article, while following a flexible modular design as Figure 2. The proje

Code and data at https://selfrefine.info/

We present Self-Refine: a novel approach that allows large language models to iteratively provide self-feedback and refine their own outputs. Self-Refine operates within a single LLM, requiring neither additional training data nor reinforce

print((int((int(eval(input()))+1)/2)))

(int(eval(input())) + 1)

num_input = eval(input())

We start by testing CoVe on a set of automatically generated questions using the Wikidata API (https://query.wikidata.org/). We create list questions of the form: "Who are some [Profession]s who were born in [City]?". For example, "Who are 

For the longform generation of biographies we also compare to several existing model results reported in [min2023factscore], in particular InstructGPT [ouyang2022training], ChatGPT (https://openai.com/blog/chatgpt) and PerplexityAI (www.per

Large language models (LLMs) have been increasingly used to interact with external environments (e.g., games, compilers, APIs) as goal-driven agents. However, it remains challenging for these language agents to quickly and efficiently learn

[Credential-related source omitted]

Programming LLM calls require strict instructions to produce function bodies _only_, due to the extensive dialogue training of the LLMs. A few programming examples are reported below with instructions highlighted in blue and templates. See 

Full examples are shown in <https://github.com/noahshinn024/reflexion>

Jacob Andreas. Language Models as Agent Models. In *Findings of the Association for Computational Linguistics: EMNLP 2022*, pages 5769--5779, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. URL `ht

Anthropic. Claude. `https://www.anthropic.com/product`, 2023. Accessed: 2023-04-03.

Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional AI: Harmlessness from AI Feedback, 2022. URL `https://arxiv.org/abs/

Collin Burns, Haotian Ye, Dan Klein, and Jacob Steinhardt. Discovering Latent Knowledge in Language Models Without Supervision. In *The Eleventh International Conference on Learning Representations*, 2023. URL `https://openreview.net/forum?

Howard Chen, Jacqueline He, Karthik Narasimhan, and Danqi Chen. Can Rationalization Improve Robustness? In *Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Te

Yanda Chen, Ruiqi Zhong, Narutatsu Ri, Chen Zhao, He He, Jacob Steinhardt, Zhou Yu, and Kathleen McKeown. Do Models Explain Themselves? Counterfactual Simulatability of Natural Language Explanations, July 2023. URL `http://arxiv.org/abs/230

Antonia Creswell and Murray Shanahan. Faithful Reasoning Using Large Language Models, August 2022. URL `http://arxiv.org/abs/2208.14271`. arXiv:2208.14271 \[cs\].

Ishita Dasgupta, Andrew K. Lampinen, Stephanie C. Y. Chan, Antonia Creswell, Dharshan Kumaran, James L. McClelland, and Felix Hill. Language models show human-like content effects on reasoning, July 2022. URL `http://arxiv.org/abs/2207.0705

Finale Doshi-Velez and Been Kim. Towards A Rigorous Science of Interpretable Machine Learning, March 2017. URL `http://arxiv.org/abs/1702.08608`. arXiv:1702.08608 \[cs, stat\].

Jacob Eisenstein, Daniel Andor, Bernd Bohnet, Michael Collins, and David Mimno. Honest Students from Untrusted Teachers: Learning an Interpretable Question-Answering Pipeline from a Pretrained Language Model. In *Workshop on Trustworthy and

Deep Ganguli, Amanda Askell, Nicholas Schiefer, Thomas I. Liao, Kamilė Lukošiūtė, Anna Chen, Anna Goldie, Azalia Mirhoseini, Catherine Olsson, Danny Hernandez, et al. The Capacity for Moral Self-Correction in Large Language Models, February

Leo Gao. Shapley Value Attribution in Chain of Thought. . URL `https://www.alignmentforum.org/posts/FX5JmftqL2j6K8dn4/shapley-value-attribution-in-chain-of-thought`.

Olga Golovneva, Moya Peng Chen, Spencer Poff, Martin Corredor, Luke Zettlemoyer, Maryam Fazel-Zarandi, and Asli Celikyilmaz. : A Suite of Metrics for Scoring Step-by-Step Reasoning. In *The Eleventh International Conference on Learning Repr

Peter Hase, Shiyue Zhang, Harry Xie, and Mohit Bansal. Simulatability: Can Models Generate Non-Trivial Explanations of Their Behavior in Natural Language? In *Findings of the Association for Computational Linguistics: EMNLP 2020*, pages 435

Denis Hilton. Social Attribution and Explanation. In Michael R. Waldmann, editor, *The Oxford Handbook of Causal Reasoning*, page 0. Oxford University Press, June 2017. ISBN 978-0-19-939955-0. doi: 10.1093/oxfordhb/9780199399550.013.33. URL

Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The Curious Case of Neural Text Degeneration. In *International Conference on Learning Representations*, 2020. URL `https://openreview.net/forum?id=rygGQyrFvH`.

Alon Jacovi and Yoav Goldberg. Towards Faithfully Interpretable NLP Systems: How Should We Define and Evaluate Faithfulness? In *Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics*, pages 4198--4205, Onl

Jaehun Jung, Lianhui Qin, Sean Welleck, Faeze Brahman, Chandra Bhagavatula, Ronan Le Bras, and Yejin Choi. Maieutic prompting: Logically consistent reasoning with recursive explanations. In *Proceedings of the 2022 Conference on Empirical M

Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large Language Models are Zero-Shot Reasoners. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, *Advances in Neural Information 

Tamera Lanham, Anna Chen, Ansh Radhakrishnan, Benoit Steiner, Carson Denison, Danny Hernandez, Dustin Li, Esin Durmus, Evan Hubinger, Jackson Kernion, Kamilė Lukošiūtė, Karina Nguyen, Newton Cheng, Nicholas Joseph, Nicholas Schiefer, Oliver

Aitor Lewkowycz, Anders Johan Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Venkatesh Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, Yuhuai Wu, Behnam Neyshabur, Guy Gur-Ari, and Vedant Misra. Solving Q

Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, et al. Holistic Evaluation of Language Models, November 2022. URL `http://arxiv.org/abs/2211.

Qing Lyu, Marianna Apidianaki, and Chris Callison-Burch. Towards Faithful Model Explanation in NLP: A Survey. . doi: 10.48550/ARXIV.2209.11326. URL `https://arxiv.org/abs/2209.11326`. Publisher: arXiv Version Number: 2.

Qing Lyu, Shreya Havaldar, Adam Stein, Li Zhang, Delip Rao, Eric Wong, Marianna Apidianaki, and Chris Callison-Burch. Faithful Chain-of-Thought Reasoning, 2023. URL `https://api.semanticscholar.org/CorpusID:256416127`.

Aman Madaan and Amir Yazdanbakhsh. Text and Patterns: For Effective Chain of Thought, It Takes Two to Tango, October 2022. URL `http://arxiv.org/abs/2209.07686`. arXiv:2209.07686 \[cs\].

Ian McKenzie, Alexander Lyzhov, Alicia Parrish, Ameya Prabhu, Aaron Mueller, Najoung Kim, Sam Bowman, and Ethan Perez. Inverse scaling prize: Second round winners, 2023. URL `https://irmckenzie.co.uk/round2`.

Hugo Mercier and Dan Sperber. Why do humans reason? Arguments for an argumentative theory. *Behavioral and Brain Sciences*, 34 (2): 57--74, April 2011. ISSN 1469-1825, 0140-525X. doi: 10.1017/S0140525X10000968. URL `https://www.cambridge.or

Sewon Min, Victor Zhong, Luke Zettlemoyer, and Hannaneh Hajishirzi. Multi-hop Reading Comprehension through Question Decomposition and Rescoring. In *Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics*, 

Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. Rethinking the Role of Demonstrations: What Makes In-Context Learning Work? In *Proceedings of the 2022 Conference on Empirical Method

Maxwell Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieber, David Dohan, Aitor Lewkowycz, Maarten Bosma, David Luan, et al. Show Your Work: Scratchpads for Intermediate Computation with Language Models

OpenAI. index for researchers. `https://platform.openai.com/docs/model-index-for-researchers`, 2023. Accessed: 2023-04-03.

Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Gray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Wel

Lorenzo Pacchiardi, Alex J. Chan, Sören Mindermann, Ilan Moscovitz, Alexa Y. Pan, Yarin Gal, Owain Evans, and Jan Brauner. How to Catch an AI Liar: Lie Detection in Black-Box LLMs by Asking Unrelated Questions, September 2023. URL `http://a

Alicia Parrish, Angelica Chen, Nikita Nangia, Vishakh Padmakumar, Jason Phang, Jana Thompson, Phu Mon Htut, and Samuel Bowman. : A hand-built bias benchmark for question answering. In *Findings of the Association for Computational Linguisti

Ethan Perez, Patrick Lewis, Wen-tau Yih, Kyunghyun Cho, and Douwe Kiela. Unsupervised Question Decomposition for Question Answering. In *Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)*, pages 

Ethan Perez, Sam Ringer, Kamilė Lukošiūtė, Karina Nguyen, Edwin Chen, Scott Heiner, Craig Pettit, Catherine Olsson, Sandipan Kundu, Saurav Kadavath, et al. Discovering Language Model Behaviors with Model-Written Evaluations, December 2022. 

Ansh Radhakrishnan, Karina Nguyen, Anna Chen, Carol Chen, Carson Denison, Danny Hernandez, Esin Durmus, Evan Hubinger, Jackson Kernion, Kamilė Lukošiūtė, Newton Cheng, Nicholas Joseph, Nicholas Schiefer, Oliver Rausch, Sam McCandlish, Sheer

Justin Reppert, Ben Rachbach, Charlie George, Luke Stebbing, Jungwon Byun, Maggie Appleton, and Andreas Stuhlmüller. Iterated Decomposition: Improving Science Q&A by Supervising Reasoning Processes, January 2023. URL `http://arxiv.org/abs/2

Cynthia Rudin. Stop Explaining Black Box Machine Learning Models for High Stakes Decisions and Use Interpretable Models Instead, September 2019. URL `http://arxiv.org/abs/1811.10154`. arXiv:1811.10154 \[cs, stat\].

William Saunders, Catherine Yeh, Jeff Wu, Steven Bills, Long Ouyang, Jonathan Ward, and Jan Leike. Self-critiquing models for assisting human evaluators, June 2022. URL `http://arxiv.org/abs/2206.05802`. arXiv:2206.05802 \[cs\].

Omar Shaikh, Hongxin Zhang, William Held, Michael Bernstein, and Diyi Yang. On Second Thought, Let's Not Think Step by Step! Bias and Toxicity in Zero-Shot Reasoning, December 2022. URL `http://arxiv.org/abs/2212.08061`. arXiv:2212.08061 \[

Mrinank Sharma, Meg Tong, Tomasz Korbak, David Duvenaud, Amanda Askell, Samuel R. Bowman, Newton Cheng, Esin Durmus, Zac Hatfield-Dodds, Scott R. Johnston, Shauna Kravec, Timothy Maxwell, Sam McCandlish, Kamal Ndousse, Oliver Rausch, Nichol

Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed Chi, Nathanael Schärli, and Denny Zhou. Large Language Models Can Be Easily Distracted by Irrelevant Context, February 2023. URL `http://arxiv.org/abs/2302.00093`. arXiv

Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R. Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, et al. Beyond the Imitation Game: Quantifying and extrapolating the capabilitie

Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V. Le, Ed H. Chi, Denny Zhou, et al. Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them, October 202

Oyvind Tafjord, Bhavana Dalvi Mishra, and Peter Clark. Entailer: Answering Questions with Faithful and Truthful Chains of Reasoning. In *Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing*, pages 2078--20

Jonathan Uesato, Nate Kushman, Ramana Kumar, Francis Song, Noah Siegel, Lisa Wang, Antonia Creswell, Geoffrey Irving, and Irina Higgins. Solving math word problems with process- and outcome-based feedback, November 2022. URL `http://arxiv.o

Boshi Wang, Sewon Min, Xiang Deng, Jiaming Shen, You Wu, Luke Zettlemoyer, and Huan Sun. Towards Understanding Chain-of-Thought Prompting: An Empirical Study of What Matters. In *ICLR 2023 Workshop on Mathematical and Empirical Understandin

Albert Webson and Ellie Pavlick. Do Prompt-Based Models Really Understand the Meaning of Their Prompts? In *Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Te

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi, Quoc V Le, and Denny Zhou. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, an

Xi Ye and Greg Durrett. The Unreliability of Explanations in Few-shot Prompting for Textual Reasoning. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, *Advances in Neural Information Processing Systems*, 2022. 

Xi Ye, Srinivasan Iyer, Asli Celikyilmaz, Ves Stoyanov, Greg Durrett, and Ramakanth Pasunuru. Complementary Explanations for Effective In-Context Learning, November 2022. URL `http://arxiv.org/abs/2211.13892`. arXiv:2211.13892 \[cs\].

Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. aR: Bootstrapping Reasoning With Reasoning. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, *Advances in Neural Information Processing Systems*, 2022. URL `

Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoc V Le, and Ed H. Chi. Least-to-Most Prompting Enables Complex Reasoning in Large Language Models. In *The Eleve

[^2]: See <https://github.com/milesaturpin/cot-unfaithfulness> for data and code.

[^7]: Accessible at <https://huggingface.co/datasets/bigbench>.

SVAMP @patel-etal-2021-nlp <https://github.com/arkilpatel/SVAMP> MIT License

GSM8k @cobbe2021gsm8k <https://github.com/openai/grade-school-math> MIT License

sNLR @liang2022holistic <https://github.com/stanford-crfm/helm> Apache License

Moral Norm @emelin-etal-2021-moral <https://github.com/demelin/moral_stories> MIT License

HuggingFace @wolf-etal-2020-transformers <https://github.com/huggingface/transformers> Apache License

[^3]: <https://github.com/debjitpaul/refiner>

Our experiments utilize GPT-3.5 and GPT-4, which are accessible via the public API at https://platform.openai.com/docs/models, as well as Llama-2, an open-source model. To facilitate reproducibility, we detail the specific kernels used, e.g

Code available at: https://github.com/yukunZhao/Self-DETECTION

Self-reflection for Large Language Models (LLMs) has gained significant attention. Existing approaches involve models iterating and improving their previous responses based on LLMs' internal reflection ability or external feedback. However,

**arXiv:** [2509.06770](https://arxiv.org/abs/2509.06770)

Our interest is in the design of software systems involving a human-expert interacting--using natural language--with a large language model (LLM) on data analysis tasks. For complex problems, it is possible that LLMs can harness human exper

**Footnote:** Our code, data, and models are released at <https://github.com/mukhal/thinkprm>.

Performing inference on large volumes of samples with large language models (LLMs) can be computationally and financially costly in industry and real-world use. We propose batch prompting, a simple yet effective prompting approach that enab

[^2]: <https://huggingface.co/docs/text-generation-inference>

[^3]: <https://huggingface.co/blog/os-llms>

In this paper, we introduce Concise Chain-of-Thought (CCoT) prompting. We compared standard CoT and CCoT prompts to see how conciseness impacts response length and correct-answer accuracy. We evaluated this using GPT-3.5 and GPT-4 with a mu

Chain-of-Thought (CoT) reasoning has emerged as a promising approach for enhancing the performance of large language models (LLMs) on complex reasoning tasks. Recently, a series of studies attempt to explain the mechanisms underlying CoT, a

- Please see the NeurIPS code and data submission guidelines (<https://nips.cc/public/guides/CodeSubmissionPolicy>) for more details.

- The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (<https://nips.cc/public/guides/CodeSubmissionPolicy>) for more details.

42. Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics <https://neurips.cc/public/EthicsGuidelines>?

[^4]: https://huggingface.co/dslim/bert-base-NER

Large Reasoning Models (LRMs) represent a breakthrough in AI problem-solving capabilities, but their effectiveness in interactive environments can be limited. This paper introduces and analyzes **overthinking** in LRMs---a phenomenon where 

Large Language Models (LLMs) have demonstrated remarkable performance in solving complex reasoning tasks through mechanisms like Chain-of-Thought (CoT) prompting, which emphasizes verbose, step-by-step reasoning. However, humans typically e

Reasoning is critical for large language models (LLMs) to excel in a wide range of tasks. While methods like Chain-of-Thought (CoT) reasoning and enhance LLM performance by decomposing problems into intermediate steps, they also incur signi

[^1]: Also available at <https://www.gitlink.org.cn/txhan/TALE>

Existing knowledge-grounded dialogue systems typically use finetuned versions of a pretrained language model (LM) and large-scale knowledge bases. These models typically fail to generalize on topics outside of the knowledge base, and requir

Large language models (LLMs) can perform complex reasoning by generating intermediate reasoning steps. Providing these steps for prompting demonstrations is called chain-of-thought (CoT) prompting. CoT prompting has two major paradigms. One

We demonstrate the utility of successive prompting using a few-shot variant of the DROP dataset [dua2019drop], selecting 300 examples for training (either fine-tuning or in-context example selection). These 300 examples are manually annotat

[^2]: <https://en.wiktionary.org/wiki/Wiktionary:Frequency_lists/PG/2006/04/1-10000>

[^3]: <https://en.wiktionary.org/wiki/Wiktionary:Frequency_lists/PG/2006/04/1-10000>

- **arXiv**: [2305.14564](https://arxiv.org/abs/2305.14564)

- **Authors**: Simeng Sun, Yang Liu, Shuohang Wang, Chenguang Zhu, Mohit Iyyer - **Affiliations**: University of Massachusetts Amherst, Microsoft Research - **Code**: https://github.com/SimengSun/pearl

- **Code**: https://github.com/SimengSun/pearl

Large language models (LLMs) have recently been shown to deliver impressive performance in various NLP tasks. To tackle multi-step reasoning tasks, few-shot chain-of-thought (CoT) prompting includes a few manually crafted step-by-step reaso

- **Affiliations**: University of Washington, MosaicML, Meta AI Research, Allen Institute for AI - **Published**: EMNLP 2023 - **Code**: https://github.com/ofirpress/self-ask

- **Code**: https://github.com/ofirpress/self-ask

Language models are increasingly being deployed for general problem solving across a wide range of tasks, but are still confined to token-level, left-to-right decision-making processes during inference. This means they can fall short in tas

**Input-output (IO) prompting** is the most common way to turn a problem input `latex $x$ ` into output `latex $y$ ` with LM: `latex $y \sim p_\theta(y | \texttt{prompt}_{{IO}}(x))$ `, where `latex $\texttt{prompt}_{IO}(x)$ ` wraps input `l

**Task Setup.** We scrape data from [4nums.com](https://www.4nums.com/game/difficulties/), which has 1,362 games that are sorted from easy to hard by human solving time, and use a subset of relatively hard games indexed 901-1,000 for testin

**Task setup.** We sample random sentences from [randomwordgenerator.com](https://randomwordgenerator.com/sentence.php) to form 100 inputs, and there is no groundtruth passage for each input constraint. As we find that GPT-4 can follow the 

**Task setup.** We scrape data from [GooBix](https://www.goobix.com/crosswords/0505/), which contains 156 games of 5x5 mini crosswords. As we observe adjacent games contain similar clues, we use 20 games with indices 1, 6, ..., 91, 96 for t

All code is available at <https://github.com/princeton-nlp/tree-of-thought-llm>.

All prompts are available at <https://github.com/princeton-nlp/tree-of-thought-llm/tree/master/src/tot/prompts>.

Trajectories are available at <https://github.com/princeton-nlp/tree-of-thought-llm/tree/master/logs>.

- **arXiv**: [2307.15337](https://arxiv.org/abs/2307.15337)

- **Authors**: Xuefei Ning, Zinan Lin, Zixuan Zhou, Zifu Wang, Huazhong Yang, Yu Wang - **Affiliations**: Tsinghua University, Microsoft Research, KU Leuven, Infinigence-AI - **Website**: https://sites.google.com/view/sot-llm

- **Affiliations**: Tsinghua University, Microsoft Research, KU Leuven, Infinigence-AI - **Website**: https://sites.google.com/view/sot-llm - **Code**: https://github.com/imagination-research/sot

- **Website**: https://sites.google.com/view/sot-llm

- **Code**: https://github.com/imagination-research/sot

Despite demonstrating emergent reasoning abilities, Large Language Models (LLMS) often lose track of complex, multi-step reasoning. Existing studies show that providing guidance via decomposing the original question into multiple subproblem

[^1]: Project: <https://allenai.github.io/adaptllm>

[^4]: <https://www.minecraft.net>

[^7]: <https://huggingface.co/meta-llama/Llama-2-70b-hf>

[^8]: <https://huggingface.co/OpenLemur/lemur-70b-chat-v1>

[^2]: <https://en.wiktionary.org/wiki/Wiktionary:Frequency_lists/PG/2006/04/1-10000>

[^3]: <https://en.wiktionary.org/wiki/Wiktionary:Frequency_lists/PG/2006/04/1-10000>

We investigate the challenge of task planning for multi-task embodied agents in open-world environments.[^2] Two main difficulties are identified: 1) executing plans in an open-world environment (e.g., Minecraft) necessitates accurate and m

- **arXiv**: [2409.17539](https://arxiv.org/abs/2409.17539)

- **Authors**: Tongxuan Liu, Wenjiang Xu, Weizhe Huang, Yuting Zeng, Jiaxing Wang, Xingyu Wang, Hailong Yang, Jing Li - **Affiliations**: University of Science and Technology of China, Beihang University, Institute of Automation Chinese Aca

- **Code**: https://github.com/HEA1OR/lot

Project Page: https://github.com/iiis-ai/cumulative-reasoning

Code and prompts for our evaluation can found here: https://github.com/SALT-NLP/chain-of-thought-bias

Despite the strength of ChatGPT, there are still several limitations (as stated in https://openai.com/blog/chatgpt/), such as failing to ask clarification questions to ambiguous user queries or refuse problematic user requests. These kinds 

Similar to other studies on prompting LLMs for dialogue applications [acl23-prompt-ood; acl23-prompt-mixed; acl23-robust], the evaluation results are likely to be sensitive to the choice of prompts. Besides, it is also likely that the desig

- Abg-CoQA (https://github.com/MeiqiGuo/AKBC2021-Abg-CoQA/tree/main/abg-coqa) is constructed based on the CoQA dataset [coqa] by truncating a partial conversation from the full conversation and selecting ambiguous questions.

- PACIFIC (https://github.com/dengyang17/PACIFIC/tree/main/data/pacific) is constructed based on the TAT-QA dataset [tat-qa], an question answering dataset in the financial domain, whose contexts contain a hybrid of tables and texts. [pacif

- OTTers is a next-turn target-oriented dialogue dataset, which requires the agent proactively generate a transition utterance to approach the designated target. We adopt the processed version (https://github.com/yyyyyyzt/topkgchat) by [col

- CraigslistBargain was created in a negotiation setting where two crowdsourced workers play the roles of the buyer and the seller to bargain the price of an item. We adopt the processed version (https://github.com/rishabhjoshi/DialoGraph_I

[^3]: Our code implementation will be released at [https://github.com/DAMO-NLP-SG/contrastive-cot](https://github.com/DAMO-NLP-SG/contrastive-cot)

[^5]: <https://platform.openai.com/docs/models>

[^6]: <https://platform.openai.com/docs/api-reference>

[^7]: <https://spacy.io/models/en#en_core_web_trf>

Large Language Models (LLMs) have recently showcased remarkable generalizability in various domains. Despite their extensive knowledge, LLMs still face challenges in efficiently utilizing encoded knowledge to develop accurate and logical re

In Large Language Models (LLMs), there have been consistent advancements in task-specific performance, largely influenced by effective prompt design. Recent advancements in prompting have enhanced reasoning in logic-intensive tasks for LLMs

While the recent Chain-of-Thought (CoT) technique enhances the reasoning ability of large language models (LLMs) with the theory of mind, it might still struggle in handling logical reasoning that relies much on symbolic expressions and rig

Large Language Models (LLMs) have achieved remarkable success in reasoning tasks with the development of prompting methods. However, existing prompting approaches cannot reuse insights of solving similar problems and suffer from accumulated

[Credential-related source omitted]

[Credential-related source omitted]

[Credential-related source omitted]

We follow the task setup proposed by Yao et. al. [yao2023tree]. We use a dataset consisting of 100 test instances. Each test instance contains 4 sentences which are randomly sampled from this website (https://randomwordgenerator.com/sentenc

Large Language Models prompting, such as using in-context demonstrations, is a mainstream technique for invoking LLMs to perform high-performance and solid complex reasoning (e.g., mathematical reasoning, commonsense reasoning), and has the

[^1]: Our code can be found at <https://github.com/Zayne-sprague/To-CoT-or-not-to-CoT>.

[^2]: <https://mistral.ai/news/mistral-large-2407/>

[^4]: <https://huggingface.co/collections/TAUR-Lab/cot-analysis-project-66bbb9e5e0156e65059895f5>

**Code:** https://github.com/Xalp/Tab-CoT

We found that in the official "parse unstructured data" demo provided by OpenAI (https://beta.openai.com/playground/p/default-parse-data), a table header is provided as part of the prompt, which is as follows: "`|Fruit|Color|Flavor|`". With

We consider two state-of-the-art large language models under the GPT-3 family [Brown et al., 2020] in our experiments, namely "code-davinci-002" and "text-davinci-002", whose APIs are made available by OpenAI (https://openai.com/api/). For 

Dataset is available at https://github.com/google-research-datasets/GSM-IC.

exec(program)

**Notes:** - Assuming each addition is correct with 90% chance, after 50 additions, the likelihood of a correct output is less than 1%. - OpenAI Codex API: https://openai.com/blog/openai-codex/

- Assuming each addition is correct with 90% chance, after 50 additions, the likelihood of a correct output is less than 1%. - OpenAI Codex API: https://openai.com/blog/openai-codex/ - XGen: https://blog.salesforceairesearch.com/xgen/

- OpenAI Codex API: https://openai.com/blog/openai-codex/

- XGen: https://blog.salesforceairesearch.com/xgen/

- SymPy library: https://www.sympy.org/en/index.html

This paper introduces 26 guiding principles designed to streamline the process of querying and prompting large language models. Our goal is to simplify the underlying concepts of formulating questions for various scales of large language mo

In the appendix, we provide additional analyses and more extensive discussions about SWE-agent, agent-computer interface (ACI) design, and model performance on various evaluation benchmarks. We also provide several thorough case studies of 

However, if an agent's action runs silently and produces no output (e.g. `rm abc.py`, `touch abc.py`), we found that simple showing no output along with a prompt can be confusing for agents to interpret, and it will often run additional, un

First is the security risks that come with executing LM-generated code on device. When evaluating on software engineering tasks that feature an incredibly diverse number of issue descriptions, running code generations on a personal computin

Project Page: https://github.com/meta-prompting/meta-prompting