Eliciting Better Multilingual Structured Reasoning
from LLMs through Code

The ability to perform complex reasoning tasks is fundamental to human intelligence, where multiple steps of thought are required. Complex reasoning remains an open-problem for large language models (LLMs), despite some recent progress. Prior works consider complex reasoning tasks specified only in English. Such an English-centric perspective provides a limited assessment of the underlying reasoning capabilities of LLMs, given any specific language is largely a surface-form representation. This motivates our first inquiry into the multilingual complex reasoning capabilities of LLMs.

We introduce the xSTREET reasoning and explanation dataset (as shown in Figure 2). xSTREET covers 4 tasks, and extends the English STREET benchmark Ribeiro et al. (2022) to 5 additional diverse languages, inheriting the source’s expert annotations and structured graphs for reasoning steps (7.8 average steps/answer). The tasks cover arithmetic, logic and science commonsense problems. We perform machine translation for the training and development data splits, and also perform human post-editing to the test sets, to ensure a high quality multilingual benchmark. We use xSTREET to evaluate several LLMs, identifying the multilingual setting as significantly challenging.

To remedy the non-English reasoning gap, we turn to the widely accepted hypothesis that LLMs trained on code are better at reasoning than those trained only on text. This code and reasoning hypothesis has been empirically corroborated by several papers Suzgun et al. (2022); Liang et al. (2023); Hendy et al. (2023). Our work takes a further step in investigating the extent to which this hypothesis holds for non-English tasks. We proceed with the insight that code can be leveraged as a structured framework to represent the underlying reasoning steps, regardless of the surface-form language of the task. We thus propose two techniques to elicit better multilingual complex reasoning from LLMs (as shown in Figure 1): at training time through a lightweight fine-tuning recipe on code, and at inference time using a novel code prompt format.

In the LLM literature, many capabilities have been characterized as ‘emergent’ with model scale Wei et al. (2022a); Patel et al. (2022). Recent work on complex reasoning has thus focused on huge (175B+) and closed-source models. In our work, we instead aim to boost performance on far smaller open-source LLMs (7B). To make our findings reproducible, we release our benchmark. Our contributions are:

1. 1.

   We collect and release the first dataset for multilingual structured reasoning, xSTREET, covering 6 diverse languages and 4 tasks (5.5K entries total).

2. 2.

   At train time: we enhance reasoning capabilities of off-the-shelf LLMs by further training on program code data where code is interleaved with non-English comments. To this end, we augment a source code corpus through translating code comments and apply low-rank parameter-efficient fine-tuning (LoRA  Hu et al. (2021)) to BLOOMZ Muennighoff et al. (2022). Our method is effective yet lightweight, while preserving general-purpose LM capabilities.

3. 3.

   At inference time: we design a code-like prompting format that mimics the structure of the reasoning tasks by interweaving function calls and multilingual text. We show this format outperforms several other prompt formats used.

4. 4.

   We evaluate multiple LLMs (BLOOMZ, GPT-3, Falcon-40b-instruct) on our benchmark, and show improved performance — even for top-performing models — across structured reasoning tasks in different languages. As our inference and training-time techniques are orthogonal, we show that they can be used in tandem to achieve the best performance.

5. 5.

   We perform qualitative and quantitative analysis to understand the roles of both the program code, and the code comments. Our findings taken together suggest that code elicits better multilingual structured reasoning by improving LLM’s adherence to the reasoning format.

We adopt the scope of complex reasoning from Ribeiro et al. (2022), and use complex reasoning and structured reasoning interchangeably. The goal is to study the reasoning process itself, and how an LLM can structure its output in steps to improve the final performance. The tasks are selected so that the knowledge to answer a question is contained with the input question and context. For ease of evaluation, the answers are multiple-choice selections, or numbers for arithmetic reasoning. Further details are given in §2.4.

We create the xSTREET dataset by translating STREET into 5 languages: Arabic (ar), Spanish (es), Russian (ru), Chinese (zh), and Japanese (ja). These languages have linguistic and script diversity; furthermore, they are the languages used in many online programming help websites.

To create the xSTREET test split, we hire expert human translators for all 5 languages through an internal team (detailed in §Data Statement). Translators are tasked with post-editing the machine translation of one sentence at a time; for context, they can refer to the entire STREET entry the sentence comes from. After receiving the translations, we re-use the reasoning graph edges, and replace English nodes with the translations to create xSTREET. This process is shown in Figure 2. We therefore extend the 914 English entries in STREET to 5484 examples in xSTREET (914 * 6 languages).

To create the xSTREET train and development splits, we use machine translation.⁵⁵5We used an online translation API (anonymized here). We then asked native speakers to evaluate the quality of 10 random sampled translations of each language. Annotators gave feedback that, despite some errors, the translations were of reasonable enough quality to use for training purposes.

Dataset statistics for the xSTREET test benchmark are given in Table 1.

Taking the idea of using code for reasoning, and comments for multilinguality a step further, we address the question: can multilingual code serve as indirect supervision for multilingual reasoning? In other words, we investigate whether the code & reasoning hypothesis holds multilingually. We therefore propose a lightweight fine-tuning recipe, which consists of creating a multilingually commented code dataset, then fine-tuning on it, which serves as indirect supervision for downstream reasoning tasks.

We hypothesize that structure, when applied to reasoning problems formulated in different languages, can abstract away some of the language-specific details, better surfacing the reasoning steps needed for a model. We thus propose the Sim (Select-and-infer multilingual comments) code prompts for complex reasoning tasks.

Sim code prompts utilize several functions. We do not provide the API definitions, instead, we expect the model to learn to use them from the in-context examples. The functions are:

• •

  select_facts(facts)

• •

  infer_new_fact(selected)

• •

  is_solved(fact, question)

• •

  make_choice(fact, choices)⁸⁸8This function is not used for non-MC tasks, i.e. GSM8k.

• •

  facts.append(fact)

select_facts and infer_new_fact are loosely inspired by Selection-Inference Creswell et al. (2023). A key difference, though, is that we use a single prompt, instead of iterative prompts. We therefore include is_solved(fact, question) as a signal for the LLM to stop generation.

Each function is annotated with its return value in an inline code comment. This is inspired by prior work Zhang et al. (2023). infer_new_fact has a string return value, i.e., the text of the new fact. We experiment with two versions of the return value of select_facts. The first, termed Sim-indexed, uses variables facts[i] to reference the facts array (similar to the indices used in linearized format). The second, termed Sim-text, directly uses each fact’s text, dereferenced from facts[i]. We find that Sim-text works best for smaller models, while Sim-indexed does for larger ones, and hence apply this going forward.

We write a rule-based Python script that converts existing structured graph annotations to Sim code prompts. Sim prompts express the exact same information as the linearized format. This property is unlike code prompts for prior work, wherein the conversion is done through in-context learning with an LLM, which can introduce errors as discussed in §2.2. The different prompting formats for LLMs are shown in Figure 3.

We report results on the xSTREET benchmark. We use the answer accuracy metric, adapting evaluation from Ribeiro et al. (2022).¹³¹³13While STREET also measure graph similarity between linearized output and reference graphs, we did not implement them for Sim prompts. This is because for the small LLMs (7B), even the linearized format had near 0 graph similarity.

Given the extensive nature of the xSTREET benchmark and our model experimentation, we highlight our findings iteratively. We first consider only BLOOMZ and BLOOMZ-Tcc, with a particular focus on ARC, where our methods are the most impactful. We then consider GPT-3 and Falcon.

The full results are given in Appendix D. Here we provide numbers for all tasks, languages, models, and prompt formats (this also includes the direct prompting format).

To further understand wherein our techniques help, or fail to help, model reasoning, we perform some manual analysis. For brevity, we focus on 2 languages, en and ar, and 2 tasks, ARC and GSM8K. We first perform error analysis on BLOOMZ, then perform a case study for each task.

We further perform 3 additional experiments, which are detailed in Appendix §E. To highlight one interesting finding, we show that training on diverse code comments, such as from the multilingual Tcc, boosts xSTREET performance in all languages including English.

We introduced xSTREET, a multilingual structured reasoning benchmark which covers 5 diverse languages , spans science commonsense, arithmetic and logical reasoning tasks, and includes high-quality intermediate reasoning steps. We found that current multilingual LLMs underperform in the non-English setting, then proposed two methods to remedy this, based on the popular hypothesis that LLMs trained on code are better reasoners. At training, we propose translating the comments of a source code dataset, to use as indirect supervision data for parameter-efficient fine-tuning. During inference, we leverage code structure to represent reasoning graphs. We perform extensive experimentation, and both of our methods better elicit underlying reasoning abilities of LLMs.

Our work brings together two areas of challenge for LLMs — multilinguality, and complex reasoning. In particular, our fine-tuning recipe shows that the code & reasoning hypothesis can apply multilingually. We suspect that improvements can be amplified if multilingual comments are included at the pre-training, instead of the fine-tuning stage. We hope our findings underscore the key role that code should play in the development of LLMs with better reasoning capabilities across languages.

One limitation is that we were unable to apply our fine-tuning recipe to the stronger LLMs. “Stronger” refers to two characteristics. First and unavoidably, we can only apply the method to weaker open-source models, as closed-source models are proprietary; nevertheless, we explored them with our inference-time Sim prompts approach, and this worked well. Second, we only were able to fine-tune a 7B parameter model due to our resource constraints, so it is to-be-determined the effectiveness of the recipe on 70B+ models.

Between the submission and publication of this work (February to August 2024), LLM development has been brisk, and several recently released $\sim$7B LLMs have shown decent performance on arithmetic reasoning. In our work, we were limited to BLOOMZ-7B, which we saw was poor at math. For followup work, therefore, we are excited to try our finetuning approach on Tcc while using these newer LLMs as base models.

Another limitation is for the xSTREET benchmark, we performed human translation on only the test set of the source STREET dataset. As we used machine translation for the train set, but also drew few-shot exemplars from these, the lower exemplar quality worsens performance compared to a gold standard exemplars. We also fine-tuned on machine-translated TCC.

While we tried to be inclusive with the languages chosen, studying 6 languages from different families and using different scripts, we acknowledge that more community effort will need to go into expanding the study of multilingual complex reasoning to lower-resource languages. We further acknowledge the limits of the translation of English reasoning tasks and intermediate steps alone, in that reasoning processes may differ for speakers of different languages. So too may a multilingual LLM respond inconsistently to queries posted in different languages Li et al. (2024), which warrants future studies into how this holds for the reasoning tasks studied in this work.

Finally, in this work, we considered only the final answer accuracy for the tasks. The original STREET tasks from Ribeiro et al. (2022) included various graph similarity metrics used to consider the intermediate reasoning steps as well – a definite strength of their structured reasoning approach vs. unstructured approaches such as CoT. We did not do this consideration due to the difficulty of reimplementing the graph similarity metric calculation for the different languages, and leave this to follow up work. Furthermore, we note that the 7B LLM we used had overall poor graph similarity (near 0 for all metrics) using the original STREET evaluation scripts and dataset.

We would like to thank Danilo Neves Ribeiro for his guidance on working with the STREET benchmark, and insightful conversations on how tackle our multilingual extension of complex reasoning. We thank several colleagues for providing annotations: Etsuko Ishii, Igor Shalyminov, Yuwei Zhang. We thank these people for discussion and feedback: Salvatore Romeo, Yi Zhang, Sam Davidson, and Sailik Sengupta.