As part of our efforts to better understand the multilingual capabilities of large language models (LLMs), we present HELM Arabic, a leaderboard for transparent and reproducible evaluation of LLMs on Arabic language benchmarks. This leaderboard was produced in collaboration with Arabic.AI.

Benchmark

HELM Arabic builds on a collection of established Arabic-language evaluation tasks that are widely used in the research community ¹. It includes the following seven benchmarks:

• AlGhafa ² — an Arabic language multiple choice evaluation benchmark derived from publicly available NLP datasets
• ArabicMMLU ³ — a native Arabic language question answering benchmark using questions sourced from school exams across diverse educational levels in different countries spanning North Africa, the Levant, and the Gulf regions
• Arabic EXAMS ⁴ — the Arabic language subset of the EXAMS multilingual question answering benchmark, which consists of high school exam questions across various school subjects
• MadinahQA ³ — a question answering benchmark published by MBZUAI that tests knowledge of Arabic language and grammar
• AraTrust ⁵ — an Arab-region-specific safety evaluation dataset consisting of human-written questions including direct attacks, indirect attacks, and harmless requests with sensitive words
• ALRAGE ¹ — an Arabic language passage-based open-ended model-graded question answering benchmark that reflects retrieval-augmented generation use cases
• ArbMMLU-HT ⁶ — a translation of MMLU to Arabic by human translators published by MBZUAI

We made the following decisions in our evaluation methodology:

• We used Arabic letters for the options in the multiple choice question answering prompt template, which better reflects LLM usage within the Arabic language context.
• We used zero-shot prompting to make the tasks more challenging; all evaluated models were instruction-tuned and did not need few-shot prompting.
• We limited the evaluation to 1000 randomly sampled instances from each subset to reduce imbalances in the number of instances across subsets.
• We disabled thinking on models with an optional thinking mode, and excluded models that had a mandatory thinking mode that could not be disabled.

Models

We evaluated a several leading models on the leaderboard. These models can be grouped into three main groups:

• Closed-weights multilingual models — Arabic.AI LLM-X, Arabic.AI LLM-S, Command A, Gemini 2.5, GPT-4.1, GPT-5.1 and Mistral Large model families
• Open-weights multilingual models — DeepSeek v3.1, Falcon3, Llama 3.3, Llama 4, Qwen2.5, and Qwen3 model families
• Open-weights models that were trained or fine-tuned specifically for the Arabic language — AceGPT-v2, ALLaM, JAIS, and SILMA model families

Results

The leaderboard results show that LLMs have made significant progress in Arabic language understanding over the last few years.

• Arabic.AI LLM-X achieved the highest mean score, as well as the highest scores for AlGhafa, EXAMS, MadinahQA, AraTrust, ALRAGE, and Translated MMLU
• Qwen3 235B A22B Instruct 2507 FP8 was the best performing open-weights model, with a mean score of 0.786.
• Open-weights multilingual models performed well relative to closed-weights multilingual models. Out of the top 10 models, 4 models were multilingual open-weights models: Qwen3 235B A22B Instruct 2507 FP8, Llama 4 Maverick (17Bx128E) Instruct FP8, Qwen3-Next 80B A3B Instruct, and DeepSeek v3.1.
• Open-weights models that were trained or fine-tuned specifically for the Arabic language underperformed compared to the other two model groups. However, this may be explained by the age of these models; the most recent model family in this group, AceGPT-v2, was released in October 2024, more than a year ago.

As with all HELM leaderboards, this leaderboard provides full transparency into all LLM requests and responses, and the results are reproducible using the HELM open source framework. We hope that this leaderboard will be a valuable resource for the Arabic NLP community.

References