FilBench: Can LLMs Understand and Generate Filipino? A Deep Dive into Tagalog and Cebuano

TL;DR

• FilBench is a comprehensive evaluation suite for Tagalog, Filipino and Cebuano, built on top of Lighteval, testing 20+ state-of-the-art LLMs across four categories: Cultural Knowledge, Classical NLP, Reading Comprehension, and Generation.
• SEA-specific open-weight LLMs (e.g., SEA-LION, SeaLLM) often achieve the highest FilBench scores for these languages, but GPT-4o remains a stronger closed-source baseline in many cases.
• Open-weight models tend to be cheaper to run than commercial models, enabling more accessible Filipino language tasks; fine-tuning with SEA-specific data yields 2–3% gains on FilBench.
• Generation tasks remain the weakest area across models, with failures including instruction translation misalignment, excessive verbosity, or hallucinating another language instead of Tagalog or Cebuano.
• FilBench is available as community tasks in the official Lighteval repository and on the FilBench leaderboard hosted by HuggingFace, with Llama 4 Maverick proposed as a practical alternative to GPT-4o for Filipino tasks.

Context and background

Filipinos are among the most active ChatGPT users globally, ranking fourth in ChatGPT traffic, yet systematic, language-specific evaluation for Philippine languages has been limited. Anecdotal evidence—such as screenshots of ChatGPT replying in Filipino—has not provided a rigorous assessment of capability across Tagalog and Cebuano. To address this gap, the FilBench project was developed as a comprehensive evaluation suite designed to measure LLM fluency, linguistic performance, translation accuracy, and cultural knowledge for Tagalog, Filipino (the standardized form of Tagalog), and Cebuano. FilBench evaluates 20+ state-of-the-art LLMs across four major categories, each comprising multiple tasks. The categories are Cultural Knowledge, Classical NLP, Reading Comprehension, and Generation, and the tasks are designed to reflect historical and current priorities in Philippine-language NLP research, spanning from sentiment analysis to translation and beyond. A single FilBench Score is produced by computing a weighted average across categories, with weights based on the number of examples in each category. The benchmark is built on top of Lighteval, an all-in-one framework for LLM evaluation, which also supports language-specific evaluation through translation pairs (for example, English to Tagalog or Cebuano for common terms such as yes (oo), no (hindi), and true (totoo)). FilBench is published as a set of community tasks in the official Lighteval repository, and the results are made publicly visible through the FilBench leaderboard on HuggingFace. This work also acknowledges support from Cohere Labs (credits for Aya model series) and Together AI for computational credits, along with contributions from the Hugging Face team and the OpenEvals community.

What’s new

FilBench introduces a structured, language-focused evaluation suite for Philippine languages, curated to reflect research trends and practical usage. The four major categories—Cultural Knowledge, Classical NLP, Reading Comprehension, and Generation—are subdivided into 12 tasks, offering a comprehensive picture of model strengths and gaps. A key design choice is to emphasize non-translated content in most tasks to preserve faithfulness to how Philippine languages are used in practice. Key implementation details include:

• Four categories with 12 tasks in total, each delivering an aggregated metric.
• A single FilBench Score computed as a weighted average across categories.
• Language-specific evaluation using translation pairs (English → Tagalog or Cebuano) for common terms to anchor evaluation templates.
• FilBench is now available as a set of community tasks in the official Lighteval repository and is accessible via the FilBench leaderboard on HuggingFace.
• The evaluation framework is designed to be cost- and compute-efficient, highlighting models on the Pareto frontier of efficiency, with open-weight LLMs often offering favorable performance per parameter. In practice, SEA-specific LLMs such as SEA-LION and SeaLLM frequently achieve the highest FilBench scores for Tagalog, Filipino, and Cebuano among models of comparable sizes. However, GPT-4o remains a strong baseline, and in some cases outperforms these open-weight SEA-specific models. The findings also point to the value of continuing to curate Filipino and SEA-specific training data for fine-tuning, which can yield additional gains of about 2–3% on FilBench. A notable insight concerns generation: across the four categories, models tend to struggle more with generation tasks, including translating instructions accurately, avoiding overly verbose outputs, and avoiding hallucinating another language instead of Tagalog or Cebuano. The results underscore that generation remains the hardest area for Filipino-language LLMs and calls for targeted improvement in this dimension. FilBench also highlights a practical takeaway for the Philippines: given limited internet infrastructure and income levels, there is a need for accessible, cost- and compute-efficient LLMs. The analysis identifies open-weight models that offer competitive performance relative to their size, supporting more affordable deployment in local contexts. For developers seeking alternatives to GPT-4o for Filipino-language tasks, the report mentions Llama 4 Maverick as a viable option to explore. For communities and organizations interested in benchmarking or improving Filipino NLP, FilBench provides a clear, reproducible framework. The FilBench leaderboard and the HuggingFace space make it straightforward to compare models, track progress, and guide research and deployment decisions.

Why it matters (impact for developers/enterprises)

The FilBench project matters because it translates general LLM capabilities into actionable insights for Philippine languages. In regions with limited bandwidth and lower incomes, cost- and compute-efficient models enable broader access to LLM-powered tools for education, government, business, and everyday tasks. Several findings matter for teams planning to deploy Filipino-language AI solutions:

• SEA-specific open-weight models often offer the best balance of performance and efficiency for Tagalog, Filipino, and Cebuano, making them attractive starting points for deployment where compute budgets are tight.
• Closed-source models like GPT-4o still set a high performance bar, so organizations may choose them for premium, mission-critical tasks where maximum accuracy is essential.
• Fine-tuning with SEA-specific instruction data can yield measurable gains (2–3%), justifying ongoing data collection and annotation efforts for regional languages.
• Generation quality remains a challenge; teams should expect to invest in instruction-following alignment, concise outputs, and cross-language consistency to improve end-user experiences.
• The FilBench framework provides a practical benchmarking path for developers to assess models before integration, and the openness of the task set supports replication and iterative improvement. From a strategic perspective, FilBench strengthens the case for region-specific NLP investments. It demonstrates that open-weight models can compete on key metrics while remaining accessible to local developers and institutions. By surfacing Pareto-efficient models, FilBench helps buyers prioritize cost-effective options without sacrificing essential capabilities. The authors also emphasize the value of continuing to collect Filipino-language data to train and fine-tune models for better performance on generation and translation tasks.

Technical details or Implementation

FilBench is built on top of Lighteval, an all-in-one framework for LLM evaluation, and defines language-specific evaluation by translating common terms from English to Tagalog or Cebuano (for example, yes → oo, no → hindi, true → totoo) to anchor evaluation templates. The four major categories and their 12 tasks are designed to reflect historical and current NLP research priorities for Philippine languages from 2006 to early 2024. Notably, most categories emphasize non-translated content to preserve the natural use of these languages. The FilBench Score is a single representative score computed as a weighted average across categories, enabling at-a-glance comparisons while preserving the nuances of each category. FilBench provides a concrete, reproducible benchmark that researchers can use to evaluate different LLMs on Filipino language tasks and similarly structured prompts. Implementation details include:

• Four categories: Cultural Knowledge, Classical NLP, Reading Comprehension, and Generation.
• Each category contains multiple tasks, with the overall score derived via a weighted average.
• Translation pairs are incorporated for language-specific evaluation (e.g., English → Tagalog or Cebuano for common terms).
• FilBench is available as a set of community tasks in the official Lighteval repository, with results visible via the FilBench leaderboard on HuggingFace.
• The project highlights the cost-efficiency of open-weight LLMs, noting that models you can download freely from HuggingFace often deliver strong performance relative to their size. The evaluation also recognizes external support, including Cohere Labs for credits to run the Aya model series and Together AI for computational credits supporting several open models. Ongoing collaboration with the OpenEvals team and Hugging Face is acknowledged for publishing and supporting the work.

Table: FilBench categories and focus

| Category | Focus

Notes

• Most categories contain non-translated content to reflect natural language use.
• FilBench is designed to be accessible to researchers and developers, with a clear path to replication via the Lighteval repository.

Key takeaways

• FilBench provides a structured, reproducible way to benchmark Tagalog, Filipino, and Cebuano across four core NLP dimensions.
• SEA-specific open-weight models frequently offer the best parameter efficiency for these languages, though GPT-4o remains a strong baseline.
• Building region-specific data and instruction-tuning yields measurable improvements (2–3% on FilBench).
• Generation challenges persist, underscoring the need for improved instruction following and cross-language consistency.
• Open-weight LLMs often deliver cost advantages without major performance penalties, supporting broader access in the Philippines.
• FilBench is openly accessible as community tasks in Lighteval and on the HuggingFace FilBench leaderboard, enabling ongoing benchmarking and improvement.

References

• https://huggingface.co/blog/filbench