AI Search Engine Multilingual Evaluation Report (v1.0)

1. Abstract

With the advent of ChatGPT, conversational search engine technology has rapidly gained widespread attention. Several general-purpose question-answering search engines, such as Perplexity and iAsk, have emerged in the market, as well as other search solutions focusing on specific vertical domains.

We believe that these conversational search products have a significant advantage in providing direct answers compared to traditional keyword-based search engines , and they may become a disruptive paradigm in the evolution of search technology. However, in actual use, we have also noticed some issues, particularly in terms of the accuracy and reliability of the answers. Inaccurate responses and so-called “hallucination answers” (i.e., answers that are irrelevant to the user’s query or completely nonsensical) frequently occur, which seriously affects the user experience.

This work tries to figure out what the best AI search engine or AI search product is? Considering the linguistic diversity of the global user base, our evaluation report selected a variety of languages, including English, Japanese, Simplified Chinese, Russian, and etc. to conduct preliminary tests and assessments on the accuracy of these question-answering search engine products.

In this report, we will detail our evaluation methodology, testing process, and the conclusions we have drawn. Our goal is to provide developers, researchers, and end-users with a comprehensive performance evaluation, to better understand the performance of these question-answering search engines in different linguistic environments, and to point out their current limitations and directions for improvement.

In this comprehensive performance evaluation of the question-answering search engine, we have arrived at the following key findings:

1. Overall, the performance of the evaluated products did not meet our expectations. However, it is noteworthy that Metaso, a company primarily serving the Chinese market, performed the best overall in the evaluation, slightly surpassing Perplexity.

Figure 1. Overall accuracy of Evaluated AI Search Products

1. The comprehensive data analysis of all products indicates that the highest accuracy is achieved in answering questions in English. By contrast, Russian questions have the lowest accuracy rate, and the accuracy for Japanese questions is also relatively low.

Figure 2. Accuracy in Languages

1. In terms of performance by language, Perplexity leads by a significant margin in answering English questions, and its performance in Simplified Chinese is also quite impressive. Metaso stands out in its performance in both Simplified and Traditional Chinese. However, both products do not reach a satisfactory level in other languages. iAsk demonstrated relatively balanced capabilities across different languages, but overall, it falls within the medium range. You.com, on the other hand, performed well only in answering questions in English.

Figure 3. Accuracy of Evaluated Products in Different Languages

Note 1: For this evaluation, the free versions of each product were chosen (the in-depth mode was selected for Metaso). The assessment of the Pro versions will be conducted later.

Note 2: This evaluation focuses solely on the accuracy of the answers, disregarding other aspects such as the language and format of the responses.

2. Product Selection

To conduct this performance evaluation of question-answering search engines, we referred to the list of leading AI search engine products provided by the aicpb.com website. After excluding traditional keyword search engines (like bing.com), we selected four of the most outstanding AI question-answering search engine products on the market for our evaluation.

These products are considered to be the best in the industry, and their technology and market performance represents the highest level of current AI search engines. Our evaluation aims to delve into the performance of these leading products, particularly in their ability to understand and respond to user queries, as well as their accuracy and reliability in handling queries in different languages.

The specific information about the aforementioned four products is as follows (referenced from statistics at aicpb.com):

3. Evaluation Data

To comprehensively assess the performance of the aforementioned AI question-answering search engine products in a multilingual environment, we selected six major languages including English, Japanese, Simplified Chinese, etc., for testing. The distribution of test cases in multiple languages is as follows:

In this evaluation, we have specifically designed five different usage scenarios, including news acquisition, local information search, technical problem, product feature inquiry, and business consultation, to simulate the types of queries users may encounter in their daily lives.

We recognize that while these five scenarios cover a range of common query types, they may not fully encompass all the potential needs of users. Therefore, we plan to continuously expand our test collection in future evaluations, by introducing more categories and scenarios to ensure that our assessment can more comprehensively reflect the actual performance and user experience of the products.

Our complete set of test cases is open-sourced and can be accessed thru link below:

GitHub – sparticleinc/ASEED: Conversational Retrieval Evaluation DatasetGitHub

4. Evaluation Method and Results

• Testing Method
  • Accuracy: Human Evaluation
  The actual answers are manually compared with the Ground truth. If the answer is a “match”, it is marked as “True”; otherwise, it is “False”.
  • Answer Correctness: refer to methods in Ragas
  The Answer correctness metric from the Ragas platform, with the GPT-4-Turbo model being used for evaluation. The formula for calculating “Answer correctness” is as follows:

Metric Description: https://docs.ragas.io/en/latest/concepts/metrics/answer_correctness.html

1. Evaluation Metrics

The results for the Accuracy evaluation have already been provided in the Abstract, and the results for the Answer Correctness evaluation are as follows.

5. Case Analysis

News

Question：最近の埼玉県行田市事故の死傷者数は何人ですか?

Ground truth：最近の埼玉県行田市の事故での死傷者数は以下の通りです:\n死亡者数: 1人\n負傷者数: 3人

Comment: Perplexity, You.com did not find relevant results. The answer provided by Metaso is basically consistent with the Ground truth. The response from iAsk is in English, but the description and source of the answer are correct.

Local Search

Question：進撃の巨人はいつ発行されましたか？

Ground truth：「進撃の巨人」は、諫山創による日本の漫画作品で、2009年9月9日に発行が開始されました。そして、2021年4月9日に連載が完結しました。最新刊（34巻）は2021年6月9日に発売されました⁵。この作品は世界中で非常に人気があり、多くの派生作品が生まれています。

Comment: In the case of answering the question about the release date of “進撃の巨人” both Perplexity and iAsk provided the correct dates, with Perplexity additionally offering more detailed information. However, the responses from Metaso and You.com were inaccurate. Perplexity’s answer better demonstrates its advantage in information gathering and processing.

Technical Consulting

Question：Anthoropic Claude のデータポリシーは？

Ground truth：ユーザーの個人データは、モデルのトレーニングに使用されないため、ユーザーデータを利用してモデルを構築することはありません。

Comment: Perplexity’s response was irrelevant to the question and did not answer it correctly. Metaso’s answer was the most accurate, providing a detailed explanation of Anthoropic Claude’s data policy. Although iAsk’s response was not relevant to the Ground truth, upon verification of its cited source, it was confirmed that its answer was still correct. You.com, on the other hand, did not find any relevant information.

Product Search

Question：갤럭시 S24의 화면 해상도는 어떻게 되나요?

Ground truth：갤럭시 S24 기본 모델의 화면 해상도는 2340 x 1080 픽셀입니다

Comment: Perplexity provided a resolution that does not exist, while iAsk offered a higher resolution, which was in fact incorrect. You.com’s answer was also incorrect. Metaso performed the best, providing accurate resolution and additional information about different models.

Business Consulting

Question：Monica.im в 2023 году имеет сколько пользователей?

Ground truth：В 2023 году у Monica.im уже более 1,5 миллионов пользователей.

Comment：Metaso provided the correct answer directly, while Perplexity and iAsk seemed more focused on explaining their search process rather than actually solving the problem. You.com’s response contained no useful information.

6. Conclusion

This evaluation conducted an in-depth multilingual performance analysis of four leading AI question-answering search engine products on the market. By testing in six language environments including English, Japanese, Simplified Chinese, and others, across five categories of scenarios such as real-time news, local search, technical consulting, product search, and business consulting, we have reached the following conclusions:

1. Overall Performance: All participating AI question-answering search engines showed potential in some areas, but the overall performance did not meet the expected level. Metaso and Perplexity had a clear lead in the tests, but there is still considerable room for improvement.
2. Language Accuracy: The accuracy rate for answers to English questions was generally higher than for other languages, with Russian and Japanese having lower accuracy rates, indicating that cross-language processing capabilities remain a challenge.
3. Product-Specific Performance: Perplexity excelled in answering questions in English, while Meta shone in handling Simplified and Traditional Chinese. iAsk performed relatively evenly across all languages, while you.com was almost unusable for non-English queries.

In the future, we plan to expand our evaluation test set, adding more languages and query scenarios to provide a more comprehensive and detailed performance assessment. We believe that with technological advancements and the enrichment of datasets, AI question-answering search engines will be able to better meet the diverse needs of global users and play an even more important role in the future of search technology.