Evaluation and Benchmarking of AI Applications in Indian Languages

The Artificial Intelligence has seen an exponential growth in the last few years. Several applications have been developed by many entities that claim various levels of accuracies. This includes various applications of AI including Generative AI, Machine Translation, ASR, TTS and so on. These applications have a direct interface with the General Public speaking the languages concerned and have the potential to influence the general public and lead them to have an opinion about something or some language.

The AI based applications are based on statistical and example-based data running statistics deep underneath. Inherently, the AI-based systems in the language domain are bound to make mistakes and should be used only with supervision for any kind of authentic works. But there is no doubt that these applications have been doing a great job in many of the Indian languages and improve the productivity in various tasks such as translation, dictation, automated systems, and various other applications where language is used as an interface (which virtually translates to any application that has an electronic interface with humans).

These applications are being used at many places, including in government websites, documents, textbooks, authenticated documentation procedures, etc. Government offices also tend to subscribe to these applications by paying for them. However, there are no standardized evaluation metrics available for such applications. Even if evaluations are available, they are reported by the developing agencies/entities themselves, which may not be considered authentic.

This conference seeks to deliberate on the methods of evaluating AI and Generative AI-based applications involving Indian Languages. The conference will also set the benchmarks and establish objective evaluation metrics for various applications.