1. Introduction

2. Objectives

The major objectives of the consortium are:

1. Become a repository of linguistic resources in the form of text and speech for all Indian languages.
2. Set appropriate standards for data collection and storage of language corpora for different research and development activities.
3. Support language technology development and sharing of tools for language-related data collection and management.
4. Facilitate creation of such databases by different organizations which could contribute and enrich the main LDC-IL repository.
5. Facilitate training and human resources development in these areas through workshops, seminars etc.
6. Create and maintain the LDC-IL web-based services that would be the primary gateway for accessing resources.
7. Design or provide help in creation of appropriate language technology, based on the linguistic data for mass use, and
8. Provide the necessary linkages between academic institutions, individual researchers and the general public.

3. Major Areas of Creation of Language Resources

• Text Corpora: monolingual and multilingual (including parallel)
• Speech corpora
• Sign language corpora
• By-products like lexicon, thesauri, WordNet etc.

4. Tasks

When we look at the major areas to be covered for developing the Language Resources and the broad objectives, the following tasks emerge before us.

1. Establishing standards
2. Creating language resources
3. Annotating language data
4. Building systems/ helping systems building
5. Creating human resources through training, workshops, on hand experience etc.,
6. Co-ordinating activities for language resources development

5. Coverage

It is kept in mind that major work in India in the area of technology, on Indian languages so far has focused more on machine translation which is an important application in a multilingual country like India. And, what is on the back burner is analyzing of language data using computational techniques to understand the language and its genres itself. Different aspects of computational linguistics provide deeper insights into the structure and functioning of each one of the languages. This power needs to be exploited to understand languages. All languages deserve attention from the researchers for their analysis and interpretation. Creation of corpora and tools for their analysis, in these languages will help in comparing and contrasting structure and functioning of Indian languages. So, corpora for minor languages will also be collected depending upon availability of text for the purpose. The same will be used for language development purpose.

6. Text Corpora - Monolingual and Multilingual (Parallel)

Monolingual / Parallel Corpora (in millions of words): Scheduled Languages

Here, the text corpora includes:

• Both balanced and full text corpora covering various domains of language use. Domains already identified and expanded as and when the need has arisen.
• Parallel corpora across different languages
• Newspaper corpora
• Historical/Inscriptional databases of Indian languages which is most important to trace not only as the living documents of Indian History but also historical linguistics of Indian languages.
• Comparative/descriptive/reference grammars shall be considered as corpora of databases.

7. Tools for Corpora Management and Analysis

These are part and parcel of corpora creation and management process. Some of them are already done and others are in the pipeline. Also, as and when the need arises, necessary ones shall be prepared. Some of them are:

1. Frequency analyzers for characters, words, sentences etc.,
2. KWIC and KWOC retrievers.
3. Tool to gather Indian Languages text (HTML) data automatically from internet.
4. Tools for Automatic transliterations across all Indian language scripts as well as Roman.
5. Parallel corpora tools for text alignment, including sentence alignment tool and chunk alignment tool as well as an interface for aligning corpora.
6. Tools for
1. Speech Data warehousing
2. Morphological analysis
3. POS tagging
4. Semantic tagging
5. Syntactic tree bank

8. Linguistic Research

The text corpora could also be used for research in

• Phonology/phonetics
• Lexical studies
• Semantics
• Pragmatics & Discourse analysis
• Sociolinguistics
• Dialectology & Variation studies
• Stylistics
• Language teaching
• Historical linguistics
• Psycholinguistics
• Social psychology
• Cultural studies

a. Phonology

• The Corpora will provide largest ever sample of Indian language data as it extends over a wide range of variation, e.g. age, gender, class, genre etc. This will help in making generalizations about the language because the corpus will definitely be representative. Thus, large scale quantitative analyses will be possible through this corpus.
• The corpora will be based on natural language and hence, the findings are more likely to reflect “real life” language.

b. Lexical studies

• The corpora will be useful for the preparation of dictionaries along with examining linguistic and non-linguistic associations of specific words such as how common are different words and their senses, whether words have systematic association with other words and does it relate to any particular register or dialect? The area of loan words can also be explored using the corpus. In lexicology, it can be used for studies of words, their meanings, elements, relations between words, word groups and the whole lexicon.
• Textual and sociolinguistic information like register, genre, domain, gender, class, age etc. allows a more accurate description of the use of a particular lexical item. Parts of speech and word sense annotation allows a precise grouping of words (i.e. phraseology).Corpus is also very helpful in studying collocation meanings.
• In producing frequency counts and dividing on various bases like which are the most common and uncommon words, where does the particular word fit on the continuum of very common to very uncommon words. This helps in understanding the patterns of use associated with a word.

c. Semantics

• It establishes an objective approach to semantics that takes account of indeterminacy and gradience. While assigning meanings to linguistic terms, the corpus can be used to provide an objective criteria as the meanings are dependent on various syntactic, morphological and prosodic contexts. Thus, an empirical objective indicator can be arrived at.
• The notion of gradience is established firmly by using the corpus as the categories are not usually clear-cut and there do exist gradients of membership connected with frequency of inclusion.
• The core area of semantics, i.e. aspect, can be investigated by using corpus. It can provide a more refined aspectual classification of verb classes and situation types

d. Pragmatics & discourse analysis

• Use of conversational corpora helps in understanding of how conversation works, with respect to lexical items and phrases which have conversational functions. It aides in identifying relevant discourse structures and also in understanding the function of many lexical and grammatical features in larger discourse contexts. Historical pragmatics also relies on corpus heavily. Thus, it helps in quantitative analyses of corpus-based approaches to pragmatics that are less explored till now.

e. Sociolinguistics

• Most of the studies have been in the area of language and gender. Other areas like sexism, femininity and sexual identity rely on the corpus. The conclusions can be tested against the existing sociolinguistic theories. General issues like language planning, language shift, language contact, language variety are also studied using the corpus. Patterns of certain use of linguistic features with dimensions like sex, age and class etc. can also be analyzed. In multilingual environment, use of corpus is very useful for all the above studies.

f. Dialectology & variation studies

• Language varieties of different geographical variation and complex co-occurrence patterns among features in different dialects and registers can be compared as well as described. This can also be used for testing the language variation theories. As the corpus is representative, the conclusions drawn will represent the whole population. This answers questions like how is speech different from writing, how do texts vary with respect to linguistic variations, how do internal parts of texts vary within a single register etc.

g. Stylistics

• Stylistics is closely related to genres and language varieties as features associated with situations of use are observed in stylistic shifts. Thus, in investigating these broader issues the corpus plays an important role. Moreover, it helps in examining an author’s particular style (vocabulary, sentences etc.) and compare one author’s work with other’s apart from investigating internally within the author’s work. It can be extended to compare various varieties too. As stylistics requires the use of quantification to endorse the judgments, corpus is not only useful but essential also. The differences between written and spoken language can be analyzed with the help of the corpus. Variations between genres can be studied using subsamples of corpora. Taking account of frequent words, sequences, and collocations, authorship attribution techniques can be modified and improve the results.

h. Language teaching

• It exposes to language learners the kind of sentences they will encounter in real life situations in different target settings, in other words, a more exact descriptions of language use. Existing syllabuses and teaching materials can be critically looked at as these materials need to be based on empirical and authentic evidence rather than tradition. Researchers can analyze relevant constructions or vocabularies both in text books and in the specific language. The teaching materials would then focus on common word forms, usage patterns and various combinations. This research will feed into the development of teaching resources for language teaching and the training of teachers.
• Apart from this, the corpora can also help in exploring domain specific language use and professional communication. Use of corpus can cover issues like theories in language teaching, methods, problems as well as solutions. Computer-assisted language learning gets easier too. Finally, open-access on-line languages data from different regional backgrounds will be an invaluable resource to be consulted by researchers, tutors and students.

i. Historical linguistics

• It can be used to look at a particular phenomena diachronically and analyzing its patterns. It helps in studying the evolution of the language through time and thus, language change in the process. It also aides in extending the “corpus” of a language.

j. Psycholinguistics

• It helps in the study of speech errors in natural conversational language that is acquired from different sources. In order to hypothesize about the human language processing, corpus is useful for analyzing language pathologies by using data from linguistically impaired people. The corpus of normal children helps in theorizing about language developmental process in humans.

k. Social psychology

• Social psychologists need naturalistic data to be tested on theories. Though most human interaction takes place through medium of speech, diaries, newspapers, company reports etc. do form part of naturally occurring texts. Moreover, conversational data can also be taken for analysis in social psychology. Thus, it helps in studying how and why people attempt to explain things and also how they regard their environment.

l. Culture studies

• Corpus of a language suggests a picture of that specific culture. It shows how words reveal commonalities and differences between cultures. The frequencies of concepts in different categories can actually tell us about different cultures. Occurrences of words in different domains can also tell us a lot about that culture.

The above areas illustrate the arena of corpus studies and also the LDC-IL shall make some attempt to do some research into some of the areas as case studies. But it does not like to fix the targets.

9. Speech Corpora

The objective of speech data collection is to primarily build speech recognition and synthesis systems for Indian languages. Voice user Interfaces for IT applications and services are very important and are valued for their ease of access, especially in telephony-based applications. In a country like India, where the rate of literacy is low, Indian language speech interfaces can provide access to IT applications and services, through internet and/or telephones, to the masses. So that people in various semi-urban and rural parts of India will be able to use telephones and Internet to access a wide range of services and information on health, agriculture, travel, etc. However, for this to become a reality, computers should be able to accept speech input in the user’s language and provide speech output. Also, in multilingual India, if speech technology is coupled with translation systems between the various Indian languages, services and information can be provided across languages more easily. Due to the lack of appropriate annotated speech databases in Indian languages robust applications have not been developed. Here, the focus is to:

1. Develop tools that facilitate collection of high quality speech data
2. Collect data that can be used for building speech recognition, speech synthesis and provide speech-to-speech translation from one language to another language spoken in India (including Indian English). Apart from these like applications in the area of text corpora, speech corpora also, main efforts are on the engineering side. So, efforts shall also be made to collect
• Child language corpora
• Pathological speech/language data and
• Speech error Data

Applications

Speech Recognition and Speech Synthesis

• Speech to Speech translation for a pair of Indian languages
• Command and control applications
• Speech-To-Text processing (Word processor or emails, Transcription of speech into mobile text messages)
• Real time voice recognition (Railway reservation / enquiry, ATM, weather information, music on demand)
• Screen Readers for Visually Impaired
• Health care (Medical Transcription)
• Multimodal interfaces to the computer in Indian languages
• Real-time Voice Writing (Court reporting, speech in the Parliament, Press conferences)
• Information delivery to illiterate / unaware of a Language (Illiterate can not read / write but can listen and understand. Who can not read / write a particular language)
• Telephony (SMS reading on cell phone, e-mail reading on cell phone, For fixed line phone: SMS can not be sent on a fixed line phone, but a TTS can convert text into speech and send on a fixed line phone)
• Pronunciation evaluation in computer-aided language learning applications
• Simple data entry (entering a credit card number, dialing a number)
• Automatic translation (Speech to speech translation)
• Robotics
• Speech enabled Office Suite etc.

The LDC-IL shall collect, transliterate (text, phonemic and phonetic level) and annotate speech data and make it available for research and development purposes to the community.

Data set - 1: The speech data set consisting of the following components for each language shall be created for collecting speech corpora for speech recognition:

1. Phonetically Balanced Vocabulary
2. Phonetically Balanced Sentences
3. Connected Text created using phonetically balanced vocabulary
4. Date Format
5. Command and Control Words
6. Proper Nouns 500 place and 500 person names
7. Most Frequent Words: 1000
8. Form and Function Words
9. News domain: news, editorial, essay - each text not less than 500 words

Number of speakers: Data will be collected from minimum of 300 (150 Male and 150 Female) speakers of each language. In addition to this, natural discourse data from various domains too shall be collected for Indian languages for research into spoken language.

Data set - 2: The speech data set consisting of the following components for each language shall be created for collecting speech corpora for speech synthesis.

1. Phonetically Balanced Vocabulary
2. Phonetically Balanced Sentences
3. Connected Text created using phonetically balanced vocabulary
4. Other texts from stories etc.,

Data for speech synthesis shall be collected from limited number of speakers - 3 male and 3 female in the studio environment. They shall invariably have very good voice quality and are professional voice givers/media announcers.

Annotation of data:

(a) Data to be used for speech recognition shall be annotated at phoneme, syllable, word and sentence levels

(b) Data to be used for speech synthesis shall be annotated at phone, phoneme, syllable, word, and phrase level. This will be in accordance with annotation guidelines already developed for the purpose.

Annotation tools: Tools will be developed for semiautomatic annotation of speech data and Praat will be used for manual annotation. These tools will also be useful for annotating speech synthesis databases.

Text Annotation Tools: POS taggers, phrase boundary markers and intonation markers etc.,

LDC-IL Coding Standard: LDC-IL has developed its own coding standard for development of tools on .NET framework. And LDC-IL will be developing different coding standards for different Linux based frameworks.

Telephone based data collection: The data set - 1 will be replicated using telephone as medium of recording.

Coverage of languages: The priority is to cover all the Scheduled languages and then take up other non-scheduled languages. The list of languages is already given in introduction. After data collection, annotation at various levels shall be taken up.

I year	II year	III year	IV year
1. Bengali	7. Manipuri	13. Maithili	19. Sindhi
2. Hindi	8. Malayalam	14. Dogri	20. Odia
3. Tamil	9. Punjabi	15. Bodo	21. Marathi
4. Telugu	10. Urdu	16. Konkani	22. Khasi
5. Assamese	11.Kannada	17. Santali	23. Tulu
6. Nepali	12. Gujarati	18. Kashmiri	24. Kodava

Pronunciation dictionaries and sound dictionaries based on the defined formats shall also be created in all the languages listed above.

Other uses of speech databases

The data set for this shall include in addition to the data set said above shall include the word list and sentence list that the Central Institute of Indian Languages has created for the purpose and standardized. Major areas of application are given below.

1. Phonetics and phonology

Annotated speech database will provide insights into production, acoustics and perception of spoken language. Corpus based phonological analysis typically involves defining a formal model, systematically testing it against data, and comparing it with other models. (In some cases, the model may be incorporated into a software system, e.g. for generating natural intonation in a text-to-speech system.) In this exploration and analysis - sorting, searching, tabulating, defining, testing and comparing - the principal task is computational.

2. Speech pathology

Developmental language disorders occur in children who do not develop functional language skills. Language disorders belong to a broad category of disorders called communication disorders that also include speech and hearing disorders. Receptive language impairments refer to a difficulty understanding language at the level of meaning. The vocabulary range is usually very limited. The purpose of simple grammatical constructions is also not properly understood.

In this field various types of pathological speech are studied, ranging from mild disorders such as hoarseness to serve disorders such as aphasia. The aim of most studies of pathological speech is to find therapies that can alleviate or cure the speech disorders of interest. Aphasia can also be subject of phycholinguistics studies. Because such language disorders can be shed some light on underlying mental process. Corpora of pathological speech are very useful for these purpose. These corpora also use full for the development of automatic classification of speech physiologies.

Speech corpora based tools are useful particularly who have one are more of the following features.

1. Sound substitutions in words, difficulty in processing sounds in to syllables and words.
2. Inappropriate use of forms, limited vocabulary development and inability to follow directions.
3. Speech corpora is helpful to list out most frequent words, phones and phonemes. With the help of multi media based speech annotated data and other speech analysis tools (like CATSCAR tools) we can give training for speech disorder people and their family members People with developmental language disorders can benefit from special education programs, usually monitored by a speech pathologist based on the speech corpora tools..
3. Language learning and language teaching

Speech corpus can help in grammar and error diagnosis. It can aid in conversational interaction and pronunciation learning. Spoken Corpus shows us how we really use language, not how we are supposed to use it or how we use it when we are writing. It reveals how very different the spoken word is from the written word. Thus, speech corpus can be used in constructing teaching materials. The students are able to study the language through language teaching materials that represent the language as it really is. One can build automatic scoring of pronunciation quality for non-native speech. Methods from pattern and speech recognition are applied to develop appropriate feature sets for sentence and word level scoring. Besides this, phoneme accuracy, duration score and recognition accuracy are identified. This can be a part of computer assisted pronunciation training (CAPT) that is specially designed for Second Language Learning. The goal of CAPT is to provide instantaneous, individualized feedback to the user on the overall quality of the pronunciation mode as in the classrooms the teachers are sometimes unable to focus on individuals. It is able to correct learner’s segmental and prosodic mistakes. CAPT also helps in providing private, stress-free and self-paced learning environment.

4. Socio-phonetics

Variation exists at every level of linguistic representation. It is now a well known fact that the study of socially conditioned variation concentrates more on phonetics. The phonetic realization of any word varies according to the speaker, the context (linguistic and social), the topic, the intention and other factors. Thus, the phonetic realizations are much layered with social meaning. This is the object of study in Socio-phonetics (the study of socially-conditioned phonetic variation in speech) .This makes a complete use of speech data. It can be used for acoustic phonetic analysis, analyzing variables, voice quality and discourse markers in different contexts. It offers help in an integrated understanding of how speech is produced, performed and perceived in its social context.

These are some of the illustrations of the use of speech databases. If time is available, efforts shall be made by the LDC-IL faculty to do some research into these aspects of speech.

10. Indian Sign Language Corpora

1. Exposure to sign by three years of age.
2. Capability and comfort with judging whether a sentence is grammatical or not.
3. Daily contact with the signed language in the deaf community (for 10+ years).

The project plan for the fieldwork will be carried out as the following, and each fieldwork fetches 3000 lexical items, 500 sentences and 10 production data enriching the corpora not only in numeric values but also in terms of regional, social, educational, and others parameters of linguistic variations:

1. Northern India : Delhi	1st year
2. Southern India: Mysore	2nd year
3. North-eastern India: Shillong	3rd year
4. Western India: Ichalkaranji	4th year
5. Eastern Indian: Kolkata	5th year

In order to avoid any potential extra erroneous influence and to assure quality, the data is verified with the bilingual consultant. The data is videographed using digital video graphy. Later, the edited data is time-line tagged as well as annotated enabling to meet the basic requirements of the applications and the tools as well as of academic and pedagogical research.

Total Indian sign language corpora set at the end of five years shall be:

Lexical items	15000
Sentences	2500
Production data	50

Standards for collecting and recording of data etc., shall be evolved in consultation with some other teams working in the area.

11. By-products like Lexicon, Thesauri, WordNet etc

1. Creation of frequency dictionaries - five per year

1st year	2nd year	3rd year	4th year	5th year
Bengali	Bodo	Assamese	Kashmiri	other Indian languages
Hindi	Dogri	Gujarati	Malayalam
Kannada	Maithili	Odia	Marathi
Manipuri	Nepali	Punjabi	Sanskrit
Urdu	Konkani	Tamil	Santali

2. Multilingual multi directional dictionary - an ongoing process
3. Aiding wordnet creation and collaborating with others for the same - an ongoing process
4. Document Management and data tracking system to manage all of the data collected, processed, annotated and distributed for LDC-IL will be created. Each document will be assigned a unique document ID and there will be link between the collected document and its associated files. This database will track information such as document ID; languages; data type; data partition; data medium; author information; date; location; source type; file format; file size; and others as required etc., The database shall also be upgraded so that the existing data is ready for use in new systems and techniques. However, the backward compatibility of the database will be of major importance.

12. Licensing Policy

Licensing is an important issue for LDC-IL. The draft policy for licensing shall be evolved through discussions within one year. The same shall be finalized within another one year by the time the annotated data is available for delivery purposes.

13. Evaluation

14. Beyond Roadmap

Above all and in addition to what LDC-IL has projected in the roadmap the LDC-IL will positively respond to the specific language data needs of the individuals, institutions and industry by taking up their requests on priority basis for licensing purposes. In the beginning the derivatives of the databases shall be licensed and after all the licensing issues are resolved the databases shall also be licensed.

15. Future plans of LDC-IL (2024-2029)

16. LDC-IL Goals

Increase Linguistic Resources:

• Develop datasets in all Indian languages
• Develop specialized datasets
• Prioritize under-resourced languages and under-resourced domains
• Publish research findings in LDC-IL

Enhance Data Quality and Standardization:

• Design data collection framework that will benefit future generation as well
• Prepare guidelines for qualitative data creation and annotation
• Develop comprehensive documentation

Improve Data Accessibility:

Make user friendly accessibility in LDC-IL website

Organize Workshops and Training Programs:

• Provide training for researchers and developers to utilize language data and technologies.
• Share best practices in NLP for Indian languages.
• Establish collaborations with academic institutions and industry partners.
• Participate in programs to understand emerging needs and priorities.
• Conduct workshops and orientation programs all over India

Develop Language Technologies:

• Develop applications that can process Indian language
• Prepare advanced techniques in natural language processing.
• Develop multilingual model.