Abstract
•Integrate n-gram based natural language processing with LSVM machine learning approach.•Achieve scalability by classifying languages and extracting from various data sources.•Achieve a sustainable environment and society by introducing a health monitoring framework that identifies COVID-19 related tweets.•Eliminate storing duplicate n-grams of different language datasets to achieve memory and bandwidth efficiency.
Due to the rapid growth of electronic documents, e.g., tweets, blogs, Facebook posts, snaps in different languages that use the same writing script, language categorization, and processing have great importance. For instance, to identify COVID-19 positive patients or people’s emotions on COVID-19 pandemic from tweets written in 35 different languages faster and accurate, language categorization and processing of tweets is significantly essential. Among many language categorization and processing techniques, character and word n-gram based techniques are very popular and simple but very efficient for categorizing and processing both short and large documents. One of the fundamental problems of language processing is the efficient use of memory space in implementing a technique so that a vast collection of documents can be easily categorized and processed. In this paper, we introduce a framework that categorizes the language of tweets using n-gram based language categorization technique and further processes the tweets using the machine-learning approach, Linear Support Vector Machine (LSVM), that may be able to identify COVID-19 positive patients. We evaluate and compare the performance of the proposed framework in terms of language categorization accuracy, precession, recall, and F-measure over n-gram length. The proposed framework is scalable as many other applications that involve extracting features and classifying languages collected from social media, and different types of networks may use this framework. This proposed framework, also being a part of health monitoring and improvement, tends to achieve the goal of having a sustainable society.