Разработка и валидация FakeKapcha: интеллектуальный инструмент для выявления дезинформации в социальных сетях
Разработка и валидация FakeKapcha: интеллектуальный инструмент для выявления дезинформации в социальных сетях
Аннотация
Быстрый рост социальных сетей ускорил распространение дезинформации, что привело к потребности в надежных методах автоматического обнаружения. В данной статье представлена FakeKapcha — программная платформа для обнаружения дезинформации в русскоязычных социальных сетях с использованием обработки естественного языка и ансамблевого машинного обучения. Предлагаемая система была обучена и проверена на наборе данных, включающем более 350 000 публикаций в «ВКонтакте», дополненном набором данных WELFake. Ансамбль из 11 моделей машинного обучения обеспечил надёжную и точную классификацию, одновременно давая вероятностные оценки дезинформации. Разработанное настольное приложение позволяет автоматизировать анализ контента социальных сетей и поддерживает практический мониторинг дезинформации. Результаты экспериментов демонстрируют высокую эффективность классификации и подтверждают результативность предложенного ансамблевого подхода для крупномасштабного анализа русскоязычных социальных сетей.
1. Introduction
The rise of digital communication platforms has transformed how information is created, shared, and consumed. Social media now serves as a key channel for news, providing users with immediate access to global events. However, the absence of strict verification processes and the rapid spread of information have led to the widespread distribution of misinformation and manipulated narratives. This can distort public perception, destabilize social discourse, and pose systemic risks online
, . The rise of unverified content highlights the shortcomings of traditional manual fact-checking methods, which are too slow and labor-intensive for the fast-paced data streams of social networks . As a result, there is a pressing need for automated systems using AI and machine learning to improve digital information integrity and safety. While significant progress has been made, most existing frameworks focus on English-language data, leaving regional social media, particularly Russian-language content, inadequately addressed due to its unique linguistic and communication challenges . Current solutions struggle on platforms like VK due to differing linguistic structures, user behaviors, and content dynamics compared to mainstream datasets.Comparison of existing methodologies for misinformation detection across diverse linguistic contexts and digital infrastructures
Methodological Approach | Data Scope | Language Focus | Limitations in Scalability | Key Advantages of the Proposed FakeKapcha Framework | Reference |
Supervised Learning | Global Datasets | Primarily English | High computational cost and data imbalance | 11-model ensemble; Russian dataset | |
Automated Heuristic Analysis | Localized Communities | Restricted Linguistic Support | Inefficient processing of non-standard vernacular and hybrid structures | Data-driven ML; VK-specific processing | |
Ensemble Machine Learning | Large-scale datasets | Multilingual/Regional | Adaptability issues regarding rapidly evolving disinformation tactics | Three-level output; integrated desktop tool |
This research develops and validates an automated framework, "FakeKapcha", for detecting misinformation in Russian-language social networks like VK. Instead of relying on manual verification or binary filters, it utilizes a scalable, multi-tiered probability assessment pipeline that addresses localized language and inauthentic behavior
. The system uses automated data collection from the VK API, compiling over 350,000 records, then text cleaning and employing high-dimensional TF-IDF feature extraction, the data undergoes processing through a stacked ensemble of eleven machine learning and deep learning algorithms. By combining linguistic signals with metadata, the framework computes a misinformation probability score and classifies content into three categories: Reliable, Potential Misinformation, or Highly Probable Fake Content presented via a user-friendly desktop GUI for real-time analysis , . This work presents significant advancements in automated text classification through several interconnected contributions. It introduces a newly curated dataset of over 350,590 publications to address gaps in open-source regional language tools. Our heterogeneous ensemble architecture combines 11 diverse machine learning and deep learning models, establishing a robust predictive baseline for high-dimensional text. This framework offers unique non-binary interpretability, effectively capturing epistemic uncertainty in live data. The findings provide insights into how large-margin linear classifiers and well-regularized probabilistic models manage semantic noise without relying on resource-intensive transformer models. Additionally, this framework is a desktop application for non-technical users like fact-checkers and policy analysts, aiding their efforts against disinformation. This paper details the workflow, pipeline mechanics, classifier performance, current system limitations, and future directions, including user behavior signals and adaptive model retraining.2. Research methods and principles
The proposed FakeKapcha system detects fake news from social network communities (e.g., VK) using machine learning and natural language processing. It functions as a sequential pipeline, converting raw social media content into structured classification results. Figure 1 illustrates the system architecture, detailing the flow from data collection to output generation.
Overall Architecture of the Proposed FakeKapcha Misinformation Detection Framework
Dataset construction and labelling: the training dataset was created from VK social network data via the official VK API, focusing on verified communities and unverified, low-credibility channels. Low-credibility communities were identified based on the lack of official verification, frequent sharing of unverified content, and manual reputation assessments. Communities focused on entertainment or advertisements were excluded. Publications were labeled: Label 0 for reliable content from verified sources and Label 1 for potentially misleading posts from unverified sources. The final dataset included 350,590 text samples, with duplicates, empty posts, ads, and multimedia-only entries removed to ensure quality.
Experimental Design and Data Partitioning: the dataset, comprising 350,590 labeled VK publications, was divided into training, validation, and test subsets using stratified sampling to preserve class distributions. Specifically, 70% was used for training, 15% for validation, and 15% for testing. This approach ensured balanced representation of both reliable and misleading classes, minimizing sampling bias.
All preprocessing, including text normalization and TF-IDF feature extraction, was performed only on the training set to avoid information leakage. Machine learning models were trained on the same partitions for fair comparisons, and performance was evaluated using Accuracy, Precision, Recall, and F1-score on the unseen test dataset.
Figure 2 - Text Preprocessing and TF-IDF Feature Matrix Construction Pipeline
Text preprocessing: raw text is preprocessed to reduce noise and ensure consistency. This involves converting text to lowercase, removing URLs, punctuation, special characters, and numbers, and then tokenizing, filtering stop words, and normalizing whitespace.
Feature extraction: the cleaned text is vectorized using TF-IDF, a method that emphasizes informative terms while reducing the weight of frequently occurring words to create suitable input for the machine learning models.
Machine learning classification: the prediction engine integrates eleven diverse machine learning classifiers, including Logistic Regression, Support Vector Machines, Random Forest, Extra Trees, Naive Bayes, XGBoost, LightGBM, CatBoost, and a Convolutional Neural Network. Each model generates an independent probability score indicating whether a publication is fake, and the final decision is made by averaging these scores:
where N equals 11
. The ensemble approach enhances stability and reduces overfitting risks compared to individual models (Fig. 3). By aggregating diverse algorithmic predictions, this stacked ensemble improves classification robustness and generalizability, this minimizes model biases and maintains a balanced diagnostic performance .
Ensemble learning architecture of the proposed FakeKapcha system
Training Time and Resource Consumption: the training of the base ensemble classifiers was conducted on a balanced dataset consisting of approximately 70,000 records. Text vectorization was performed using TF-IDF, which created 10,000 features based on unigrams and bigrams, taking between 2 and 5 minutes. The complete sequential training and validation pipeline for all 11 classical models on a CPU required between 10 and 30 minutes. Experimental fine-tuning of transformer-based architectures, such as BERT and RoBERTa, resulted in a slight increase in accuracy (approximately 95–96%). However, this process demanded several hours of CPU computation and increased model storage to 4,4 GB. As a result, a classical machine learning ensemble was chosen for the production version to effectively balance predictive precision with low hardware requirements.
Result generation and visualization: the system outputs the predicted class, probability score, confidence level, original text, and metadata. According to P. Kulkarni et al. (2021), the use of a three-level output scheme: reliable information, potential misinformation, and fake information, enhances interpretability for uncertain cases
. All results are stored in a local SQLite database and presented through a graphical user interface for real-time analysis and visualization. The desktop interface loads the model ensemble in up to 60 seconds. For classifying individual publications, the process includes text preprocessing, vectorization, and model voting, with core latency between 100 and 500 milliseconds. If automatic translation through an external API is needed, processing takes 1 to 3 seconds. In batch processing of community feeds, the system can handle over 5 posts per second, and re-analyzed texts return cached results instantly.3. Results and Discussion
The FakeKapcha system was validated using a dataset of 350,590 publications from VK communities via the VK API. Eleven machine learning algorithms were compared under identical conditions using TF-IDF features, with performance assessed by Accuracy, Precision, Recall, and F1-score. The Passive Aggressive algorithm achieved the highest score (0,972), followed by LinearSVC and LightGBM (0,971), XGBoost (0,963), and CNN (0,961). Logistic Regression and CatBoost were around 0,955, while Extra Trees and Random Forest scored between 0,922 and 0,928. Naive Bayes had the lowest performance at 0,896. These results highlight the effectiveness of ensemble learning in creating a strong classification framework that outperforms individual models, as shown by the comparative heat map in Figure 4 ,
.
Figure 4 - Comparative evaluation of machine learning models based on Accuracy, Precision, Recall, and F1-score
The radar chart (Fig. 5) displays the distribution of Accuracy, Precision, Recall, and F1-score for the top three classifiers: Passive Aggressive, LinearSVC, and LightGBM. Passive Passive Aggressive demonstrates the highest overall performance across all evaluation metrics, indicating superior classification quality, while LinearSVC and LightGBM have slightly lower values. This comparison illustrates that all models maintain balanced performance across metrics, essential for reliable classification and preventing bias. These results align with literature indicating that ensemble architectures, especially with robust TF-IDF representations, enhance discriminatory power compared to individual classifiers
, .
Figure 5 - Radar chart comparison of the highest-performing machine learning classifiers
PR curves provide a more diagnostic view than ROC plots for the imbalanced VK corpus, as evidenced by the significant AP difference of around seven percentage points between the upper curves and the Naive Bayes trajectory
. Passive Aggressive and Logistic Regression demonstrate robustness due to their mathematical foundations: Passive Aggressive effectively adapts to high-dimensional TF-IDF vectors, while Logistic Regression models stable, calibrated probabilities across thresholds. In contrast, Naive Bayes struggles with rare co-occurring terms . Overall, the PR visualization supports the view that large-margin linear classifiers and well-regularized probabilistic models maintain consistent classification quality, while those relying on strong independence assumptions perform poorly as recall increases, highlighting the need for residual-based and ensemble architectures in the FakeKapcha pipeline .
Precision–Recall performance analysis of representative classification models
Figure 7 - Graphical interface of the FakeKapcha system demonstrating misinformation detection results
With the increasing costs of server-grade hardware and limited access to paid cloud AI APIs, the proposed tool offers significant practical benefits for everyday use. Operating as an autonomous desktop application, it allows for real-time verification of VK social media streams without the need for cloud infrastructure, monthly API subscriptions, or expensive GPU hardware. This system serves as a lightweight, privacy-preserving decision-support tool for editors, community moderators, and general users who want to verify local content on VK.
Ethical consideration
Even though automated classification tools offer valuable support, they should not be misconstrued as instruments for determining absolute truth
. Machine learning models rely on statistical patterns and may produce errors when handling satire, irony, opinion-based articles, rapidly changing news, or content that lacks sufficient context. As a result, these systems function as decision-support tools that assist users rather than supplanting human judgment. This distinction is vital due to the risks of false positives, which can erode trust in legitimate sources, and false negatives, which may allow misleading content to persist. Transparency remains an ethical imperative; users need to understand that the results are probabilistic estimates, not definitive verifications. Thus, incorporating confidence indicators, probability scores, and a "Potential Misinformation" category helps convey the uncertainty that accompanies automated predictions .Future Directions
Further research and development of the FakeKapcha tool should assertively expand beyond VK, creating a unified platform capable of processing content from other platforms such as Telegram, X, Facebook, Reddit, and other online news portals
, , . Thereby, our integration of transformer-based language models specifically designed for Russian-language content is expected to improve detection accuracy.The tools should be allowed to tap into and expand a dataset and self-model retrain, which will ensure that the system remains resilient and adaptive to evolving information ecosystems.
To maintain long-term reliability, future versions of the system will incorporate continuous performance monitoring using data drift indicators and predictive quality metrics. The ensemble performance will be periodically evaluated through Accuracy, Precision, Recall, and F1-score on newly labeled validation samples. A decrease in the F1-score of more than 5% relative to the baseline performance or statistically significant changes in feature distributions will be used as indicators for initiating ensemble retraining. The retraining process can be planned after accumulating sufficient newly verified data from supported platforms, ensuring that the framework remains robust against evolving misinformation patterns.
4. Conclusion
Modern information environments increasingly require automated verification technologies. This article demonstrates the successful application of machine learning to detect misinformation in Russian-language social media content. The FakeKapcha system implements a comprehensive pipeline, including automated data collection via the VK API, text preprocessing, TF-IDF feature extraction, ensemble classification, and result visualization. Using a custom dataset of over 350,000 publications, we confirmed that this ensemble strategy enhances prediction stability and robustness for heterogeneous textual data. A key innovation is a three-level classification mechanism that categorizes content as Reliable Information, Potential Misinformation, or Fake Information, improving interpretability by enabling nuanced, non-binary assessments alongside transparent confidence scores. The system effectively integrates automated data acquisition, natural language processing, and user-oriented design to assess information reliability on social media. Future development aims to expand platform support, create cross-platform mobile applications, and incorporate multimodal analysis and advanced language models to broaden the system's applicability.