arXiv:2511.16397v1 Announce Type: new 
Abstract: While web data quality is crucial for large language models, most curation efforts focus on filtering and deduplication,treating HTML-to-text extraction as a fixed pre-processing step. Existing web corpora rely on heuristic-based extractors like Trafilatura, which struggle to preserve document structure and frequently corrupt structured elements such as formulas, codes, and tables. We hypothesize that improving extraction quality can be as impactful as aggressive filtering strategies for downstream performance. We introduce MinerU-HTML, a novel extraction pipeline that reformulates content extraction as a sequence labeling problem solved by a 0.6B-parameter language model. Unlike text-density heuristics, MinerU-HTML leverages semantic understanding and employs a two-stage formatting pipeline that explicitly categorizes semantic elements before converting to Markdown. Crucially, its model-based approach is inherently scalable, whereas heuristic methods offer limited improvement pathways. On MainWebBench, our benchmark of 7,887 annotated web pages, MinerU-HTML achieves 81.8\% ROUGE-N F1 compared to Trafilatura's 63.6\%, with exceptional structured element preservation (90.9\% for code blocks, 94.0\% for formulas). Using MinerU-HTML, we construct AICC (AI-ready Common Crawl), a 7.3-trillion token multilingual corpus from two Common Crawl snapshots. In controlled pretraining experiments where AICC and Trafilatura-extracted TfCC undergo identical filtering, models trained on AICC (62B tokens) achieve 50.8\% average accuracy across 13 benchmarks, outperforming TfCC by 1.08pp-providing direct evidence that extraction quality significantly impacts model capabilities. AICC also surpasses RefinedWeb and FineWeb on key benchmarks. We publicly release MainWebBench, MinerU-HTML, and AICC, demonstrating that HTML extraction is a critical, often underestimated component of web corpus construction.

يتناول المقال تقديم MinerU-HTML، وهو خط أنابيب جديد للاستخراج مصمم لتحسين جودة استخراج HTML إلى نص لنماذج اللغة الكبيرة. على عكس المستخرجين التقليديين المعتمدين على القواعد، يستخدم MinerU-HTML نهجًا قائمًا على النموذج يعزز الفهم الدلالي ويصنف العناصر قبل تحويلها إلى Markdown. تهدف هذه الطريقة إلى الحفاظ على بنية الوثيقة وتحسين الأداء العام لنماذج اللغة.

El artículo presenta MinerU-HTML, un nuevo pipeline de extracción diseñado para mejorar la calidad de la extracción de HTML a texto para grandes modelos de lenguaje. A diferencia de los extractores basados en heurísticas tradicionales, MinerU-HTML utiliza un enfoque basado en modelos que mejora la comprensión semántica y categoriza los elementos antes de convertirlos a Markdown. Este método busca preservar la estructura del documento y mejorar el rendimiento general de los modelos de lenguaje.

L'article présente MinerU-HTML, un nouveau pipeline d'extraction conçu pour améliorer la qualité de l'extraction HTML-texte pour les grands modèles de langage. Contrairement aux extracteurs basés sur des heuristiques traditionnelles, MinerU-HTML utilise une approche basée sur un modèle qui améliore la compréhension sémantique et catégorise les éléments avant de les convertir en Markdown. Cette méthode vise à préserver la structure des documents et à améliorer les performances globales des modèles de langage.

The article discusses the introduction of MinerU-HTML, a novel extraction pipeline designed to improve the quality of HTML-to-text extraction for large language models. Unlike traditional heuristic-based extractors, MinerU-HTML utilizes a model-based approach that enhances semantic understanding and categorizes elements before converting them to Markdown. This method aims to preserve document structure and improve the overall performance of language models.

AICC: Parse HTML Finer, Make Models Better -- A 7.3T AI-Ready Corpus Built by a Model-Based HTML Parser

arXiv:2511.16416v1 Announce Type: cross 
Abstract: This study explored whether supervised machine learning and deep learning models can effectively distinguish perceived lower-quality news articles from perceived higher-quality news articles. 3 machine learning classifiers and 3 deep learning models were assessed using a newly created dataset of 1,412,272 English news articles from the Common Crawl over 2018-2024. Expert consensus ratings on 579 source websites were split at the median, creating perceived low and high-quality classes of about 706,000 articles each, with 194 linguistic features per website-level labelled article. Traditional machine learning classifiers such as the Random Forest demonstrated capable performance (0.7355 accuracy, 0.8131 ROC AUC). For deep learning, ModernBERT-large (256 context length) achieved the best performance (0.8744 accuracy; 0.9593 ROC-AUC; 0.8739 F1), followed by DistilBERT-base (512 context length) at 0.8685 accuracy and 0.9554 ROC-AUC. DistilBERT-base (256 context length) reached 0.8478 accuracy and 0.9407 ROC-AUC, while ModernBERT-base (256 context length) attained 0.8569 accuracy and 0.9470 ROC-AUC. These results suggest that the perceived quality of worldwide news articles can be effectively differentiated by traditional CPU-based machine learning classifiers and deep learning classifiers.

تستكشف هذه الدراسة فعالية نماذج التعلم الآلي الخاضعة للإشراف ونماذج التعلم العميق في التمييز بين المقالات الإخبارية التي تُعتبر ذات جودة منخفضة وتلك ذات جودة عالية. باستخدام مجموعة بيانات تضم 1,412,272 مقالة إخبارية باللغة الإنجليزية من 2018 إلى 2024، تقيم الدراسة ثلاثة مصنفات للتعلم الآلي وثلاثة نماذج للتعلم العميق. تشير النتائج إلى أن المصنفات التقليدية مثل Random Forest ونماذج التعلم العميق مثل ModernBERT-large يمكن أن تحقق دقة كبيرة في تصنيف جودة الأخبار.

Este estudio investiga la efectividad de los modelos de aprendizaje automático supervisado y de aprendizaje profundo para distinguir entre artículos de noticias percibidos como de menor y mayor calidad. Utilizando un conjunto de datos de 1,412,272 artículos de noticias en inglés de 2018 a 2024, la investigación evalúa tres clasificadores de aprendizaje automático y tres modelos de aprendizaje profundo. Los hallazgos indican que clasificadores tradicionales como Random Forest y modelos de aprendizaje profundo como ModernBERT-large pueden lograr una precisión significativa en la clasificación de…

Cette étude examine l'efficacité des modèles d'apprentissage supervisé et d'apprentissage profond pour distinguer les articles de presse perçus comme de qualité inférieure et supérieure. En utilisant un ensemble de données de 1 412 272 articles de presse en anglais de 2018 à 2024, la recherche évalue trois classificateurs d'apprentissage automatique et trois modèles d'apprentissage profond. Les résultats indiquent que des classificateurs traditionnels comme Random Forest et des modèles d'apprentissage profond tels que ModernBERT-large peuvent atteindre une précision significative dans la class…

This study investigates the effectiveness of supervised machine learning and deep learning models in distinguishing between perceived lower-quality and higher-quality news articles. Using a dataset of 1,412,272 English news articles from 2018 to 2024, the research evaluates three machine learning classifiers and three deep learning models. The findings indicate that traditional classifiers like Random Forest and deep learning models such as ModernBERT-large can achieve significant accuracy in classifying news quality.

AICC: Parse HTML Finer, Make Models Better -- A 7.3T AI-Ready Corpus Built by a Model-Based HTML Parser

Was this article worth reading? Share it

DocsParse

Bytefitz

Jsonify