Inferring the Size of Large Language Models From Popular Text Memorization
A new study proposes a method to infer the size of large language models (LLMs) based on their text memorization capabilities, addressing the common issue of developers withholding parameter counts. This approach utilizes the accuracy of next-word predictions from widely circulated texts to estimate a model's memorization limits, which correlate with its total parameter count.
WPN Brief
- What Happened
A new study proposes a method to infer the size of large language models (LLMs) based on their text memorization capabilities, addressing the common issue of developers withholding parameter counts. This approach utilizes the accuracy of next-word predictions from widely circulated texts to estimate a model's memorization limits, which correlate with its total parameter count.
- Why It Matters
Understanding the size of LLMs is crucial for interpreting their capabilities and associated costs, especially as these models become integral to various applications in artificial intelligence. By providing a means to estimate model size, this research could enhance transparency and inform future developments in LLM technology.
- The Bigger Picture
The study highlights ongoing discussions in the AI community regarding model evaluation, personalization, and the balance between exploration and exploitation in decision-making tasks. As LLMs evolve, their ability to accurately predict and generate text remains a focal point, influencing their application across diverse fields, including education, communication, and data analysis.
Related Reports
More coverage on this story
10 reports across the wire
On the Persistent Effects of Lexicality in Large Language Models
A recent study published on arXiv investigates the persistent effects of lexicality in large language models (LLMs), revealing that lexical overlap significantly influences the structure of representations extracted from these models, often overshadowing semantic content. The research employs adversarial semantic stress tests to quantify this influence across various architectures and training regimes.
Are Large Language Models Suitable for Graph Computation? Progress and Prospects
Recent research has explored the suitability of large language models (LLMs) for graph computation, focusing on their ability to reason over structured relationships and perform algorithmic operations. The study identifies two paradigms: LLMs as executors, which solve graph tasks directly, and LLMs as planners, which formulate problems and decompose reasoning steps. This comprehensive review aims to clarify the role of LLMs in graph-solving pipelines.
Finding the Minimal Parameter Budget for Implicit Reasoning: A Data Complexity Driven Scaling Law for Language Models
A recent study published on arXiv investigates the minimal parameter budget necessary for language models (LMs) to perform implicit reasoning, which involves inferring new facts from existing knowledge without explicit supervision. The research identifies a scaling law that connects the optimal parameter budget to a measure of graph search entropy, demonstrating that appropriately sized LMs can effectively reason over specific information amounts.
How reliable are LLMs when it comes to playing dice?
A recent study investigated the probabilistic reasoning capabilities of large language models (LLMs) through a controlled benchmarking study on discrete probability problems. The research evaluated eight state-of-the-art models, revealing an average accuracy of 0.96 on standard problems but only 0.59 on counterintuitive ones, indicating significant limitations in their reasoning abilities.
Should You Use Your Large Language Model to Explore or Exploit?
A recent study evaluates the effectiveness of large language models (LLMs) in assisting decision-making agents with exploration-exploitation tradeoffs. The research highlights that while reasoning models show potential for exploitation tasks, they are often too costly or slow for practical use. Conversely, non-reasoning models can enhance performance on medium-difficulty tasks, although they still underperform compared to simple linear regression methods.
Skip a Layer or Loop It? Learning Program-of-Layers in LLMs
Recent research has introduced a flexible program-of-layers (PoLar) approach for large language models (LLMs), allowing for dynamic execution of pretrained layers, which can be skipped or looped based on input requirements. This method aims to enhance inference efficiency and accuracy by customizing layer execution for various inputs.
Analysing Differences in Persuasive Language in LLM-Generated Text: Uncovering Stereotypical Gender Patterns
A recent study analyzed the differences in persuasive language generated by large language models (LLMs), focusing on how factors such as recipient gender, sender intent, and output language influence the effectiveness of persuasive communication. The research evaluated 13 LLMs across 16 languages, revealing significant gender differences in the generated persuasive language.
UnpredictaBench: A Benchmark for Evaluating Distributional Randomness in LLMs
UnpredictaBench has been introduced as a benchmark aimed at evaluating the distributional randomness capabilities of large language models (LLMs). This evaluation is crucial as LLMs are increasingly utilized in simulations that require capturing the unpredictability of real-world systems, rather than converging on a single plausible answer. The benchmark includes 448 problems that assess the models' ability to sample from various target distributions.
Data-Constrained Language Model Pretraining: Improved Regularization and Scaling Laws
A recent study published on arXiv explores data-constrained language model pretraining, focusing on improved regularization and scaling laws. The research highlights the challenges faced as training compute outpaces the availability of natural language data, leading to a regime where models are trained over finite datasets for multiple epochs. The study introduces masked-input regularization (MIR) to enhance autoregressive pretraining without altering the model architecture.
Re-Centering Humans in LLM Personalization
A recent study published on arXiv investigates the personalization capabilities of large language models (LLMs) using human data, revealing significant limitations compared to synthetic data. The research involved analyzing 550 human conversations and 5,949 judgments on user attributes, highlighting challenges in extracting relevant information and generating personalized responses.