Transformers vs. Recurrent Models for Estimating Forest Gross Primary Production

The paper introduces AdamHuberDecay, a new adaptive optimizer designed for pre-training large transformer-based generative models. This optimizer replaces the traditional $ ext{l}_2$ penalty used in AdamW with a decoupled smooth Huber regularizer. The proposed method allows for quadratic decay of parameters below a certain threshold while applying linear decay for larger values. This approach aims to improve regularization gradients, maintain invariance to second-moment rescaling, and enhance sparsity for overgrown weights.

This study investigates the performance of 3D Convolutional Neural Networks (3D CNNs) and Long Short-Term Memory (LSTM) networks for real-time American Sign Language (ASL) recognition. The evaluation is based on a dataset of 1,200 ASL signs across 50 classes, focusing on accuracy, computational efficiency, and latency. Results show that 3D CNNs achieve 92.4% recognition accuracy but require more processing time per frame compared to LSTMs, which maintain 86.7% accuracy with lower resource consumption. A hybrid model demonstrates decent performance, highlighting the importance of architecture s…

The article discusses advancements in probabilistic forecasting, emphasizing the limitations of point predictions in capturing sudden changes in time series data. It introduces a novel approach using Deep Lattice Networks (DLN) to forecast non-parametric cumulative distribution functions (CDFs). This method aims to enhance the accuracy of forecasts by allowing for implicit quantile regression, addressing the historical reliance on parametric models in CDF forecasting.

Mental health issues pose a significant global socioeconomic challenge, worsened by COVID-19, which has increased the demand for telehealth services. While large language models (LLMs) like GPT-2 provide potential solutions through constant availability and non-judgmental interactions, they often lack the necessary contextual and emotional awareness for effective therapeutic dialogue. This study explores the use of supervised fine-tuning and reinforcement learning to improve GPT-2's ability to generate therapeutic conversations by processing contextual information and emotional states simultan…

The accurate retrieval of vegetation biophysical variables from satellite imagery is essential for ecosystem monitoring and agricultural management. This study introduces a physics-informed Transformer-VAE architecture designed to invert the PROSAIL radiative transfer model, enabling the simultaneous estimation of key canopy parameters from Sentinel-2 data. The model, trained solely on simulated data, demonstrates performance comparable to state-of-the-art methods that utilize real imagery, successfully retrieving leaf area index and canopy chlorophyll content from real-world datasets.

This paper presents a transformer-based approach for classifying hope expressions in text. It compares three architectures: BERT, GPT-2, and DeBERTa, for binary classification (Hope vs. Not Hope) and multiclass categorization (five hope-related categories). The initial BERT implementation achieved 83.65% binary and 74.87% multiclass accuracy. BERT outperformed others in extended comparisons, requiring fewer resources. GPT-2 had the lowest accuracy, while DeBERTa showed moderate results but at a higher computational cost. Error analysis highlighted architecture-specific strengths.