HighlightCurated by Aramai Editorial

Inductive Representation Learning on Large Graphs

Presents GraphSAGE, an inductive framework that generates node embeddings by sampling and aggregating features from a node's local neighborhood.

Most node-embedding approaches are transductive, requiring all nodes present during training and failing to generalize to unseen nodes. GraphSAGE is a general inductive framework that leverages node feature information to generate embeddings for unseen data, learning a function that samples and aggregates features from a node's local neighborhood rather than training per-node embeddings. It outperforms strong baselines on three inductive node-classification benchmarks and generalizes to entirely unseen graphs in a protein-protein interaction dataset.

Based on: Inductive Representation Learning on Large Graphs · Neural Information Processing Systems

HighlightCurated by Aramai Editorial

LoRA: Low-Rank Adaptation of Large Language Models

Proposes LoRA, which freezes pre-trained weights and injects trainable low-rank decomposition matrices into Transformer layers for efficient adaptation.

Full fine-tuning of large pre-trained language models becomes impractical at scale — deploying independent fine-tuned instances of GPT-3 175B is prohibitively expensive. LoRA freezes pre-trained weights and injects trainable rank decomposition matrices into each Transformer layer, cutting trainable parameters for downstream tasks by 10,000x and GPU memory by 3x versus fine-tuning GPT-3 with Adam. LoRA matches or beats fine-tuning quality on RoBERTa, DeBERTa, GPT-2, and GPT-3, with higher training throughput and, unlike adapters, no added inference latency.

Based on: LoRA: Low-Rank Adaptation of Large Language Models · International Conference on Learning Representations

HighlightCurated by Aramai Editorial

LLaMA: Open and Efficient Foundation Language Models

Introduces LLaMA, foundation language models (7B-65B) trained solely on publicly available data, with LLaMA-13B outperforming GPT-3 on most benchmarks.

LLaMA is a collection of foundation language models ranging from 7B to 65B parameters, trained on trillions of tokens. The work shows that state-of-the-art models can be trained using publicly available datasets exclusively, without proprietary or inaccessible data. LLaMA-13B outperforms the 175B GPT-3 on most benchmarks, LLaMA-65B is competitive with Chinchilla-70B and PaLM-540B, and all models are released to the research community.

Based on: LLaMA: Open and Efficient Foundation Language Models · arXiv.org

HighlightCurated by Aramai Editorial

Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification

Proposes the PReLU activation and a rectifier-aware initialization, achieving 4.94% top-5 ImageNet error — first to surpass human-level performance.

This work studies rectifier neural networks for image classification from two angles. It proposes the Parametric Rectified Linear Unit (PReLU), which generalizes the rectified unit and improves fitting at nearly zero extra cost with little overfitting risk, and derives a robust initialization method accounting for rectifier nonlinearities, enabling very deep rectified models to be trained from scratch. The models achieve 4.94% top-5 test error on ImageNet 2012 — a 26% relative improvement over GoogLeNet (6.66%) and the first result surpassing reported human-level performance (5.1%).

Based on: Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification · IEEE International Conference on Computer Vision

HighlightCurated by Aramai Editorial

DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs

Introduces DeepLab for semantic segmentation, combining atrous convolution, atrous spatial pyramid pooling, and fully connected CRFs.

DeepLab addresses semantic image segmentation with three contributions. Atrous convolution controls the resolution of feature responses and enlarges filters' field of view without extra parameters or computation; atrous spatial pyramid pooling (ASPP) probes features at multiple sampling rates to capture objects and context at multiple scales; and a fully connected CRF on the final DCNN layer improves boundary localization. DeepLab set a new state of the art on PASCAL VOC-2012 (79.7% test mIOU) and advanced results on PASCAL-Context, PASCAL-Person-Part, and Cityscapes.

Based on: DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs · IEEE Transactions on Pattern Analysis and Machine Intelligence

HighlightCurated by Aramai Editorial

Statistical Learning Theory

A chapter on statistical machine learning covering SVMs, k-nearest neighbor, Naive Bayes, artificial neural networks, and instance-based learning.

Statistical machine learning systems learn from data rather than the environment in general. This chapter presents techniques including Support Vector Machines for pattern recognition and classification, SVM prediction of structured objects, the k-nearest neighbor method, and Naive Bayes classifiers. It also introduces artificial neural networks — error-correction rules, Boltzmann learning, the Hebbian rule, competitive learning, deep learning — treats instance-based learning in detail, and concludes with a summary and practice exercises.

Based on: Statistical Learning Theory · Technometrics

HighlightCurated by Aramai Editorial

Sequence to Sequence Learning with Neural Networks

Presents an end-to-end sequence-to-sequence learning approach using multilayered LSTMs to encode inputs to a fixed vector and decode target sequences.

Deep neural networks perform well on difficult tasks given large labeled training sets but cannot map sequences to sequences. This paper presents a general end-to-end sequence learning approach that uses a multilayered LSTM to encode the input sequence into a fixed-dimensional vector and a second deep LSTM to decode the target sequence. On WMT-14 English-to-French translation it reaches 34.8 BLEU versus 33.3 for a phrase-based SMT system, and 36.5 when reranking that system's 1000 hypotheses; reversing source word order markedly improved performance.

Based on: Sequence to Sequence Learning with Neural Networks · Neural Information Processing Systems

HighlightCurated by Aramai Editorial

Training language models to follow instructions with human feedback

Introduces InstructGPT: GPT-3 fine-tuned on demonstrations and human feedback rankings via RL to align language models with user intent.

Larger language models are not inherently better at following user intent and can produce untruthful, toxic, or unhelpful outputs. The authors align models by fine-tuning GPT-3 on labeler-written demonstrations, then further fine-tuning with reinforcement learning from human feedback using rankings of model outputs, producing InstructGPT. In human evaluations, outputs from the 1.3B-parameter InstructGPT are preferred over the 175B GPT-3 despite 100x fewer parameters, with improved truthfulness, less toxic generation, and minimal regressions on public NLP datasets.

Based on: Training language models to follow instructions with human feedback · Neural Information Processing Systems

HighlightCurated by Aramai Editorial

Explaining and Harnessing Adversarial Examples

Argues neural nets' adversarial vulnerability stems from their linear nature and derives a fast method to generate examples for adversarial training.

Machine learning models, including neural networks, consistently misclassify adversarial examples — inputs perturbed by small, worst-case changes that cause confident wrong answers. Countering earlier explanations based on nonlinearity and overfitting, the authors argue the primary cause is the models' linear nature, which explains why such examples generalize across architectures and training sets. This view yields a simple, fast method for generating adversarial examples; using them for adversarial training reduces the test error of a maxout network on MNIST.

Based on: Explaining and Harnessing Adversarial Examples · International Conference on Learning Representations

HighlightCurated by Aramai Editorial

A Theoretical Extension of the Technology Acceptance Model: Four Longitudinal Field Studies

Extends the Technology Acceptance Model (TAM2) with social influence and cognitive instrumental processes, validated in four longitudinal field studies.

Develops TAM2, an extension of the Technology Acceptance Model explaining perceived usefulness and usage intentions via social influence (subjective norm, voluntariness, image) and cognitive instrumental processes (job relevance, output quality, result demonstrability, ease of use). It was tested with longitudinal data on four systems at four organizations (N=156), spanning voluntary and mandatory usage at three measurement points. TAM2 was strongly supported, explaining 40-60% of the variance in usefulness perceptions and 34-52% of the variance in usage intentions.

Based on: A Theoretical Extension of the Technology Acceptance Model: Four Longitudinal Field Studies · Management Sciences

HighlightCurated by Aramai Editorial

LAMMPS - A flexible simulation tool for particle-based materials modeling at the atomic, meso, and continuum scales

Definitive reference for LAMMPS, the open-source molecular dynamics simulator for particle-based materials modeling from atomic to continuum scales.

Since its 2004 open-source release, LAMMPS has become a widely used tool for particle-based materials modeling from atomic to mesoscale to continuum length scales, growing from fifty thousand lines of code to a million. The paper describes several of its fundamental algorithms and the design strategies that keep it flexible for users and developers. It also highlights recently added capabilities, including dynamic load balancing, on-the-fly visualization, magnetic spin dynamics models, and quantum-accuracy machine learning interatomic potentials.

Based on: LAMMPS - A flexible simulation tool for particle-based materials modeling at the atomic, meso, and continuum scales · Computer Physics Communications

HighlightCurated by Aramai Editorial

Prototypical Networks for Few-shot Learning

Proposes Prototypical Networks for few-shot classification, learning a metric space where classes are predicted by distance to class prototypes.

Prototypical Networks address few-shot classification, where a classifier must generalize to classes unseen in training given only a few examples of each. They learn a metric space in which classification is performed by computing distances to prototype representations of each class, a simpler inductive bias than recent approaches that benefits the limited-data regime. Analysis shows simple design decisions yield substantial gains over approaches with complicated architectures and meta-learning, and a zero-shot extension achieves state-of-the-art results on CU-Birds.

Based on: Prototypical Networks for Few-shot Learning · Neural Information Processing Systems