Highlight

Swin-Unet: Unet-like Pure Transformer for Medical Image Segmentation

Swin-Unet is a pure Transformer with a U-shaped encoder-decoder and skip connections for medical image segmentation.

Swin-Unet is a pure Transformer with a U-shaped encoder-decoder architecture for medical image segmentation, addressing the limited global context of convolutional networks. Tokenized image patches pass through a hierarchical Swin Transformer encoder with shifted windows, and a symmetric decoder with patch-expanding layers restores resolution, linked by skip connections for local-global feature learning. On multi-organ and cardiac segmentation tasks, this convolution-free design outperforms methods based on full convolution or hybrid transformer-convolution architectures.

Based on: Swin-Unet: Unet-like Pure Transformer for Medical Image Segmentation · ECCV Workshops

Curated by Aramai Editorial

Read summary →
Highlight

Flow Matching for Generative Modeling

Flow Matching introduces a simulation-free method to train continuous normalizing flows by regressing vector fields of conditional probability paths.

Flow Matching is a simulation-free method for training Continuous Normalizing Flows (CNFs) by regressing vector fields along fixed conditional probability paths. It covers a general family of Gaussian paths subsuming diffusion paths, and using diffusion paths yields more robust, stable training. Using Optimal Transport displacement interpolation instead gives more efficient paths with faster training and sampling and better generalization. On ImageNet it beats diffusion baselines in likelihood and sample quality while enabling fast sampling with standard ODE solvers.

Based on: Flow Matching for Generative Modeling · International Conference on Learning Representations

Curated by Aramai Editorial

Read summary →
Highlight

In-datacenter performance analysis of a tensor processing unit

In-datacenter evaluation of Google's Tensor Processing Unit, a custom ASIC for neural network inference, versus contemporary CPUs and GPUs.

This paper evaluates the Tensor Processing Unit (TPU), a custom ASIC deployed in Google datacenters since 2015 to accelerate neural network inference. Built around a 65,536 8-bit MAC matrix-multiply unit delivering 92 TeraOps/s and 28 MiB of on-chip memory, its deterministic execution suits strict 99th-percentile latency needs better than CPUs or GPUs. Running production TensorFlow workloads (MLPs, CNNs, LSTMs), the TPU averaged 15-30X faster than a contemporary Haswell CPU or K80 GPU, with 30-80X better performance per watt.

Based on: In-datacenter performance analysis of a tensor processing unit · International Symposium on Computer Architecture

Curated by Aramai Editorial

Read summary →
Highlight

The I-TASSER Suite: protein structure and function prediction

The I-TASSER Suite predicts protein 3D structure and function via threading, structure clustering, and template-based ligand-binding annotation.

The I-TASSER Suite predicts protein structure and function through an integrated pipeline. Low-energy conformations are found by structure clustering, then refined via reassembly simulation and atomic-level energy minimization, with quality scored by confidence measures and the new ResQ method. Function is annotated by matching models to the BioLiP database to infer ligand-binding sites, EC numbers, and GO terms via COFACTOR, TM-SITE, and S-SITE. In CASP10 and CAMEO tests it produced correct folds and accurate ligand-binding predictions.

Based on: The I-TASSER Suite: protein structure and function prediction · Nature Methods

Curated by Aramai Editorial

Read summary →
Highlight

OPTICS: ordering points to identify the clustering structure

Introduces OPTICS, which builds an augmented ordering of a database representing its density-based clustering structure across a broad range of parameters.

Cluster analysis is a key database-mining method, but nearly all clustering algorithms need input parameters that are hard to set yet strongly affect results, with no global setting capturing intrinsic structure. Instead of an explicit clustering, OPTICS builds an augmented ordering of the database representing its density-based clustering structure, encoding density-based clusterings across a broad range of parameters. It supports automatic and interactive analysis, extracting representative points and arbitrary-shaped clusters, with visualizations for large datasets.

Based on: OPTICS: ordering points to identify the clustering structure · SIGMOD Conference

Curated by Aramai Editorial

Read summary →
Highlight

QLoRA: Efficient Finetuning of Quantized LLMs

Presents QLoRA, finetuning a 4-bit quantized LLM through LoRA adapters to train a 65B model on one 48GB GPU while preserving 16-bit performance.

QLoRA is an efficient finetuning method that cuts memory enough to tune a 65B model on one 48GB GPU while matching 16-bit performance. It backpropagates gradients through a frozen, 4-bit quantized model into Low-Rank Adapters (LoRA) via 4-bit NormalFloat (NF4), an information-theoretically optimal type for normal weights; double quantization; and paged optimizers for memory spikes. Guanaco models reach 99.3% of ChatGPT on Vicuna after 24 hours of tuning. Across 1,000+ models, data quality is shown to outweigh scale, and chatbot benchmarks prove unreliable.

Based on: QLoRA: Efficient Finetuning of Quantized LLMs · Neural Information Processing Systems

Curated by Aramai Editorial

Read summary →
Highlight

On Pixel-Wise Explanations for Non-Linear Classifier Decisions by Layer-Wise Relevance Propagation

Proposes layer-wise relevance propagation, decomposing nonlinear classifier decisions pixel-wise into heatmaps to explain image classifications.

Interpreting automated image classifiers helps verify their reasoning, yet most machine-learning methods act as black boxes. This work proposes a general method for understanding classification decisions via pixel-wise decomposition of nonlinear classifiers. It visualizes each pixel's contribution to predictions for both kernel-based classifiers over Bag-of-Words features and multilayered neural networks, as heatmaps that let experts verify decisions and focus analysis. It is evaluated on PASCAL VOC 2009, synthetic shapes, MNIST, and a pre-trained ImageNet model.

Based on: On Pixel-Wise Explanations for Non-Linear Classifier Decisions by Layer-Wise Relevance Propagation · PLoS ONE

Curated by Aramai Editorial

Read summary →
Highlight

Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer

Introduces the Sparsely-Gated Mixture-of-Experts layer, scaling model capacity over 1000x via conditional computation and a trainable gating network.

A network's capacity is bounded by its parameter count; conditional computation, activating only parts of a network per example, promises greater capacity without proportional compute but faces algorithmic and performance obstacles. This work realizes it with a Sparsely-Gated Mixture-of-Experts (MoE) layer of thousands of feed-forward sub-networks, where a trainable gating network picks a sparse combination per example. Applied between stacked LSTMs with up to 137B parameters, it yields over 1000x capacity gains and beats state-of-the-art LM and translation at lower cost.

Based on: Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer · International Conference on Learning Representations

Curated by Aramai Editorial

Read summary →
Highlight

Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecasting

Introduces Autoformer, a decomposition Transformer with an Auto-Correlation mechanism for accurate, efficient long-term time series forecasting.

Autoformer targets long-term time series forecasting, where intricate future patterns hinder Transformer self-attention and sparse point-wise attention causes an information bottleneck. It is a decomposition architecture that makes series decomposition a basic inner block with progressive capacity. Its Auto-Correlation mechanism exploits series periodicity to discover dependencies and aggregate representations at the sub-series level, beating self-attention in efficiency and accuracy. Autoformer reaches state-of-the-art accuracy with a 38% relative gain on six benchmarks.

Based on: Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecasting · Neural Information Processing Systems

Curated by Aramai Editorial

Read summary →
Highlight

Survey of Hallucination in Natural Language Generation

Surveys hallucination in natural language generation: metrics, mitigation, and progress across summarization, dialogue, QA, and machine translation.

Natural language generation has advanced rapidly with sequence-to-sequence and Transformer-based language models, yielding more fluent output for tasks like summarization, dialogue, and data-to-text. However, such models often hallucinate unintended text, degrading performance and user trust. This survey comprehensively reviews hallucination in NLG in two parts: a general overview of metrics, mitigation methods, and future directions; and task-specific progress in abstractive summarization, dialogue generation, generative QA, data-to-text, and machine translation.

Based on: Survey of Hallucination in Natural Language Generation · ACM Computing Surveys

Curated by Aramai Editorial

Read summary →
Highlight

DARTS: Differentiable Architecture Search

Introduces DARTS, a differentiable neural architecture search that relaxes the search space to be continuous, enabling efficient gradient-based search.

DARTS tackles the scalability of neural architecture search by casting the task in a differentiable form. Instead of evolution or reinforcement learning over a discrete, non-differentiable space, it uses a continuous relaxation of the architecture representation so search runs efficiently via gradient descent. Experiments on CIFAR-10, ImageNet, Penn Treebank, and WikiText-2 show it finds high-performance convolutional architectures for image classification and recurrent ones for language modeling, orders of magnitude faster than non-differentiable methods.

Based on: DARTS: Differentiable Architecture Search · International Conference on Learning Representations

Curated by Aramai Editorial

Read summary →
Highlight

SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis

Presents SDXL, a latent diffusion text-to-image model with a 3x larger UNet, dual text encoders, and a refinement model for higher-fidelity synthesis.

SDXL is a latent diffusion model for text-to-image synthesis that scales up Stable Diffusion with a UNet backbone three times larger, driven by more attention blocks, a larger cross-attention context, and a second text encoder. It adds novel conditioning schemes, training on multiple aspect ratios, and a refinement model that boosts visual fidelity via a post-hoc image-to-image step. SDXL substantially outperforms prior Stable Diffusion versions and rivals black-box state-of-the-art generators. Code and model weights are released openly.

Based on: SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis · International Conference on Learning Representations

Curated by Aramai Editorial

Read summary →