Small Language Model Quantization: AWQ vs. GPTQ 4-Bit Efficiency and Perplexity Evaluation
Evaluating activation-aware weight quantization versus generalized post-training quantization on edge accelerators and consumer workstations.
Quantization Perplexity Estimator
Benchmark model degradation across bit-width reductions
Empirical Analysis & Findings
## Empirical Analysis of Post-Training Quantization (PTQ)
Running local inference on edge devices necessitates aggressive compression of model weights from 16-bit floating point down to 4-bit integers. Maintaining instruction-following fidelity requires quantization techniques that avoid degrading salient transformer attention layers.
### 1. Perplexity and Memory Comparison on Llama-3.1-8B
Evaluated on Wikitext-2 and GSM8k reasoning benchmarks using an Apple M3 Max (36GB Unified Memory):
| Quantization Method | Bit Width | Model Size (GB) | Wikitext-2 Perplexity | GSM8k Accuracy | Memory Bandwidth (GB/s) |
|:-------------------|:---------:|:---------------:|:---------------------:|:--------------:|:-----------------------:|
| FP16 Baseline | 16-bit | 15.8 GB | 6.14 | 82.4% | 280.0 |
| AWQ (GEMM) | 4-bit | 4.9 GB | 6.32 | 80.9% | 148.5 |
| GPTQ (ActOrder) | 4-bit | 4.9 GB | 6.48 | 79.1% | 152.0 |
| GGUF (Q4_K_M) | 4.5-bit | 5.3 GB | 6.28 | 81.2% | 155.2 |
Linked Entities in the Autonomous Knowledge Graph
Frequently Asked Questions
Why does AWQ outperform GPTQ on smaller models (under 7B parameters)?
AWQ identifies salient weight channels by observing activation distributions, preserving the top 1% critical weights in FP16 to prevent catastrophic perplexity spikes.
Can 4-bit quantized models run concurrently in 16GB unified memory?
Yes. A 4-bit AWQ 8B parameter model consumes approximately 5.2 GB of VRAM, leaving over 10 GB for KV cache context windows and OS operations.
Related Research & Briefs
Production RAG Architecture: Dense Retrieval, HyDE, and Cross-Encoder Re-Ranking
Architecting low-latency Retrieval-Augmented Generation systems with hybrid BM25 search, Hypothetical Document Embeddings, and cross-encoders.
Generative Engine Optimization (GEO): LLMs.txt Manifests, Entity Graph Topology, and AI Search Citations
Empirical analysis of Generative Engine Optimization (GEO) strategies, measuring citation frequency in ChatGPT, Claude, and Perplexity using standardized llms.txt manifests and Schema.org JSON-LD graphs.
LLM Speculative Decoding: Throughput Benchmarks, Draft Model Ratios, and GPU Memory Bandwidth
Empirical benchmarks comparing speculative decoding performance across vLLM, TensorRT-LLM, and SGLang using Llama-3.3-70B and quantized draft models.
Vector Database Benchmarks: HNSW vs. IVF-PQ Index Scalability, Recall, and Memory Footprint
A deep empirical comparison of Hierarchical Navigable Small World graphs versus Inverted File Product Quantization across 100M vector workloads.