This article discusses the unique difficulties in hardware design for large language model inference, particularly during the autoregressive Decode phase. It identifies memory and interconnect issues as primary challenges and proposes four research directions to improve performance, focusing on datacenter AI but also considering mobile applications.