Modern techniques have emerged since the original "Attention Is All You Need" paper to optimize transformer architectures, focusing on reducing memory usage and computational costs during inference. Key advancements include Group Query Attention, Multi-head Latent Attention, and various architectural innovations that enhance performance without significantly compromising quality. These methods aim to improve the efficiency of large models in practical applications.

NN-Former introduces a novel approach to neural architecture representation by combining the strengths of Graph Neural Networks and transformers while addressing their limitations. It emphasizes the importance of sibling nodes in the architecture topology and proposes new mechanisms for predicting accuracy and latency, achieving improved performance in learning Directed Acyclic Graph topology.