Transformers process prompts as a flat sequence of tokens, which can lead to loss of meaningful relationships, especially with structured text. RePo introduces a learned module that assigns each token a real-valued position based on its semantics. This approach helps preserve important relationships that traditional methods overlook. The authors demonstrate that RePo improves performance across various tasks, particularly in noisy contexts, structured data, and long contexts, addressing limitations posed by rigid positional structures like RoPE and NoPE. The article highlights that cognitive load affects model performance, similar to human cognitive processes. By allowing the model to reshape context geometry for attention, RePo enables better focus on relevant information, even when it’s distanced from the current token. In experiments with OLMo-2, the authors found RePo consistently outperformed standard methods. For example, in noisy context evaluations, RePo scored 55.68 on average, surpassing RoPE by 11.04 points. In scenarios where structured inputs are flattened into text, preserving relational integrity becomes challenging. RePo shows a slight edge over RoPE with an average improvement of 1.94 EM points in structured data tasks. Notably, NoPE performed best on the graph dataset, indicating that traditional positional strategies may not fit well with graph-structured inputs. This suggests that while RePo enhances the model's ability to manage context, the effectiveness of positional encoding can vary based on the structure of the input data.