Latte is a framework designed for collaborative test-time adaptation of vision-language models within federated learning environments. It allows each client to maintain a local memory of historical test data and share class prototypes with similar clients, enhancing model performance while addressing the challenges of limited test data and preserving personalization. Experimental results demonstrate Latte's effectiveness in improving adaptability and performance in decentralized settings with minimal communication costs.