New paper involving AWARE: https://www.sciencedirect.com/science/article/pii/S1071581926001539
Highlights
- Leveraged smartphone screen text to predict and explain user behaviours.
- Demonstrated interpretability of fine-tuned large language models (LLMs).
- Linked digital screen content with offline, real-world user activities.
- Developed a framework for explainable smartphone sensing research.
Abstract
Smartphones are essential to daily life, and their rich data streams have been used to study how people use their phones, and more broadly human behaviour. While previous research has largely focused on app usage and keystroke dynamics to predict smartphone use, these analyses are typically limited to making predictions rather than providing explanations or reasoning for observed behaviours. In this exploratory study, we investigate the potential of leveraging screen text and large language models (LLMs) to uncover insights and reasoning about user behaviour. Using a dataset of over 100 million on-screen words collected from 21 participants over two weeks, we explore multiple ways to use screen text and LLMs for three tasks: predicting the next app a user will open, inferring what real-world activities they are engaged in, and understanding how they interact within apps. Orthogonally, we demonstrate the interpretive capabilities of LLMs, highlighting their potential to explain the reasoning behind observed user actions. Our findings suggest that screen text holds promise for providing deeper insights into both digital and real-world human behaviour. We discuss the broader implications of our findings, including enhancing user experience and enabling privacy-preserving, on-device analysis, while proposing future research directions in screen text analysis.