A Pilot Study to Evaluate Success Criteria for Symbiotic AI

Artificial Intelligence (AI) is transforming humans’ activities through continuous collaboration. In a context where AI is becoming pervasive, the focus is shifting toward the creation of Symbiotic AI (SAI) systems that support humans rather than replace them allowing them to intervene on system’s behavior while providing explanations. This contribution presents a set of Success Criteria (SC) that can be used to evaluate SAI systems by assessing whether the requirements of symbiosis are met. This work also presents the protocol for the user study to validate the SC in real-world contexts. To avoid potential issues that can affect the validity of the user study, a pilot survey was performed involving 5 participants in which they were asked to assess the SC based on three metrics (Comprehensibility, Relevance, and Feasibility) with respect to an AI-based system. According to the preliminary results, the SC achieved positive ratings, but the protocol requires refinements since participants often prioritized the usability of the system over the assessment of the SCs.