AI Research - Lead Researcher

     The current project we are working on at User Behavioristics, Inc. is a Comparative AI Study.  As we have access to a lot of high-quality research data due to running multiple mock studies, we could use it to compare professional researcher feedback and AI insights. As of this moment, we have run a Pilot study that only uses one AI, but we plan to expand to 5 of them for the full-scale study. The purpose of the study is to examine the strengths and weaknesses of the AI output and determine where it would be most useful for human researchers.  Both the AI and the human team started with the same initial data - 20 playtest recordings and survey results. 

    After conducting the study, we came to several interesting conclusions about the weaknesses and strengths of the AI agent. The main issue present in the AI video analysis was the software’s inability to pick up on the footage and detect non-verbal behavior. The only time the AI did so successfully was when the player's speech matched their nonverbal actions, such as a player saying, "I am about to go through the gate." Another big issue that often arose when using the AI was the lack of focus on the most important details. As the AI used the entire transcript as input, all player feedback was deemed equally important, which prevented it from focusing on critical issues. 

    Still, there was certain functionality that the software provided that would be extremely useful for a UX researcher. Despite not always being 100 percent accurate, transcriptions can provide an easily searchable script of the playtest, allowing the researcher to easily find the correct moment in the video. Lastly, the speed at which the AI performs its analysis is significantly faster than human research, taking less than an hour instead of over 1 week of a standard User Behavioristics study.  This shows that AI is best at time-consuming, monotonous work, allowing researchers to save valuable time to focus on creative insights that AI is not as good at.