AI Advice Depth and Breadth in Technical Customer Support

This study investigates the impact of integrating AI advice on technical customer support, a critical and resource-intensive function that minimizes delays and costs. The field experiment will be conducted with 54 technicians from a technical customer support team solving approximately 180 problems weekly, focusing on their problem-solving efficiency and effectiveness. Technicians currently face limitations in accessing stored knowledge and prefer varied advice styles, leading to unclear impacts on their problem-solving performance.
The research aims to demonstrate how AI advice influences problem-solving compared to existing practices. It involves two subexperiments in the field. In the first one, it examines different two AI chatbot response styles - concise and detailed - in a general chat not directly related to a specific problem and their effects on technician satisfaction, working modes and learning. In a second one, it examines four different chatbot settings - one specific or three suggestions based on two different information retrieval options - and their effects on resolution time, technician satisfaction, resolution quality, working modes and learning. The study further considers different problem types and technicians' backgrounds.
The interventions build on an AI system that integrates knowledge from internal databases, e.g., previously resolved incident, manuals, wikis, using retrieval-augmented generation. In the first experiment, via the system prompt two different AI chatbots that vary in their information depth - concise and detailed - will be randomly be available for each problem-solving task. The second experiment, directly connects problems to the internal databases and initiates the chat with suggestions of potential solutions randomly generated.
Previous studies in the same context and feedback from technicians brought up unclear implications on which AI advice style will be more helpful and effective. We expect different impacts on resolution time, chatbot interaction patterns and working modes depending on the collaboration with the different chatbot versions. We also expect different impact depending on the problem type and technician's experience.