AI Vendor Performance Assessment in the Mining Industry: A Monte Carlo and LLM Approach

Abstract

This paper examines the methodologies for assessing AI vendor capabilities in the context of Roy Hill, a prominent iron ore mining company in Western Australia. Through the use of Monte Carlo simulations and Large Language Models (LLMs), including innovative techniques like the Multi-Persona LLM (MP-LLM), we evaluate potential AI vendors to identify those that align with Roy Hill's strategic and operational goals. A robust vendor evaluation framework was developed, integrating survey data with independent assessments of LLMs and their products. The MP-LLM framework was specifically tested for its problem-solving ability and demonstrated enhanced performance when combined with tailored prompt engineering and curated personas. To mitigate challenges such as information asymmetry and confirmation bias, vendor feedback was incorporated, and evaluation metrics were refined using both LLMs and Monte Carlo simulations. The study contributes to AI vendor selection methodologies in the mining sector, emphasizing the importance of adaptive strategies in a rapidly changing technological landscape. Future work will explore advanced techniques such as knowledge graphs and expanded persona libraries to improve AI capability assessments and support operational excellence across industries.