Use of Expert Judgments to Inform Bayesian Models of Insider Threat Risk
To promote effective detection and mitigation of insider threats, research has sought to identify, validate, and integrate cyber and behavioral (sociotechnical) indicators into comprehensive models of insider threat risk. Because validation of proposed indicators is hampered by a lack of appropriate real-world data, innovative approaches have used expert judgments as an initial step in developing and evaluating threat assessment models. For probabilistic models such as Bayesian networks, assigning probability values to posterior evidence is particularly challenging because it often relies on subjective base-rate (prior) and conditional probabilities estimates that are difficult to obtain and fraught with human errors and biases. The purpose of the present study was to test the efficacy of an expert knowledge elicitation method that does not rely on probability judgments in supporting development of probabilistic as well as non-probabilistic/risk-based predictive models of insider threat. We compared previously obtained expert judgments of threat/risk levels for a large set of indicators within a comprehensive ontology of technical and behavioral indicators of insider threats with corresponding likelihood ratio estimates that we obtained in the present study, concluding that the observed high correlation between the risk versus probability judgments demonstrates the efficacy of acquiring expert judgments of threat/risk levels as a practical alternative to the difficult and unreliable methods of acquiring conditional probability estimates from human experts. Based on these results, we created a Bayesian model of insider threat that incorporates all (200) individual factors specified in the ontology and compared the performance of the Bayesian and risk-based models in predicting the judgments of experts, as proxies for real data and ground truth. Results indicated that the Bayesian model performed slightly better than a risk-based model that had been proposed and examined in prior research. This research demonstrated benefits of cross-fertilization of methods used in developing non-probabilistic/risk-based and probabilistic models in the insider threat domain. Implications of these findings for advancing insider threat predictive analytics, and future research needs, are discussed.