Transformer Based Bayesian Network Structure Learning from an Information Theory Perspective

The pivotal challenge of handling uncertainty in artificial intelligence has prompted a growing focus on learning {Bayesian} network ({BN}) structures from data in recent years. Nevertheless, most of the existing methods such as hill-climbing, variational auto-encoder and graph neural network-based algorithms continue to encounter the issues like local optimum and exponential computational complexity, etc. In response to these challenges, we propose a Transformer-based approach for learning the {BN} structure (T-{BNSL}) from an information theory perspective. Specifically, we first establish the graph skeleton of a {BN} by employing conditional independence tests grounded in mutual information. Subsequently, we propose a hill-climbing method with decomposable BIC scoring function to generate potential directed acyclic graphs (DAGs) and evaluate the BIC scores of a subset of these DAGs. Lastly, we use the Transformer framework with a refined attention module to predict the BIC scores of the remaining DAGs, enabling the efficient identification of the {DAG} with the highest score. Experimental findings demonstrate that our approach significantly surpasses state-of-the-art competitors in terms of accuracy and efficiency by several orders of magnitude when learning the {BN} structure.