Evaluating Credibility and Political Bias in LLMs for News Outlets in Bangladesh

AbstractLarge language models (LLMs) are widelyused in search engines to provide direct an-swers, while AI chatbots retrieve updated infor-mation from the web. As these systems influ-ence how billions access information, evaluat-ing the credibility of news outlets has becomecrucial. We audit nine LLMs from OpenAI,Google, and Meta to assess their ability to eval-uate the credibility and political bias of the top20 most popular news outlets in Bangladesh.While most LLMs rate the tested outlets, largermodels often refuse to rate sources due to in-sufficient information, while smaller modelsare more prone to hallucinations. We create adataset of credibility ratings and political iden-tities based on journalism experts’ opinions andcompare these with LLM responses. We findstrong internal consistency in LLM credibil-ity ratings, with an average correlation coeffi-cient (ρ) of 0.72, but moderate alignment withexpert evaluations, with an average ρ of 0.45.Most LLMs (GPT-4, GPT-4o-mini, Llama 3.3,Llama-3.1-70B, Llama 3.1 8B, and Gemini 1.5Pro) in their default configurations favor theleft-leaning Bangladesh Awami League, givinghigher credibility ratings, and show misalign-ment with human experts. These findings high-light the significant role of LLMs in shapingnews and political information