Web literacy, and specifically the ability to discern website credibility, is increasingly important in the age of highly accessible information on the Internet. While credibility checking tools exist, they often require the user to be proactive about checking potentially suspicious websites. CredBot is a Chrome extension chatbot that automates the credibility-checking process. It presents the credibility of a website with credibility signals and also warns about sensitive topics or low-credibility websites. Users are able to converse with the chatbot and understand the reasoning behind credibility measures. This version of CredBot was iteratively created until it achieved moderate capabilities. A working prototype was then tested on its effectiveness for users in a pilot study. Overall, the promising findings suggest that with refinement to the underlying prompt engineering powering the chatbot and continual design iterations, CredBot will be able to improve users’ web literacy when searching the web.
With factors such as biased perspectives or deeply embedded ads that undermine the reliability and trustworthiness of its content, it is difficult to discern website credibility. The average web user often expects a streamlined search query process, but existing methods of verifying a website’s credibility are cumbersome and only employed at the user’s discretion [8]. Because many users don’t have a firm understanding on what makes a website credible, if the credibility checking tool relies on the user proactively entering a website they find suspicious, all websites users falsely deem trustworthy escape scrutiny.
To solve these issues, we propose a conversational user interface Chrome extension that automates and customizes the credibility-checking process as a latent assistant [4]. Our artifact will primarily detect when users search up sensitive topics (or topics prone to misinformation) or websites with low credibility, and provide credibility warnings to deter the consumption of misinformation and converse with the user and provide the reasoning behind warnings or the credibility measures chosen based on Wellesley’s CredLab research [1]. Our tool serves as education-on-demand on the credibility and trustworthiness of websites. Those who surf the web can benefit through our tool that prompts consistent consideration of credibility criterias that determine a website’s trustworthiness. The customization and non-intrusive design of our tool additionally places more control into the user without placing burdens on the user to check website credibility, thus encouraging user-retention, trust in our tool, and the continuous development of web literacy [5, 7, 9].
CredBot can be compared to existing credibility tools and chatbots for its functions and design. Bing Chat is a similar chatbot embedded in the web-browsing experience for users [3]. While Bing Chat is a general companion for search engines and would require specific prompting to evaluate a website’s credibility, it similarly is able to answer questions about the page the user is on. NewsGuard likewise provides credibility ratings, specifically for news and information websites [6]. Their ratings were created by “trained journalists'' based on nine journalistic criteria, for which each site received a trust score and “Nutrition Label” review. While CredBot will not generate a credibility score for users, as we hope to encourage conversations with the chatbot for development of web literacy, CredBot will produce a credibility evaluation in a manner similar to NewsGuard’s nutrition label. CredBot builds upon the credibility-checking capabilities of existing tools such as NewsGuard in an innovative chatbot Chrome extension format similar to Bing Chat in order to serve as a helpful educational tool in the web-searching process.
By evaluating the effectiveness of a developing tool, such as CredBot, which utilizes LLM automation and customizes its evaluation of a website’s credibility according to credibility signal criteria for users, we aim to create a tool which is user-friendly and non-intrusive without compromising its effectiveness [7, 9]. A vision of a future exists in which LLM will be integrated into the way we surf the web [2]. Our study provides key insights for this vision through demonstration of how the LLM integration can function as both an educational resource and safety measure as users navigate an increasingly complex web.
JavaScript is the backbone of the chatbot and was used because of it being a standard for developing Chrome extensions. It enables dynamic content manipulation and user interaction handling within the browser environment. The scripts chatbot4.js and contentScript.js primarily use JavaScript for scripting behavior, handling events, and interacting with web pages, and are responsible for essential functionalities in the extension such as responding to user actions, manipulating web page content, and communicating with external APIs like OpenAI's GPT-4. For the structure of the chatbot interface, HTML was used in conjunction with CSS following the format of our low-fidelity sketch. This integration of HTML and CSS ensures the chatbot is not only functional but also aesthetically appealing and user-friendly. Upon initialization, our extension injects the chatbot interface directly into the web page's DOM. This process involves creating and appending HTML elements, styling them with CSS, and activating JavaScript event listeners.
The extension integrates with OpenAI's GPT-4 API, a crucial element for enabling advanced natural language processing and machine learning capabilities. This integration allows the chatbot to generate intelligent, context-aware responses and conduct credible website evaluations. The API's sophisticated algorithms analyze user inputs and website content, aiding in delivering accurate and relevant information in response to user queries. Rrapid iteration was needed for the integration of GPT-4 to have the correct capabilities for Credbot. The first step was ensuring that OpenAI’s API was successfully connected to our chrome extension, and that it was generating context-aware responses through GPT-4’s framework. However, this version of the chatbot did not know the URL of the website it was on, as the public version of OpenAI's API, including ChatGPT, does not currently have the capability to browse the internet. Therefore, for the second iteration, the URL of the website the user was currently on was retrieved and passed as a user prompt to our chatbot. Because the public version of OpenAI’s API cannot perform live searches and access internet content from a URL, the inner text of the web page also had to be passed to our chatbot in order for it to perform analyses on its contents.
Manifested in the manifest.json file, the extension is tailored for Chrome browsers, following the Manifest V3 structure. This structure governs the extension's permissions, ensuring secure access to necessary resources like active tabs and external APIs. It also manages the injection of scripts and resources into web pages, facilitating the seamless integration of the chatbot into the user's browsing experience without disrupting the functionality of the web pages.
Prompt engineering, in the context of this chatbot's integration with OpenAI's GPT-4 API, is a critical aspect of the technology overview. For CredBot, the system prompt informed the GPT model that “You are a professional assistant meant to help users determine the credibility of the website they are currently on” and gave specific instructions on how to format and generate their credibility evaluation based. These evaluations are based on the url link and webpage’s inner html that is fed to the model as a user prompt.
CredBot activates based on URL patterns, ensuring it does not interfere with or load on websites such as personal emails. The Chrome extension will appear on the bottom right corner of the webpage as a permanent pop-up window. Once CredBot evaluates the webpage’s credibility, the pop-up window notifies the user and invites them to view CredBot’s results with a “see more” button. If the webpage was deemed to have low credibility, a red warning banner will accompany this notification, and if it was deemed to have medium credibility, a yellow warning banner will accompany this notification.
The chatbot is equipped with a feature to detect and warn users about whether or not medical information is present on a webpage, due to its sensitive nature. By scanning for specific keywords and phrases related to health and medicine, the chatbot can alert users to take caution and advise consulting healthcare professionals for accurate information. It is presented as a blue warning banner at the top of the pop-up window once the credibility evaluation is generated. This feature enhances the chatbot's utility in safeguarding user health and safety.
User interaction with the chatbot is facilitated through a text input field and a send button. These elements capture and process user messages, which are then displayed within the chat interface. JavaScript's event handling mechanisms monitor these inputs, ensuring that user queries are promptly captured and responded to. In connection to the OpenAI API, JavaScript’s asynchronous communication patterns, like fetch API and promises, are employed to send requests and handle responses. This design enables an interactive and continuous dialogue between the user and the chatbot.
One of the chatbot's primary functions is to assess the credibility of websites. This is achieved by analyzing website content against predefined criteria developed by Professor Eni Mustafaraj’s Credibility Lab, including disclosure of authorship, disclosure of ownership, volume of ads, volume of promoted content, and type of organization (non-profit, for-profit, governmental etc.) [1]. The chatbot then displays an assessment message, categorized as low, medium, or high credibility, informing users about the potential trustworthiness of the content they are viewing, according to these five credibility signals.
The chatbot's aesthetic appeal and responsive design are achieved through CSS styling. The design ensures that the chatbot interface is not only visually pleasing but also adaptable to various screen sizes and user interactions. The use of modern CSS features like flexbox and media queries contributes to a fluid and responsive user experience, accommodating a wide range of devices and display environments.
Future work will focus on enhancing CredBot's user interface, making it more aesthetically pleasing and intuitive. This improvement aims to foster greater user engagement and satisfaction. Assessing the consistency of CredBot's responses is also a crucial step forward, ensuring reliability and trustworthiness in its assessments. With more intensive prompt engineering, we will eliminate the need to hard-code credibility evaluations for websites and specify exactly how CredBot should apply credibility signal criteria for a website’s credibility evaluation. Furthermore, with more robust parsing of the innerHTML of a webpage, we may be able to systemically pass in information regarding ads, hyperlinks, and text content beyond the current passing of only the inner text. A long-term study with a diverse user base is envisaged to gain deeper insights into CredBot's long-term usability and effectiveness. Such a study will be invaluable in understanding the evolving dynamics of user interaction with AI tools for information assessment. Further, enhancing the AI model's accuracy and addressing emerging privacy concerns will be pivotal in ensuring CredBot's relevance and efficacy in real-world scenarios.
[1] Ace Wang, Liz Maylin De Jesus Sanchez, Anya Wintner, Yuanxin Zhu, and Eni Mustafaraj. 2023. Assessing Google Search’s New Features in Supporting Credibility judgments of Unknown Websites. In Proceedings of the 2023 Conference on Human Information Interaction and Retrieval (CHIIR '23). Association for Computing Machinery, New York, NY, USA, 303–307. https://doi-org.ezproxy.wellesley.edu/10.1145/3576840.3578277
[2] Azeem Azhar. 2023. Azeem on AI: Are Large Language Models the Future of the Web? Podcast. (24 May 2023. Retrieved October 15, 2023 from https://hbr.org/podcast/2023/05/azeem-on-ai-are-large-language-models-the-future-of-the-web
[3] Bing. [n. d.]. Bing Chat. https://www.microsoft.com/en-us/edge/features/bing-chat?form=MA13FJ
[4] Demetris Paschalides, Alexandros Kornilakis, Chrysovalantis Christodoulou, Rafael Andreou, George Pallis, Marios Dikaiakos, and Evangelos Markatos. 2019. Check-It: A plugin for Detecting and Reducing the Spread of Fake News and Misinformation on the Web. In IEEE/WIC/ACM International Conference on Web Intelligence (WI '19). Association for Computing Machinery, New York, NY, USA, 298–302. https://doi-org.ezproxy.wellesley.edu/10.1145/3350546.3352534
[5] Farnaz Jahanbakhsh, Yannis Katsis, Dakuo Wang, Lucian Popa, and Michael Muller. 2023. Exploring the Use of Personalized AI for Identifying Misinformation on Social Media. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (CHI '23). Association for Computing Machinery, New York, NY, USA, Article 105, 1–27. https://doi-org.ezproxy.wellesley.edu/10.1145/3544548.3581219
[6] NewsGuard Ratings. [n. d.]. NewsGuard. https://www.newsguardtech.com/solutions/newsguard/
[7] Pavel Smutny and Petra Schreiberova. Chatbots for learning: A Review of educational chatbots for the Facebook Messenger. Computers & Education 151 (2020), 103862. https://doi.org/10.1016/j.compedu.2020.103862
[8] The Mind Online. 2019. (May 2019). Fake News: Finding It, Fighting It. Podcast. Retrieved October 15, 2023 from https://open.spotify.com/episode/5cAKhM3YlJryRUxLBL3lvS?si=536a97ffc60a43fa
[9] The Radical AI Podcast. 2020. Emoji Design, White Accountability, and the Ethical Future of Chatbots with Miriam Sweeney. Podcast. (June 2020). Retrieved October 15, 2023 from https://open.spotify.com/episode/48DjWEX2tf2nIPvvWZvtAC?si=9632e3cfd5034808