Undiksha Virtual Assistant (Shavira) : Integration Frequency Asked Question with Rasa Framework

Nowadays, the implementation of chatbots in information systems is getting popular to increase user experience. In our previous research, we developed Undiksha Virtual Assistant (Shavira). Shavira is a knowledge management system (KMS) as Frequency Asked Question (FAQ) system to facilitate users to get an answer about their questions. The challenge on the existing Shavira system is that the users must open a website and type queries to get a list of answers. The text query must match with the keyword in systems. To address this problem and to avoid cognitive load, in this research we implemented a chatbot in Shavira. The purpose of the chatbot implementation in Shavira is to make information easier to be accessed and to increase user experience. The users can ask a question in natural language and get an answer immediately on their chat application such as: Telegram, Facebook, or Messenger. We used Rasa Framework as a chatbot engine in Shavira. Rasa Framework is an open-source virtual assistant engine based on artificial intelligence. The challenge in conducting this research is in integrating the existing FAQ system with Rasa Framework. This research consists of four phases, they are: adaptation data structure in the existing FAQ with data structure requested by Rasa framework, data integration and mapper, and evaluation. The result of this research is a new Shavira that is integrated with a chatbot and can be accessed from chat applications such as Telegram. We evaluated the accuracy of Shavira powered by Rasa Framework with ten topics about academic information. The results showed that the accuracy of Shavira is 90%. We also evaluated the usability of Shavira with Software Usability Scale (SUS) Questionnaire. The usability evaluation showed that Shavira satisfied users with value 81 (threshold 68) and categorized Excellent, Acceptable and Prometer.


INTRODUCTION
A good information quality in terms of time, accuracy and relevancy is very important to be given to users as a part of a service. Information quality helps users to achieve their goals by utilizing the services provided. In the previous research, we developed a web-based Undiksha FAQ system to help users get answers of the frequently asked questions. The FAQ stores data on frequently asked questions and answers to those questions. With the FAQ, it is hoped that users will get answers easily to the questions.
Searching information through website requires more effort by users to find the exact information. The user has to open the browser, then open the website address, type the question keyword and finally get the answer. These steps are less efficient if the user wants to get a faster answer. One of the most popular technological innovations today to provide automatic answers to users is Chatbot technology.
Currently, many organizations use technology to provide services to customers automatically, namely Chatbot technology (Almansor & Hussain, 2020). According to Sandbank, Shmueli-Scheuer, Herzig, Konopnicki, Richards & Piorkowski (2018), about 80% company plan to implement chatbot. The main benefit of using a chatbot in an organization is that it can automatically answer customer questions quickly. In addition, the use of chatbots also reduces the operating costs of providing customer service. Chatbots have been widely used in both the private and public sectors such as: banks, government, tourism, the stock market and so on. Chatbot also can use in education area (Clarizia et al., 2018). Chatbot technology is now equipped with voice technology including Siri, Alexa, Google Assistant and Cortana (Androutsopoulou et al., 2019). The method to search for answers by the chatbot utilizing machine learning technology to perform a sophisticated retrieval process, where responses are generated based on analysis from web searches (Cahn, 2017). Chatbot also can used in Veterinary consultation (Huang & Chueh, 2021 (Singh et al., 2019). In addition, developers can also develop from scratch by utilizing NLP and Machine learning technologies. The choice of chatbot technology is tailored to the needs or design of the chatbot application.
In this research, the FAQs that have been developed previously will be integrated with Rasa Framework. RASA is an open source framework consists of stack machine learning consists of libraries for developers to create contextual chatbots. The FAQ serves as a data source that will be linked to the RASA Framework via the webservice API, then the data will be trained on the RASA framework to get a chatbot model. The model is used as a service that is connected to the presentation layer such as: Telegram and Facebook.

Figure 1. FAQ Undiksha User interface
Undiksha Virtual Assistant (Shavira) is an intelligence system based on natural language processing (NLP) that can help users to get information with natural language. On our previous research, we have developed FAQ Undiksha as a part of Shavira with User Centered Design approach (Resika Arthana et al., 2021). FAQ Undiksha is a web-based application used to facilitate the users to get answer related to the questions they have ( Figure 1). FAQ Undiksha is developed with PHP programming language and MySQL database. FAQ Undiksha can be used by any organizations and any activities at Undiksha. The organizations that have an activity can manage their lists of questions and answers. The overall workflow of FAQ Undiksha depicted in Figure 2.
Jurnal Sains dan Teknologi | 266 Design for chatbots represents a transition from the design of visual layout and interaction mechanisms to the design of conversation (Følstad & Brandtzaeg, 2017). In this research, we proposed improvisation capability of Shavira with the implementation of chatbot. The purposed implementation chatbot in Shavira is providing access information with online chat conversation in natural language. With Shavira powered by chatbot, the users will be easier to access information.
In this research, we use RASA Framework as the engine of chatbot. Rasa is an open source machine learning framework for automated text and voice-based conversations. Rasa capable to understand messages, hold conversations, and connect to messaging channels and APIs (Rasa, 2021).  (Rasa, 2021) The message that is sent to Rasa will be received by Natural Language Understanding (NLU).
NLU consists of Interpreter. Interpreter correspond to convert text to dictionary consist of text, the intent and any entities. Then, the message will be passed to the Tracker. Tracker correspond to keep track of conversation state. The action based on text chooses by policy, then logged to the tracker and send response to the user by web page or other service. In this research, we use telegram as interface chatbot by users.
There are many previous researches that implemented virtual assistant with rasa Framework. For example, Deepika at all design the implementation of intelligent virtual laboratory using RASA framework (Deepika et al., 2021). The purpose of this research is to help students to get answer about mistakes in C programming language with virtual assistant. (Jiao, 2020) integrated RASA NLU with neural network (NN) methods implements the system based on entity extraction after intent recognition. (Windiatmoko et al., 2021) using RASA Framework for University Enquiries.

METHODS
In this research, we improved Shavira with the implementation of Chatbot technology based on the existing FAQ Undiksha. FAQ stored frequently asked questions and the answers. Figure 4 shows the implementations of the research workflow. Due to different data format between FAQ Undiksha and Rasa Framework, we must make data mapping from FAQ Undiksha and NLU. After data mapping, we trained the data in Rasa Framework to get Chatbot Models. The Chatbot Models used when there is an interaction in Chatbot.

Data Integrator and Mapper
In the existing FAQ Undiksha system, we stored data as single pair question and answer. Due to Rasa Framework needs more than one questions as alternatives and allowed to give alternative answers, we modified existing schema database in existing FAQ Undiksha. The new schema database depicted in Figure 5.

Figure 5. Modification data structure FAQ to comply Rasa Framework
Developing chatbot in Rasa Framework required config at least 3 files in ".yml" format. These files consists of nlu.yml, domain.yml, rules.yml. The nlu.yml file store list of questions and intent as category (Figure 6. nlu.yml). Every intent consist of one or more questions. The domain.yml file store responses based on the intents as action from questions (Figure 7. domain.yml). We also can put image as responses. In the domain.yml file, we can put one or more texts answer on every intents. The rules.yml file store rules or paths between intent and action (Figure 8. rules.yml). If we need to the conversation between user and chatbot in sequential ask and action, we can configure in stories.yml.

Chatbot Development
We provide access to Shavira chatbot via Telegram application. Telegram is free and opensource instant messaging (IM) system. We use Telegram as interaction layer because Telegram is one of the most popular messaging applications and most of our staff and students use that application to communicate. The integration of Rasa Framework with Telegram consists of several steps. First, we published service Rasa via https then we created Bot on Telegram via botFather. The final step is setup a webhook between telegram with Rasa using webhook URL. After that, we can communicate to chatbot via Telegram as show in Figure 10.

Integration
As show on Figure 11, we modified form input on FAQ Undiksha to support more than one alternative questions. The alternative questions is useful to make training data more reliable. Users input data of question and answer then the data stored in database.

Figure 11. Input Question Answer FAQ Undiksha
Integrator and Mapper publish data based on data on database as a web service. The webservice can be accessed by endpoint url. As depicted at Figure 12,the response from web service formatted as json.

Figure 12. JSON output from web service
Data sync and scheduler retrieve data from web service. Scheduler retrieve data based on the time that has been set before. Data JSON received by dispatcher and write nlu.yml ( Figure  13), domain.yml( Figure 6) and rules .yml ( Figure  7) On Rasa framework, we configure some parameter to make chatbot as FAQ Assistants (Figure 14).
First one, we updated config.yml in policies parameter. We used RulePolicy as policies, so the chatbot will respond the same way every time the same type of question is being asked. Next, we set ResponseSelector in NLU pipeline. The ResponseSelector will build a single retrieval model for all retrieval intents

Evaluation
We design evaluation chatbot by developing interactive conversation to confirm that the answer of chatbot is correct or not. The users ask something to chatbot, then the chatbot will answer and ask to the user whether the answer is correct or not. Correct answer means that answer of chatbot appropriate with context of questions. With this method, we can get the accuracy of chatbot.
We also evaluated usability of chatbot with Software Usability Scale (SUS) to get users satisfaction level. SUS consists of 10 question with 5 options, from strongly agree to strongly disagree (Sharfina & Santoso, 2016) (Arthana et al., 2019.

RESULTS AND DISCUSSION
We use 15 sample questions as data in this research. The questions are about academic information in Undiksha. We input the data in FAQ system, then we synchronize to RASA Framework. After synchronization has done, then the RASA train data and save as model. The model loaded by RASA CORE and provide access to telegram. The users can interact with bot via telegram.
Data from FAQ Undiksha is extracted via web service formatted as JSON Format ((shows in Figure 12). Data from JSON mapped and saved as files that required by RASA Framework. The files are "nlu.yml", "domain.yml" and "rules.yml" (Figure 6,  Figure 7, Figure 8).
Jurnal Sains dan Teknologi | 270 After the data have been stored in "nlu.yml" then it is trained in Rasa Framework. After the train has been done automatically by Rasa Framework as config that defined in "config.yml" (Figure 15). When data train completed, Rasa Framework saved model in file model. File model will be loaded when Rasa was run.
Integration with Telegram application done by hooking between Telegram API with Rasa Endpoint. We used https protocol that defined in action endpoint as required by Rasa Framework.

Figure 16. Confirmation correct answer by chatbot
The users can interact with Shavira by Telegram Apps. First, the users search or add chatbot name, then the users click on start button. After that, users can ask something and it will be answered automatically by the Shavira. The topic that can be asked by users is limited by specified context in topic list as shown at Table 1.
The challenged on Chatbot is need more data in training. More data collections make the chatbot more intelligent. Limited response cause by limited data training reduce user satisfied.
We evaluated accuracy of Shavira by asking users to ask to Shavira via telegram application. The Shavira will answer automatically and request feedback to the users whether the answer is appropriate with the context of the questions or not. After users confirm, the chatbot will saved feedback in databases based on users and topic. The results of feedback users shown in Table 2. According to the evaluation results as shown in Table 2, overall accuracy Shavira is 90%. Based on log activity conversation between users with chatbot, we found that some questions are not answered based on the context due to misleading term. For example, when users asked about Kartu Rencana Studi (KRS) the users use term 'Mata Pelajaran'. They use synonym term that is not defined in data training (nlu.yml). In the future, the collection of questions on each topic should be more varied to suit the user's questions.

Rasa Framework not supported
Indonesian language yet. We cannot utilize spacy_sklearn pipelines in config.json to improve interaction between Shavira and users. This limitation potential to reduce quality of conversations. On future research, we must implement Indonesian language in chatbot like what have been done in previous research by (Elcholiqi & Musdholifah, 2020).  Figure 17. SUS Scale Position (modified from (Sauro, 2018)) In addition to evaluation of accuracy, we also evaluate usability. We evaluate user satisfaction with Software Usability Scale (SUS).
The Questions of SUS consists of.
1. I think that I would like to use this system frequently. 2. I found the system unnecessarily complex. 3. I thought the system was easy to use. 4. I think that I would need the support of a technical person to be able to use this system. 5. I found the various functions in this system were well integrated. 6. I thought there was too much inconsistency in this system. 7. I would imagine that most people would learn to use this system very quickly. 8. I found the system very cumbersome to use. 9. I felt very confident using the system. 10. I needed to learn a lot of things before I could get going with this system.
The respondent of SUS Questionnaire is 15 respondents as shown at Table 3. The results of SUS questionnaire analysis showed that users satisfaction level in Shavira interaction is 81. According to (Jeff Sauro, 2011), the threshold of results SUS questionnaire is 68. So, the user satisfaction users in Shavira categorized as Excellence, Acceptable and Promoter ( Figure 17).

CONCLUSION
We have improved capability of Shavira from web-based FAQ to implementation with chatbot powered by Rasa Framework. The challenge in this research are data integration between existing FAQ and Rasa framework done by data Integrator and Mapper. Data from FAQ that is provided by web service and caught by Data Scheduler and Dispatcher then saved as file that required by RASA Framework. To make access to chatbot easier, we integrated with Telegram Application. The evaluation show that accuracy of Shavira is 90% and the users satisfied with Excellence, Acceptable and Promoter category.