Journal of Ergonomics
Open Access

Evaluating Usability for e-Learning “Madrasati” Platform in Saudi Arabia

Jamilah M. Alamri^* and Sarah S. Almoaiqe^{1 *}Correspondence: Jamilah M. Alamri, Department of Computer Science, Computer and Information Sciences, King Saud University, Riyadh, Saudi Arabia, Email:

Author info »

Introduction

One of the essential requirements of success and sustainability in websites is quality and to achieve it measurements is needed. Therefore, software usability evaluation is an important concept in the discipline of Human Computer Interaction (HCI). Evaluating usability offers a number of benefits for both future users of a product as well as the company or developers creating that product. Therefore, empirical methods are to help evaluate and validate the research results, such as: controlled experiments, case studies, survey research, ethnographies and action research [1].

Software interface should be designed to be easy to use, easy to learn, easy to memorize and these are the attributes of software usability evaluation [2]. To achieve that both accessibility and usability are used in relation to user interfaces and more recently to websites. Due to the importance of accessibility the World Wide Web Consortium (W3C) in 1997 established the Web Accessibility Initiative (WAI) to promote the accessibility of the Web, gives a general definition of Web accessibility as “people with disabilities can use the Web more specifically they can perceive, understand, navigate, and interact with the Web” [3].

Dumas et al. [4] stated the benefits of usability testing in improving the process that an organization needs to develop a product which has much better long-term impact on usability than improving the product. Moreover, the right process of usability testing has great effect on software development life cycle by saving time and efforts which will provide better user experience [5]. There are various usability evaluation techniques [6] and in this research heuristic evaluation guidelines by Nielsen [7] will be applied to analyze a user interface for its compliance with established usability principles, i.e., the heuristics.

This study examines and evaluates the Madrasti platform [8] in Saudi Arabia which was founded in the sake of providing online education during the COVID-19 pandemic. The main reason of establishing this platform is to enhance integration between community institutions and the Ministry of Education in support of online education and e-learning programs. However, the Ministry was able to provide the Madrasati platform in a short period of time with the efforts of talented Saudi developers, to provide comprehensive educational services to students, teachers, supervisors, school leaders and parents. Therefore, evaluation of e-learning systems is important to ensure successful delivery, efficient use, and positive impacts on learners.

The focus of this research is to evaluate the usability and the content accessibility for the Madrasti platform [8] in Saudi Arabia since many users have been complaining about the accessibility and usability of the platform. Furthermore, the correlation between the two of them will be discussed to emphasize the impact one makes on the other so that future designers are aware of any potential quality improvements. In this evaluation a framework by [9] will be applied to measure the usability and the content accessibility.

Background and rationale

Usability evaluation of the software interfaces is one of the prominent concepts that encouraged the researchers in the field of Human Computer Interaction for a long period of time. In this section, several studies have discussed evaluating the accessibility and usability of websites from different categories.

Many researches had been conducted on evaluating e-learning websites in Saudi Arabia [10]. For instance, the authors in [11] have studied and analyzed the internal and external usability attributes of 11 university websites that offer distance education courses in Saudi Arabia, they found that this websites are reliable but violate basic usability guidelines. In [12,13] the writers examined the usability of JUSUR LMS by applying usability technique was based on satisfaction surveys, the results show that students liked JUSUR LMS and find it easy to use. However, the system suffers from some technical and functional problems that affect its usability. The authors of [14] evaluated the design user interface in order to prove their hypothesis that ‘Blackboard LMS is accessible and usable by the teachers from different faculties for the delivery of content in the King Saud University, navigational features, and ease of use to answer the research questions by conducting questionnaires. Their result was supporting the hypothesis that the software is easy to access and use. Algafis et al., [15] just published a study in Saudi Arabia conducting an empirical research on Saudi students’ preference for Moodle and Blackboard systems. The authors used three comparison metrics in their study: user interface, download and upload service and materials organization. As a result, for this study the students favored the Moodle learning management system instead of Blackboard in terms of usability.

The purpose of evaluating e-learning website is to understand the mental model of the user and bond it with designer’s perception; Norman [16] stated that the gap between the mental model and designer’s perception is considered as vital disadvantage, which leads to weak usability. There have been some papers evaluating usability in e-learning websites, for instance, [9] investigates the quality of e-learning websites in the Middle East in term of usability and content accessibility. The authors applied their study on eleven websites from eleven countries. They proposed a framework that addresses two assessing e-learning websites quality metrics by using a methodology starting with problem identification followed by Web diagnostic tools and data sampling. As a result, the authors stated that the websites are considered no accessible for a large group of users; therefore, the developers of the target websites do not follow usability and accessibility standards. Moreover, the writers in this paper [17] discussed how the absence of assistive technology support in e-learning websites could stimulate negative user experiences. They developed an e-learning usability evaluation model based on user’s mental model which will reduce the gap between user’s mental model and designer’s perception. To enhance the accessibility in order to enable people with disabilities to take part in the educational and learning processes, Bocevska et al, [18] analyzed the accessibility of some e-learning websites by applying the Web Content Accessibility Guidelines (WCAG) 2.0. Their findings were that each LMS has some fundamental features that cannot be modified and variable features which can be adapted for the specific accessibility requirements of people with disabilities.

Some researchers [19] emphasize the relationship between content accessibility and usability. For instance, Waddell et al, proposed that accessibility problems might be a subset of usability problems. Therefore, analyzing the correlation will be conducted in this evaluation. In [20-22] the authors stated that accessibility is subset of usability and having a high rating in accessibility does not mean that a website is also rated high in usability.

After investigating the issues with the quality of e-learning websites, there are some researches on enhancing the concept of quality of e-learning systems. For instance, the authors of [23] proposed a new framework called TICS (Technology, Interaction, Content, Services), which focuses on the most vital aspects to be taken into consideration when designing or evaluating an e-learning system. They showed a huge interest in user-system interaction by deriving guidelines from TICS aspects called eLSE (e-Learning Systematic Evaluation). This methodology combines Abstract Tasks (ATs), inspection technique with user-testing to allow inspectors with minimum experience in evaluating e-Learning systems to perform accurate evaluations. The result shows an advantage of the AT inspection over the other usability evaluation methods, demonstrating that Abstract Tasks are effective and efficient tools to drive evaluators and improve their performance.

The authors in [24] used two research approaches to accomplish qualitative results; acmeological and competency-based to provide a model of e-learning management. Using these methods helped to analyze the features of foreign and Russian LMS platforms; developing tools for the e-learning system development; and measure the extent to which e-learning tools meet the preferences of high-potential users. Shurygin et al. stated that the framework of e-learning education is more effective and popular among the representatives of new generations. Moreover, traditional classroom-based teaching added to LMS platforms delivers additional benefits and opportunities in higher education. The authors of this study proposed a number of resources that will help to develop elearning in higher education given the requirements imposed by globalization.

Methodology

A framework by [9] will be followed in this research to answer the research questions, which will assess e-learning websites quality metrics in regard to accessibility and usability. The reason behind applying the framework in this evaluation is that it is consist of the most commonly known Nielsen usability guidelines [25] which will be used for usability evaluation which will assess uploading speed, main page size, and number of broken links and WCAG guidelines [26], which are used for website content accessibility evaluation, and automatic evaluation tools will be used to measure websites metrics. Hammad et al., methodology was to start from identifying the problem then by using some web diagnostic tools and sampling the data.

To start the process Madrasti platform [8] were selected to evaluate its accessibility and usability. The evaluation was conducted on 5 main pages which are mentioned in Table 1. Several statistical analysis techniques were employed to answer and analyze the research questions. Currently, there are many tools that assess the content accessibility evaluation online and for this evaluation WAVE [27] has been used. Furthermore, Broken Link Checker [28] to measure usability and app.telemetry [29] tool to monitor page load speed. For evaluating the platform website’s performance, the Chrome DevTools [30] will be used, which will help to analyze runtime performance. To plan the research, a set of research questions were needed such as:

Table 1: Selected pages.

Does Madrasti platform rate high in terms of the usability standards?

How accurately

Is the content accessibility standards followed in Madrasti platform?

How are the results of the content accessibility evaluation of Madrasati platform related to the results of the usability evaluation?

Data collection and analysis

The Madrasti platform was selected to evaluate its accessibility and usability since it is the main platform used in Saudi Arabia by over 6 million users at the current outbreak of Coronavirus disease [30]. The Ministry of Education divided entry to the portal into two periods: for the middle and high school in the morning from 9:00 am until 2:00 pm in the afternoon and the elementary stage from 3:00 pm in the afternoon until 6:30 pm in the evening. The experiments were conducted using a student account on 29 November 2020; the evaluation was executed in the two periods: between 11:00 am and 1:00 pm and between 4:00 am and 6:00 pm.

Results

Usability assessment

To test the usability, Nielson usability guidelines will be applied, to measure the website’s speed, page size and broken links. In this experiment, we used the Chrome DevTools tool to evaluate the websites’ pages size and app. telemetry for load time. The Broken Link Checker tool is used to test the broken links for the targeted pages. The broken links percentage was evaluated using this formula.

Where, BL_{Link checker} is the percent of broken links, ValidURL is the number of valid links; Invalid_URL is the number of invalid links. A summary of the collected results is presented in Table 2, which shows the mean, standard deviation, minimum, and maximum values of the percentage of the broken links, page size, and load time for the Madrasti platform. Fortunately, the percentage shown in Table 2 of the broken links is zero, which reflects a good technical administration of the platform. Considering Nielsen proposition in order to achieve desired response times for a website of 8 K size, the ideal response times is 1 second [31]. In Table 2 shows that the Madrasti platform assessment results on average are huge for optimal response time. However, the latest report from HTTPArcive [32] stated that the mean size of a website is 2062.4 KB, thus the results is considered adequate.

Table 2: Madrasati usability assessment results.

The answer of the first research question, Q1: Does Madrasti platform rate high in terms of the usability standards? Is no, from the findings above it is clear that Madrasti did not meet Nielsen proposition with regard to the usability and it has some design problems.

Accessibility assessment

The Madrasti’s content is evaluated according to WCAG 1.0. The WCAG has three priorities, and each checkpoint has a priority level assigned by the Working Group based on the checkpoint’s impact on accessibility, as follows [22].

First priority (P1): A web content developer must satisfy this checkpoint. Otherwise, one or more groups will find it impossible to access information in the document. Satisfying this checkpoint is a basic requirement for some groups to be able to use Web documents.

Second priority (P2): A web content developer should satisfy this checkpoint. Otherwise, one or more groups will find it difficult to access information in the document. Satisfying this checkpoint will remove significant barriers to accessing Web documents [24].

Third priority (P3): A web content developer may address this checkpoint. Otherwise, one or more groups will find it somewhat difficult to access information in the document. Satisfying this checkpoint will improve access to Web documents.

In order to measure the content accessibility, WAVE tool has been used for the 5 pages, the mean and standard deviation for number of accessibility errors and alerts found for the selected pages in the three standards accessibility priorities are shown in Table 3 below [30]. Table 3, demonstrates that the Madrasati platform failed to fulfill the acceptable level of accessibility. According to WCAG standards a website is considered to be accessibly approved if there are no Priority 1 accessibility problems detected. For the selected pages, each one has at least one error in the priority 1 level, and the average number of errors in P1 is 23, which is a large number of errors. As a result, we can say that all selected pages are not approved in term of accessibility.

Table 3: Content accessibility errors and alerts.

The number of errors is shown in Figure 1 for each page in the three levels of priority. As illustrated in Figure 1, the Priority 2 has the greatest number of errors. As presented the number of accessibility failures for the same website relatively scaled over the P1 and P2.

Figure 1: Number of content accessibility errors.

However, Figure 2 shows that in most selected pages, number of warnings for a page is relatively close in the first and second priority levels, whereas it appears that the selected pages are clear of any neither errors nor warnings regarding P3.

Figure 2: Number of content accessibility warnings.

To conclude, the answer of the second research question, Q2: How accurately are the content accessibility standards followed in Madrasti platform? It is not followed properly since there are too many violations for WCAG standards.

Usability and content accessibility correlation

Table 4 demonstrate the correlation values between number of content accessibility errors in the three priorities with the broken links percentage, page size and load time. The correlation values between the web page size and P1 and P2 errors are 0.653 and -0.102 respectively. The platform developers added more contents without following accessibility standards which led to increasing the page size. Moreover, the highest correlation occurred with number of errors in P1, and then decreased in P2. This indicates that the website developers are violating the most important checkpoints while they are building the platform, and that is the answer to the last research question [31].

Table 4: Correlation values.

Discussion

The aim of this research is to evaluate the usability and the content accessibility for the Madrasti platform [8] in Saudi Arabia. The evaluation was conducted on 5 main pages; the findings show that there are no broken links, which reflect a good technical administration of the platform. Moreover, shows that the Madrasti platform assessment results on average are too big for optimal response time. As for the content accessibility standards for the selected pages, the average number of errors in P1 is 23, which is a large number of errors thus all selected pages are not approved in term of accessibility. Furthermore, the findings show that there is a correlation between the number of content accessibility errors and the usability assessment results which indicates that the platform developers are violating the most important checkpoints while they are building the platform. These findings need to be addressed by the developer in order to increase the efficiency of the platform to serve a wider audience especially people with disabilities and allow them to preserve, understand, navigate and interact with the platform in an optimal way [32].

A similar study has been done by Hammad et al. evaluating Noor website and their result was that the website had failed to fulfil the content accessibility and usability standards. This study was based on student account access to Madrasti.

Conclusion and Future Work

It is important to remember that a mix of factors such as aesthetics, ease of use, learning and memorability etc. is included in usability. This paper evaluates the content accessibility and usability of Madrasti platform in Saudi Arabia. The evaluation results demonstrate that the developers of the platform do not follow usability and accessibility standards. Unfortunately, Madrasti platform is considered to be not accessible for some users with disabilities. Furthermore, Madrasti is reliable but considered low in term of usability. Based on statistic comparison the findings show that there is a correlation between the number of content accessibility errors and the usability assessment results and that should encourage the developers in Madrasti platform to focus more on them in their work to achieve the best user experience.

As for future work, proper evaluation will be conducted to test other functionalities targeting parents and faculty in a special usability lab on the platform end to end, also to engage participants to evaluate the platform using surveys.

References

1. Easterbrook S, Singer J, Storey MA, Damian D. Selecting empirical methods for software engineering research guide to advanced empirical software engineering. Adv Empir Softw Eng. 2008; 13:285-311.
2. Tao Y. Work in progress-introducing usability concepts in early phases of software development. IEEE Exp. 2005.
3. Thatcher J, Burks MR, Heilmann C, Lawton SH, Kirkpatrick A, Lauke PH, et al. Web Standards and Regulatory Compliance. Press. 2006.
4. Dumas JS, Redish J. A practical guide to usability testing. Intellect books. 1999.
5. Kucheriavy A. Good ux is good business: how to reap its benefits. Forbes. 1973.
6. Shneiderman B, Plaisant C, Cohen m, Jacobs S, Elmqvist N, Diakopoulos N. Designing the user interface: strategies for effective human-computer interaction. Pearson. 2016.
7. Nielsen J,  Molich R. Heuristic evaluation of user interfaces. CHI. 1990; 90:249-256.
8. Al-Omar K. Evaluating the internal and external usability attributes of e-learning websites in Saudi Arabia. Adv Comput Int J. 2017; 8(3):12-18.
9. Al-Omar K. Automated usability evaluation of elearning websites in saudi Arabia. Feedback. 2017; 7(9): 01-10.
10. Benaida M, Namoun A, Taleb A. Evaluation of the impact of usability in Arabic university websites: Comparison between Saudi Arabia and the UK. Int J Adv Com Sci App. 2018; 9(8): 53-57.
11. Alturki UT, Aldraiweesh A, Kinshuck D. evaluating the usability and accessibility of lms ‘blackboard’ at king saud university. Contemp Issues Educ Res. 2016; 9(1):33-44.
12. Alghafis A, Alrasheed A, Abdulghany A. A study on the usability of moodle and blackboard-Saudi students perspectives.  Int J Interact Mob Technol. 2020; 14(10):159-165.
13. Norman DA. Emotional design: why we love (or hate) everyday things. Learn Tech Lib. 2004.
14. Ain Q, Asalam M, Muhammad S, Awan S, Pervez MT, Naveed N, et al. A Technique to Increase the Usability of E-Learning Websites. Pak J Sci. 2016; 68(2):164-170.
15. Bocevska A, Savoska S, Ristevski B, Tabakovska BN. Analysis of Accessibility of the e-Learning Platforms According to the WCAG 2.0 Standard Compliance. UKLO. 2018. 26-31.
16. Shneiderman B. Promoting universal usability with multi-layer interface design. Comput Phys Handicap. 2002; 73:1-8.
17. Shneiderman B, Hochheiser H. Universal usability as a stimulus to advanced interface design. Taylor. 2000; 43(5): 84-91.
18. Ma THY, Zaphiris P. The usability and content accessibility of the e-government in the UK. 2003; 20:760-764.
19. Lanzilotti R, Ardito C, Costabile MF, De Angeli A. Else methodology: a systematic approach to the e-learning systems evaluation. Educ Technol Soc. 2006; 9(4):42-53.
20. Shurygin VY, Berestova AV, Litvinova TM, Kolpak EP, Nureyevà A. The usability of learning management systems as a platform for distance learners. Talent Dev Excell. 2020;12(1):3984-3997.
21. Web Content Accessibility Guidelines (WCAG) Overview. Web Accessibility. 2018.