Journal of Information Technology & Software Engineering
Open Access

Usability Testing of Android Applications- Survey and Case Study

Neha Chaudhary and Aditya Upadhyay^{* *}Correspondence: Aditya Upadhyay, Department of Computer Science and Engineering, Manipal University, Jaipur, Rajasthan, India, Email:

Author info »

Introduction

Industry of mobile application is growing very rapidly [1]. By 2020 mobile applications are forecast to make around 189 billion U.S. dollars in revenues [2,3]. As of March 2017, 2.8 million mobile applications were available at Google Play Store and there were 2.2 million apps available in Apple’s App Store. There are approximate 4.5 billion subscribers in the world. The most popular app category among Apple iOS users is utilities, social networking, photo and video and games. In 2017, WhatsApp has 1.2 billion monthly active users [2].According to study Smart phone users in the United States spend 86% of their mobile usage time exclusively on apps [4]. An additional study shows that mobile users spend 85% of their mobile app usage time using just 5 apps [3]. Many applications which fails, once launched, due to poor usability issues [5]. The area of mobile application testing is very challenging and complex due to different types and configurations of mobile devices.

Usability Testing

Usability Testing is done to determine whether the newly made software/app is easy to use or not. Usability Testing is a main phase of Software testing that can save money by timely identification and improvement of errors [5]. The main objective of usability testing is to find errors in user interface of mobile application [6]. Usability testing is a non-functional testing technique. It is also known as User Experience (UX) testing.

Usability attributes

Usability attributes are the features to measure the quality of application [7,8]. According to Harrison et al. usability of mobile application is represented by three attributes; effectiveness, efficiency, satisfaction and cognitive load [9]. Nielsen’s identified five attribute of usability defined below [10].

• How it is easy to complete basic tasks the first time by the users.

• How easily can they perform tasks after using it before?

• When users come back to the application after time, how easily can they recall functionality?

• How easily users can recover from the errors?

• How pleasurable is it to use the application design?

• - It states to the mental processing power needed to use your application and it affects how easily users complete tasks.

International Organization for Standardization (ISO) defined usability considering only three attributes satisfaction, efficiency and effectiveness [11]. ISO standard does not consider the few attribute like Error, Memorability and Learnability to test usability of applications.

• Effectiveness means completeness and accuracy with which users accomplish tasks.

• User Satisfaction means Self-determination, and positive attitudes toward the use of the application Figure 1.

Figure 1: A framework for Usability testing of mobile application.

Process of usability testing

The process of Usability testing consist of following steps [12]:

Step-1:- Planning and Preparation: In this, the objective of usability test is determined.

Step-2:-Recruiting test users: Select the desired number of test users according to your usability test plan. 5 users are sufficient to uncover approx. 80% of problems.

Step-3:- Perform Usability Test: In this, testing tasks are executed. Testing team assign the tasks to different test users with short description of application.

Step-4:- Test outcomes examination: After task completion, Data is analyzed to derive useful inferences and give recommendations to correct the usability of your application.

Step-5:- Apply modifications: The finest recommendations and the changes are incorporated in application to improve its performance. Finally, the updated application is tested again.

Usability testing methods

There are two methods to perform Usability Testing [7,13].

Laboratory usability testing: In this method, testing is performed in a specific laboratory room in guidance of the observers. The testers are allotted tasks to execute. The duty of the observer is to monitor the behavior of the test users and report the outcome of testing to the observer.

Field usability testing: In this method, observers and test users are remotely located. There is no lab setup for testing. Think aloud protocol is used during the testing.

Literature Review

Usability testing of applications software’s designed for smart mobile devices faces a various challenges due to distinct features of mobiles such as wireless network, different OS, different screen size and resolution, limited bandwidth and the changing context [7]. Testing methods of usability testing of desktop applications may not be used for mobile applications. Piotr Chynal, et al, perform usability testing of Facebook mobile version site using eye tracking glass and first with the real phone placed on the handle, second using You Wave emulator [14]. The result shows that eye-tracking played an important role data collection.

Balagtas et al. explained the challenges of usability evaluation for mobile applications due to restrictions of device, and absence of supporting tools, and lack of techniques to collect the usability data [15]. They addresses two techniques for data collection either external camera or logging. They have introduced a framework EvaHelper (Evaluation Helper) based on a logging technique. D. Zhang et al. addressed that the exclusive features of mobile phones such as mobile context, wireless connectivity, small screen size, display resolutions, and many more, become challenges for usability testing of mobile applications [7]. Mona Masooda et al. focused on the importance of usability testing for Mobile Educational application designed for 4-5 years children [16]. Usability testing was conducted for a free MEA available for Android users, Out of 10 only 6 Jakob Nielsen’s Usability Heuristics Principles were considered. This study shows that there is a gap in the mobile learning industry, where there is a need to acquire User Centered Design knowledge and skills.

Betiol et al. perform the comparative analysis of usability testing of mobile interfaces based on testing evaluation approaches: (i) computer based mobile phone emulator- Openwave emulator inside the lab (ii) using a real mobile device inside the lab (iii) using a mobile device connected with a wireless camera [17]. Selma Limam et al. introduce a mobile application for weight loss with three interfaces- (i) A messaging system, (ii) a personal goal achievement system,(iii) a group goal achievement system and perform usability testing of application with native speakers of Arabic and one gender to avoid biases [18]. After the usability testing of application, localization of application was also performed.

Wei et al. performed usability testing on library mobile application of Chongqing University for improving its usability issues [3]. A field study was conducted to test the usability of app in a real environment. The study concludes that the library application was effective; however, the efficiency of the application needs more improvements. Rachel Harrison et al. explained Nielsen usability model, ISO usability model with specific attributes and introduce new model PACMAD with additional attribute cognitive load [9]. PACMAD model consider three major factors or attribute which affected the usability of a mobile application: User, Task and Context of use. PACMAD model consider 7 usability attributes.

Fetaji et al. addresses the lack of research about effectiveness, efficiency and usability of m-learning systems and proposed a prototype MobileView (Mobile app) for testing purposes [19]. The proposed strategy focused on qualitative information for the usability and benefits of the environment. Swaid and Suid et al. perform study focused on recognizing the usability heuristics to apply when valuing the usability of m-commerce applications and compare usability heuristics defined by Google android design guidelines, Nielsen, and Apple human interface guidelines [20]. Usability testing of two m-commerce applications, LetGo and OfferUp, were performed. These mobile apps offers services to user’s like- sell and buy secondhand items locally using the GPS feature. After performing testing, they also discussed the problems encountered during the usability testing in both applications.

Kallio and Kaikkonen et al. performed comparative analysis between the laboratory and field based testing to analyze the effects of the testing environment on usability result [21]. The result shows that field based testing is complex and time consuming. Hanifah Muslimah Az-zahra et al. performed usability testing on two applications GoEat and LetsEat [22]. These two applications are differ only in color and interaction style with different widgets and consider 3 usability attributes- Effectiveness, learnability, satisfaction. Rodríguez et al. establish a usability lab using free tools using screen recording tools, Teamviewer etc. and evaluate usability of municipality mobile web app [23]. The effectiveness achieved by the test users was 35% and 71.42% of the users correctly finished tasks. Borys, Milosz et al perform usability testing of a mobile Sales force automation app using eye-tracking glass in quasi real condition [24]. Milosz et al. perform usability testing of mobile app for controlling access to rooms using smart mobile devices and data was collected using thinking aloud protocol, eye tracking glasses [25]. After analysis of test results, usability expert team introduced 13 recommendations for better usability Table 1.

Table 1: Summary of usability studies of mobile applications.

Case study usability testing of university info app

In this section, the usability testing of JUezzy app is performed [26]. JUezzy is an application which provide information of a university campus to the students or other members of the university Figure 2.

Figure 2: Home Page of JUezzy application.

It provide information such as- food mess menu (Breakfast, Lunch and Dinner menu), contact details of the faculties members and their sitting location. The usability of this app was performed with team of 20 participants, different branch and streams. The field based testing approach was used to analyze the experience and behavior of users with the methodology of pre-tests questionnaire, accomplished tasks and post-test surveys [27].

Method of usability testing of juezzy

The process of usability testing include the following steps.

Step-1:-Preparation stage: Before starting the testing, the researcher introduced and explained the purpose of performing usability testing. The participants were clarified that purpose of the testing was not to evaluate their skills [28].

Step-2:-Pre-test questionnaire: A set of questions was designed for determining the knowledge of test-participants in using a mobile application. The questionnaire include the following basic questions about test participant knowledge:

Step-3:-Performing tasks: five tasks were considered based on the functionality provided by the application.

• Task 1:- Find the contact details of a particular faculty member.

• Task 2:- Check whether the stationary shop is open or closed?

• Task 3:- Find the location of computer lab 9.

• Task 4:- Check the menu of today lunch and evening refreshment.

Step-4:-Post-test survey: After performing the five tasks, the test participants evaluated their satisfaction with the task and share their experience and suggestions with the researcher.

Test results and users suggestions

After the experiment, Data was collected from all test users and analyze them. The participants shared their experience with suggestion to improve the application.

• App should provide the information of list of holidays or academic calendar and syllabus for various courses.

• The sitting information of faculty must be provided with the contact number and e-mail id of faculty, it also mentions the department of faculty. Library timings must also be included.

• Sometimes the food menu was wrong; it is advised to upload it carefully with timely manner. Contact details of Doctor and timings of Medical room must be included for emergency.

• It is difficult to understand the location of a lab or lecture theater for new students. So, text must be there to explain the location clearly or navigation must be provided.

• Information and date sheet of Midterm test and End semester exam must be included with notification for increasing usefulness.

• Navigation to login for ERP portal, Parent portal and Service portal for notices should be provided. It will help to increase the number of users per year.

• Lecture notes or previous year question papers must also be included.

The pre-test questionnaire result shows that all test participant were using android smartphones with different latest versions and all were familiar with how to use mobile app. Three main attributes of usability were measured in this usability test. Effectiveness was considered as the failure or success of completing the task; Efficiency was considered as the amount of time spent in completing the task; User satisfaction was measured through an attitude and experience of participants Table 2.

Table 2: Task completion during usability testing.

The test results shows that, three tasks (tasks 1,4) were completed by all participants and the remaining three tasks (task 2,3 and 5) were finished by less than 75 percent of the participants due to bad interface. Task 1 and Task 4 were very simple and easy that’s why all test participants completed the task with average time of 12.37s and 12.5s respectively. Task 5 was difficult because most participants did not know how to find location of faculty. Task 2 and Task 3 were less difficult in comparison to Task 5. After the evaluation of test data and analysis of participant suggestions, a new modified interface of mobile application was designed and that new interface design was proposed to the developer team [29].

Threats to validity

The usability testing was performed on Android smart phones with certain test participant team size. If the experiment was performed on android tablets then the results may differ because tablet normally have larger screens than mobiles, operations on tablets are different from smart phones. Field based testing method was used during usability testing. If lab based testing method with change in test team size may also give different results. During usability testing on smart phones, fat finger problems are more commonly occurs. All test participants were university students who were aware with operations of smart mobiles and knew how to use a particular application. However, usability depends on users’ experience and preferences. The usability problems identified by participants may differ, if they have different experience and preferences [30].

Conclusion

Mobile application industry is growing very rapidly. Every day, lot of mobile applications are launched and failed due to poor interface design or user experience. Based on studies, there are two testing methods for usability testing- lab based testing or field based testing and both testing methods have some advantages and disadvantages. Lab based testing is costly option but give more accurate results as compared to field based testing. Most of the educational institutes developed mobile application for their students having information like- faculty details, course details and syllabus, updated time table, exam notifications and many more things. So it is very important to perform usability test on that application because effective usability testing is very important for the success of application. In case study, Usability testing was performed on android mobile app developed for students. On the basis of result of testing some suggestions and new interface design were advised to the development team to improve the user experience of students. In future, the usability testing will be performed again with modified application having multiple testing team size for better results and reviews. Comparative study will also be performed in between lab based and field based testing using the modified mobile application.

References

1. Gao J, Bai X, Tsai W, Uehara T. Mobile application testing: A tutorial. Computer. 2014;46–55.
2. Wei Q, Chang Z, Cheng Q. Usability study of the mobile library app: an example from Chongqing University. Libr. Hi Tech. 2015;33(3):340–355.
3. Sterling G. Report: Mobile Users Spend 80 Percent Of Time In Just Five Apps. Mobile Marketing. 2015.
4. Bruegge B, Dutoit AH. Object-oriented software engineering using UML. Patterns and Java- (Required). Prentice Hall .2004.
5. Zhang D, Adipat B. Challenges, methodologies, and issues in the usability testing of mobile applications. International Journal of Human-Computer Interaction. 2005;18(3):293–308.
6. Harrison R, Flood D, Duce D. Usability of mobile applications: literature review and rationale for a new usability model. J Interact. Sci. 2013;1:1–16.
7. ISO 9241: Ergonomics Requirements for Office Work with Visual Display Terminals (VDTs) International Standards Organisation, Geneva. 1997.
8. Mohan H. Testing the usability of mobile apps: A quick-starter guide on MOBILE UX TESTING. 2017.
9. Mifsud J. Usability Testing Of Mobile Applications: A Step-By-Step Guide. 2020.
10. Chynal P, Szymanski JM, Sobecki J. Using Eye tracking in Mobile application usability testing. ACIIDS. 2012;7198:178–186.
11. Balagtas-Fernandez F, Hussmann H. A Methodology and Framework to Simplify Usability Analysis of Mobile Applications. IEEE. 2009;520-524.
12. Masood M, Thigambaram M. The Usability of Mobile Applications for Pre-schoolers. Procedia-Social and Behavioral Sciences. 2015;197:1818-1826.
13. Betiol AH, Cybis WA. Usability testing of mobile devices: A comparison of three approaches. INTERACT. 2005;470-481.
14. Mansar SL, Jariwala S, Shahzad M, Anggraini A, Behih N, AlZeyara A. A Usability Testing Experiment For A Localized Weight Loss Mobile Application. Procedia Technology. 2012;839-848.
15. Fetaji M, Dika Z, Fetaji B. Usability testing and evaluation of a mobile soft- ware solution: A case study. 2008;501–506.
16. Kallio T, Kaikkonen A, Cankar M, Kallio T. Kankainen A. Usability testing of mobile applications: A comparison between laboratory and field testing. Journal of Usability Studies. 2005;1:4–16.
17. Az-Zahra HM, Pinandito A, Tolle H. Usability evaluation of mobile application in culinary recommendation system. IEEE Asia Pacific Conference on Wireless and Mobile. 2015;89- 94.
18. Rodríguez RA, Vera PM, Vallés GY, Martínez MR. Creating a Usability Lab for Testing on Mobile Devices Using Free Tools. IEEE CACIDI 2016 - IEEE Conference on Computer Sciences. 2016.
19. Borys M, Milosz M. Mobile application usability testing in quasi-real conditions. 8th International Conference on Human System Interactions (HSI). 2015;381-387.
20. Borys M, Milosz M. Mobile application usability testing in quasi-real conditions- the synergy of Using different methods. 11th International Conference on Human System Interactions (HSI). 2018; 362-368.
21. Muccini H, Francesco DA, EspositoP. Software testing of mobile applications: challenges and future research directions. In: Proceedings of the 2012 7th International Workshop on Automation of Software Test (AST).2012;29–35.
22. Kitchenham BA, Charters SM. Guidelines for Performing Systematic Literature Reviews in Software Engineering. School of Computer Science and Mathematics.2007.
23. Petersen K, Feldt R, Mujtaba S, Mattsson M. Systematic mapping studies in software engineering. In: Proceedings of the 12th International Conference on Evaluation and Assessment in Software Engineering.2008; 68–77.
24. Ghazizadeh FZ, Vafadar S. A Quantitative Evaluation of Usability in Mobile Applications: An Empirical Study. International Symposium on Computer Science and Software Engineering Conference (CSSE). 2017.