There has been extensive research in the use of eMedia [or more specifically digital media (dMedia)] delivered interventions for a variety of psychiatric disorders. Mobile apps delivering Psycho-social Interventions fall under the term mHealth. The term “mHealth” (mobile Health) is commonly used to describe the practice of medicine through mobile devices that are connected to internet.

Reviews in this area have so far focused on effects of eMedia delivered interventions using a variety of platforms [1-3]. However, more recently the reviewers have started examining the literature for usability, acceptability and feasibility of these interventions [4]. As far as we are aware there is no published literature that has reported development and initial testing of an eMedia delivered intervention in a scientific and systematic manner. Similarly, reported literature do not describe following a set of guidelines for quality control to develop eMedia delivered interventions. Only a few studies report details of the intervention developers, and an even smaller number of interventions tested in RCTs (Randomised Controlled Trials) are available in the public arena.

For replicability to be possible interventions should be clearly described. Poor description of interventions threatens the potential for good science in trials and this would seem an area of research where pioneering trialists could leave their work difficult to replicate through inexact descriptions of interventions. There is currently an interest in standardizing interventions in this area, and there have been attempts to establish criteria to self certify eMedia delivered interventions [5]. Similarly, there is an interest in evaluation of risk assessment of such interventions [6].

Surprisingly, the regulatory agencies have not paid sufficient attention to eMedia-delivered interventions until recently. Therefore, only little guidance is available for clinicians or end users. The Food and Drug Administration (FDA) released guidance recently, focusing primarily on mHealth apps (applications) that transform the mobile platform into a regulated medical device [7]. The remaining eMedia will be subject to no regulation [8]. In Australia guidance is available from the Therapeutic Goods Administration for health care practitioners, but this is very limited [9]. The Business Standards Institution (BSI) in England, has developed “PAS 277 Health and wellness apps - Quality criteria across the life cycle - Code of practice” [10] in conjunction with Innovate UK (www.innovateuk.gov.uk). There are however, no concrete guidelines available in any of the above systems on developing, initial testing and reporting of the mobile apps for Psycho-social Interventions. The mHealth delivered Psycho-social Interventions might offer unique opportunities and challenges, e.g., they might be delivered as stand alone interventions for common mental health problems, without medical support or supervision. Therefore, there is a need to develop strict protocols and quality standards that the app developers in this area should follow.

We are suggesting possible pathway to development and testing mobile apps for Psycho-social Interventions, using an example of drug approval by the FDA.

The pharmaceuticals follow fixed procedures to ensure quality of new drugs, and in order to report its efficacy, safety and dosage. Any new drug in the United States must go through an extensive review and approval process by the (FDA) before it can be prescribed to patients. A drug is any product that is meant to be used in diagnosing, treating, or preventing diseases by affecting the structure or function of the body. Prescription drugs, which must be obtained with the written authorization of a licensed health care professional, are regulated by the FDA, and some over-the-counter drugs are regulated by the FDA as well [11].

Pre-clinical trial stage

When a substance is considered to be a potential drug, it is initially tested in a laboratory. This is carried out in two steps. During the first step lab studies are conducted to test the drug’s effects on living cells, as well its toxic effects. The second step involves laboratory testing on animals to investigate the drug’s safety, and its efficacy.

If the results of the laboratory and animal studies are encouraging, the drug sponsor submits an IND (Investigational New Drug) application to the FDA. The IND application summarizes information from the laboratory and animal testing and provides a proposal for obtaining clinical data from human patients [11]. The FDA team reviewing the application can be composed of doctors, scientists, and statisticians.

Clinical trial stage (safety, dosage and efficacy)

If the FDA approves the IND application, clinical trials involving human participants start. The purpose of these trials is to test the safety, dosage and efficacy of the new drug. Clinical trials are divided into the following phases, although in real life one phase might test more than one outcome,

Phase 1 trials - normally involve a small number of participants, who are healthy volunteers. The focus is on drug safety.

Phase 2 trials - involve a reasonable number of participants with a disease or condition and focus on optimal dose and the ability of the drug to treat the condition.

Phase 3 trials - involve even a higher number of participants with a disease or condition. The focus of these trials is safety and efficacy. These trials use an RCT design and commonly compare the drug with a placebo or a gold standard drug. These trials look into more in-depth questions, such as effects on certain groups of patients.

Once the clinical trials are completed, the pharmaceutical submits an NDA (New Drug Application), which requests approval of the drug for marketing in the United States. If the drug is approved by the FDA, the official label for the drug is written, which describes the indications, known side effects and warnings about the product [11].

The goals of the NDA are to provide sufficient information to permit FDA to establish the followings: [1] whether the drug is safe and effective when used as directed? [2] are these benefits more than the risks? [3] Is the drug’s labelling appropriate, and what information should it contain? [4] Did the pharmaceuticals use appropriate methods in developing, testing and reporting the drug? and [5] Does the label tells the complete story of drug development, psychopharmacology, toxicity etc.

The label may be updated to include information about the drug’s new indications and the side effects. The pharmaceutical is required to submit safety updates, and both the prescribers and the patients can also report serious events related to the drug to the FDA. Some drugs are withdrawn if they cause very serious side effects. Further trials are conducted to test the effectiveness (pragmatic trials) of the drug, to measure the degree of beneficial effect under “real world” clinical settings (Table 1).

Table 1: Comparison of drug and mobile apps for psycho social interventions: Development and testing.

An end to end testing process begins with engaging the testing team upfront while requirements are being defined. These should include non-functional, regulatory and performance requirements. These requirements must be inspected by all the stakeholders and signed off. It is mandatory for all the participating business/engineering teams to follow step-by-step processes, that can be summarized in following steps;,

1. Plan and prepare (Requirement Traceability Matrix, Entry/Exit/ Success Criteria, Metrics/Measurements/Key Performance Indicators). (Requirement inspections-These inspections should be conducted by the business, engineering, solution architecture, testing and implementation teams), (Requirement Tractability Matrix-Requirements must be linked to one or more test cases as needed. Defects must also be lined with failed test cases), (Key Performance Indicators- definition of success and key objectives)

2. Design (Test Scenarios, Test Cases, Test Steps, Test Data)

3. Execute (Unit Testing, System Integration Testing, Performance Testing, Business Acceptance Testing, Alpha Testing, Beta Testing) [Unit Testing - Conducted by the engineering Team, (System Integration Testing - Conducted by Testing Team), (Performance Testing - Conducted by Performance Engineers to measure an application against defined “Load Conditions”), (Alpha Testing - Conducted by engineering/testing teams) and (Beta Testing - Conducted by real end users)

4. Transparent reporting to all the stakeholders

5. Continuous Improvements - Improvements to the application based on user's feedback

Pre-alpha stage

The pre-alpha stage refers to all activities performed during the software project before testing. These activities can include requirements analysis, software design, software development, and unit test

Alpha testing

The alpha phase is the first step to software testing. In this phase, the app is tested using a variety of techniques, most notably white box techniques, that test the internal structures or workings of an app [12], and the black box techniques that test the functionality of an app. Additionally, the gray box testing combine both black and white box techniques, by another testing team [13].

Beta testing

Once the app is feature complete, the beta testing is carried out. This can reveal unknown defects or bugs. This is normally tests usability or acceptability of the app. The process of delivering a beta version to the users is called “beta release” and this is typically the first time that the software is available outside of the organization that developed it. When new features and functionality are constantly added to the app it is called “perpetual beta”. The beta testing can either be “closed beta” in which beta testing is done by a closed group of individuals or “open beta” in which testing is done by a larger group.

Release of the app

Once the beta testing is complete, the app is called a release candidate (RC) or the final product. In this stage of product stabilization, all product features have been designed, coded and tested through one or more beta cycles with no known “show-stopper-class bug”. This release is also called “code complete”. Once released, the software is generally known as the "stable release".

Support and end-of-life

During its supported lifetime, software is sometimes subjected to service releases, or service packs, sometimes also called "interim releases". When software is no longer sold or supported, the product is said to have reached end-of-life.

The way forward: A proposed pathway to development and testing of the mobile apps

We suggest a pathway to develop and test apps that deliver Psychosocial Interventions, that can follow the following sequence;

Development stage

In addition to the steps described in pre alpha testing stage, the initial development stage should also consider the type of the interventions. The following types of interventions are normally delivered by eMedia; Type 1 [An intervention delivered by a human therapist through eMedia (e.g., telephone delivered problem solving by a therapist, Avatar Therapy)], Type 2 [An intervention based on a manualized, well established therapy delivered through eMedia (e.g., CBT delivered by a website that is based on a manual, or thought diaries)], and, Type 3 [A new intervention that did not exist before, and is not based on previous theory or on therapeutic principles (e.g, electronic dispensing), or if it is, it is based loosely in theory]. It, therefore, follows that both Type 1 and 2 interventions will require less rigorous testing. Although, type 1 might only be used by a therapist (on prescription versus over the counter apps).

Alpha testing

The process of app testing can start normally with alpha testing (white and black box testing).

Artificial intelligence (AI) testing

Artificial Intelligence is about making machines intelligent [14]. There has been a recent increase in interest in the application of Artificial Intelligence (AI) techniques to Software Engineering (SE) problems. The work in this field is typified by recent advances in Search Based Software Engineering, but also by long established work in Probabilistic reasoning and machine learning for Software Engineering [14]. This is an area that is full of controversies. To start with, for example, there is no agreement on definition or function (s) of intelligence, among scientists. There are published reports of intelligent Real Time Therapy (iRTT), using experiential sampling methods (ESM) [15], however, none of the available interventions in this area can be considered intelligent using the current criteria for artificial intelligence. This is, however an area that can be considered in future app testing.

Clinical testing

This can typically focus on efficacy, but the secondary objectives include, testing for dosage and safety. This stage can also include acceptability [User Acceptance Testing (UAT) how the end user experience the app] and feasibility [whether the intervention can be implemented given time, financial, legal, personal, and social constraints]. This can be divided into the following steps:

Beta testing

We recommend that Beta testing should be conducted in two stages, i.e., closed and open beta testing.

Closed Beta testing (phase 1 trial)

This phase can involve a small number of healthy volunteers to test the safety (side effects, privacy and security). There is evidence that Psycho-social Interventions cause adverse outcomes [16]. Furthermore, there may be risks particular to the eMedia such as reduction in social activities, or even addiction to eMedia [6]. However, we are not aware of published literature in this area. Delivery through eMedia complicates and possibly increases risks, in terms of security and privacy of data [17]. Additionally, preliminary data on acceptability and feasibility can be collected.

Open Beta testing (Phase 2 trial)

This can typically test dosage (we believe that the concept of dosage is important here, to offer advice on duration of the app use, in order to avoid risks of side effects), acceptability, and feasibility. This phase can further test efficacy by the end users in a pre and post test design.

Gamma testing (Phase 3 trials)

This phase can focus solely on efficacy using RCT design, that compare the app with either a placebo (for example, a game app that has no activity) or an already approved app that is known to work for the condition being treated. Secondary objectives can include testing safety and acceptability.

Download history and user feedback

In addition to the Phase 2 and 3 trials, the number of downloads and feedback from the participants should be essential part of the evaluation

Marketing and Post marketing stage and effectiveness trials

Once the initial testing has been carried out, the app can be launched. The app at this stage should provide “information sheets” similar to those provided by the drug company (Please see Box 1). Future trials can focus on effectiveness, including cost effectiveness.

The increasing number of mobile apps requires careful consideration of how they are developed, tested and marketed. There is a need for professionals from both the IT and the Health background to work jointly in these areas. Mobile apps for Psycho-social Interventions can be developed using a process similar to the discovery, development, testing and marketing of drugs. There are notable differences between the two, for example the apps have a relatively short life, compared with the drugs, due to rapid developments in the field. This, however, also highlights a further complication, due to an ongoing process of improvement based on technological advances and user feedback, the app might change radically after some time. This also means, the period for app

development and testing should be relatively short, as compared with testing and marketing of drugs. Researchers and clinicians might also like to consider developing a mobile version of new Psycho-social Interventions. However, there are some areas that need further research. For example, the concept of safety (side effects, privacy and security), acceptability and feasibility should be further explored.