FFQIS: Food Frequency Questionnaire with Intense Sweeteners; CVI: Content Validity Index; I-CVI: Item Content Validity Index; S-CVIave: Scale Content Validity Index; Kappa: Cohen`s Kappa; IQR: Interquartile Range

Artificial sweeteners comprise several vegetal origins or synthetically formed substances used in the food industry because of their high potential sweetness and low caloric value. To date, the Food and Drug Administration (FDA) has approved six artificial sweeteners and in Europe, twelve are already commercialized. Although international and most of the countries regulatory agencies consider these products safe, their risks are still a matter of controversy as new evidence has emerged on their effect on metabolism and have created controversies among the scientific community [1,2].

The increase of consumption of non-caloric sweeteners in food and beverages has come in parallel with an abundancy of processed products in the market place. Ng et al. identified 85,451 formulated consumer packaged food products in the United States between 2005 and 2009; from these, 1% contained non-caloric sweeteners and 6% had a mixture of caloric and non-caloric additives [3]. Consumers trust that the amounts of additives for each particular product is within the Acceptable Daily Intake set by international regulators [4], but it is clear that consumers are not aware of the mixed amount of intense sweeteners they eat in a daily basis. According to an American Dietetic Association press release [5], “consumers do not distinguish amid the various kinds of sweeteners contained in foods and beverages they consume”. The regulatory agencies do not require manufacturers to include on labels the actual amount of non-caloric sweeteners contained when the product is labeled as “Generally Recognized as Safe” (GRAS) [6].

Beside, social organizations have overturned their efforts on raising awareness on the calorie restriction consumption and have put aside potential risks by overconsuming non-caloric sweeteners. The disparity between excessive information on natural sweeteners such as sucrose, glucose or fructose and the limited communication on intense sweeteners has made a gap of information despite their increasing consumption worldwide [5].

Intense sweeteners are mainly used in the production of industrial foods with low caloric value, and these are commonly known as light as they maintain the original sweetness of the product. Their caloric content is below that of many other food products, offering a real possibility for weight control among consumers. However, new studies in young adults have revealed association between non-caloric sweeteners intake and the increase of accumulated abdominal fat, metabolic diseases, hypertension, insulin resistance and diabetes type 2, [7-9] which have not been confirmed by others [10].

Soda products are the most consumed among sweetened beverages with intense sweeteners as they have been positioned as a healthy alternative to regular food and drinks [11-14]. Teenagers are a particular group described as important partaking in the excessive sweetened fluids intake. From the total beverage intake per day, it is estimated between 4 and 18% corresponds to drinks prepared with non-caloric sweeteners. Likewise, among the US population, estimates are as high as 15%, indicating beverages consumption with intense sweeteners has increased in the general population and doubled among children over the last 10 years [15-17].

Considering the extent to which non-caloric sweetener products are increasing, there is a need for more intervention studies in order to evaluate consumption quantity and effects, and to make recommendations based on the evidence related to the use of non-caloric sweeteners [18].

Nowadays, intake of intense sweeteners in industrialized food products is at its highest, therefore there is a need to have accurate instruments to estimate total intake of intense sweeteners, however, no validated instruments exists to objectively quantify the level of consumption of foods and drinks with added non-caloric sweeteners, therefore, the aim of the study was to design a questionnaire to assess the content validity and reliability of a food frequency questionnaire for intense sweeteners (FFQIS) among a Hispanic population.

We carried out a study separated into two sections, first using a cross-sectional design for validation and second a prospective cohort design for test retest. Studies were approved by the research and ethics committee in our hospital, and all participants provided informed consent (Figure 1).

Figure 1: Flow diagram of study process: methods for validity and reliability.

Validity participants and measurement

Content validity was assessed by inviting experts to participate; specialists (n=40) with a master’s degree or doctorate in nutrition, medicine or other related field were contacted through a network. Those agreeing to participate received an e-mail invitation to rate the FFQIS items with regards to construct, study population, and purpose on a 4-point Likert scale (highly relevant =4, quite relevant =3, somewhat relevant =2, not relevant =1). Comprehensiveness was measured by asking the specialists whether the items covered the entire construct measured.

Statistical analysis

The Item Content Validity Index (I-CVI), defined as the proportion of experts who rated the content as valid (relevance rating of 3 or 4), was calculated for each item. Items were rated excellent when the I-CVI value was greater than 0.78; items considered excellent (I-CVI ≥ 0.78) were thus retained for Study 2. Items on the threshold (I-CVI=0.75, having 2 raters) were individually assessed. For complete scale validation, all I-CVI values were averaged to calculate a Scale Content Validity Index (S-CVI), for which a value exceeding 0.90 is considered excellent. Data were analyzed using SPSS version 22.0.

Reliability participants and measurement

To validate the reliability of the questionnaire, a Hispanic population random sample was taken, comprising participants recruited at the General Hospital of Mexico “Dr. Eduardo Liceaga”, in the period between March and July 2016.

Two individuals were calculated to participate per each item included in the food consumption frequency, there were 141 foods to be evaluated accounting for 282 individuals plus 5% losses. At the end, 300 people agreed to participate in the test retest study.

Questionnaire design and study procedure

Reliability was assessed by comparing estimates from the food with non-caloric sweeteners frequency questionnaire, applied twice, with 2 weeks apart, among a Spanish speaking sample (test-retest agreement).

FFQIS captured consumption habits for food containing non-caloric sweeteners during the previous 2 weeks from each individual. Consumption patterns by categories (6 food groups) were supported with images of each food. To validate the reliability of the FFQIS, we implemented a food diary to be filled out by the participant for each time the questionnaire was applied.

Statistical analysis

Sample size was calculated considering a 5% alpha value, 95% confidence level and a z value of 1.96, including at least 300 healthy volunteers.

The level of agreement between the test-retest was evaluated using a Cohen`s kappa coefficient (k) and interquartile range (IQR). The values obtained for k were interpreted according to the cut-off points proposed by Landis and Koch (1977) [11]; thus, values of less than 0.21 indicate poor agreement, 0.21 to 0.40 fair agreement, 0.41 to 0.60 moderate agreement, 0.61 to 0.80 substantial agreement, and greater than 0.80 almost perfect agreement.

Data entry and analysis were performed with SPSS Version 22.0 (Chicago, Ill, USA). Descriptive statistics and Kolmogorov-Smirnov (for testing normality) were used in this study. A p<0.05 was considered as statistically significant.

Content validation

Forty out of 65 specialists invited to participate in this research completed and returned the survey questionnaire. Respondents were primarily men (79%), with a mean age of 33 ± 5 years and a mean of 10 ± 8 years of experience in nutrition or research activities.

The initial FFQIS included 141 foods derived from a survey which identified the food offer with non-caloric sweeteners offer at the main food stores. Of which the specialist considered unnecessary 47 items and necessary but no presents in the questionnaire 9 items, finally were included 103 foods divided into 6 groups, with corresponding images of each food. The CVI in whole FFQIS was 0.92 (range 0.60-1.00). In group 1 we included, solid and drinking yogurt and had CVI=0.90, in group 2, ice cream, jam, gelatin, maple syrup and chocolate flavoring and had CVI=0.95, for group 3 chocolate bars and chewing gum has CVI=0.90, in group 4 nectars, flavored squash, soft drinks and teas had CVI=0.95, in group 5 bread, biscuits and corn flakes had CVI=0.90, and finally in group 6 artificial sweeteners and others had CVI=0.60 (Table 1). In conclusion, the experts considered the 103 foods in the questionnaire relevant and suitable for application in the prospective cohort study.

Table 1: The table values indicate I-CVI included items in the FFQIS.

Reliability of questionnaire

Three hundred out of 320 participants recruited, completed the study and were included in the analyses. Two hundred and eight were female (69.3%) and ninety two were male (30.7%); their mean age was 28 years old (range 18-57). Excluded participants were people who did not follow the study protocol indications or failed to deliver the 24- hour recall or the Food Frequency diary.

Table 2 shows average kappa values between test-retest of FFQIS, as reported in each dietary questionnaire. The mean k value for group 1) yogurt and yogurt smoothie drink k=0.33 ± 0.27; for group 2) ice cream, jam, jellies and maple flavored syrups k=0.64 ± 0.20; for group 3) chocolate bars and chewing gum k=0.66 ± 0.06; for group 4) nectars, flavored waters, soft drinks, teas k=0.53 ± 0.23; for group 5) bread, biscuits and corn flakes k=0.40 ± 0.30 and for group 6) sweeteners and other foods k=0.59 ± 0.24.

Table 2: Representative kappa values for test-retest of FFQIS.

The comparative analysis between the second FFQIS and the food diary showed a significant k value (k=0.412; P< 0.05).

Likewise, strong and significant correlation was found between intakes of intense sweeteners in the test-retest of FFQIS (r=0.892; P<0.05).

This study describes the content validity and reliability of a food frequency questionnaire that deals with intense sweeteners (FFQIS), among a Hispanic population. The validity of the questionnaire was determined with respect to its content and reliability, achieving an I-CVI of 0.92 and k=0.53, respectively. At first sight, this FFQIS has a good and acceptable appearance in terms of content validity, and reliability.

In the first study, experts judged the majority of the items to be relevant. Groups of sweeteners and others were the only groups with a low I-CVI (0.60) value, indicating that these items are apt for measuring the construct; however, we needed to include more items, thus more foods were included in the final FFQIS. In the other groups, the FFQIS demonstrated excellent content validity, proving the relevance of all foods included in this instrument for measuring consumption habits for food with added non-caloric sweeteners. This FFQIS showed satisfactory characteristics in terms of reliability, structural validity, and concurrent criterion-related validity, according to the criteria provided by Davis and Polit et al. who provide evidence of content validity for instruments through to content validity index, based on expert ratings of item relevance [19-21]. We have obtained the I-CVI of 103 foods into 6 groups and concluded that our FFQIS shows an I-CVI value mean of 0.86 for all groups (Table 1). These results appear to endorse good content validity, because researchers recommend that a scale with excellent content validity should be composed of I-CVIs of 0.78 or higher [20,21].

In the second study, we measured the reliabilities of functional status of FFQIS, using an interval of 1 to 2 weeks between assessments, because during this time, consumption habit is unlikely to change appreciably among consumers of artificial sweeteners. In the test-retest, agreement ranged from 0.16 to 1.00 for all foods included in each group, weighted kappa values were between 0.33 and 0.66 and for the majority of groups, the kappa values, showed moderate to substantial agreement (Table 2). We use kappa because it is mainly used to test inter-rater agreement among observers who rate dichotomous categories of data, in this case consumption or non-consumption of each of the foods in the FFQIS. A minimally acceptable kappa of 0.60 for inter-rater agreement was recommended by Gelfand and Hartmann, but Landis and Koch provided benchmarks for various levels of kappa magnitude and strength of agreement; a positive kappa indicating that inter-rater agreement occurred more frequently than would be expected to occur by chance [18,22,23]. Our kappa values in the test-retest denote the ability of the FFQIS to sufficiently evaluate non-caloric sweetener intake in the Hispanic population and the strong correlation between the intakes of non-caloric sweeteners in the test-retest of FFQIS. Finally, the comparative analysis between the second FFQIS and the food diary showed a significant kappa value. These results indicate that this FFQIS is reliable and that measurements manifest low standard errors.

In this study, we validated a questionnaire that made it possible to identify FFQIS in the Hispanic population. The present study validated a fist FFQIS in the Hispanic population.

Ethics approval and consent to participate

The study was approved by the committee of research and ethics in our Hospital with the number DI/16/301/03/022.

Availability of data and material

All materials described in the manuscript, including all relevant raw data, will be freely available to any scientist wishing to use them for non-commercial purposes, without breaching participant confidentiality.

Competing interests

The authors declare that they have no competing interests.

Funding

The study was funded by the National Council on Science and Technology (CONACYT) with the number 261575.

Melendez-Mier provided the initial idea for the study. Bueno- Hernández established the data file and provided background information. Melendez-Mier and Bueno-Hernández were responsible for study design, analyses, and drafting the report. All authors oversaw the study design, contributed to interpretations of the findings, reviewed literature, helped in writing the report and have seen and approved the final version of the paper.

We thank all participants and experts who took part in this study: Adriana Hernández Alarcón, Ana P. Carbajal Méndez, Denisse Castillo Herrasti, Eliud S. Aguilar Barrera, Elizabeth Solís Pérez, Gabriela Olguín Contreras, Gabriela Quiroz Olguín, Grecia L. Blanco Melo, Guisela Mena Mora, Iván F. Celaya Correa, Ivonne Escalona Nandez, Jesús Pérez Garrido, Julián Vázquez Talavera, Karla J Arcos Rabago y Macías, Laura G. Chico Barba, Lizbeth Arce Granados, Loredana Tavano Colaizzi, Luis Ortiz Hernández, Manuel A. Romero Mijares, María del Mar González Nava, María E. Meza Delgado, María F. Rivas Aguirre, Mariel Vera Amaya, Marisa Y. Quiroz Salcedo, Martha Núñez Aldana, Mayela Martínez Saavedra, Oliver A. Sollano Trejo, Paola V. Miranda Alatriste, Renata L. Rivera Flores, Rocío Ramírez González, Sandra Hernández Valencia, Sandra Pacheco Veintiau, Victoria E. Bolado García. We particularly thank the study participants for their important contributions.