Study design
We conducted a choice-based conjoint analysis (CBCA), a method for preference elicitation that uses a survey instrument to force participants to trade-off between attributes, allowing their underlying preferences to be estimated through statistical analysis53,54,55. The basic principle of CBCA is that the profiles that constitute a choice set share the same attributes but are differentiated by the attributes’ levels, which are controlled experimentally. Participants make choices in a series of choice sets, and the pattern of responses allows us to quantify the impact of changes in attribute levels on choice56. There are several types of CBCA, with the two main ones being traditional CBCA, which presents a fixed choice task with a set combination of different product attributes, and Adaptive Choice-Based Conjoint (ACBC), in which the tasks are adjusted based on the respondent’s preferred attributes identified from the initial choice57,58. In this study, we adopted the traditional CBCA approach of presenting fixed-choice tasks. The methodological framework of this study (including instrument, design, deployment, and data analysis) was consistent with ISPOR—the Professional Society for Health Economics and Outcomes Research Good Research Practice for Conjoint Analysis59.
Development of attributes and levels
The attributes and levels in this study were designed based on the evaluation criteria of the ‘National Healthcare App Accreditation Program in Korea.’ The accreditation program divides the types of certifications into three categories (chronic disease management, lifestyle modification management, and simple information delivery) according to the characteristics of mobile applications (see Supplementary Table 2). According to previous research, where many studies have used six attributes, we have also set the number of attributes in our study to six60. There are a total of 14 indicators in the accreditation program; we selected four indicators that are evaluated in all three certification types: (1) the number of studies on effectiveness, (2) frequency of information delivery, (3) cybersecurity and data safety, (4) user satisfaction. We also included two additional variables, ‘accreditation’ and ‘cost’, resulting in six attributes (see Table 4). To determine the levels for the selected attributes, we either directly adopted the indicators from the accreditation program or referred to the levels utilized in previous studies22,23,26,61,62.
Table 4 Description of Attributes and levelsThe number of studies on effectiveness
This attribute was divided into levels of 0, 1, 2, and 3. In previous studies, the ‘number of studies concerning safety and effectiveness’ levels were set at ‘0, 1, 2, 3, ‘ while the levels for ‘proven effectiveness’ were set as ‘yes, not yet’22,23. In our study, we opted for polytomous levels instead of binary levels to derive more meaningful analyses.
Frequency of information delivery
The levels for the frequency of information delivery were set at 2, 4, and 6 times per month. Due to the lack of prior research on this attribute, these levels were aligned with the evaluation criteria of the Korean accreditation program.
Cybersecurity and data safety
We categorized cybersecurity and data safety into three levels – low, medium, and high – due to the difficulty of assigning a precise numerical score to this attribute. Similarly, previous research has classified the attributes of ‘security and privacy’ into three levels: no security assurance, some assurance, and complete security assurance26.
User satisfaction
User satisfaction is categorized into under 80, between 80–89, and above 90. In previous studies, attributes such as ‘the app has been recommended by other healthcare professionals’ or ‘the ratings of the app’ were used, with levels defined as ‘yes or no’ or ‘3.2, 4.0, 4.8 (out of 5), ‘ respectively23,26. We maintained three levels but a more intuitive and familiar scale to obtain more insightful data, dividing scores in 10 out of 100 increments.
Accreditation
Following previous research, the accreditation level was set as binary, with ‘yes’ or ‘no’ options23.
Cost
Our previous research found that WTP for mobile health services in South Korea ranged from $9.3 to $16.0, depending on service experience and type37. Based on these findings, we categorized the cost levels into multiples of 5, with a minimum value of 0 and a maximum value of 15.
We utilized the software provided by Sawtooth (Utah, USA) to design our CBCA, generating 15 paired questions to present the attributes and levels of the service. Sawtooth is widely recognized in the field of conjoint analysis, allowing us to efficiently model and measure respondents’ trade-offs between different service attributes63. According to previous research, the choices typically used include binary options, where respondents choose between two alternatives, or ternary options, where three alternatives are provided64. Following prior studies, by presenting two pairs of choice, we could manage the cognitive load on respondents while still gathering robust data22,23,26,65.
Participants
An online/mobile survey targeting healthy individuals aged 20 years or older (n = 407), patients with hypertension or diabetes (n = 589), and physicians (n = 97) was conducted. The inclusion criteria for this study were as follows: (1) adults aged 19 years or older; (2) individuals without any diagnosed chronic disease, patients diagnosed with hypertension or diabetes, and physicians with a medical license number; and (3) no difficulty in completing the online questionnaire. Exclusion criteria are (1) those who did not consent to the study and (2) those who lacked the decision-making capacity to complete the online survey.
For healthy individuals and patients with hypertension or diabetes, e-mails were sent to online panel members registered with Gallup Korea (Seoul, Korea) to provide information on the survey outline and to participate if desired. For the healthy individual group, we asked whether they had ever been diagnosed with a chronic disease, using hypertension and diabetes as examples. We only included those who reported no diagnosis. In the case of the patient group, we asked whether they had been diagnosed with hypertension (ICD-10: I10–I15) or diabetes (ICD-10: E10–E14), and those who responded that they had been diagnosed were included in the patient group. Healthy individuals were recruited through proportional allocation based on sex, age, and region in Korea as of 2022. Patients with hypertension or diabetes participated in the survey on a first-come, first-served basis because it was difficult to determine the parameters for proportional allocation.
The physician participants were recruited through an announcement posted on a physician community website with approximately 23,000 registered physicians. The website verifies the license numbers of its members to ensure they are practicing physicians. The website allows physicians to chat, use anonymous message boards, provide lecture content, and post job listings. It also offers a service for surveying physicians, which we used to recruit participants and conduct the survey.
Healthy individuals and patients who participated in the survey received $2.50 (1 USD = 1200 KRW) as incentives, and doctors received $16.67. A survey of healthy individuals and patients with chronic disease was conducted between January 16 and February 2, 2023, and a survey of physicians was conducted between August 10 and 14, 2023. No follow-up reminders were sent to participants.
Online survey procedure
The participants first responded to questions about their demographic characteristics (sex, age, and place of residence). Only the patients then answered additional questions regarding the duration of their illness, medication, family history of hypertension or diabetes, and complications. Healthy individuals and physicians did not answer these questions and proceeded directly to the next step. Next, they read an explanation of mobile-based healthcare services. This service assists individuals in managing their health by using personal data entered through a smartphone, such as steps, weight, and meal records. Based on this data, the service creates personalized exercise and diet plans. It also provides support through regular messages that offer tailored counseling, education, and advice, helping users keep healthy habits and make informed health decisions.
Subsequently, the participants were asked to choose their preferred service from the two paired services, each presenting different levels of a given attribute, as shown in the example in Fig. 2. They completed a total of 15 such questions. To maximize the information collected and reduce interpretation bias, we did not provide an opt-out option and required participants to select one of the two service options. We also enforced a rule requiring participants to spend a minimum of 15 seconds on each question, ensuring they had sufficient time to consider the attributes and levels. The required time was set in consultation with Gallup Korea’s members, recognizing that it is easy for participants to respond quickly without providing adequate consideration in online research. In addition, an explanatory text, as presented in Table 4, was provided to help participants to understand the factors accurately. It was continuously presented at the top of the survey page so they could view it while answering the survey. In addition, to help participants identify the differences in levels for each attribute, we used different font colors to highlight the differences between the two choices. The questionnaire items were randomly rotated and presented to reduce order bias. Gallup Korea developed the survey pages.
Fig. 2: The flow of choice tasks and examples of options.
In this survey, participants selected one of two service options with differing attributes across 15 separate questions. They had to choose between the two options without the ability to skip or opt-out. A 15-second minimum time limit was enforced for each question to encourage thoughtful consideration of the choices. Descriptions of the attributes were persistently displayed at the top of the page for clarity. The order of the questions was randomized to mitigate the risk of bias from the sequence of questions.
Statistical analysis
For this study, we analyzed participant preferences using a conditional logit model implemented in STATA version 16. We coded all attributes as categorical variables, except for cost, which was treated as continuous to identify trade-offs in WTP for attributes of the healthcare service app23. We estimated a main effect model for each group. A WTP analysis was also conducted to understand how participants are willing to trade one attribute for another. It is crucial in this study because differences in preferences cannot be compared solely based on beta coefficients from a main effects analysis, given that the study involves three different populations. As shown in Eq. (1), the WTP for each attribute was calculated by dividing attribute’s beta coefficient by the cost attribute’s beta coefficient, enabling us to estimate the WTP for specific attributes23.
$${\rm{WTP}}i=\beta i\div\beta {\rm{cost}}$$
(1)
WTPi represents the willingness to pay for attribute i.
βi is a beta coefficient for attribute i.
βcost is a beta coefficient for the cost attribute.
We further divided into subgroups based on age and gender and calculated each group’s WTP. This analysis reveals how much each subgroup will pay for a particular attribute.
Ethics approval
All procedures were conducted according to ethical standards and the principles outlined in the Declaration of Helsinki. This study was approved by the Institutional Review Board of Yonsei University (4-2022-1517). After reading the explanation page, which included information about the study’s goals, participants, and data storage time, all participants consented. There was no preregistration for this study.