Objectives:As the evidence base and public demand for integrative health and complementary and alternative medicine continue to grow, the need for pragmatic data to guide its integration into standard medical practice is also growing. A cost-effective way of gathering this data is in conjunction with randomized controlled trials of interventions of interest. Unable to find brief, generalizable instruments to collect willingness to pay and ideal treatment frequency data from the patient perspective in conjunction with clinical trials, we developed the Intervention Frequency and Cost Survey to fill this gap. The current study describes this survey’s development and psychometric properties.

Methods: Based on a bibliographic review of existing instruments, we developed the first survey version which was modified in an iterative process according to a priori standards for face and content validity, measured using Lawshe’s content validity ratio. Second, Pearson’s correlations were used to measure test-retest reliability in two demographically-distinct groups of massage study completers with knee pain due to osteoarthritis. Finally, all completers of a massage therapy clinical trial (n=179) were invited to complete the survey in order to collect survey administration feasibility data.

Results: Face validity data exceeded a priori standard of 90% on its first presentation to a panel of 10 naïve raters. Content validity, assessed by 10 experts, exceeded a priori standards for each survey item (Content Validity Ratio>0.62), and test-retest reliability was excellent (Population 1: r = .9035, p< .0001, n=28; Population 2: r = .9635, p<.0001, n=10). Response rate was 66%, and average time to complete the survey was 2.1 min (SD=±2.0).

Conclusion: The Intervention Frequency and Cost Survey demonstrated high face validity, solid content validity, and excellent test-retest reliability. It is a practical instrument for measuring patient willingness to pay and ideal frequency of massage therapy. Future studies should evaluate use of this instrument for efficient collection of cost and frequency data for additional health interventions.

Keywords: massage, complementary and alternative medicine, cam, osteoarthritis, knee, chronic pain, integrative medicine, health policy, willingness to pay, out of pocket expense, clinical decision making, online surveys

IH, integrative health; CAM, complementary and alternative medicine; RCT’s, randomized control trials; IFACS, intervention frequency and cost survey; NCCAM, national centre for complementary and alternative medicine; CVR, content validity ratio; SMEs, subject matter experts

As the evidence base for integrative health (IH) and complementary and alternative medicine (CAM) increases, and public demand for IH/CAM continues to grow, the need for pragmatic data to guide integration of these interventions into standard medical practice is also growing. A cost-effective way of gathering this needed data is in conjunction with randomized control trials (RCT’s). To fill the gap of practical, reliable and valid instruments to be used in translational and implementation research of specific IH/CAM therapies, this study developed and validated the Intervention Frequency and Cost Survey (IFACS). Specifically, the IFACS enables data collection from participants after clinical trials of any intervention regarding patient willingness to pay and ideal treatment frequency.

High rates of CAM usage by Americans are changing national health care priorities. According to the 2007 National Health Interview Survey of more than 32,800 Americans, 38.2 percent of adults in the United States and nearly 12 percent of children used some form of CAM within the previous 12months.¹ At the time, CAM usage was estimated to account for 1.5 percent of all United States health care expenditures ($2.2trillion),² but almost all of the money spent on CAM was out of pocket ($33.9billion).³ In other words, even though almost no CAM was covered by insurance, well over a third of American adults felt it was important enough to their health to pay for it out of their own pocket. In particular, reported usage of massage therapy increased 3.3% from 2002 to 2007, 1.5% more than the next most-used CAM approach included in the survey: yoga.⁴ {Blackwell, 2014, Summary health statistics for U.S. adults: national health interview survey`, 2012} (Because complementary health approaches included in the 2007 NHIS were not consistent across years, NHIS results from 2012 are not reported here). “Demand” may be defined as, “An economic principle that describes a consumer's desire and willingness to pay a price for a specific good or service.⁵” Using this definition, the 2007 NHIS findings suggest substantial consumer demand for CAM, particularly for massage therapy.

In response to this public demand, national health research priorities are changing. The National Institutes of Health’s National Center for Complementary and Integrative Health (NCCIH), formerly known as the National Center for Complementary and Alternative Medicine (NCCAM), has responded to the high rate of CAM use by adjusting their strategy to include more studies informing real-world implementation of CAM. For example, NCCIH’s current strategic plan states, “As in the past, our plan emphasizes the importance of basic and clinical research as the core of building the evidence base for CAM. But in this plan, we give increased emphasis to translational research and bringing the methods of effectiveness and outcomes research to the real world where public use is extensive.⁶ Accordingly, NCCIH’s current third strategic aim is to “increase understanding of ‘real world’ patterns and outcomes of CAM use and its integration into healthcare and health promotion”.

This study supports the current national research priorities emphasized by NCCIH by developing and evaluating a practical instrument to enable data collection on consumer demand that works hand in glove with randomized clinical trials. Furthermore, development of this practical instrument is aligned with NCCIH’s emphasis on “understanding real world patterns of CAM use to guide its integration into standard healthcare” as it will allow for data collection that can improve decisions that affect medical care at the levels of both policy and the individual.⁷

Presently, there are very few studies of massage therapy that include willingness to pay and/or ideal frequency data, which is important for implementing massage therapy in real-world settings. Those few that have asked participants about their willingness to pay for massage therapy have shown conflicting results.⁸ None of the studies identified specific out of pocket amounts. Furthermore, to our knowledge, no studies have used an instrument like IFACS, designed to be administered several months after completion of an RCT of a particular treatment, to investigate whether study completers sought out the treatment on their own and the real-world decisions they made with regard to treatment frequency and payment.

Overview

Evaluation of IFACS took place in two phases to test instrument psychometrics and feasibility (see Table 1). Steps of survey development included creation of the initial survey based on study aims and literature review followed by an iterative process of face validity testing and survey update to meet a priori standards. A similar, iterative process was used for content validity testing. Once the standards were met for face and content validity and the final, online survey had been reviewed to eliminate errors and ensure proper function, test-retest reliability was tested in two separate populations according to the following methods, reviewed and approved by the Duke University Institutional Review Board.

Phase I

Survey development and psychometric testing: Consistent with this study’s aim to provide an efficient, reliable and valid tool for collecting data about the value placed by patients on a particular CAM treatment, we wanted to create a survey that was easy to understand and practical to administer while maximizing the usefulness of the data collected with it. Previous studies demonstrate higher response rates for web-based surveys for which an invitation was sent by email which included the survey URL⁹ and higher rates of survey completion for shorter surveys.¹⁰ Hence, to maximize response rates and survey completion, IFACS was designed to be an online survey that can be completed in about two minutes. We selected a simple, two-module design to gather data about both actual and ideal treatment scenarios to ensure the instrument captured the value placed on the treatment by the participant, regardless of whether or not they were able to receive it.

In order to maximize clinical relevance, IFACS was designed to be administered after an RCT of the treatment of interest. Administering the IFACS after completion of a related RCT has the following advantages: 1) since the IFACS is sent to study completers, separate sampling strategies and recruitment procedures are not needed, contributing to data collection efficiency; 2) using a relatively homogeneous sample limits residual variance (error), allowing individual differences in the variable of interest to be more easily discerned; and 3) it allows the treatment frequency and willingness to pay data to be linked to outcomes data from the RCT. The latter linking contributes to data collection efficiency because information such as demographics and primary study outcome does not need to be collected again. Linking willingness to pay and treatment frequency preferences after study completion to treatment response during the core study provides useful translational data to better understand uptake of CAM therapies as well as inform healthcare implementation strategies. Also, because it is administered to all study completers, the sampling strategy of the IFACS is determined by the parent RCT. Thus, the generalizability of results obtained using the IFACS is the same as that of the parent study, and so the results have the same predictive power of the parent RCT to which it is linked.

Before generalizing use to other CAM treatments, we first sought to establish reliability and validity of the IFACS for a single treatment in a single condition, specifically for massage therapy when used for chronic knee pain. Thus, IFACS reliability was tested among completers of an RCT of massage therapy for chronic pain due to osteoarthritis of the knee.

Face validity: A panel of ten non-experts was used to assess face validity, defined as the degree to which a test "looks like" it is going to measure what it is supposed to measure.¹¹ To assess this, the instrument was presented to 10 individuals who were naïve to clinical research and psychometric measurement. These raters were asked, “Would the answers to these questions tell us whether a person received or wanted to receive massage therapy after completion of the EMBARK massage study, how often they received or would like to receive massage, and what they paid or would like to pay for massage therapy?” A positive response from at least ninety percent of the raters was deemed a priori to demonstrate reasonable face validity.

Content validity: A second, non-overlapping panel of 10 healthcare research experts was used to assess content validity of the IFACS using C.H. Lawshe’s method.¹² The method gauges agreement among subject matter experts (SMEs) regarding how essential a particular item is for the expressed intention of the instrument. SMEs were required to respond to the following question for each item: "Is the skill or knowledge measured by this item 'essential,' 'useful, but not essential,' or 'not necessary'?”.¹²

The content validity ratio (CVR) was calculated for each IFACS item as follows:

where

CVR= content validity ratio,

ne= number of SME panelists indicating "essential", and

N= total number of SME panelists.

Thus, the CVR can range from-1 (no panelists found the item to be essential), to 1 (all panelists found it essential). Using the updated critical values table for Lawshe’s CVR, an alpha level of 0.05, and 10 panelists a CVR greater than 0.62 was required (level set a priori) to meet significance for content validity.¹³ If the CVR for a particular item did not meet this criterion, the item was removed from the questionnaire.

Reliability: Because our primary concern was that each IFACS item elicited a consistent response from individuals, rather than whether the instrument as a whole measured a single underlying construct such as depression or anxiety, we focused on test-retest reliability rather than internal-consistency measures such as Cronbach’s alpha to demonstrate IFACS reliability.¹⁴ To assess the stability of the measure across time and population, test-retest reliability was assessed in samples drawn from two distinct populations of patients with OA knee pain. The first sample consisted of randomly selected completers (n=28) of the EMBARK study, with inclusion/exclusion criteria described elsewhere.¹⁵ During their participation in the EMBARK study, participants received up to thirty 60-minute full-body Swedish massages over 52weeks as the therapeutic intervention for OA of the knee. The second sample for the IFACS reliability trial was composed of completers of a supplemental pilot to the EMBARK Study conducted with U.S. Veterans who met the same entry criteria and received the same therapeutic protocol for OA of the knee as noted above for the first sample (n=10). Participants in both samples were administered the questionnaire twice, 1-3weeks apart, to evaluate temporal stability. Pearson’s correlations between time points were calculated for each individual and averaged separately for each sample.

Phase II

Evaluation of survey feasibility: At least two months after completing a 52-week efficacy and dosing RCT of massage therapy for OA knee pain, those participants who were not randomly selected for reliability testing were invited to answer a short questionnaire about their massage-seeking behavior since study completion. All parent study participants who provided an email address were sent an IRB-approved email therapy since completing the massage study (n=179). The email made it clear that participation was strictly voluntary, that no compensation was offered, and that clicking on the included survey link indicated their understanding of and consent to these terms. The link led to a presentation of IFACS, prepared using an online survey tool (Qualtrics Research Suite), which has a function that indicates whether emails were opened. Means, standard deviations, t-tests, Chi-square, used for validity and reliability estimates were calculated using SAS statistical software (Cary, NC).

Phase 1

Survey development: IFACS met criteria for both content and face validity as well as test-retest reliability (see Table 2). Using Lawshe’s method, all items met the CVR criterion for content validity individually (greater than 0.62 for 0.05 alpha levels). Therefore no items were removed from the questionnaire, and the mean CVR for the whole instrument was calculated as 0.70 (SD=±0.19). One-hundred percent of our naive rater panelists (n=10) responded “yes” to the face validity question; “Would the answers to these questions tell us whether a person received or wanted to receive massage therapy after completion of the EMBARK massage study, how often they received or would like to receive massage, and what they paid or would like to pay for massage therapy?”. This exceeded our a priori criterion of 90% positive responses from naive raters.

Table 3 shows characteristics of the two samples to which IFACS was administered for test-retest reliability. The populations differed notably with respect to age, employment status, sex, and race with Sample 2 being younger (42 versus 55years old), predominantly unemployed (70%), and male (70%). A greater percentage of Sample 2 self-identified as Black/African American (50% versus 11%). Test-retest reliability of IFACS was established in both Sample 1 (r = 0.90, p< .0001, n=28) and Sample 2 (r = 0.96, p<.0001, n=10).

Phase II

Evaluation of survey feasibility: Of those invited to participate (n=179), 117 completed the final IFACS survey (see Figure 1 & Table 4) for a 66% response rate. Characteristics of these 117 participants are described in Table 5. They were mostly white (92%) women (90%) with mean age of 57years and mean BMI of 32. About half of them (52%) reported being employed at the time of the study.

Only two questionnaires (1.7%) contained missing data. In both cases, the participant only responded to the first question. As shown in Table 6,Figure 1. the mean number of days between the time the initial email invitation with survey link was and the time the participant began taking the survey was 2.3days (SD=±2.8). Once started, the mean time to complete the survey was 2.1minutes (SD=±2.0).

Figure 1 Intervention Frequency and Cost Survey (IFACS) flow. Items in brackets could be adapted for interventions other than massage therapy. *There was no self-identified American Indian/Alaska Native, Asian, Native Hawaiian or Pacific Islanders participants in the study..

IFACS is a new, online tool to collect willingness to pay and ideal treatment frequency data from the patient perspective. It was designed to be administered to completers of an RCT of a particular treatment several months post-study. The instrument demonstrated excellent face validity and solid content validity,¹⁶ evaluated by a panel of experts. Test-retest reliability estimates (over 0.90 with p<0.0001 for two demographically disparate samples) were excellent, better than those for similar instruments tested with similar sample sizes.^17‒19 When email invitations including the link to complete IFACS was sent to 179 completers of a massage therapy RCT, the response rate was 66%. Of these, over 98% had no missing values, and the average time to complete the survey was just over two minutes. Thus, the survey was very efficient to administer and yielded results that were both timely and reliable.

Limitations of online surveys in general include concerns about sampling error, validity, and reliability of data collected in this manner. A typical example is the internet survey used by Hideo Yasunaga’s group in Japan to collect data about patient willingness to pay for health services related to the common cold, retinal detachment, and myocardial infarction.²⁰ Both this survey and IFACS were internet-based, took less than five minutes to complete, and were used to collect data about willingness to pay using multiple choice response categories. However, the two surveys differed in two important ways. First, in contrast to Yasunaga’s survey, IFACS underwent reliability and validity testing as described in the methods and results sections of this paper. Second, the studies differed in their sampling strategy. Yasunaga’s survey was sent to 2,500 people who were randomly selected from over 50,000 volunteers who had registered to take online surveys; IFACS was sent to all 179 completers of a particular RCT, so its sampling strategy was constrained to that of the parent study. Although Yasunaga’s sample was well-matched with ours for age and sex, it is impossible to completely remove potential bias associated with either study’s sampling method. While Yasunaga’s study had a much larger sample size, and therefore may have yielded results more generalizable to the overall population, it was limited to those who registered to take surveys. In contrast, the generalizability of results obtained using IFACS is determined by the RCT it follows. In addition, since IFACS was designed to be administered to RCT completers, it may maximize the investigator’s ability to assess individual differences in the measures of interest (frequency and willingness to pay) by controlling for residual variance through selection of a relatively homogeneous population (i.e., all patients with pain from knee OA who had at least enough interest in massage to enroll in the RCT).

Our approach of linking IFACS survey administration to an RCT has advantages over standard online survey methodology. Most important among these are higher response rates (more than double Yasunaga’s survey), which we hypothesize to be due to the participants’ familiarity with the treatment of interest from the parent study, and the ability to link data obtained using IFACS to the rich data obtained from the parent study. The latter strategy will allow identification of meaningful associations between clinical and pragmatic measures. For example, because the data collected with IFACS is linked to a parent massage therapy RCT, it is possible to investigate if the level of pain relief experienced by participants in the massage therapy RCT is predictive of their preferred frequency of treatment and/or the amount they are willing to pay for it.

Future directions of this work include reliability testing for other modes of administering IFACS such as over the telephone, in person, or in hardcopy as well as for interventions other than massage therapy. These future studies might also include a qualitative component to gather feedback on the process of taking the survey, which may be useful for analyzing patterns of missing data, for instance.

 In the future, it is our hope that IFACS will prove to be a commonly-used tool for investigators conducting RCT’s of IH/CAM treatment interventions, as well as suitable non-CAM interventions. Using IFACS as an add-on to already planned RCT’s supports NCCIH’s increased emphasis on “bringing the methods of effectiveness and outcomes research to the real world.⁶

IFACS is a practical, valid, and reliable instrument with which to collect post-study data about patient willingness to pay for an ideal frequency of specific IH/CAM treatments like massage therapy.

Special thanks to Dr. Partap Khalsa and the NIH’s National Center for Complementary and Alternative Medicine for supporting this work via grant 3 R01 AT004623-05S2.

Author declares there are no conflicts of interest.

None.

1. Barnes PM, Bloom B, Nahin RL. Complementary and alternative medicine use among adults and children: United States, 2007. Natl Health Stat Report. 2018;12:1–23.
2. www.cms.hhs.gov/NationalHealthExpendData
3. Nahin RL, Barnes PM, Stussman BJ, et al. Costs of complementary and alternative medicine (CAM) and frequency of visits to CAM practitioners: United States, 2007. Natl Health Stat Report. 2009;18:1–14.
4. Blackwell DL, Lucas JW, Clarke TC. Summary health statistics for U.S. adults: national health interview survey, 2012. Vital Health Stat. 2014;260:1‒161.
5. Heakal R . Economic basics: supply and demand. Cnada: Investopedia: Financial Dictionary; 2016.
6. National Center for Complementary and Alternative Medicine. NIOH: Third Strategic Plan (2011-2015): Exploring the Science of Complementary and Alternative Medicine. In: USDoHaH, Bethesda, USA. 2015.p.3–37.
7. Sox HC, Goodman SN. The methods of comparative effectiveness research. Annu Rev Public Health. 2013;33:425–445.
8. Sherman KJ, Cherkin DC, Connelly MT, et al. Complementary and alternative medical therapies for chronic low back pain: What treatments are patients willing to try? BMC Complement Altern Med. 2004;4:9.
9. Hewson C YP, Laurent D, Vogel C.) Internet research methods: A practical guide for the social and behavioural sciences. New technologies for social research. United Kingdom, Sag. 2003.
10. Fan W YZ. Factors affecting response rates of the web survey: A systematic review. Computers in Human Behavior. 2010;26(2):132–139.
11. Gravetter FJ, Forzano L. Research Methods for the Behavioral Sciences. 4th edn. Wadsworth, USA:  Belmont, CA; 2012.pp.3–125.
12. Lawshe CH . A quantitative approach to content validity. Personnel Psychology. 1975;28:563–575.
13. Wilson FR, Pan W, Schumsky DA. Recalculation of the critical values for Lawshe's content validity ratio. Measurement and Evaluation in Counseling and Development. 2012;45:197‒210.
14.  What does Cronbach's alpha mean? Institute for Digital Research and Education. UCLA, Statistical Consulting Grou p, USA. 2013.
15. Perlman AI, Ali A, Njike VY, et al. Massage therapy for osteoarthritis of the knee: a randomized dose-finding trial. PLoS One. 2012;7(2):e30248.
16. Davidshofer KR, Murphy CO. Psychological Testing: Principles and Applications.  6th edn. Upper Saddle River, USA. 2005.
17. Arcara G, Bambini V. A Test for the Assessment of Pragmatic Abilities and Cognitive Substrates (APACS): Normative Data and Psychometric Properties. Front Psychol. 2016;7:70.
18. Ibrahim H, Lindeman B, Matarelli SA, et al. International Residency Program Evaluation: Assessing the Reliability and Initial Validity of the ACGME-I Resident Survey in Abu Dhabi, United Arab Emirates. J Grad Med Educ. 2014;6(3):517‒520.
19. Sandhu S, Killaspy H, Krotofil J, et al. Development and psychometric properties of the client's assessment of treatment scale for supported accommodation (CAT-SA). BMC Psychiatry. 2016;16:43.
20. Yasunaga H, Ide H, Imamura T, et al. Willingness to pay for health care services in common cold, retinal detachment, and myocardiac infarction: an internet survey in Japan. BMC Health Serv Res. 2016;6:12.