Traditional Medicine and Complementary/Alternative Medicine

Subjective scales for the evaluation of therapeutical effects and their use in complementary medicine

A. LIVERANI
Istituto di Igiene e Medicina Preventiva, Facoltà di Medicina e Chirurgia dell'Università di Milano
liverani@dsi.unimi.it

E. MINELLI
Esperto WHO Collaborating Centre for Traditional Medicine - Università degli Studi di Milano
minwhomi@tin.it

A. RICCIUTI
Direttore Scientifico Rivista Medicina Naturale, Associazione Italiana Ricerca sui Sistemi
alberto.ricciuti@iol.it

Abstract

Using existing systems for appraising  the efficacy of various therapies, regardless of their typology, the authors considered about ten methods commonly used to evaluate quality of life. They give guidelines on how  to build another index for judging quality of life taking into account the patient's conditions of life. These indexes share similar problems in finding objective measures for assessing personal situations without regard to patient's or observer's subjective impressions.
Nevertheless, the authors demonstrate that for randomized controlled clinical trials and for meta-analyses that this deficiency is also a problem, at least qualitatively, for scientific methods that are considered quite objective. Therefore, according to the methodological manifesto of  the World Health Organization (WHO), the authors conclude by suggesting the adoption, for complementary medicines' studies, of methods already known for their use in conventional medicine, notwithstanding general limited validity methods of validation.

Introduction

Discussion about the effectiveness of alternative therapies is still inconclusive.  In the following paper "scientific" medicine will be referred to as "conventional" to identify it from "complementary" and "alternative" therapies.
International emphasis, in particular by the World Health Organization (WHO), on costs of therapies has increased public interest in complementary medicine as, if effective, it is much cheaper. This consideration has strengthened the interest of the international scientific community in finding methods to evaluate the effects of non conventional treatments stimulating WHO in 1995 to start a research program to achieve a universal valuation system of Quality of Life (QOL). The ambitious program triggered discussion of how to evaluate therapy effects. This entails critical examination of the methodologies and instruments basic to conventional medicine. Such an examination highlights the differences between the different approaches of the different medicines.
The present "biomedical model" (Engel G.L. 1977, Capra F. 1982) reveals difficulties, that can partially be overcome by an emergent theory of "systematic thought" (von Bertalanffy L. 1969, Miller J.G. 1978, Morin E 1980, Delattre P. 1982, Capra F. 1996) in human health:

1. Effectiveness of utilized instruments to produce important biological effects (cellular level)
2. Effectiveness in producing therapeutic effects for specific pathologies (organismic level)
3. Effectiveness in improving health
   1. of individual (person relationship level)
   2. of society (psycho-social and socio-economic level)

This paper will focus on point 3.a. to examine the use of different instruments already validated (tests for evaluation of QOL) for recording changes of a patient's condition from a subjective perspective. This type of evaluation is not linked to the type of therapy used. However, it is appropriate when complementary therapies are used, with or without the conventional therapies.evaluation of the QOL, independent of the instruments used to improve it, is increasingly used for chronic-degenerative diseases, for which there are no sure treatments. This is an area where combination of conventional and complementary therapies is gaining popularity. A domain of possible observations accepted by the supporters of different medical approaches needs to be identified.

WHOQOL-100 and NIH methodological manifesto

Different resolutions made by WHO (Consultation on Acupuncture, Milan, Nov. 1996), take the stance that, in those countries where conventional biomedicine is the basis of medical courses, the ethical use of complementary medicines must be backed by controlled clinical tests of its effectiveness.
Some have strong doubts that conventional research methods, such as the randomized controlled trial (RCT), can be used to evaluate the effectiveness of complementary therapies. Two opposite positions can be identified: the first is in the work of WHO, the other is summarized by the Methodological Manifesto of the Working Group on Quantitative Methods in Alternative Medicine of NIH (Levin et al., 1997).  In 1995, WHO, with the elaboration of the WHOQOL-100, created a system to evaluate the impact of complementary therapies on QOL that has the characteristic of defining the effects of a therapy, following as evaluation criteria, the impact on the patient in his psychological and physical globality rather than the impact on disease. For example, with this evaluation system, radiotherapy and radical surgery can have the same effectiveness concerning a specific cancer, but if one of the two methods implies a higher quality of the patient's life, that one would be preferred.
The work of WHO in evaluating QOL originated for different reasons. The most important was to widen the criteria for measuring health beyond the traditional indicators such as mortality and morbidity data and includes the impact of the disease on the quality of life. Second, the different methods elaborated in the USA and in the UK to measure health condition have produced instruments which have proved unsatisfactory in different contests. Third, the growing mechanistic model of medicine, based only on the complete defeat of disease, has strengthened the need for a humanistic approach to health.
The main aim of WHO was to develop an internationally valid method for measuring life quality to promote a global approach to health.

WHOQOL-100

Fig 1 - Classification of the domains related to the quality of life (Division of Mental Health WHO. Field Trial WHOQOL-100, Facet Definitions and Questions. MNH/PSF/95.1.B, Geneva, Switzerland, 1995)

Conversely, some groups of scientists, even if skeptical of RCTs and "reductionist" approaches, think that alternative therapies are not so unusual and can be studied with existing methodologies.  A strong position was enunciated by the Working Group on Quantitative Methods in Alternative Medicine of the NIH (Levin et al., 1997), which resulted in a kind of Methodological Manifesto, the aim of which is to focus on problems and solutions and guidelines for the evaluation of CAM (Complementary and Alternative Medicine). In this document, in contrast to statements from many researchers and doctors involved in CAM, the validity of existing research methodology and analytical procedures is underlined, while also comparing the non-conventionality of the hypotheses and of the therapy conclusions of CAM.
The document is divided into two parts, one describing the principal methodological problems and one describing guidelines and recommendations.

The Methodological Manifesto

The principal methodological problems reflect the opinio of many non-conventional practitioners, that their therapies or proposed mechanisms are so unusual they cannot be evaluated or investigated with conventional or existing tools. These can be summarised as follows:

1. Complex individualized interventions
2.  Identification of therapeutic effects
3. Specific effects vs. systemic perturbations
4. Long term effects
5. Re-conceptualization of the human body
6. Multifactorial aetiologies

The Methodological Manifesto consists of seven guidelines and recommendations completed by the Working Group on Quantitative Methods in Alternative Medicine of the NIH.

1. CAM Research Problems need different methodological and analytical approaches.
2. Researchers should use the strongest most rigorous research model and the best statistical procedures for problems that arise from the study. Some types of investigation may not be amenable to usual procedures of medical research (double-bind, controlled studies vs. placebo RCT), but may require different methodological and statistical approaches to solve the problems particular to the study.
3. Clinical trials are not the only possible studies.
4. The results of observational studies may suggest new approaches in the design of intervention trials. New medical knowledge is often reached through observation of cohort studies or a simple series of cases.
5. The "black box" approach. For example, a clinical trial on acupuncture or on herbal therapy may be performed without considering how multifactorial the intervention is nor how unusual the proposed healing mechanism, provided there is a clear biomedical endpoint. This is similar to the test of pharmacological agents whose mechanisms of action are not yet known and understood.
6. Existing quantitative procedures are usually adequate for alternative therapy research and complementary medical systems.
7. Complex complementary medical systems can be studied as "gestalts". This, a study on the effectiveness of, for example, macrobiotic therapy, would consider it as a whole and not as one particular element of the global intervention.

Finally, the Working Group on Quantitative Methods in Alternative Medicine emphasized that randomization is an important principle for ensuring that studies and their results are representative and generalizable.

Weakness of controlled randomized clinical trials

The randomized clinical trials (RCT) is the principle tool of clinical research and to document the effect (positive or negative) of a therapy we rely on the work of Theodore Pincus (1997).  It must be observed that the RCT is based on the standard medical record. One inadequacy of this approach is its inability to record professional "acta" more than their utility or failure. Additionally, it is not designed to evaluate the effectiveness of a long term treatment.  These limits can be linked to the evolution of the medical record within the "biomedical model", the fundamental paradigm of medicine of the XX century  (Engel, 1977, Holman, 1976, Sagan, 1987).
Briefly, in this model the record of the patient is considered "subjective" and therefore "not scientific."  Quantitative data and information (lab analysis, radiographs, etc.) are considered "objective" as they can be recorded as precise and defined standards. Quantification yields unique measuring references and reliable measures for monitoring the evolution of pathology. Psychological conditions and other symptoms of well-being or malaise, and also pain, are often considered of minor importance.

The rise of controlled randomized clinical trials

To overcome these limitations the randomised controlled clinical trials, designed and developed in the early '50s (Daniels & Hill, 1952), are regarded the optimal method for the evaluation of  the effectiveness of any intervention.  RCTs are effective in evaluating treatments in the acute phases of diseases that evolve in short periods (with easily identifiable end-points) and for documenting side effects of drugs. However, for the predominant problems of our time, the study of chronic diseases, characterised by problems that evolve in the medium and long term, where end-points are not easily identifiable, and that need "control", "care" and "palliation", the RCT method is not adequate.  The rigid and limiting procedures imposed by the theoretical criteria underlying these studies can, in some cases, prevent or compromise the patient's long term optimal treatment.  For example, studies on patients affected by rheumatoid arthritis, showed that drugs considered effective in treating this pathology in the short run, but with high toxicity and secondary effects (metotrexate, D-penicillamine, sulfasalasine, ect) in some cases lead to interruption of the treatment, have the same, if not lowered, effectiveness in the long term as drugs considered to be less effective in short term clinical trials.

Limits of controlled randomised clinical trails

The limitations of the RCTs may be grouped into two categories:

1. those resulting from "pragmatic" design of medical clinical research
2. limitations "intrinsic" to the adopted methodology

Six pragmatic limitations of randomised clinical trails.

1. Exclusion criteria in randomized trials

In all randomized clinical trials, there are exclusion criteria that restrict the selection of certain patients to minimize the interference of factors that can bias the results. It follows that many interventions are tested on a minority of patients. The exclusion criteria are defined by the researcher and may be exigent (high number of exclusion criteria) or practical (low number of exclusion criteria).

2. The relatively short observation period in most of the clinical trials of chronic diseases

In acute diseases the observation period of a few months may be sufficient to recognise the effectiveness of the treatment and to identify possible secondary effects. However, in chronic diseases, a few months, one year or few years (the observations periods for most clinical trials) is to short too evaluate fully the effectiveness of the treatment.

3. Standard dosages imposed by protocols

Most randomized controlled trials establish rigid doses to guarantee "scientific" criteria. A patient who reports drug efficacy but severe gastrointestinal distress might be withdrawn from the trial, even if, in everyday practice, the dosage might be reduced and therapy continued effectively without secondary effects. In a trial the idea of personalized therapy, the maximum possible dosage for a particular patient and not the maximum possible dosage is ignored. Therefore, rigidly designed clinical trials risk underestimating the effectiveness of the drug and overestimating the toxicity. The clinical behavior imposed by rigid protocols does not reflect everyday clinical practice.

4. The use of markers that in several cases are not appropriate for tracing the evolution of the pathology

In one trial of one or a few years it is not possible to evaluate the long term outcomes of a chronic disease, like death or disability. Often the markers used in trials for chronic diseases are not suitable and reliable for tracing the future of the pathology (for example the carcinoembryonic antigen [CEA] test), and therefore are not a good guide for reaching the long term objectives of the medical treatment.

5. Significant statistical results are not necessarily clinically important and viceversa.

The effectiveness of a clinical trial is evaluated by the statistical significance of the difference of results of the studied treatment versus placebo or  different treatments. However, significant statistical results may not correspond to a clinical point of view (for example whwn any improvement of a physical parameter goes on with a reduction of quality of life) . Moreover, the need for large numbers in order to reach statistical significance implies problems for rare pathologies.

6. Tendency to ignore important factors not previewed in the research but that affect the results

In many clinical trials, important factors linked to patients bias the results more than the treatment (or of the placebo) for which the patients where randomised (many examples cn be found in researches on antihyperlipidemia dugs that do not consider the nutritional habits of the patient). However, this information is generally ignored or not emphasised.

Four intrinsic considerations.

1. The design of a clinical trial may influence the results despite the inclusion of a control group

The design of an RCT may considerably shift the probability that an intervention will or will not appear to be more effective than a placebo. The RCT is not by definition neutral and the existence of a control group does not necessarily eliminate biases which are intrinsic in the design of a study

2. Clinical trials results generally refer to groups of patients and ignore the variability between individuals

Consider a typical trial in which 60% of individuals respond more favourably to drug A than to drug B, 20% respond more favourably to drug B, and 20% find both drugs of equal effectiveness. The interpretation of the results is that drug A is superior to drug B for all patients, although a more precise interpretation might be that drug A is superior to drug B for certain individuals while the opposite is true for others.Many hospital stock only one single drug due to economic constraints, regardless of the fact that each drug is effective only for certain individuals.

3. Interpretation of side effects is not standardised and introduces biases into results of clinical trials

4. The design of clinical trials distorts the placebo effect by informing the patients that they are under study and may not receive the "best" therapy

Patients who participate in clinical trials are fewer than 0.001% of all patients treated by drugs (Pincus, 1997). The exclusion criteria usually mean that the sample under investigation is not representative of the universe of patients with the disease nor of patients under treatment. Moreover, costs of clinical trials cannot easily be surmounted. It is important to emphasise the value of clinical trials in documenting long term safety, or the absence of safety, in contrast to effectiveness.

Weaknesses of meta-analysis models

The critique of RCTs and of recent statistic models led to the development of meta-analytic models. Meta-analysis can reach significant results when different sampling is carried out, for example cross-country sampling grouped to one particular research variable. Meta-analysis was designed as a tool to link non-coordinated research on similar topics carried out in different places. The effectiveness of the result is arguable as the newly elaborated indices tend to lose specificity.
Multicenter research is the area in which meta-analysis can be of considerable advantage. In this case the method guarantees that if the collected information is found significant, it is regarded as methodologically sound by the scientific community, for example the study on the effects of homeopathic drugs (Linde et al., 1997).

Patient Self-Report Questionnaires

The same difficulties can be overcome by using patients self report-questionnaires (PSRQ) to monitor pathology and long term effectiveness during each meeting with the patient. They may also be helpful to the practitioner at the intellectual, professional and ethical level. These questionnaires can be used to evaluate the effectiveness of treatments in conventional, complementary or  integrated medicine, and are an adequate tool to evaluate the general effectiveness of the therapeutic treatment on the patient.
In the biomedical model the term "scientific" is considered as applying only to high technology procedures or to RCT, but not to usual clinical care. However, any scientific study is characterised by several prerequisites: an hypothesis; a protocol to standardize the way in which quantitative data are collected; a methodology to record observations; a defined criteria to evaluate analysis; and data analysis techniques.  Any clinical encounter in which the practitioner does not know which one is the "best" therapy can add to scientific research. So any effort to collect systematic information that could be useful to define the evolution of the patient’s condition can be considered legitimate and necessary for scientific research.
The proposal is to ask the patients at each medical encounter to fill up a simple questionnaire to be entered into a database.

1. Hypothesis: all the interventions carried out by the health professional are effective and minimally harmful relatively to the conditions under treatment.
2. Protocol: a standardized collection of information through the self report questionnaires to be handed to the patient. This questionnaires can also be designed so to include the collection of lab data, radiographs, other physiologic measures, additional questionnaires.
3. Methodology of data collection: the methodology consists of recording data from each patient to avoid having a selection of patients.
4. Result: must be defined at the beginning of the observation phase and can consider traditional indicators such as death, blood pressure, the reduction of swollen joints as well as biopsychosocial indicators such as function status changes, pain, depression, helplessness, optimism, social support, stress.
5. Analysis techniques: these depend on the type of database, but a computer is not absolutely necessary. Need for a computer depends on the type of questionnaires, the quantity of data and the problem under survey.

A questionnaire of this kind could be a useful tool to control the quality of professional activity. The costs of implementing such a system of questionnaires is approximately 1-3% of the total costs of treatment. This cost is justified in terms of the practitioner professional ethics and responsibility toward the patient and toward himself.

Comparative analysis of subjective scales

When comparing different subjective scales, the researchers considered the following categories, for giving different weight to the information collected:

• parameters apparently quantifiable even by the most conditioned patient (mobility, working capacity, alimentation, personal care, sleep, …)
• reaction to the disease of patients. These data are the most directly affected by the patient. If any reliable information is  to be collected, no absolute evaluation must be asked of the patient, but only relative to the previous state (if the patient is chronic this enables the history to be tracked)
• Intermediate parameters, meaning how the patient senses his or her condition relative to external parameters (eg he works but he has lost satisfaction with the work)

The large number and the typology of subjective scales for clinical use is well known. It is basically an attempt to codify the impression of the practitioner and/or of the patient toward the treatment ie to describe a real situation.  It is therefore quite hard to collect all the material produced unless limiting the research, for example, to one specific pathology. This is what has been done by on the work of M. Tamburini in evaluating cancer treatments.

The following methods have been evaluated.

KPS: the Karnofsky Performance Status Scale, proposed in 1948, is the best known of the tests in use. Even if it is not used for measuring functional integrity, the symptoms and/or the signs of disease and the toxicity of the treatment can be assessed but, more generally, the quality of life. The KPS uses a method of evaluation in which the maximum score corresponds to the normal status of health.

QL index: The quality of life index proposed by Spitzer in 1981, introduces a global measure of quality of life for cancer patients. The scales that are used in five activity areas have been criticized in practice. However, the QL index has the merit of proposing a mediate evaluation and therefore relates well to patient's status connected to quality of life.

RSCL: The Rotterdam Symptom Checklist is the most recent, developed in 1990 (De Haels et Al.), as a tool to measure globally the quality of life for cancer patients. The RSCL is a very innovative instrument and is still under study for some methodological and administrative aspects.

SCI: The SCI proposed by Tamburini in 1991 has the objective of measuring pain during chemotherapy and is based on the idea of counting the number of treatment days of pain. The method builds on methods of similar questionnaires and is easy to complete.

SDS: The Symptom Distress Scale (McCorkle 1978) aims to evaluate the degree of suffering caused by cancer patients' symptoms according to 13 linked aspects evaluated on a five point ordinal scale. The test demonstrates the problem of patient's learning process that can bias the data when repeated

TIQ: The therapy Impact Quality (Ventafridda et Al 1990), conceived for evaluating the effects of therapy for terminal patients, evaluates the quality of life in four domains using 36 items plus 1 general item. The test can be used by non specialized personnel.

VAS/LASA: The VAS/ Linear Analogue Self Assessment (Scott et Al. 1976, Priestman et Al 1976, Carlsson 1983, Jensen et Al 1986) are general tools, well established for medical and psychological parameters. A great amount of research has been done to evaluate and compare this tool to other analytical instruments (verbal and numerical scales).

It is easy to observe that the different methodological approaches may be grouped into three types:

• Immediate evaluation of a specific pain status
• Global evaluation of pain status
• Identification of patient's global condition, extended also to social aspects.

The first category can be used in all pain situations, as it is not specific. However, it can strongly depend on the type of observed subject and psychic component.

The second category instruments shows that to reach some result linked to the therapy the inventory must be repeated.

The third class methods try to identify some objective measure of distress and to compare the state of two patients.

Weaknesses common to all include:

• Variability of observed subjects
• Variability in time of single subject
• Bias from the investigator (even unconscious)
• Learning effect in the process of repeated administration
• Limits of quantitative analysis of observations

Weakness of subjective tools

Spurious objectivity masking environmental and psychological conditions cannot be eliminated. It is the underlying structural weakness of all these observation methods.  The only recourse is to define and limit as well as possible the purpose of the observation and of the research.  The use of ordinal scales aims to overcome the risks of imprecision in questionnaires by use of binary answers. Two approaches are used: a) the patient is left free to choose within a well specified domain, b) the patient is given the choice between three or more options defined a priori.  Without focusing on the statistical implications, the apparently greater precision of the first alternative rapidly vanishes when the effects implied by such a choice make it difficult to compare the choices of more individuals. Moreover, in the second alternative the discussion is on the number and scaling of the choices from which the patient has to select. Too many choices leads to low comparison, few choices result in poor information. Moreover, imprecise definition of the choices leads to loss of information.  These investigation tools are worthy of further study, even if some points of weakness exist, when an effort is made to identify binary answer questionnaires. That is where the investigated situation are identified with the greatest precision and to the proposed questions, even if numerous, a yes / no answer can be given.

References

1. Bertalanffy L.von 1969, General System Theory (Tr. it.: Teoria generale dei sistemi, Mondadori, Milano 1983)
2. Capra F. 1982, The Turning Point, Simon and Schuster, New York (Tr. it.: Il punto di svolta, Feltrinelli, Milano 1984)
3. Capra F. 1996, The Web of Life, Doubleday-Anchor Book, New York (Tr.it.: La rete della vita, Rizzoli, Milano 1997)
4. Daniels M & Hill AB. 1952, “Chemotherapy of Pulmonary Tuberculosis in Young Adults: An Analysis of the Combined Results of Three Medical Research Council Trials”, Br. Med. J., 1:1, 162-8
5. Delattre P. 1982, “Théorie des systèmes et épistémologie”, in La notion de système dans les sciences contemporaines, tomo II, Epistémologies, Librairie de l’Université, Aix-en-Provence (Tr. it.: Teoria dei sistemi ed epistemologia, Einaudi, Torino 1984)
6. Division of Mental Health World Health Organization Geneva 1995. “Field Trial WHOQOL-100, Facet Definitions and Questions” MNH/PSF/95.1.B
7. Engel GL. 1977, “The Need for a New Medical Model: A Challenge for Biomedicine”, Science, 196: 129-36
8. Holman HR. 1976, “The ‘Excellence’ Deception in Medicine”, Hosp. Pract. 11: 11-21
9. Levin S., Glass T., Kushi L., Schuck J., Steele L., Jonas W. 1996. “Quantitative Methods in Research on Complementary and Alternative Medicine: a Methodological Manifesto” Medical Care (inpress)
10. Linde K. et Al. 1977, “Are the clinical effects of homoeopathy placebo effects? A meta-analysis of placebo-controlled trials”, Lancet, 350: 834-43
11. Miller J.G. 1978, La teoria generale dei sistemi viventi, Franco Angeli, Milano (Testo, tradotto da Angelo Beretta, rielaborato dall’Autore sulla base di tre articoli apparsi nel 1965 sulla rivista Behavioral Science)
12. Morin E. 1980, La vie de la vie, Editions du Seuil, Paris (Tr. it.: La vita della vita, Feltrinelli, Milano 1987)
13. O.M.S. - Ginevra 1983, Traditional medicine and Health Care Coverage. A Reader Health Administrators and Practitioners (Tr.it.: Il ruolo delle medicine tradizionali nel sistema sanitario. Valutazioni scientifiche e antropologiche, Edizioni RED, 1984)
14. Pincus T. 1997. “Analyzing Long-term Outcomes of Clinical Care without Randomized Controlled Clinical Trials: The Consecutive Patient Questionnaire Database”, Advances - The Journal of Mind-Body Health, 13: 2, 3-32
15. Ricciuti A. 1990, “La validazione scientifica: verso il riconoscimento di differenti razionalità?”, Atti Convegno Medicine ‘Diverse’: situazione e prospettive in Italia, Milano 23-24 marzo ‘90
16. Sagan LA. 1987, The Health of Nations: True Causes of Sickness and Well-being, NY: Basic Books
17. Tamburini M. 1997, Quality of Life Assessment in Medicine, CD-ROM per Windows, Glamm Interactive, Milano
18. The WHOQOL Group 1995. “The World Health Organization Quality of Life Assessment (WHOQOL): Position Paper from the World Health Organization”, Soc. Sci. Med. 41: 10, 1403-1409
19. WHO-Guidelines on basic training and safety in acupuncture, WHO/EDM/TRM/99.1, 1999

Print this page

Edited by Aldo Campana,