Electronic Letters to:

Commentaries: Warren S. Browner Predicting Fracture Risk: Tougher Than It Looks BoneKEy-Osteovision 2007; 4: 226-230 [Full text] [PDF]

eLetters: Submit a response to this article

Electronic letters published:

Re: Predicting Fracture Risk: Easily Misled

Warren S Browner   (19 October 2007)

Predicting Fracture Risk: Easily Misled

John A Kanis, Anders Oden, Helena Johansson and Eugene V. McCloskey   (19 October 2007)

Re: Predicting Fracture Risk: Easily Misled 19 October 2007
Warren S Browner California Pacific Medical Center Research Institute, San Francisco, California, USA Send letter to journal: Re: Re: Predicting Fracture Risk: Easily Misled warren{at}cpmcri.org Warren S Browner	I read the letter by Dr. Kanis and his colleagues with great interest and more regret. It wasn’t my intent to make disingenuous comments about their paper. I was asked by the editors to try to explain why risk prediction is hard, even when there are strong risk factors for an outcome. Their letter reiterates some of the difficulties I mentioned and suggests a few more. First, I agree that risk prediction models for hip fractures in women should include the risk of death (1). However, low bone density and many other causes of hip fracture are also associated with increased mortality (2-6). So accounting for death in a risk model reduces that model’s ability to distinguish those who fracture, since some women at high risk of fracture die before they have a chance to break a bone. Second, it looks like my opinion about ROC areas was not clear. I don’t think areas under ROC curves help much when evaluating risk prediction models, because—as I stated in the Commentary—“they ignore the absolute risk of what is being predicted.” Let’s take the example of age as a predictor of hip fracture, from Table 1 in their letter. (I’m not sure I fully understand the Table, since the usual way to calculate the area under an ROC curve for a continuous variable like age would have one value, rather than four, for each length of follow-up. I think what Kanis et al. did was use a series of age cutoffs to select the population of interest and calculate separate areas for each one.) Looking at Table 1, for example, the area under the age-hip fracture ROC curve has an area of 0.833 for age ≥ 50 years and a follow-up period of one year. That means I could get rich betting that among a pair of randomly selected women older than 50 years of age—one with, and one without, a hip fracture in a given year—the one with the fracture will be older: I’d be right 83.3% of the time. So far, so good. But despite this impressive ROC area, age would make a poor risk prediction model for hip fracture, and I would become poor betting that any particular woman ≥50 years old would have a fracture, even at a 100 to 1 pay-off. (For those who are mathematically inclined, there are approximately 48 million women ages 50 and older in the U.S., in whom about 240,000 hip fractures occur each year, so the expected result of a $1 wager is a loss of 50 cents.) As to the specific argument that centers on Figure 1 in their letter, Kanis et al. are right that a substantial increase in the relative risk gradient (say, from 1.6 to 2.6 per SD change in a risk factor) improves risk prediction by identifying more higher-risk women. Again, so far, so good. But most of these “higher-risk” women will still not suffer a fracture and most of the fractures will occur in “low-risk” women. I also agree that increasing a model’s relative risk from 3.68 to 4.23 per SD (e.g., by adding clinical risk factors for fracture to a model based on bone density alone) isn’t very helpful. Finally, as mentioned in the Commentary, I agree that the final decision about who should receive an intervention is, alas, even more important—and complicated—than the “simple” task of identifying those at high risk. References: 1. Nguyen ND, Ahlborg HG, Center JR, Eisman JA, Nguyen TV. Residual lifetime risk of fractures in women and men. Bone Miner Res. 2007;22:781-8. 2. Browner WS, Seeley DG, Vogt TM, Cummings SR. Non-trauma mortality in elderly women with low bone density. Lancet. 1991; 38:355-358. 3. Browner WS, Pressman AR, Nevitt MC, Cummings SR. Mortality following fractures in older women. The Study of Osteoporotic Fractures. Arch Intern Med. 1996; 156:1521-5. 4. von der Recke P, Hansen MA, Hassager C. The association between low bone mass at the menopause and cardiovascular mortality. Am J Med. 1999;106:273-8. 5. Nguyen ND, Center JR, Eisman JA, Nguyen TV. Bone loss, weight loss, and weight fluctuation predict mortality risk in elderly men and women. J Bone Miner Res. 2007 22:1147-54. 6. Ensrud KE, Ewing SK, Taylor BC, et al. Frailty and risk of falls, fracture, and mortality in older women: the study of osteoporotic fractures. J Gerontol A Biol Sci Med Sci. 2007;62:744-51.
Predicting Fracture Risk: Easily Misled 19 October 2007
John A Kanis WHO Collaborating Centre for Metabolic Bone Diseases, University of Sheffield Medical School, UK, Anders Oden, Helena Johansson and Eugene V. McCloskey Send letter to journal: Re: Predicting Fracture Risk: Easily Misled w.j.pontefract{at}shef.ac.uk John A Kanis, et al.	We were interested to read the Commentary published recently by Browner in BoneKEy (1) on our paper describing the clinical use of risk factors for the prediction of fracture risk (2). Whereas he makes some valuable points in his tutorial on risk models in general, the comments on our paper are disingenuous. A major aim of our paper was to demonstrate whether the clinical risk factors chosen added significantly to fracture risk prediction with or without the use of bone mineral density (BMD) measurements: As termed by Browner, our results showed that we had ‘the ante required to play the hand’. A second, and perhaps more important aim, was to determine whether this finding in our source cohorts could be reproduced in the independent validation cohorts. The paper was not intended to play the game or indeed to show how to play the game. The value of fracture risk prediction lies in the likelihood of correctly categorising those who will or will not sustain a fracture over a given time. The probability of fracture will depend not only on the fracture hazard (reported in our paper), but also on the death hazard (not reported) and the calibration of risk scores for death and fracture to the epidemiology of the region of interest (reported elsewhere (3)). Moreover, in our article we showed only the predictive power of the combination of variables without including age. In other prediction models (e.g. for cardiovascular events or for fracture), age is included in a simple way with other variables (4;5). There are statistical reasons for not including age in the assessment of the predictors, but of course age is one of the variables to be used, together with the other variables, for the assessment of fracture probability. In this context it is relevant to ask what the area under the ROC curve is for age alone. That depends of course on the age intervals considered and the time frame. Table 1 shows areas under the ROC curve for age alone. The variable used is the probability of hip fracture that incorporates the hip fracture hazard based on 34,902 women who sustained a hip fracture in the population of Sweden (6). The area under the ROC curve for age as a predictor is the probability that the age at start of follow up for a randomly chosen individual who sustains a hip fracture exceeds the corresponding age for an individual without hip fracture during the period. The population size at different ages is included in the calculations. Table 1. The area under the ROC curve for age as a predictor of hip fracture among women with different follow up periods. Age range (years) 1 year follow up 2 years follow up 10 years follow up ≥50 0.833 0.827 0.769 ≥60 0.790 0.782 0.710 ≥70 0.734 0.725 0.640 ≥80 0.676 0.665 0.578 The interpretation of areas under ROC curves in clinical testing has been considered by Hosmer and Lemeshow (7) and is summarised in Table 2. Browner considers areas of 0.78 as a ‘valuable’ ROC area and considered that as a limit, which has to be crossed by prediction models in order to have a discriminatory ability. Table 2. Discriminatory value of areas under the ROC curve Area under ROC curve Value =0.5 no discrimination ≥0.7<0.8 acceptable discrimination ≥0.8<0.9 excellent discrimination ≥0.9 outstanding discrimination It is evident that age alone passes the limit for a valuable discriminatory ability under certain conditions. Age combined with the other predictors will have an even higher area under the ROC curve. Thus the comments by Browner on the receiver operating characteristic (ROC) curves reported in our paper are misplaced when translated to a clinical context. Our paper focussed on the gradient of risk for fracture provided by BMD, the clinical risk factors and the combination. It is therefore but one step in the creation of probability models. Notwithstanding, for the prediction of hip fracture, the clinical risk factors performed at least as well as peripheral BMD (2;8). In general, the combination of the clinical risk factors with BMD gave higher gradients of risk (RR/SD) than the use of BMD alone, though the effect is modest when age is not considered. However, even modest improvements in the gradient of risk can have marked consequences for the identification of patients. Consider, for example, two tests, the one with a gradient of risk of 1.6 and the other with a gradient of risk of 2.6. Imagine, for the sake of argument that we wished to identify for treatment individuals at a given age who have a 3-fold increase in fracture risk. The test with a gradient of risk of 1.6 would identify 0.5% of the population with a RR of 3.0 or more. The test with a gradient of risk of 2.6 would, in contrast, identify ten times as many individuals (5% of the population) above this risk (see Figure 1). Moreover, the average risk in the latter group (RR=4.9) would exceed that in the group identified by the test with the lower gradient of risk (RR=3.5). The mathematical basis for these calculations is provided elsewhere (9), but the example illustrates the impact of even small increments in gradient of risk. The yield in sensitivity from a given improvement in the gradient of risk is greater the lower the gradient of risk (10). Conversely, there is little to be gained by improving gradients of risk above 3 or 4/SD. Thus the example cited by Browner (comparing a gradient of risk of 3.68 with 4.23) is at the same time a statement of the obvious and very misleading. Ultimately, the usefulness of prediction models has to be based on health economic arguments rather than the simple labels like acceptable, excellent, or valuable discriminatory ability. That the models identify individuals with a high fracture probability (11) who respond to intervention (12) and with a risk above a cost-effectiveness threshold (13) should be the ultimate arbiter. References 1. Browner WS. Predicting fracture risk: Tougher than it looks. BoneKEy. 2007: 8; 226-230. 2. Kanis JA, Oden A, Johnell O, Johansson H, De Laet C, Brown J, Burckhardt P, Cooper C, Christiansen C, Cummings S, Eisman JA, Fujiwara S, Gluer C, Goltzman D,Hans D, Krieg MA, La Croix A, McCloskey E, Mellstrom D, Melton LJ 3rd, Pols H, Reeve J, Sanders K, Schott AM, Silman A, Torgerson D, van Staa T, Watts NB, Yoshimura N. The use of clinical risk factors enhances the performance of BMD in the prediction of hip and osteoporotic fractures in men and women. Osteoporos Int. 2007;18:1033-46. 3. Kanis JA, Johnell O, De Laet C, Jonsson B, Oden A, Ogelsby AK. International variations in hip fracture probabilities: implications for risk assessment. J Bone Miner Res. 2002;17:1237-44. 4. Conroy RM, Pyorala K, Fitzgerald AP, Sans S, Menotti A, De Backer G, DeBacquer D, Ducimetiere P, Jousilahti P, Keil U, Njolstad I, Oganov RG, Thomsen T, Tunstall-Pedoe H, Tverdal A, Wedel H, Whincup P, Wilhelmsen L, Graham IM; SCOREproject group. Estimation of ten-year risk of fatal cardiovascular disease in Europe: the SCORE project. Eur Heart J. 2003;24:987-1003. 5. Black DM, Steinbuch M, Palermo L, Dargent-Molina P, Lindsay R, Hoseyni MS, Johnell O. An assessment tool for predicting fracture risk in postmenopausal women. Osteoporos Int. 2001;12:519-28. 6. Johnell O, Kanis JA, Jonsson B, Oden A, Johansson H, De Laet C. The burden of hospitalised fractures in Sweden. Osteoporos Int. 2005;16: 222-8. 7. Hosmer DW and Lemeshow S. Applied Logistic Regression. Second edition. John Wiley & sons, New York, 2000, page 162. 8. Marshall D, Johnell O, Wedel H.Meta-analysis of how well measures of bone mineral density predict occurrence of osteoporotic fractures. BMJ. 1996;312:1254-9. 9. De Laet C, Oden A, Johansson H, Johnell O, Jonsson B, Kanis JA. The impact of the use of multiple risk indicators for fracture on case-finding strategies: a mathematical approach. Osteoporos Int. 2005;16: 313-8. 10. Kanis JA, Johnell O, Oden A, De Laet C, Jonsson B, Dawson A. Ten-year risk of osteoporotic fracture and the effect of risk factors on screening strategies. Bone. 2002;30:251-8. 11. Johansson H, Oden A, Johnell O, Jonsson B, de Laet C, Oglesby A, McCloskey EV, Kayan K, Jalava T, Kanis JA. Optimization of BMD measurements to identify high risk groups for treatment—a test analysis. J Bone Miner Res. 2004;19:906-13. 12. McCloskey E, Johansson H, Oden A, Aropuu S, Jalava T, Kanis JA. Efficacy of clodronate on fracture risk in women selected by 10-year fracture probability. J Bone Miner Res. 2007;22 (Suppl 1):S17. 13. Kanis JA, Borgstrom F, Zethraeus N, Johnell O, Oden A, Jonsson B. Intervention thresholds for osteoporosis in the UK. Bone. 2005;36:22-32.