ORIGINAL ARTICLE

 

Validation of a model to predict six-month mortality in incident elderly dialysis patients

 

Josefina Santos^1,2, Pedro Oliveira^3,4, Jorge Malheiro^1,2, Andreia Campos¹, Sofia Correia¹, António Cabrita¹, Luísa Lobato^1,2, Isabel Fonseca^1,2,3

¹ Nephrology Department, Hospital de Santo António, Centro Hospitalar Universitário do Porto (CHUP), Porto, Portugal

² Unit for Multidisciplinary Research in Biomedicine, Instituto de Ciências Biomédicas Abel Salazar Porto, Universidade do Porto, Porto, Portugal

³ EPI Unit, ISPUP – Institute of Public Health, Universidade do Porto, Porto, Portugal

⁴ Department of Population Studies, Instituto de Ciências Biomédicas Abel Salazar Porto, Universidade do Porto, Porto, Portugal

 

Correspondence to:

 

ABSTRACT

Background and objectives: To evaluate RRT benefits and risks and to inform patients and their families about ESRD treatment options, we have developed a prognostic score to predict 6-month mortality in elderly ESRD patients initiating dialysis. Five independent predictors were identified and a point system was constructed: age 75 years or older (2 points), coronary artery disease (2 points), cerebrovascular disease with hemiplegia (2 points), time of nephrology care before dialysis [< 3.0 months (2 points); ≥ 3 to < 12 months (1 point)], serum albumin levels [3.0 - 3.49 g/dL (1 point); < 3.0 g/dL (2 points)]. Model performance was good in both discrimination and internal validation. Before adopting our risk score into practice, our aim is to externally validate this initial predictive model by assessing its performance on a new data set. Methods: We apply the predictive score developed in a cohort of CKD patients, aged 65 years and over who started dialysis between 2009 and 2016, to an independent cohort of ESRD patients, aged 65 years and over who started dialysis between 2017 and 2019, in our Nephrology department. The performance of the prediction equation created in development cohort, was assessed using discrimination and calibration metrics in the validation cohort. Results: Our validation study cohort included 168 individuals, with a mortality rate of 12.5% (n=21) within 6-months of dialysis initiation. Model performance in the validation cohort had an acceptable discrimination [AUC of 0.79; (95% confidence interval, 0.70 to 0.88)]. The Hosmer and Lemeshow goodness-of-fit test was not statistically significant, indicating good calibration of the model (χ2, 5 degrees of freedom = 2.311; P = 0.805). Conclusions: Our predictive simple score based on readily available clinical and laboratory data demonstrates a good performance when externally validated, namely with respect to discrimination and calibration. Model validation is crucial for adequately informing patients and their families about ESRD treatment options and providing a more patient-centered overall approach to care. Before we start general implementation in clinical practice, our score needs further validation in larger patient cohorts.

Key Words: Prognosis Score; End-Stage Renal Disease; Elderly; Decision Making

 

INTRODUCTION

Mortality in chronic kidney disease (CKD) remains high, particularly among the elderly, who represent the most rapidly growing segment of the end-stage renal disease (ESRD) population in Western countries^1,2.

One of the challenges to clinicians caring for older CKD patients expected to progress to ESRD lies in the evaluation of the overall benefit of offering them renal replacement therapy (RRT). Thus, for evaluating RRT benefits and risks and informing patients and their families about ESRD treatment options based on a shared decision-making process, several scoring systems have been developed^3-7. One of the concerns related to the available predictive scores is that those may be unsuitable for widespread application due to unproven generalizability.

Portugal has the one of the highest incidences and prevalence of ESRD in the world^8,9. Considering the need to develop prognostic models adapted to the specificities of each population, we have recently developed a prognostic score for predicting early death in elderly ESRD patients initiating dialysis in a cohort of Portuguese patients¹⁰.

This score had a good performance and it was internally validated using bootstrapping methods¹¹. If possible, before adopting a risk score into practice, the prognostic score should be externally validated and tested in a group of patients different to the sample used to develop the score¹¹.

Therefore, the objective of this study is to validate the previously developed prognostic score in an independent dataset and compare its performance with other known scoring systems⁴.

METHODS

A prospective cohort study was performed for external validation of our prognostic score10. The sample included all patients aged 65 years and over referred to the Nephrology Department of Centro Hospitalar Universitário do Porto (CHUP), who started dialysis as their first RRT between January 2017 and December 2019.

The study was performed in accordance with the Declaration of Helsinki and approved by CHUP’s Institutional Review Board.

Data was collected primarily from electronic clinical records and through information from dialysis centers. Demographic, clinical and functional variables were recorded. Glomerular filtration rate (GFR) was estimated using the Chronic Kidney Disease Epidemiology (CKD-EPI) 2009 creatinine equation¹²; all serum creatinine measurements were performed in the same laboratory using a calibrator for automated systems (Roche Diagnostics). Etiological diagnosis of CKD was based on the patient’s history, kidney ultrasound, and kidney biopsy, when available.

Cognitive status was evaluated using the Mini Mental State Examination (MMSE)13 with cognitive impairment defined for scores lesser or equal to 23. Functional dependency was defined as requiring assistance for transfer, classified as totally dependent or need assistance for transfer; otherwise, patients were classified as autonomous.

A modified version of the Charlson comorbidity index (mCCI)¹⁴, i.e., by excluding subject’s age and presence of kidney disease, was calculate and subdivided into three subgroups (0-2, 3-4, ≥5).

The outcome of interest was all-cause mortality within first 6 months of dialysis therapy initiation. In the validation cohort, vital status was checked until 30 December 2019.

The prognostic score that we intend to validate was developed in patients from the same center who started dialysis between January 2009 and December 2016. The design and detailed methodology used in the development of the prognostic model has been described previously¹⁰.

Statistical Analysis

Data are reported as medians and interquartile range (IQR) or frequencies and proportions whenever appropriate.

Comparisons between groups for categorical data were made using the chi-square test. Continuous data were compared using the Mann-Whitney test for non-normally distributed variables.

The discriminative power of the prognostic score (i.e., the ability to identify patients at highest risk of dying within the first 6 months of starting dialysis) was assessed by calculating the area under the receiver operating characteristic (ROC) curves (AUC). Calibration of the risk score reflecting the link between predicted and observed risk was evaluated by the Hosmer-Lemeshow goodness-of-fit test (a P-value above 0.05 indicates acceptable calibration). The developed risk score in our work¹⁰ and Couchoud score4 was calculated for each patient to determine the performance of each scoring system in predicting mortality. The discrimination of each scoring system was assessed and compared using AUC.

A P value < 0.05 was considered statistically significant for all analyses. Data were analyzed using the STATA 13.0 and SPSS 26.0 (SPSS, Inc., Chicago, IL) statistical software.

Model Development

Briefly, our score10 was developed using data from a cohort (development cohort) of 421 patients, aged 65 years and over who started dialysis between 2009 and 2016, in our Nephrology Service. Demographics and clinical variables were included as potential predictors.

The predictive score was developed using a multivariable logistic regression analysis. A bootstrapping method15,16 was used for internal validation.

Five independent predictors were identified and a point system was constructed: age 75 years or older (2 points), coronary artery disease (2 points), cerebrovascular disease with hemiplegia (2 points), time of nephrology care before dialysis [< 3.0 months (2 points); ≥ 3 to < 12 months (1 point)], serum albumin levels [3.0 - 3.49 g/dL (1 point); < 3.0 g/dL (2 points)] (Table I).

 

 

Model performance was good in both discrimination [AUC of 0.793; (95% confidence interval, 0.73 to 0.86)] and internal validation [concordance statistics of 0.791 (95% confidence interval, 0.73 to 0.85)].

With our model10, we made a risk assessment questionnaire for clinicians’ and patients’ use, illustrated in Figure 1, exposing a simple understandable method for establishing a patient’s risk for the outcome depending on an individual’s status for the five variables included in the tool.

 

 

RESULTS

Baseline Characteristics of Study Participants

The validation cohort included 168 individuals aged 65 years or older. Baseline patient characteristics from the development and validation cohorts are summarized in Table II.

 

 

Compared to patients from the development cohort, patients from the validation set had lower eGFR at dialysis initiation and had fewer hospitalizations within 6-months prior to dialysis. Furthermore, patients included in the validation sample were more functionally autonomous and were referred earlier to nephrology care prior to dialysis.

Independent Validation

Among patients in the validation cohort, there were 21 deaths (12.5%) within the first 6 months of dialysis initiation.

In the validation set (n=168), the performance of the prognostic score is shown in Figure 2, with an AUC of 0.79 (95% CI 0.70-0.88) indicating acceptable (nearly good) discrimination. The Hosmer and Lemeshow goodness-of-fit test was not statistically significant, indicating good calibration of the model (χ2, 5 degrees of freedom = 2.311; P = 0.805).

 

 

Comparison with Alternative Risk Score

Couchoud score4 was calculated for all patients in the validation cohort according to corresponding formula, with an AUC of 0.766 (95% CI 0.65–0.88). In this cohort, the performance of our score was higher than Couchoud score, but not statistically significant (P = 0.63) (Figure 3).

 

 

DISCUSSION

Incorporating predictive models into CKD management for older patients may help to inform patients and their families about ESRD treatment options and provide a more patient-centered overall approach to care.

Risk prediction models are based on equations designed on the basis of prognostic factors and clinical outcomes, available at the time the prediction is made, and collected in specific and representative cohorts of individuals followed up for a given period of time^17,18.

The performance of a risk prediction model is commonly assessed by testing its calibration and discrimination. Calibration describes the agreement of observed and predicted event rates¹⁹. Discrimination expresses the ability of the prediction model to distinguish individuals who will develop the outcome of interest from those who will not²⁰.

Another important question for physicians to consider is whether the score accurately predicts outcomes in people like their patients.

So, validation of prognostic models is a determinant step before we start implementation in clinical practice. Models should be internally and especially externally validated to obtain reliable estimates of model performance¹¹.

Internal validation implies assessment of model performance directly in the derivation cohort. This approach yields an optimistic estimate of model performance^17,18. To minimize this limitation, the model can be developed on the whole dataset and data reuse methods, such as cross-validation and bootstrapping, applied to assess performance^11,17,18.

In the derivation of our score10 we performed a bootstrapping procedure (5000 bootstrap samples) to internally validate the risk score, which generated a concordance statistics of 0.791 (95% confidence interval, 0.73 to 0.85) and an optimism of 0.002.

Even with a good performance achieved in the same cohort as the one that was used to develop the model, before adopting a risk score into practice, clinicians need to decide whether the score accurately predicts outcomes in a sample similar to their patients but belonging to a different source population; therefore, validation in an independent sample is required¹¹.

In the past years, several mortality scores have been developed on the basis of various combinations of comorbidities and laboratory data, but only a few of them have focused on short-term survival including only elderly CKD patients^3-7. Also, only a few of the models were externally validated^21-23.

Portugal has one of the highest unadjusted incidences of ESRD among European countries8. About 64% of the incident dialysis patients in 2018 were over 65 years with a mean age of 67.2 years for prevalent patients²⁴, above the mean age of the European registry⁸.

Differences in patients’ profiles, namely distinct sociodemographic and clinical characteristics between the cohorts used to derive those scores, reinforce the need to develop predictive scores adapted to the specificities of each population. With this in mind, we have recently developed a prognostic score for predicting early death in elderly ESRD patients initiating dialysis that has been derived and internally validated in a cohort of Portuguese patients¹⁰. This score is based on simple and readily available information.

With respect to model performance, the proximity of the AUC generated by bootstrapping procedure to the observed AUC and a very acceptable optimism indicate a good discrimination ability of our score. The good performance (its calibration and discrimination) of our model on the new data (validation group), indicated that the model was likely not overfit, and demonstrated its predictive accuracy.

In the development cohort, the performance of our risk score was significantly higher than Couchoud score⁴, which reflects the different characteristics of the populations involved in derivation of the models. Also, in the validation cohort, although not statistically significant, the performance of our score was higher than Couchoud score⁴.

Bansal et al.²¹ developed a prediction equation for 5-year risk of mortality for older people with CKD stages 3-5 not treated with dialysis.

The equation included nine readily available clinical variables (age, sex, race, eGFR, urine albumin-to-creatinine ratio, smoking, diabetes mellitus, and history of heart failure and stroke), and it was externally validated in a large cohort of elderly CKD patients. This model has an acceptable calibration and discrimination in both the development (C-statistic = 0.72; 95% confidence interval, 0.68 to 0.74) and validation cohort (C-statistic = 0.69; 95% confidence interval, 0.64 to 0.74).

However, one of its limitations is that the validation cohort did not fit the frailty phenotype associated with CKD²⁶ because the authors enrolled well-functioning men and women, and it has been well established that frailty is an additional risk factor for mortality in CKD patients²¹.

It is important to highlight any differences that might affect model translation between the validation sample and the original study sample. The differences in the baseline characteristics between our validation and development population are shown in Table II. Patients from the validation set had lower eGFR at dialysis initiation, had fewer hospitalizations within 6-months prior to dialysis, were more functionally autonomous and were referred earlier to nephrology care than patients from the development cohort. These differences may be due to the difference in timing of dialysis initiation, as the validation cohort was more recent than the development cohort.

Even so, our model achieved a good performance in the validation cohort, which confirms its predictive accuracy in a different source population, i.e.; it is independently validated.

Floege et al.²³ have published another risk prediction model developed in a European hemodialysis cohort with a mean age of 64 years old, using objective measurements. This model was then validated in an external cohort of the Dialysis Outcomes and Practices Patterns Study (DOPPS) and exhibited a moderate discrimination (C-statistic of 0.68 to 0.79). Nevertheless, contrary to our model¹⁰, the Floege et al. score²³ has not been developed nor validated in a cohort of elderly dialysis patients. In addition, because the development cohort includes only patients who survived the first 3 months, whereas the validation cohort of DOPPS includes mainly prevalent patients, it is still not a perfect risk predictor for frail elderly, in which the risk of short-term mortality is what needs to be predicted.

The Couchoud et al. model⁴ was externally validated in a US population²²; although investigators modified the score; poor performance was observed with respect to prediction of 6-month mortality in older patients with ESRD commencing dialysis. Although the sample size of our validation cohort has a limited pool of subjects compared to the development cohort, inherent to a single-center validation study, the validation sample included data on all the variables in the derivation model¹⁰.

Simplicity of models and reliability of measurements are important criteria in developing clinically useful prognostic models¹¹. Our predictive score¹⁰ includes variables that are well defined, measurable, and readily available; in other words; our model is clinically useful.

There are some limitations in our study. First, this is a single-center study, with a relatively small sample size. Secondly, our population consisted of incident dialysis patients that were referred to nephrologists.

Those who were not referred, not selected for, or not accepted for dialysis initiation, were not included. Our model may, therefore, not be generalizable to the entire population of elderly ESRD patients.

In conclusion, after development, our score was independently validated in a new dataset of patients indicating acceptable discrimination to predict early mortality for elderly CKD patients who initiate dialysis. This simple and accurate prediction score based on readily available data can be an easily implemented tool to apply in daily practice to guide patient care.

 

References

1. Pippias M, Jager KJ, Kramer A, et al. The changing trends and outcomes in renal replacement therapy: Data from the ERA-EDTA Registry. Nephrol Dial Transpl 2016; 31:831–841.         [ Links ]

2. Saran R, Li Y, Robinson B et al. US Renal Data System 2014 Annual Data Report: Epidemiology of Kidney Disease in the United States. Am J Kidney Dis 2015; (66):1:S1–S306.         [ Links ]

3. Cohen LM, Ruthazer R, Moss AH, Germain M. Predicting six-month mortality for patients who are on maintenance hemodialysis. Clin J Am Soc Nephro 2010; 5:72–79.         [ Links ]

4. Couchoud C, Labeeuw M, Moranne O, et al. A clinical score to predict 6-month prognosis in elderly patients starting dialysis for end-stage renal disease. Nephrol Dial Transplant 2009; 24:1553–1561.         [ Links ]

5. Couchoud C, Beuscart J, Aldigier J, et al. Development of a risk stratification algorithm to improve patient-centered care and decision making for incident elderly patients with end-stage renal disease. Kidney Int 2015; 88:1178–1186.         [ Links ]

6. Wick JP, Turin TC, Faris PD, et al. A clinical risk prediction tool for 6-month mortality after dialysis initiation among older adults. Am J Kidney Dis 2017; 69:568–575.         [ Links ]

7. Thamer M, Kaufman JS, Zhang Y, et al. Predicting early death among elderly dialysis patients: development and validation of a risk score to assist shared decision making for dialysis initiation. Am J Kidney Dis 2015; 66:1024–1032.         [ Links ]

8. Kramer A, Boenink R, Noordzij M, et al. The ERA-EDTA Registry Annual Report 2017: A Summary. Clin Kidney J 2020; sfaa048.         [ Links ]

9. United State Renal Data System. http://www.usrds.org. (21 August 2018, date last accessed)

10. Santos J, Oliveira P, Malheiro J, et al. Predicting 6-month mortality in incident elderly dialysis patients: a simple prognostic score. Kidney and Blood Pressure Research 2020; 45:38–50.         [ Links ]

11. Altman DG, Vergouwe Y, Royston P, Moons KG. Prognosis and prognostic research: validating a prognostic model. British Medical Journal 2009; 338:1432–1435        [ Links ]

12. Levey SA, Stevens LA, Schmid CH, et al. CKD-EPI (Chronic Kidney Disease Epidemiology Collaboration). A new equation to estimate glomerular filtration rate. Ann Intern Med 2009; 150: 604–612.         [ Links ]

13. Folstein MF, Folstein SE, McHugh PR. “Mini-Mental State”. A practical method for grading the cognitive state of patients for the clinician. J Psychiatr Res 1975; 12:196–198.

14. Charlson ME, Pompei P, Ales KL, MacKenzie CR. A new method of classifying prognostic comorbidity in longitudinal studies: development and validation. J Chron Dis 1987; 40:373–383.         [ Links ]

15. Moons KG, Donders AR, Steyerberg EW, Harrell FE. Penalized maximum likelihood estimation to directly adjust diagnostic and prognostic prediction models for overoptimism: a clinical example. J Clin Epidemiol 2004; 57:1262–1270.         [ Links ]

16. Harrell FE Jr, Lee KL, Mark DB. Multivariable prognostic models: issues in developing models, evaluating assumptions and adequacy, and measuring and reducing errors. Stat Med 1996; 15: 361–387.         [ Links ]

17. Steyerberg EW, Vickers AJ, Cook NR, et al. Assessing the performance of prediction models: a framework for traditional and novel measures. Epidemiology. 2010 Jan;21(1):128-38.         [ Links ]

18. Tripepi G, Heinze G, Jager KJ, et al. Risk prediction models. Nephrol Dial Transplant 2013 Aug;28(8):1975-80.         [ Links ]

19. Tripepi G, Jager KJ, Dekker FW, Zoccali C. Statistical methods for the assessment of prognostic biomarkers (part II): Calibration and re-classification. Nephrol Dial Transplant 2010;25(5):1402-5.         [ Links ]

20. Tripepi G, Jager KJ, F. W. Dekker, Zoccali C. Statistical methods for the assessment of prognostic biomarkers (part I): discrimination. Nephrol Dial Transplant 2010; (25):399–1401.

21. Bansal N, Katz R, De Boer IH, et al. Development and validation of a model to predict 5-year risk of death without ESRD among older adults with CKD. Clin J Am Soc Nephrol 2015; 10(3):363–371.         [ Links ]

22. Cheung KL, Montez-Rath ME, Chertow GM, et al. Prognostic stratification in older adults commencing dialysis. J Gerontol A Biol Sci Med Sci 2014; 69(8):1033–1039.         [ Links ]

23. Floege J, Gillespie IA, Kronenberg F, et al. Development and validation of a predictive mortality risk score from a European hemodialysis cohort. Kidney Int 2015; 87(5):996–1008.         [ Links ]

24. Substitutive Renal Therapy of Chronic Renal Disease in Portugal. Available at http://www.spnefro.pt/comissoes_Gabinete_registo_2018/registo_2018.         [ Links ]

25. Pippias M, Kramer A, Noordzij M, et al. The European Renal Association - European Dialysis and Transplant Association Registry Annual Report 2014: A Summary. Clin Kidney J 2017; 10:154–169.         [ Links ]

26. Shlipak MG, Stehman-Breen C, Fried LF, et al. The presence of frailty in elderly persons with chronic renal insufficiency. Am J Kidney Dis 2004; 43(5):861–867.         [ Links ]

 

Correspondence to:

Josefina Santos

Nephrology Department, Centro Hospitalar Universitário do Porto

Largo Prof. Abel Salazar 4099-001 Porto, Portugal.

E-mail: josefina.sts@gmail.com

 

Ethical Statement

The study was performed in accordance with the Declaration of

Helsinki and approved by the CHUP Institutional Review Board. The

subjects have given their informed consent.

 

Author Contributions

The authors contributed to this article in the following way: Study design: JS, IF; data collect: JS, AC, SO; data analysis: JS, PO, IF; methodology: JS, PO, AC, LL, IF; manuscript preparation: JS, IF.

 

Disclosure of potential conflicts of interest: none declared

 

Received for publication: Aug 18, 2020

Accepted in revised form: Sep 25, 2020