FormalPara Key Points

A computerized algorithm using a combination of International Classification of Diseases, Tenth Revision T codes for drug toxicity and K-71 codes for toxic liver injury was developed to identify idiosyncratic drug-induced liver injury (DILI) cases from the electronic medical record (EMR).

The DILI-case-searching algorithm performed well and identified 121 true positives and 61 false positives and had a positive predictive value of 66.5% (95% confidence interval 62.7–70.3).

Future studies incorporating natural language processing may further improve the efficiency of this algorithm in identifying high-causality DILI cases from the EMR.

1 Introduction

Idiosyncratic drug-induced liver injury (DILI) is a rare but important cause of liver disease [1]. In most instances, DILI is not related to the dose or duration of suspect medication use and occurs in fewer than 1 in 10,000 treated patients [2, 3]. In contrast, acetaminophen is a well-known dose-dependent, intrinsically hepatotoxic drug, with at least 60,000 cases of acetaminophen hepatotoxicity reported each year in the USA [4]. In addition to being a leading cause of acute liver failure in the USA, idiosyncratic DILI is also the most common reason for US FDA regulatory actions regarding approved medications [5]. To better understand the etiologies, risk factors, and outcomes with DILI, a reliable and efficient means of identifying cases is needed.

Idiosyncratic DILI is largely a clinical diagnosis of exclusion that requires stepwise evaluation for alternative causes of liver injury and improvement in liver biochemistries after drug discontinuation [6,7,8,9,10]. Identifying idiosyncratic DILI cases in administrative claims databases and the electronic medical record (EMR) has proven very challenging. Pharmacoepidemiological studies using International Classification of Diseases, Ninth Revision (ICD-9) codes to identify probable cases of DILI have had a low yield (i.e., < 1%), largely because of the lack of DILI-specific codes [11]. ICD-9 codes also have limited positive predictive value (PPV) for identifying cases of severe idiopathic acute liver injury [12, 13]. In 2015, the more precise ICD-10 diagnostic coding system, which has more than three times as many individual codes than the ICD-9 system, was introduced in the USA [14,15,16]. Although the ICD-10 system contains over 60 primary liver-specific diagnostic codes, data on the yield and accuracy of ICD-10-based searching algorithms for identifying DILI cases in the EMR are limited [17,18,19]. The aim of the current study was to develop a searching algorithm using a combination of ICD-10 codes for adverse drug events (T codes) and toxic liver injury (K codes) to identify individual patients with idiosyncratic DILI for further investigations.

2 Methods

2.1 Patient Selection

A waiver to conduct this retrospective chart review study was obtained from the institutional review board at the University of Michigan. The inpatient and outpatient EMRs at the University of Michigan Health systems in the Epic Care system (Epic Systems, Madison, WI, USA) were searched using DataDirect, a self-serve proprietary tool that allows access to clinical data such as diagnoses, encounters, procedures, medications, and laboratory results on more than four million unique patients [20]. Following the implementation of the ICD-10 coding system, all encounters with an ICD-10 T code for 1 of 15 categories of specific drug poisoning/toxicity from 1 October 2015 to 30 September 2018 were identified using DataDirect [14]. Next, we identified encounters with an ICD-10 K-71 code for 1 of 11 toxic liver injury diagnoses during that time period (see Supplemental Tables 1 and 2). We then identified encounters that had at least one ICD-10 T code combined with an ICD-10 K-71 code that also met one of our predefined laboratory criteria for clinically significant liver injury. Acetaminophen hepatotoxicity cases were excluded by eliminating cases with the ICD-10 code T39.1 for 4-aminophenol derivatives. Clinically significant liver injury was defined as an aspartate aminotransferase (AST) level ≥ 5 × the upper limit of normal (ULN), alanine aminotransferase (ALT) ≥ 5 × ULN, alkaline phosphatase (ALP) ≥ 2 × ULN, total bilirubin ≥ 2.5 mg/dL or international normalized ratio (INR) ≥ 1.5 based on the Drug-Induced Liver Injury Network (DILIN) prospective study [1]. For each identified case, demographic data (age, sex, race/ethnicity), suspect drug dose and duration, laboratory testing for alternative causes of liver injury (hepatitis A immunoglobulin M [IgM], hepatitis B surface antigen, hepatitis B core IgM, hepatitis C antibody, antinuclear antibody, anti-smooth muscle antibody), liver imaging (ultrasound, computerized tomography, and/or magnetic resonance imaging), and liver biopsy results were extracted and entered into our study database. Inpatient and outpatient physician notes regarding the liver injury episode were also manually reviewed.

2.2 Pattern and Severity of Liver Injury

The pattern of liver injury was classified as hepatocellular, cholestatic, or mixed according to the R ratio, which compares the serum ALT and ALP in multiples of their ULN based on the values at DILI onset. The R ratio was calculated using the formula R = (ALT/ULN)/(ALP/ULN). R > 5 defined hepatocellular injury, R < 2 cholestatic injury, and 5 > R > 2 mixed pattern of injury. Severity of DILI episode was categorized on a scale of 1–5 as follows: (1) mild = elevated ALT and/or ALP levels but bilirubin < 2.5 mg/dL and INR < 1.5; (2) moderate = elevated ALT and/or ALP levels and bilirubin ≥ 2.5 mg/dL or INR ≥ 1.5; (3) moderate–severe = elevated ALT, ALP, bilirubin, and/or INR levels and patient hospitalized or an ongoing hospitalization prolonged because of DILI; (4) severe = elevated ALT and/or ALP levels and bilirubin ≥ 2.5 mg/dL and at least one of the following: (i) hepatic failure (INR ≥ 1.5, ascites or encephalopathy), (ii) other organ failure believed to be due to DILI; and (5) fatal = patient died or underwent liver transplantation because of DILI.

2.3 Causality Assessment

The DILIN expert opinion causality score ranges from 1 (definite) to 5 (unlikely): (1) definite =  > 95% likelihood, (2) highly likely = 75–95% likelihood, (3) probable = 50–74% likelihood, (4) possible = 25–49% likelihood, and (5) unlikely =  < 25% likelihood [21]. The likelihood of a drug causing a liver injury was assessed by drug latency (time from initiation to DILI onset), dechallenge (time from discontinuation to laboratory improvement), and evaluation for competing causes of liver injury (viral and autoimmune serologies, liver imaging, liver pathology). In addition, the phenotype and clinical features of the DILI episode were compared with what is known of the hepatotoxicity profile of the suspect agent. In this study, the following extensive clinical data were extracted from the EMR and entered into a study database: laboratory test results at DILI onset; serial results over time; results of tests for hepatitis A, B, C, and autoimmune hepatitis; alcohol use; liver imaging results; use of corticosteroids; ursodiol; presence of eosinophilia, rash, and fever; and liver pathology reports. All cases were initially adjudicated by one of the physician investigators, and then a final DILIN causality score was achieved via consensus of at least three physicians. For analytical purposes, patients were categorized as “probable DILI” when they had a DILIN causality score of definite/highly likely/probable (1/2/3), whereas those with an alternative cause of liver injury and DILIN expert opinion score of possible/unlikely (4/5) were deemed “non-DILI.” Causality was also assessed with the Roussel Uclaf Causality Assessment Method (RUCAM) instrument, which provides a semiquantitative assessment of causality by assigning − 3 to + 3 points to each of six domains [22]. A final summary score allowed categorization of cases as highly probable (> 8), probable (5–8), possible (1–4), or unlikely (< 1).

2.4 Data Analysis

Descriptive statistics were reported as mean ± standard deviation or median (range) for normally and non-normally distributed data, respectively. Between-group comparisons were performed using chi-squared or Fisher’s exact test. Differences were considered significant at p < 0.05. All statistical analyses were performed using Microsoft Excel (Microsoft Corporation, Redmond, WA, USA).

3 Results

Between 1 October 2015 and 30 September 2018, a computerized search of 1,211,787 patient encounters identified 35,624 instances with at least one of the 15 T codes for drug toxicity/poisoning and 659 encounters with at least one of the 11 K-71 codes for toxic liver injury (Fig. 1). In addition, 517 encounters had both a T code and a K-71 code. Among these 517 encounters, 257 unique patients met at least one laboratory criteria for clinically significant liver injury. Of these 257 patients, 75 had acetaminophen hepatotoxicity, and these cases were excluded using the T39.1 code for 4-aminophenol derivatives. Therefore, the final study sample included 182 patients with potential idiosyncratic DILI whose charts were manually reviewed for causality assessment. After applying the DILIN causality assessment method, 121 of the 182 (66.5%) cases were categorized as “probable DILI”, whereas 61 (33.5%) had an alternative cause of liver injury that was more likely, as shown in Table 1 (i.e., non-DILI). There were no statistically significant differences in patient age, sex, or race in the probable DILI versus non-DILI group. Although the mean serum AST and ALT levels were higher at presentation in the patients with probable DILI, the between-group differences were not statistically significant because of the wide range of values. However, the pattern of liver injury was more likely to be hepatocellular (56.2 vs. 39.3%, p = 0.04), and the patients with probable DILI were more likely to be hospitalized (87.1 vs. 49.2%, p < 0.001) than the non-DILI group, indicative of more severe illness.

Fig. 1
figure 1

Flow diagram of ICD-10-based computerized searching algorithm for idiosyncratic DILI cases. In the final study sample of 182 patients, 121 (66.5%) were adjudicated as “probable DILI” and the remaining 61 (33.5%) were diagnosed with alternative causes of liver injury or “non-DILI”. DILI drug-induced liver injury, ICD-10 International Classification of Diseases, Tenth Revision

Full size image
Table 1 Presenting clinical features and outcomes of probable DILI and non-DILI cases
Full size table

Table 2 shows the proportion of cases attributed to each of the 15 individual T codes used in the searching algorithm. The highest proportion of probable DILI cases was attributed to antineoplastic agents (25.6%), followed by systemic antibiotics (23.1%) and diuretics and other drugs not otherwise specified (23.1%). Not surprisingly, the most frequently identified suspect drugs in the probable DILI group were nivolumab (ten), trimethoprim-sulfamethoxazole (eight), pembrolizumab (seven), and amoxicillin-clavulanate (six). The most common alternative medical diagnoses for the non-DILI cases included sepsis (26%), ischemic hepatitis (23%), and liver cancer (15%) (Fig. 2).

Table 2 Individual T codes associated with the probable DILI and non-DILI casesa
Full size table
Fig. 2
figure 2

Alternative causes of liver injury identified in the 61 patients with non-DILI. DILI drug-induced liver injury

Full size image

3.1 RUCAM Scores

A significantly higher proportion of probable DILI cases had a RUCAM score of probable or higher compared with the non-DILI cases (89.3 vs. 41.0%, p < 0.01), lending face validity to the expert opinion causality assessment method (Table 3). A strong inverse correlation between the overall RUCAM scores and DILIN expert opinion scores was seen in both the probable DILI and the non-DILI cases (Fig. 3).

Table 3 Distribution of RUCAM scores in cases of probable DILI and non-DILI
Full size table
Fig. 3
figure 3

Correlation between RUCAM and DILIN causality scores in the 182 adjudicated cases. There was a strong inverse relationship between the RUCAM and DILIN expert opinion causality scores (R2 = 0.412). DILIN Drug-Induced Liver Injury Network, RUCAM Roussel Uclaf Causality Assessment Method

Full size image

4 Discussion

Idiosyncratic DILI is an uncommon but potentially serious cause of liver injury and a leading reason for FDA regulatory actions involving approved medications. For these reasons, substantial efforts and resources have gone into research to better understand host factors that may increase susceptibility to DILI [23]. However, identifying patients with probable DILI for enrollment into mechanistic, pharmacoepidemiological, and clinical studies has proven to be very challenging because idiosyncratic DILI is a rare clinical diagnosis of exclusion that even experienced clinicians find difficult to establish.

To improve DILI detection, researchers have used several strategies to search administrative databases for potential DILI cases with varying success. Searching algorithms based on ICD-9 codes have consistently had low PPV for identifying DILI cases [11, 12, 17]. A recent meta-analysis of ICD-9-based searching algorithms reported a pooled PPV estimate of only 14.6% for identifying DILI cases [13]. The low yield with ICD-9 codes was likely because of the lack of DILI-specific codes and the potential for coding errors by medical billers. The ICD-10 coding system was introduced in 2015 and has more detailed codes and greater specificity and accuracy for many diseases than the ICD-9 coding system. For instance, ICD-10-based algorithms to identify patients with systemic lupus erythematosus in the EMR had significantly higher PPVs than ICD-9-based algorithms [24]. Similarly, ICD-10 codes had a PPV of > 90% for identifying true cases of cirrhosis in the Veterans Administration patient population [25]. ICD-10 coding also has particularly good sensitivity and specificity in identifying a multitude of adverse drug reactions, including liver toxicity [15, 16].

In the current study, a computerized algorithm based on ICD-10 codes for specific drug poisoning/toxicity diagnoses (T36–T65) and toxic liver injury (K71.0–71.9) was developed to search for probable DILI cases in a widely used commercial EMR (Epic) at a large tertiary care center. These two sets of ICD-10 codes were selected to maximize the sensitivity of the algorithm. In addition, predefined minimal laboratory criteria were used that have previously been shown to identify DILI cases more reliably than less stringent laboratory criteria [26]. Over the 3-year study period, 182 potential idiosyncratic DILI cases were identified from a database of over 1.2 million unique patient encounters after excluding all acetaminophen hepatotoxicity cases (Fig. 1). Following causality assessment of these 182 individual cases, 121 true positives, 61 false positives, and an overall PPV of 66.5% (95% confidence interval 62.7–70.3) was noted. This PPV is more than four times higher than the pooled PPV of ICD-9-based algorithms for DILI case identification (i.e., 66.5 vs. 14.6%) [13]. These findings suggest that a computerized algorithm search using ICD-10 codes is highly reliable and accurate in identifying probable DILI confirmed by expert opinion case adjudication. The high PPV observed with our searching methodology also suggests a high degree of coding accuracy for the DILI cases under study wherein misdiagnosis is frequently reported [27].

Other recently published studies have begun to explore the utility of ICD-10-based searching algorithms to identify patients with acute liver injury who were prescribed antidepressants [17, 18]. In one multicenter European study, multiple ICD-10 codes performed substantially better than the ICD-9 codes for liver injury, and they also performed better in inpatients versus outpatients [17]. However, the methodology in these other studies only involved searching for various K codes associated with liver injury (without T codes), and no causality assessment for a specific drug using semiquantitative expert opinion methods was undertaken. Nonetheless, the PPVs for the algorithms employed were generally good at 8–80%, and they were substantially better than those seen with ICD-9 algorithms.

4.1 Limitations

Limitations of this study include the lack of a laboratory-based gold standard for verifying a diagnosis of DILI. However, other studies have demonstrated that structured expert opinion-based causality assessment is more accurate and reliable than RUCAM scores and is currently the clinical gold standard for establishing a diagnosis of DILI [28]. Furthermore, it is possible that our searching algorithm did not pick up all DILI cases that arose during the study period. To address this concern, the search algorithm will need to be repeated in other datasets to improve generalizability and determine which specific K-71 codes can provide the largest number of cases (Supplemental Table 2). In addition, the benefit of checking other hepatic failure liver injury codes such as K-72 requires further study. A final limitation of our study was the need for manual review of individual patient notes to adjudicate cases. The use of bioinformatics tools such as natural language processing may further improve the yield and efficiency of searching the EMR for bona fide cases of DILI [29, 30].

5 Conclusions

Overall, this study showed that an ICD-10-based algorithm to search for DILI in the EMR had a high PPV for identifying DILI cases. Furthermore, the accuracy was excellent and better than that achieved in previously published studies using ICD-9 codes [9, 10]. The strong correlation between the expert opinion causality scores and RUCAM scores lends credence to the expert opinion causality assessment methods employed in this study. In addition to validation in other patient populations, incorporating bioinformatics tools such as natural language processing will likely improve the accuracy and efficiency of this computerized searching algorithm to facilitate future research into the pharmacoepidemiology and pathogenesis of DILI. Lastly, this DILI-searching algorithm may facilitate rapid detection of postmarketing hepatotoxicity signals for newly approved drugs.