Calculating the **95% confidence interval** for the **AUC** of the **ROC curve** allows one to infer the range of values that the AUC would take from the results of data obtained 'by chance'.

Confidence intervals are therefore always presented in scientific papers.

This article describes **how to find the 95% confidence interval of the ROC-AUC** and shows **how the calculation can be implemented** by the Python language.

## Online automatic calculations

The **AutoCalc** automatic calculation web application of Ushitora Lab. allows you to calculate **confidence intervals** for possible **AUC** values of **ROC** curves **online** based on the method described in this article.

Various other calculation functions are available for you to try out!

## What is ROC-AUC?

**ROC Curve** (Receiver Operating Characteristic Curve) is a method of examining the classification performance of a test by plotting the **trade-off** between the sensitivity of the test (**true positive rate**) and the **false positive rate**.

The area under the curve is called the **AUC** and a test can be considered excellent if the AUC of the ROC curve (**ROC-AUC**) exceeds approximately **0.8**.

The nature of the ROC curve and how it is used in conjunction with the **PR curve** is described in more detail in the following article.

## Confidence interval for ROC-AUC

### What is confidence interval?

**Confidence Interval** (**CI**) is defined in principle as,

Repeated sampling from a

populationthat follows anormal distribution, each time calculating the interval in a certain way.When

x% of the multiple intervals obtained at this time contain the population mean within the interval,the interval obtained by this method is called the

x% confidence interval.

However, outside of situations such as writing a statistics paper, it may be easier to think of it in a simpler way.

The **95% confidence interval** of the ROC-AUC, which is the subject of this article, should be used as an indicator to mean that the ROC-AUC is roughly around this value.

### Formula for confidence interval of ROC-AUC

The value of the ROC-AUC calculated from the data obtained is written as \(A\) .

Also, write the **standard error** of \(A\) as \(\mathrm{SE}(A)\) (the calculation method is shown later).

In this case, the **x% confidence interval** of the ROC-AUC \(C(x)\) is,

$$C(x)=A\pm z\left(\frac{1-x}{2}\right)\mathrm{SE}(A)$$.

Here, \(z(a)\) is a function that returns **the upper a% point of the normal distribution**.

When determining the **95% confidence interval** ( \(x=0.95\) ), which is often used as a reference in statistics, from the **normal distribution table**, we can use

$$z\left(\frac{1-x}{2}\right)=z(0.025)=1.96$$

and therefore,

$$C(x)=A\pm 1.96\mathrm{SE}(A)$$

can be obtained.

### Standard error of ROC-AUC

\(\mathrm{SE}(A)\) used in the confidence interval formula, according to Hanley et al. (1982), can be calculated as

$$\mathrm{SE}(A)=\sqrt{\frac{A(1-A)+(n_P-1)(Q_1-A^2)+(n_N-1)(Q_2-A^2)}{n_P\cdot n_N}}$$.

where \(n_P,n_N\) is, respectively, of the data,

- \(n_P\): The number of people who should have truly
**positive**results (= e.g. the number of people sick) - \(n_N\): The number of people who should have truly
**negative**results (i.e. healthy)

These will be referred to as **positive samples** and **negative samples** from now on.

\(Q_1,Q_2\) is a probability value calculated from the test that produced the ROC curve.

- \(Q_1\):The probability that when
**two positive samples**are randomly selected, both will be judged "**more strongly suspected positive**" by the test than**one negative sample**randomly selected. - \(Q_2\): The probability that when
**two negative samples**are randomly selected, both of them will be judged "**strongly suspected negative**" by the test than**one positive sample**randomly selected.

Hanley and McNeil showed us that these two probabilities can be calculated as

$$Q_1=\frac{A}{2-A}$$

$$Q_2=\frac{2A^2}{1+A}$$

## Implementation in Python

Following the above theory, a programme to calculate the 95% confidence interval of the ROC-AUC is implemented in Python.

The variables in the code have the following meanings.

`y_true`

: an array of**labels**recording whether the data are positive or negative samples.`y_score`

: an array of**positive probabilities**that the test has determined for the data.- Example: the value of the test value, etc.

`positive`

: the letter of the label representing a positive example- Default settings assume positive =
`1`

, negative =`0`

.

- Default settings assume positive =

The following code calculates the ROC curve using functions from the `sklearn`

library and simultaneously outputs the 95% confidence interval of the ROC-AUC.

## Reference

[1] J. A. Hanley and B. J. McNeil. *The Meaning and Use of the Area under a Receiver Operating Characteristic (ROC) Curve.* Radiology, 1982.

## Comments