Clinical validation of the CE-IVD marked Therascreen MGMT kit in a cohort of glioblastoma patients

Abstract

BACKGROUND:

Pyrosequencing is recognized as a strong technique to analyze the MGMT status of glioblastoma patients. The most commonly used assay, quantifies the methylation levels of CpGs 74 to 78. A more recent CE-marked In Vitro Diagnostic Medical Device (CE-IVD) assay, Therascreen, analyzes CpGs 76–79.

METHODS:

We performed a comparison of these two assays to evaluate the potential impact of this shift in analyzed CpGs. Therascreen analysis was centrally performed for 102 glioblastoma patients, who were part of a prospective multicenter trial.

RESULTS:

A strong correlation was observed for the mean values of the 4 or 5 analyzed CpGs, with lower values recorded using the Therascreen assay, especially for values greater than 20%. When considering a classification in 3 categories ( $>$ 12%: methylated; $⩽$ 8%: unmethylated; 9–12%: grey zone), 93% of patients were identically classified between the two assays. Using a binary classification, 95% and 97% of patients were identically classified with cut-offs of 8% and 12%, respectively. A strong prognostic significance was observed for both assays: median overall survival were 15.9 months and 34.9 months for respectively unmethylated and methylated patients with either test.

CONCLUSION:

The results demonstrate that these assays may be used interchangeably.

1. Introduction

Qiagen currently sells numerous assays for quantitative measurements of MGMT (methylguanine methyltransferase) methylation status using the pyrosequencing technique. Hs_MGMT_01_PM PyroMark CpG assay (ref PM 00149702) analyzes 7 CpGs located upstream of DMR1 and DMR2 regions, where methylation has significantly been correlated with expression [1]. This assay must therefore be avoided for a clinical purpose. Two additional assays (ref 970032 and 972032) quantify the methylation levels of CpGs 74 to 78. They are respectively optimized for use with the PyroMark Q24 and Q96. These two similar assays are currently the most widely used and validated pyrosequencing assays [2]. They will be further referred to as PSQ. In 2011, a fourth assay was launched: the Therascreen MGMT Pyro Kit (Thera). This assay, in contrast to the others, is a CE-marked In Vitro Diagnostic Medical Device (CE-IVD) that meets all requirements of the EC Directive 98/79.

Several countries around the world have adopted the international standard ISO 15189, which specifies requirements for quality and competence in medical laboratories. Among the preferred procedures described in this international standard are those specified in the instructions for use of in vitro medical devices. Furthermore, as the steps of validation into the laboratory are less extensive for CE-IVD assays, laboratories tend to favor this type of assay. Thera appears to be a suitable choice for MGMT testing and some studies have reported strong analytical performances for this assay [3, 4]. However, this assay quantifies the methylation levels of CpGs 76 to 79 instead of CpGs 74 to 78. One of the strengths of PSQ for MGMT testing relies on the several independent studies that are concordant with the threshold levels that discriminate glioblastoma (GBM) patients as being good or poor responders to Temozolomide (TMZ) treatment [2]. As a heterogeneous pattern of methylation can be observed for some tumors, the shift in the CpGs analyzed between the two assays could potentially impact the result ( $=$ average methylation percentage of the tested CpGs). It is therefore mandatory to validate the cut-off for the Thera assay.

We have recently performed a prospective dedicated multicenter trial, which allowed us to validate the use of PSQ in a daily practice. For the present study, we analyzed 102 frozen GBM patients from this trial with the Thera assay and compared the results to those obtained previously with the standard PSQ test.

2. Materials and methods

2.1 Patients and samples

Samples were analyzed from patients enrolled in a prospective study dedicated to the validation of two techniques to assess MGMT status [5]. Patients were enrolled for this study between the dates of March 11, 2009 and June 29, 2011 from 8 French centers. Eligible patients had histologically confirmed de novo-glioblastoma, between the ages of 18–70 and presented with no contraindications, as dictated by the Stupp protocol. The protocol was approved by the Rennes medical ethics committee and informed consent was obtained from each patient.

DNA was extracted from 3 primary cell lines (RNS85/96 and 175), which were used as quality controls in each series of tests.

2.2 MGMT promoter methylation analysis

DNA extractions from frozen clinical samples and sodium bisulfite treatment were performed at each center according to local procedures. Samples with a histologically estimated tumor cell content below 40% were excluded from the study. The Thera test was centrally performed on any remaining bisulfite treated DNA following completion of the main portion of the project. Thera was performed using the Therascreen MGMT Pyro Kit (ref 971061, Qiagen, France) according to the manufacturer’s instructions. The average percentage of the 4 CpGs tested was considered to calculate the cut-off.

2.3 Statistical analysis

Statistical analysis was performed using R statistical software (version 2.13.0, http://www.Rproject.org). The function risksetAUC (package risksetROC) in the R statistical software was used to obtain the area under the ROC curve. Additionally, the Harrell’s C index [6] was calculated using the validate function (in Design package). To study OS and PFS, cumulative event curves (censored endpoints) were established using the Kaplan-Meier method.

3. Results

3.1 Study population

Among the 112 samples of patients initially analyzed with PSQ [5], 102 were available for the Thera analysis. The median Progression Free Survival (PFS) for these 102 patients was 9.5 months (8.8–11.2; 95% CI) and the median Overall survival (OS) was 20.6 months (18.7–23.0; 95% CI).

3.2 Analysis of intra-laboratory reproducibility of Thera

Each control was evaluated in 6 different series. The mean values were 4% for RNS85 (range: 4–4%), 16% for RNS175 (range: 15–17%) and 33% (range: 32–34%) for RNS96. The reproducibility CVs were 5% for RNS175 and 3% for RNS96. All the results were identical for RNS85, which is close to the limits of quantification previously published for pyrosequencing (4%).

Table 1

Mean, median and extreme values for each CpG and for the mean of the 4 or 5 CpGs analyzed with the PSQ and Thera assays

	Mean	Median	Min	Max
Thera1 (CPG76)	16	4	0	90
Thera2 (CPG77)	16	6	0	66
Thera3 (CPG78)	13	6	1	67
Thera4 (CPG79)	20	10	1	89
Thera mean	16	7	1	65

	Mean	Median	Min	Max
PSQ1 (CPG74)	23	7	0	86
PSQ2 (CPG75)	25	9	1	85
PSQ3 (CPG76)	22	8	1	85
PSQ4 (CPG77)	20	7	0	87
PSQ5 (CPG78)	17	7	1	86
PSQ mean	21	8	1	84

Table 2

Comparison of the prognostic impacts when evaluating MGMT promoter methylation with therascreen pyrosequensing (Thera) and “standard” pyrosequencing (PSQ). The previously determined cut-offs of 8% and 12% were tested to determine the associated Hazard ratio (HR) and the level of significance (represented by the $p$ value, which is to compare to 1.3/1000 with the multiple comparison correction of Bonferroni), after adjustment on age and Karnofsky score. The prediction errors were globally evaluated and reported as the Area Under the ROC Curve (AUCROC) and the Harrell’s C index

Variable	Cut-off (%)	HR	$p$	AUCROC	CHarrel
THERA	8	0.30	1.00 E ${}^{-06}$	0.68	0.69
THERA	12	0.30	1.00 E ${}^{-06}$	0.68	0.68
PSQ	8	0.25	$<$ 1.00 E ${}^{-06}$	0.69	0.69
PSQ	12	0.27	$<$ 1.00 E ${}^{-06}$	0.69	0.69

Table 3

Agreement between the two assays. Number of patients identically classified dichotomizing patients in 3 classes with cut-offs at 8 and 12% (A) or in two classes with cut-offs at 8% (B) or 12% (C)

A
[dir=NW,width=3cm]TheraPSQ	$⩽$ 8	9–12	$>$ 12
$⩽$ 8	51	4	1
9–12	0	2	2
$>$ 12	0	0	42

B
[dir=NW,width=1.75cm]TheraPSQ	$⩽$ 8	$>$ 8
$⩽$ 8	51	5
$>$ 8	0	46

C
[dir=NW,width=1.75cm]TheraPSQ	$⩽$ 12	$>$ 12
$⩽$ 12	57	3
$>$ 12	0	42

Figure 1.

Comparison between Thera and PSQ. Agreement between the two tests for CpG76, CpG77, CpG78 and the mean of the 4 or 5 CpGs is described using scatter and Bland-Altman plots.

Table 4

Results for samples classified differently by PSQ and Thera assays. For patients without event occurrence, time to death or time to recurrence are written in italics

Patient’s		PSQ1	PSQ2	PSQ3	PSQ4	PSQ5		PSQ mean	Time to recurrence	Time to death
number				Thera1	Thera2	Thera3	Thera4	Thera mean	(months)	(months)
	PSQ	10	34	12	14	4		15
1017	Thera			10	12	4	12	10	181	642
	PSQ	9	11	9	8	8		9
2154	Thera			11	8	4	10	8	594	1256
	PSQ	9	9	9	7	10		9
7056	Thera			5	4	7	10	7	84	84
	PSQ	17	9	9	11	6		10
2022	Thera			4	7	6	11	7	279	402
	PSQ	6	16	17	7	7		11
8004	Thera			13	6	5	8	8	454	821
	PSQ	20	22	15	11	12		16
7125	Thera			10	9	12	11	10	347	1190
	PSQ	33	39	27	23	31		30
4109	Thera			1	1	1	2	1	1379	1379

Figure 2.

Kaplan-Meier analysis of overall survival (OS) according to MGMT promoter methylation status. M: patients with a value above the calculated cut-off and therefore considered as “methylated”; UM: patients with a value below or equal to the calculated cut-off and therefore considered as “unmethylated”.

3.3 Comparison of MGMT methylation results obtained with the 2 kits

The median percentages of methylation for the studied population were 7% when using Thera (range 1–65%) and 8% when using PSQ (range 1–84%) (Table 1). A strong correlation was observed for CpG76, 77, 78 and CpG mean analyzed using the PSQ and Thera assays. However, almost systematically, lower results were recorded with Thera, especially for values greater than 20% (Fig. 1).

3.4 Validation of the pre-defined cut-offs 8% and 12% for Thera

We recently recommended a classification of MGMT

promoter methylation status into three subgroups: “unmethylated” (0–8%), “methylated” (13–100%) and a grey zone for patients with intermediate values (9–12%) [5]. The cut-offs 8 and 12% were tested in this series of patients. The percentages of patients classified as “methylated”, when employing a cut-off of 8% and 12% for PSQ, were 50% and 44% with AUCROC values of 0.69 for OS (Table 2). These data are almost identical to those obtained for the overall population ( $n=$ 112 patients, 49%, 44% and AUCROC values of 0.69 and 0.70), indicating the absence of bias in the selection of the 102 patients for the present cohort. The percentages of patients classified as “methylated” when employing a cut-off of 8% and 12% for Thera were 45% and 41%, with AUCROC values of 0.68 for OS (Table 2). With a classification in 3 categories (methylated/unmethylated/grey zone), 93% of patients were identically classified using the two assays. With a binary classification (methylated/unmethylated), 95% and 97% of patients were identically classified with cut-offs of 8% and 12% respectively (Table 3). Table 4 shows the pyrosequencing results, as well as the time to death and time to progression for the patients differently classified by the two assays.

Kaplan-Meier survival curves displaying the OS of patients dichotomized according to these cut-offs are presented in Fig. 2. At a methylation cut-off of 8%, median OS were 34.7 months (95% CI 23.0–39.4) versus 15.9 months (95% CI 13.2–19.0) for respectively methylated and unmethylated patients analysed with Thera, with a p value of 1.3 10 ${}^{-7}$ . Results were 30.0 months (95% CI 22.9–39.1) versus 15.9 months (95% CI 13.1–19.0) with a p value of 2.6 10 ${}^{-9}$ for PSQ. At a methylation cut-off of 12%, results were 34.9 months (95% CI 24.5–43.8) versus 17.0 months (95% CI 13.7–19.5) with a p value of 3.2 10 ${}^{-7}$ for Thera and 34.9 months (95% CI 24.5–40.8) versus 16.4 months (95% CI 13.7–19.1) with a p value of 1.4 10 ${}^{-8}$ for PSQ. However, only few samples had methylation between 9–12% and it was not possible to individualize them in our study.

4. Discussion

MGMT promoter methylation is recognized as an effective predictor of response to TMZ for newly diagnosed GBM patients. Among the different techniques to analyze MGMT status, pyrosequencing is regarded as a very robust technique and its clinical utility has been validated in several independent studies [7, 8, 9, 10, 11, 12, 13, 14]. Pyrosequencing provides the percentage of methylated alleles of each CpG site analyzed and generally the average of the different sites is used to classify patients as “methylated” or “unmethylated”. The 5 most commonly analyzed CpGs are CpGs 74 to 78 and commercial kits are available to assess them, allowing reproducible and comparable results from one laboratory to another. The more recently launched Thera kit quantifies the methylation levels of CpGs 76 to 79. We have previously described that methylation can be heterogeneous from one site to the other [15]; analyzing different CpGs could therefore have an impact on the mean result. As a corollary the thresholds optimized for a combination of CpGs may not be optimal for additional combinations.

A comparison of CpGs one by one demonstrated a strong correlation between the results obtained with PSQ and Thera for common CpGs 76, 77 and 78, but with lower results with Thera for values above 15–20%. In our own experience, bisulfite treated DNA is very stable, so the delay of 2–3 months between Thera analysis and bisulfite treatment can’t explain these lower results. The same observation was performed for the mean values. As we propose threshold values of 8 and 12%, this difference has a minor impact for patient’s classification as “methylated” and “unmethylated”. With cut-offs at 12% and 8%, 97% and 95% of patients were identically classified according to the two techniques. When considering a three-class classification, among the 7 patients differently classified, 6 would have been classified as methylated with one technique and as being in the grey zone with the alternate technique. Specific authors have proposed higher cut-offs between 25% and 35% to classify patients [8, 11, 16]. In our study, the higher the cut-off increased, the higher the percentage of discordant cases. For example, values of 25%, 35% and 50% associated with 6%, 9% and 21% of patients having been differently classified within the two techniques.

To validate a clinical cut-off, we tested values of 8 and 12%. Forty-one percent of patients had values $>$ 12% with Thera compared to 44% with PSQ. Fifty-five percent of patients had values $⩽$ 8% with Thera compared to 50% with PSQ. In all discordant cases except one, result were between 9% and 12% with one assay and $>$ 12% or $⩽$ 8% with the other assay. A very good prognostic significance was observed for both assays and both cut-offs. The reduced risk of death in the case of results above 12% was slightly higher with PSQ compared to Thera (HR: 0.27, $p<$ 1.00E ${}^{-06}$ versus HR: 0.30, $p=$ 1.00E ${}^{-06}$ ), as we had already reported using a non-company producted PSQ test [15]. For this study, we have not been able to establish the prognosis of patients whose tumors present with percentages of methylation comprise between 9% and 12%. Since it is for this category of patients that we observed the higher number of discordant cases between the two techniques, we recommend, as for PSQ, to consider patients with a mean methylation percentage $⩽$ 8% as unmethylated, those with a mean methylation percentage $>$ 12% as methylated and those with percentages of methylation between 9% and 12% as being in a grey zone. An additional study analyzing a series of GBM FFPE samples with Thera found that the optimal cut-off value to dichotomize patients was $⩾$ 8% [17], providing confidence in our choice of thresholds.

In conclusion, Thera and PSQ may both be used to analyze MGMT status in glioblastomas. Intra-laboratory reproducibility for Thera was good and others have previously reported a high analytical performance of this kit, including inter-laboratory reproducibility [3]. The same cut-offs can be applied for the two kits, although they do not interrogate exactly identical CpGs.

Conflict of interest

The experiments were undertaken with the understanding and written consent of each subject. The study was conformed with the code of ethics of the word medical association.