The aim of this work is to provide insights into multiple metrics clinical validation of deformable image registration and contour propagation methods in 4D lung radiotherapy planning. The following indices were analyzed and compared: Volume Difference (VD), Dice Similarity Coefficient (DSC), Positive Predictive Value (PPV) and Surface Distances (SD). The analysis was performed on three patient datasets, using as reference a ground-truth volume generated by means of Simultaneous Truth And Performance Level Estimation (STAPLE) algorithm from the outlines of five experts. Significant discrepancies in the quality assessment provided by the different metrics in all the examined cases were found. Metrics sensitivity was more evident in presence of image artifacts and particularly for tubular anatomical structures, such as esophagus or spinal cord. Volume Differences did not account for position and DSC exhibited criticalities due to its intrinsic symmetry (i.e. over- and under-estimation of the reference contours cannot be discriminated) and dependency on the total volume of the structure. PPV analysis showed more robust performance, as each voxel concurs to the classification of the propagation, but was not able to detect inclusion of propagated and ground-truth volumes. Mesh distances could interpret the actual shape of the structures, but might report higher mismatches in case of large local differences in the contour surfaces. According to our study, the combination of VD and SD for the validation of contour propagation algorithms in 4D could provide the necessary failure detection accuracy.
ColganR.McClellandJ.McQuaidD.EvansP. M.HawkesD.BrockJ.LandauD.WebbS.Planning lung radiotherapy using 4D CT data and a motion model. Phys Med Biol53 (20), 5815–30 (2008). DOI: 10.1088/0031–9155/53/20/017.
4.
ChaoM.LiT.SchreibmannE.KoongA.XingL.Automated contour mapping with a regional deformable model. Int J Radiat Oncol Biol Phys70 (2), 599–608 (2008). DOI: 10.1016/j.ijrobp.2007.09.057.
5.
ChaoM.XieY.XingL.Auto-propagation of contours for adaptive prostate radiation therapy. Phys Med Biol53 (17), 4533–42 (2008). DOI: 10.1088/0031–9155/53/17/005.
6.
EzhilM.ChoiB.StarkschallG.BucciM. K.VedamS.BalterP.Comparison of rigid and adaptive methods of propagating gross tumor volume through respiratory phases of four-dimensional computed tomography image data set. Int J Radiat Oncol Biol Phys71 (1), 290–6 (2008). DOI: 10.1016/j.ijrobp.2008.01.025.
7.
LinL.ShiC.LiuY.SwansonG.PapanikolaouN.Development of a novel post-processing treatment planning platform for 4D radiotherapy. Technol Cancer Res Treat7 (2), 125–32 (2008). PubMed PMID: 18345701.
8.
LuW.OliveraG. H.ChenQ.ChenM.RuchalaK. J.Automatic re-contouring in 4D radiotherapy. Phys Med Biol51 (5), 1077–99 (2006). DOI: 10.1088/0031–9155/51/5/002.
9.
van der PutR. W.KerkhofE. M.RaaymakersB. W.Jürgenliemk-SchulzI. M.LagendijkJ. J.Contour propagation in MRI-guided radiotherapy treatment of cervical cancer: The accuracy of rigid, nonrigid and semi-automatic registrations. Phys Med Biol54 (23), 7135–50 (2009). DOI: 10.1088/0031–9155/54/23/007.
10.
CrumW. R.HartkensT.HillD. L.Non-rigid image registration: Theory and practice. The Br J Radiol77, Spec No. S140-53 (2004). DOI: 10.1259/bjr/25329214.
11.
MaintzJ. B.ViergeverM. A.A survey of medical image registration. Med Image Anal2 (1), 1–36 (1998). DOI: 10.1016/S1361–8415(01)80026-8.
12.
KleinA.AnderssonJ.ArdekaniB. A.AshburnerJ.AvantsB.ChiangM.ChristensenG. E.CollinsD. L.GeeJ.HellierP.SongJ. H.JenkinsonM.LepageC.RueckertD.ThompsonP.VercauterenT.WoodsR. P.MannJ. J.ParseyR. V.Evaluation of 14 nonlinear deformation algorithms applied to human brain MRI registration. NeuroImage46 (3), 786–802 (2009). DOI: 10.1016/j.neuroimage.2008.12.037.
13.
KashaniR.HubM.BalterJ. M.KesslerM. L.DongL.ZhangL.XingL.XieY.HawkesD.SchnabelJ. A.McClellandJ.JoshiS.ChenQ.LuW.Objective assessment of deformable image registration in radiotherapy: A multi-institution study. Med Phys35, 5944–5953 (2008). DOI: 10.1118/1.3013563.
14.
BrockK. K.Deformable registration accuracy consortium, Results of a multi-institution deformable registration accuracy study (MIDRAS). Int J Radiat Oncol Biol Phys76, 583–596 (2010). DOI: 10.1016/j.ijrobp.2009.06.031.
15.
MurphyK.van GinnekenB.ReinhardtJ. M.Evaluation of registration methods on thoracic CT: The EMPIRE10 challenge. IEEE Trans Med Imaging30, 1901–1920 (2011). DOI: 10.1109/TMI.2011.2158349.
16.
EzhilM.StarkschallG.MohanR.CoxJ.KomakiR.Validation of a model-based segmentation approach to propagating normal anatomic regions of interest through the 10 phases of respiration. Int J Radiat Oncol Biol Phys71 (3), 900–6 (2008). DOI: 10.1016/j.ijrobp.2008.02.025.
17.
CaldwellC. B.MahK.UngY. C.DanjouxC. E.BaloghJ. M.GanguliS. N.EhrlichL. E.Observer variation in contouring gross tumor volume in patients with poorly defined non-small-cell lung tumors on CT: The impact of 18FDG-hybrid PET fusion. Int J Radiat Oncol Biol Phys51 (4), 923–31 (2001). DOI: 10.1016/S0360–3016(01)01722-9.
18.
EliasziwM.YoungS. L.WoodburyM. G.Fryday-FieldK.Statistical methodology for the concurrent assessment of interrater and intrarater reliability: Using goniometric measurements as an example. Physical Therapy74 (8), 777–88 (1994). PUBMED ID: 8047565.
19.
GeetsX.DaisneJ.ArcangeliS.CocheE.De PoelM.DuprezT.NardellaG.GrégoireV.Inter-observer variability in the delineation of pharyngo-laryngeal tumor, parotid glands and cervical spinal cord: Comparison between CT-scan and MRI. Radiother Oncol77 (1), 25–31 (2005). DOI: 10.1016/j.radonc.2005.04.010.
20.
MurphyM. J.WeiZ.FatygaM.WilliamsonJ.AnscherM.WallaceT.WeissE.How does CT image noise affect 3D deformabile image registration for image-guided radiotherapy planning?Med Phys35 (3), 1145–53 (2008). DOI: 10.1118/1.2837292.
21.
RemeijerP.RaschC.LebesqueJ. V.van HerkM.A general methodology for three-dimensional analysis of variation in target volume delineation. Med Phys26 (6), 931–40 (1999). DOI: 10.1118/1.598485.
22.
BeaucheminM.ThomsonK.The evaluation of segmentation results and the overlapping area matrix. Int J Remote Sensing18 (18), 3895–9 (1997). DOI: 10.1080/014311697216720.
23.
WarfieldS. K.DenglerJ.ZaersJ.GuttmannC. R.WellsW. M.3rdEttingerG. J.HillerJ.KikinisR., Automatic identification of grey matter structures from MRI to improve the segmentation of white matter lesions. J Image Guid Surg1 (6), 326–338 (1995). DOI: 10.1002/(SICI)1522–712X(1995)1:6<326::AID-IGS4>3.0.CO;2-C.
24.
WarfieldS. K.ZouK. H.WellsW. M., Simultaneous truth and performance level estimation (STAPLE): An algorithm for the validation of image segmentation. IEEE Trans Med Imaging23 (7), 903–92 (2004). DOI: 10.1109/TMI.2004.828354.
25.
KabusS.KlinderT.MurphyK.Van GinnekenB.LorenzC.PluimJ.Evaluation of 4D-CT lung registration, Lecture Notes in Computer Science, 5761 LNCS (PART 1), 747–754 (2009). DOI: 10.1007/978–3–642–04268-3_92.
26.
MurphyK.van GinnekenB.KleinS.StaringM.de HoopB. J.ViergeverM. A.PluimJ. P.Semi-automatic construction of reference standards for evaluation of image registration. Med Image Anal15 (1), 71–84 (2011). DOI: 10.1016/j.media.2010.07.005.
27.
PaganelliC.PeroniM.RiboldiM.SharpG. C.CiardoD.AlterioD.OrecchiaR.BaroniG.Scale invariant feature transform in adaptive radiation therapy: A tool for deformable image registration assessment and re-planning indication. Phys Med Biol58 (2), 287–99 (2013). DOI: 10.1088/0031–9155/58/2/287.
28.
AspertN.Santa-CruzD.EbrahimiT.MESH: Measuring error between surfaces using the Hausdorff distance. Proceedings of the IEEE International Conference on Multimedia and Expo 2002 (ICME), vol. I, 705–708 (2002). DOI: 10.1007/978–3–642–04268-3_92.
BertC.MetheanyK. G.DoppkeK. P.TaghianA. G.PowellS. N.ChenG. T.Clinical experience with a 3D surface patient setup system for alignment of partial-breast irradiation patients. Int J Radiat Oncol Biol Phys64 (4), 1265–74 (2006). DOI: 10.1016/j.ijrobp.2005.11.008.
31.
AspertN.Santa-CruzD.EbrahimiT.MESH: Measuring error between surfaces using the Hausdorff distance, in Proceedings of the IEEE International Conference on Multimedia and Expo (ICME), vol. I, 705–708 (2002). DOI: 10.1109/ICME.2002.1035879.
32.
Rodriguez-VilaB.GayaF.Garcia-VicenteF.GomezE. J.Three-dimensional quantitative evaluation method of nonrigid registration algorithms for adaptive radiotherapy. Med Phys37 (3), 1137–1145 (2010). DOI: 10.1118/1.3302916.
33.
RohlfingT.Image similarity and tissue overlaps as surrogates for image registration accuracy: Widely used but unreliable. IEEE Trans Med Imaging31 (2), 153–63 (2012). DOI: 10.1109/TMI.2011.2163944.
34.
DiceL. R.Measures of the amount of ecologic association between species. Ecology26 (3), 297–302 (1945). DOI: 10.2307/1932409.
35.
IsambertA.DhermainF.BidaultF.CommowickO.BondiauP.MalandainG.LefkopoulosD.Evaluation of an atlas-based automatic segmentation software for the delineation of brain organs at risk in a radiation therapy clinical context. Radiother Oncol87 (1), 93–9 (2008). DOI: 10.1016/j.radonc.2007.11.030.
36.
SimsR.IsambertA.GrégoireV.BidaultF.FrescoL.SageJ.MillsJ.BourhisJ.LefkopoulosD.CommowickO.BenkebilM.MalandainG.A pre-clinical assessment of an atlas-based automatic segmentation tool for the head and neck. Radiother Oncol93 (3), 474–8 (2009). DOI: 10.1016/j.radonc.2009.08.013.
GerigG.JomierM.ChakosM.VALMET: A new validation tool for assessing and improving 3D object segmentation. In MIC-CAI 2001 Lecture Notes in Computer Science516–523 (2001). DOI: 10.1007/3–540–45468-3_62.
39.
RogeljP.KovacicS.GeeJ.Validation of a non-rigid registration algorithm for multi-modal data. In Proc SPIE, vol. 4684, 299–307 (2002). DOI:10.1117/12.467170.
40.
ShacklefordJ. A.KandasamyN.SharpG. C.On developing B-spline registration algorithms for multi-core processors. Phys Med Biol55 (21), 6329–6351 (2010). DOI: 10.1088/0031–9155/55/21/001.
41.
LouieA. V.RodriguesG.OlsthoornJ.PalmaD.YuE.YaremkoB.AhmadB.AivasI.GaedeS.Inter-observer and intra-observer reliability for lung cancer target volume delineation in the 4D-CT era.,Radiother Oncol95 (2), 166–71 (2010). DOI: 10.1016/j.radonc.2009.12.028.
42.
ZijdenbosA. P.DawantB. M.MargolinR. A.PalmerA. C.Morphometric analysis of white matter lesions in MR images: Method and validation. IEEE Trans Med Imaging13, 716–24 (1994). DOI: 10.1109/42.363096.
43.
ShusharinaN.SharpG.Analytic regularization for landmark-based image registration. Phys Med Biol57, 1477–1498 (2012). DOI:10.1088/0031–9155/57/6/1477.
44.
CiardoD.PeroniM.RiboldiM.OrecchiaR.BaroniG.The role of regularization in deformable image registration for head and neck adaptive radiotherapy, Technol Cancer Res Treat, in Press (2013).
