Accurate tracking of tissue motion is critically important for several ultrasound elastography methods. In this study, we investigate the feasibility of using three published convolution neural network (CNN) models built for optical flow (hereafter referred to as CNN-based tracking) by the computer vision community for breast ultrasound strain elastography. Elastographic datasets produced by finite element and ultrasound simulations were used to retrain three published CNN models: FlowNet-CSS, PWC-Net, and LiteFlowNet. After retraining, the three improved CNN models were evaluated using computer-simulated and tissue-mimicking phantoms, and in vivo breast ultrasound data. CNN-based tracking results were compared with two published two-dimensional (2D) speckle tracking methods: coupled tracking and GLobal Ultrasound Elastography (GLUE) methods. Our preliminary data showed that, based on the Wilcoxon rank-sum tests, the improvements due to retraining were statistically significant (p < 0.05) for all three CNN models. We also found that the PWC-Net model was the best neural network model for data investigated, and its overall performance was on par with the coupled tracking method. CNR values estimated from in vivo axial and lateral strain elastograms showed that the GLUE algorithm outperformed both the retrained PWC-Net model and the coupled tracking method, though the GLUE algorithm exhibited some biases. The PWC-Net model was also able to achieve approximately 45 frames/second for 2D speckle tracking data investigated.
ShiinaTNightingaleKRPalmeriMLHallTJBamberJCBarrRG, et al. WFUMB guidelines and recommendations for clinical use of ultrasound elastography: part 1: basic principles and terminology. Ultrasound Med Biol. 2015;41(5):1126-47.
2.
ItohAUenoETohnoEKammaHTakahashiHShiinaT, et al. Breast disease: clinical application of us elastography for diagnosis. Radiology. 2006;239(2):341-50.
3.
BurnsideESHallTJSommerAMHesleyGKSisneyGASvenssonWE, et al. Differentiating benign from malignant solid breast masses with us strain imaging. Radiology. 2007;245(2):401-10.
4.
HallTJZhuYSpaldingCS.In vivo real-time freehand palpation imaging. Ultrasound Med Biol. 2003;29(3):427-435.
5.
HeinIAO’BrienWD.Current time-domain methods for assessing tissue motion by analysis from reflected ultrasound echoes—a review. IEEE Trans Ultrason Ferroelectr Freq Control. 1993;40(2):84-102.
6.
JiangJPengB. Ultrasonic Methods for Assessment of Tissue Motion in Elastography. Hoboken, NJ: John Wiley & Sons.
7.
LubinskiMAEmelianovSYO’DonnellM.Speckle tracking methods for ultrasonic elasticity imaging using short-time correlation. IEEE Trans Ultrason Ferroelectr Freq Control. 1999;46(1):82-96.
8.
PesaventoAPerreyCKruegerMErmertH.A time-efficient and accurate strain estimation concept for ultrasonic elastography using iterative phase zero estimation. IEEE Trans Ultrason Ferroelectr Freq Control. 1999;46(5):1057-67.
9.
ViolaFWalkerWF.A spline-based algorithm for continuous time-delay estimation using sampled data. IEEE Trans Ultrason Ferroelectr Freq Control. 2005;52(1):80-93.
10.
WalkerWFTraheyGE.A fundamental limit on delay estimation using partially correlated speckle signals. IEEE Trans Ultrason Ferroelectr Freq Control. 1995;42(2):301-308.
Pellot-BarakatCFrouinFInsanaMFHermentA.Ultrasound elastography based on multiscale estimations of regularized displacement fields. IEEE Trans Med Imag. 2004;23(2):153-63.
13.
RivazHBoctorEForoughiPZellarsRFichtingerGHagerG.Ultrasound elastography: a dynamic programming approach. IEEE Trans Med Imag. 2008;27(10):1373-77.
14.
JiangJHallTJ.A generalized speckle tracking algorithm for ultrasonic strain imaging using dynamic programming. Ultrasound Med Biol. 2009;35(11):1863-79.
15.
LorenzAErmertHGarcia-SchuermannMSommerfeldH.-JSengeTPhilippouS.On the use of a modified optical flow algorithm for the correction of axial strain estimates in ultrasonic elastography in medical diagnosis. J Acoust Soc Am. 1999;105(2):1208-09.
16.
ZakariaTQinZMauriceRL.Optical-flow-based B-mode elastography: application in the hypertensive rat carotid. IEEE Trans Med Imaging. 2010;29(2):570-78.
17.
PanXGaoJTaoSLiuKBaiJLuoJ.A two-step optical flow method for strain estimation in elastography: simulation and phantom study. Ultrasonics. 2014;54(4):990-96.
18.
HashemiHSRivazH.Global time-delay estimation in ultrasound elastography. IEEE Trans Ultrason Ferroelectrics Freq Contr. 2017;64(10):1625-36.
19.
DosovitskiyAFischeryPIlgEHäusserPHazırbaşCGolkovV, et al. FlowNet: learning optical flow with convolutional networks. In: Proceedings of the 2015 IEEE International Conference on Computer Vision (ICCV). ICCV ’15, pp. 2758-66. Washington, DC: IEEE Computer Society. Available from https://arxiv.org/abs/1504.06852.
20.
RanjanABlackMJ.Optical flow estimation using a spatial pyramid network. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2720-29. Available from https://arxiv.org/abs/1611.00850.
21.
IlgEMayerNSaikiaTKeuperMDosovitskiyABroxT. FlowNet 2.0: evolution of optical flow estimation with deep networks. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Available from https://github.com/lmb-freiburg/flownet2.
22.
HuiTWTangXLoyCC. LiteFlowNet: a lightweight convolutional neural network for optical flow estimation. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8981-89. Available from https://github.com/twhui/LiteFlowNet.
23.
SunDYangXLiuMYKautzJ. PWC-Net: CNNs for optical flow using pyramid, warping, and cost volume. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8934-43. Available from https://arxiv.org/abs/1709.02371.
24.
KibriaMGRivazH. GLUENet: Ultrasound elastography using convolutional neural network. In: StoyanovDTaylorZAylwardSTavaresJMRSXiaoYSimpsonA, et al., eds. Simulation, Image Processing, and Ultrasound Systems for Assisted Diagnosis and Navigation. Cham: Springer International Publishing, pp. 21-28.
25.
WuSGaoZLiuZLuoJZhangHLiS. Direct reconstruction of ultrasound elastography using an end-to-end deep neural network. In: FrangiAFSchnabelJADavatzikosCAlberola-LópezCFichtingerG, eds. Medical Image Computing and Computer Assisted Intervention—MICCAI 2018. Cham: Springer International Publishing, 2018, pp. 374-82.
26.
WangYHelminenEJiangJ.Building a virtual simulation platform for quasistatic breast ultrasound elastography using open source software: a preliminary investigation. Med Phys. 2015;42(9):5453-66.
27.
JensenJASvendsenNB.Calculation of pressure fields from arbitrarily shaped, apodized, and excited ultrasound transducers. IEEE Trans Ultrason Ferroelectrics Freq Contr. 1992;39(2):262-267.
28.
JiangJHallTJ.A coupled subsample displacement estimation method for ultrasound-based strain elastography. Phys Med Biol. 2015;60(21):8347-64.
29.
PengBWangYHallTJJiangJ.A GPU-accelerated 3-D coupled subsample estimation algorithm for volumetric breast strain elastography. IEEE Trans Ultrason Ferroelectrics Freq Contr. 2017;64(4):694-705.
30.
LuoJBaiJHePYingK.Axial strain calculation using a low-pass digital differentiator in ultrasound elastography. IEEE Trans Ultrason Ferroelectrics Freq Contr. 2004;51(9):1119-27.
31.
JiangJHallTJSommerAM.A novel performance descriptor for ultrasonic strain imaging: a preliminary study. IEEE Trans Ultrason Ferroelectr Freq Control. 2006;53(6):1088-1102.
32.
SongXPogueBWJiangSDoyleyMMDehghaniHTostesonTD, et al. Automated region detection based on the contrast-to-noise ratio in near-infrared tomography. Appl Opt. 2004;43(5):1053-62.
33.
WangYNasiefHGKohnSMilkowskiAClaryTBarnesS.Three-dimensional ultrasound elasticity imaging on an automated breast volume scanning system. Ultrason Imaging. 2017;39(6):369-92.
34.
WangYJiangJHallT.A 3-D region-growing motion-tracking method for ultrasound elasticity imaging. Ultrasound Med Biol. 2018;44(8):1638-53.
35.
VargheseTOphirJ.A theoretical framework for performance characterization of elastography: the strain filter. IEEE Trans Ultrason Ferroelectr Freq Control. 1997;44(1):164-72.
36.
BilgenM.Dynamics of errors in 3D motion estimation and implications for strain-tensor imaging in acoustic elastography. Phys Med Biol. 2000;45(6):1565-78.
ShelhamerELongJDarrellT.Fully convolutional networks for semantic segmentation. IEEE Trans Pattern Anal Mach Intell. 2017;39(4):640-51.
39.
CuiSMaoLJiangJLiuCXiong. Automatic semantic segmentation of brain gliomas from MRI images using a deep cascaded neural network. J Healthc Eng. 2018. Available from: https://www.hindawi.com/journals/jhe/2018/4940593/.
WangYPengBJiangJ.Building an open-source simulation platform of acoustic radiation force-based breast elastography. Phys Med Biol. 2017;62(5):1949-68.
42.
BarbonePEBamberJC.Quantitative elasticity imaging: what can and cannot be inferred from strain images. Phys Med Biol. 2002;47(12):2147-64.
43.
HallTJOberaitAABarbonePESommerAMGokhaleNHGoenezentS, et al. Elastic nonlinearity imaging. Conf Proc IEEE Eng Med Biol Soc. 2009:1967-70.
44.
GoenezenSDordJSinkZBarbonePEJiangJHallTJ, et al. Linear and nonlinear elastic modulus imaging: an application to breast cancer diagnosis. IEEE Trans Med Imaging. 2012;31(8):1628-37.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
