Video magnification techniques can reveal subtle temporal variations that are difficult or even impossible to see with the naked eye and display them in an indicative manner. State-of-the-art approaches rely on hand-designed filters or precise prior knowledge of the target signal and produce magnified outputs that are always bounded by the spatial support of the related pyramids. This paper proposes a method for adaptively magnifying subtle video changes by directly solving three key optimization problems. To efficiently model the magnification transformation, alternating direction method of multipliers is employed to solve convex variation-detection optimization problems. Following the transformation step, the perturbation problem is innovatively solved using a forward additive iterative approach to iteratively minimize the dissimilarity between the original and magnified sequences based on the enhanced correlation coefficient. The proposed method can be applied to videos overlaid with different types of temporal change to obtain motion and color magnification. Quantitative and qualitative experimental comparison of the proposed method with state-of-the-art techniques reveals that it produces magnified videos with improved visual quality and substantially fewer artifacts or blurring.
DavisABoumanKLChenJGRubinsteinMBüyüköztürkODurandF, et al. Visual Vibrometry: Estimating Material Properties from Small Motions in Video. IEEE Transactions on Pattern Analysis & Machine Intelligence.2017; 39(4): 732-45.
2.
WuHYRubinsteinMShihEGuttagJDurandFFreemanW. Eulerian video magnification for revealing subtle changes in the world. Acm Transactions on Graphics. 2012; 31(4): 1-8.
3.
ChaY-JChenJBüyüköztürkO. Output-only computer vision based damage detection using phase-based optical flow and unscented Kalman filters. Engineering Structures.2017; 132: 300-13.
4.
BalakrishnanGDurandFGuttagJ, editors. Detecting Pulse from Head Motions in Video. Computer Vision and Pattern Recognition (CVPR), 2013; IEEE Conference on; 2013.
5.
LiuCTorralbaAFreemanWTDurandFAdelsonEH. Motion magnification. ACM transactions on graphics (TOG).2005; 24(3): 519-26.
6.
TulyakovSAlameda-PinedaXRicciEYinLCohnJFSebeN, editors. Self-adaptive matrix completion for heart rate estimation from face videos under realistic conditions. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016.
7.
BaiJAgarwalaAAgrawalaMRamamoorthiR. Selectively de-animating video. ACM Trans Graph.2012; 31(4): 1-10.
8.
WadhwaNRubinsteinMDurandFFreemanWT. Phase-based video motion processing. ACM Transactions on Graphics (TOG).2013; 32(4): 80.
9.
WadhwaNRubinsteinMDurandFFreemanWT, editors. Riesz pyramids for fast phase-based video magnification. 2014 IEEE International Conference on Computational Photography (ICCP), 2014: IEEE.
10.
ElgharibMHefeedaMDurandFFreemanWT, editors. Video magnification in presence of large motions. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2015.
11.
KooijJFvan GemertJC, editors. Depth-aware motion magnification. European Conference on Computer Vision, 2016; Springer.
12.
ZhangYPinteaSLVan GemertJC, editors. Video acceleration magnification. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017.
13.
TakedaSOkamiKMikamiDIsogaiMKimataH, editors. Jerk-Aware Video Acceleration Magnification. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018.
14.
OhT-HJaroensriRKimCElgharibMDurandFFreemanWT, et al., Learning-based video motion magnification. Proceedings of the European Conference on Computer Vision (ECCV), 2018.
15.
ChenWMcDuffD. DeepMag: Source Specific Motion Magnification Using Gradient Ascent. arXiv preprint arXiv180803338, 2018.
16.
AdeliHGhosh-DastidarSDadmehrN. A spatio-temporal wavelet-chaos methodology for EEG-based diagnosis of Alzheimer’s disease. Neuroscience Letters.2008; 444(2): 190-4.
17.
AhmadlouMAdeliHAdeliA. Spatiotemporal analysis of relative convergence of EEGs reveals differences between brain dynamics of depressive women and men. Clinical EEG and Neuroscience.2013; 44(3): 175-81.
18.
KilmerMEMartinCD. Factorization strategies for third-order tensors. Linear Algebra and its Applications.2011; 435(3): 641-58.
19.
BakerS. lain Matthews. Lucas-Kanade 20 Years On: A Unifying Framework: Part 1. CMU-RI-TR-02-16.
20.
SanchezAMorenoABVelezDVélezJF. Analyzing the influence of contrast in large-scale recognition of natural images. Integrated Computer-Aided Engineering.2016; 23(3): 221-35.
21.
KoziarskiMCyganekB. Image recognition with deep neural networks in presence of noise–dealing with and taking advantage of distortions. Integrated Computer-Aided Engineering.2017; 24(4): 337-49.
22.
WangPBaiX. Regional parallel structure based CNN for thermal infrared face identification. Integrated Computer-Aided Engineering.2018; 25(3): 247-60.
23.
HsuW-Y. Automatic pedestrian detection in partially occluded single image. Integrated Computer-Aided Engineering.2018; 25(4): 369-79.
24.
Vera-OlmosFPardoEMeleroHMalpicaN. DeepEye: Deep convolutional network for pupil detection in real environments. Integrated Computer-Aided Engineering.2019; 26(1): 85-95.
25.
KimDRhoSJunSHwangE. Classification and indexing scheme of large-scale image repository for spatio-temporal landmark recognition. Integrated Computer-Aided Engineering.2015; 22(2): 201-13.
26.
LacabexBCuesta-InfanteAMontemayorASPantrigoJJ. Lightweight tracking-by-detection system for multiple pedestrian targets. Integrated Computer-Aided Engineering.2016; 23(3): 299-311.
DelibasisKKGeorgakopoulosSVKottariKPlagianakosVPMaglogiannisI. Geodesically-corrected Zernike descriptors for pose recognition in omni-directional images. Integrated Computer-Aided Engineering.2016; 23(2): 185-99.
29.
Benito-PicazoJDomínguezEPalomoEJLópez-RubioEOrtiz-de-Lazcano-LobatoJM. Motion detection with low cost hardware for PTZ cameras. Integrated Computer-Aided Engineering.2019; 26(1): 21-36.
30.
EsteppJRBlackfordEBMeierCM, editors. Recovering pulse rate during motion artifact with a multi-imager array for non-contact imaging photoplethysmography. 2014 IEEE International Conference on Systems, Man, and Cybernetics (SMC); 2014: IEEE.
ZhangCLiuJLiangCXueZPangJHuangQ. Image classification by non-negative sparse coding, correlation constrained low-rank and sparse decomposition. Computer Vision and Image Understanding.2014; 123: 14-22.
33.
CuiXHuangJZhangSMetaxasD, editors. Background subtraction using group sparsity and low rank constraint. Proc. 12th European Conf Computer Vision (ECCV), 2012.
34.
ShenXWuY, editors. A unified approach to salient object detection via low rank matrix recovery. 2012 IEEE Conference on Computer Vision and Pattern Recognition, 2012: IEEE.
35.
ZhangTGhanemBLiuSAhujaN, editors. Low-rank sparse learning for robust visual tracking. European conference on computer vision, 2012; Springer.
36.
ZhangZMatsushitaYMaY, editors. Camera calibration with lens distortion from low-rank textures. CVPR. 2011; 2011: IEEE.
37.
ChengBLiuGWangJHuangZYanS, editors. Multi-task low-rank affinity pursuit for image segmentation. 2011International Conference on Computer Vision, 2011; IEEE.
38.
CarrollJDChangJ-J. Analysis of individual differences in multidimensional scaling via an N-way generalization of “Eckart-Young” decomposition. Psychometrika.1970; 35(3): 283-319.
39.
HarshmanRA. Foundations of the PARAFAC procedure: Models and conditions for an “explanatory” multimodal factor analysis, 1970.
40.
JiHLiuCShenZXuY, editors. Robust video denoising using low rank matrix completion. 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2010: IEEE.
41.
PengYGaneshAWrightJXuWMaY. RASL: Robust alignment by sparse and low-rank decomposition for linearly correlated images. IEEE Transactions on Pattern Analysis and Machine Intelligence.2012; 34(11): 2233-46.
42.
BoydSVandenbergheL. Convex optimization: Cambridge University Press, 2004.
43.
ChanSHKhoshabehRGibsonKBGillPENguyenTQ. An augmented Lagrangian method for total variation video restoration. IEEE Transactions on Image Processing.2011; 20(11): 3097-111.
44.
GeemZWKimJHLoganathanGV. A new heuristic optimization algorithm: harmony search. Simulation.2001; 76(2): 60-8.
45.
ParejoJARuiz-CortésALozanoSFernandezP. Metaheuristic optimization frameworks: a survey and benchmarking. Soft Computing.2012; 16(3): 527-61.
46.
GloverF. Heuristics for integer programming using surrogate constraints. Decision Sciences.1977; 8(1): 156-66.
47.
IshibuchiHYoshidaTMurataT. Balance between genetic search and local search in memetic algorithms for multiobjective permutation flowshop scheduling. IEEE Transactions on Evolutionary Computation.2003; 7(2): 204-23.
48.
WolpertDHMacreadyWG. No free lunch theorems for optimization. IEEE Transactions on Evolutionary Computation.1997; 1(1): 67-82.
49.
ChengJZhangGCaraffiniFNeriF. Multicriteria adaptive differential evolution for global numerical optimization. Integrated Computer-Aided Engineering.2015; 22(2): 103-7.
50.
IaccaGCaraffiniFNeriF. Multi-strategy coevolving aging particle optimization. International Journal of Neural Systems.2014; 24(1): 1450008.
51.
BoydSParikhNChuEPeleatoBEcksteinJ. Distributed optimization and statistical learning via the alternating direction method of multipliers. Foundations and Trends® in Machine learning.2011; 3(1): 1-122.
52.
YangWHHanD. Linear convergence of the alternating direction method of multipliers for a class of convex optimization problems. SIAM Journal on Numerical Analysis.2016; 54(2): 625-40.
53.
HongMLuoZ-Q. On the linear convergence of the alternating direction method of multipliers. Mathematical Programming.2017; 162(1-2): 165-99.
54.
MonteiroRDSvaiterBF. Iteration-complexity of block-decomposition algorithms and the alternating direction method of multipliers. SIAM Journal on Optimization.2013; 23(1): 475-507.
55.
MooreBEGaoCNadakuditiRR. Panoramic Robust PCA for Foreground – Background Separation on Noisy, Free-Motion Camera Video. IEEE Transactions on Computational Imaging.2019; 5(2): 195-211.
56.
ParagiosNChenYFaugerasOD. Handbook of mathematical models in computer vision: Springer Science & Business Media, 2006.
57.
HornBKlausBHornP. Robot vision: MIT press, 1986.
58.
AnandanP. A computational framework and an algorithm for the measurement of visual motion. International Journal of Computer Vision.1989; 2(3): 283-310.
59.
ShumH-YSzeliskiR. Construction of panoramic image mosaics with global and local alignment. Panoramic vision: Springer, 2001; pp. 227-68.
60.
PsarakisEZEvangelidisGD, editors. An enhanced correlation-based method for stereo correspondence with subpixel accuracy. Tenth IEEE International Conference on Computer Vision (ICCV’05); Volume 1, 2005; IEEE.
61.
HagerGDBelhumeurPN. Efficient region tracking with parametric models of geometry and illumination. IEEE Transactions on Pattern Analysis & Machine Intelligence.1998; (10): 1025-39.
SemerciOHaoNKilmerMMillerE, editors. An iterative reconstruction method for spectral CT with tensor-based formulation and nuclear norm regularization. The Second International Conference on Image Formation in X-Ray Computed Tomography, 2012.
64.
ZhangZElyGAeronSHaoNKilmerM, editors. Novel methods for multilinear data completion and de-noising based on tensor-SVD. Proceedings of the IEEE conference on computer vision and pattern recognition, 2014.
65.
CaiJ-FCandèsEJShenZ. A singular value thresholding algorithm for matrix completion. SIAM Journal on Optimization.2010; 20(4): 1956-82.
66.
LewisAS. The convex analysis of unitarily invariant matrix functions. Journal of Convex Analysis.1995; 2(1): 173-83.
67.
KoldaTGBaderBW. Tensor decompositions and applications. SIAM Review.2009; 51(3): 455-500.
68.
OseledetsIV. Tensor-train decomposition. SIAM Journal on Scientific Computing.2011; 33(5): 2295-317.
69.
BramanK. Third-order tensors as linear operators on a space of matrices. Linear Algebra and its Applications.2010; 433(7): 1241-53.
70.
KilmerMEBramanKHaoNHooverRC. Third-order tensors as operators on matrices: A theoretical and computational framework with applications in imaging. SIAM Journal on Matrix Analysis and Applications.2013; 34(1): 148-72.
71.
MudenagudiUBanerjeeSKalraPK. Space-time super-resolution using graph-cut optimization. IEEE Transactions on Pattern Analysis and Machine Intelligence.2010; 33(5): 995-1008.
72.
KimB. Numerical optimization methods for image restoration, Stanford University, 2002.
73.
NgMK. Iterative methods for Toeplitz systems: Numerical Mathematics and Scie, 2004.
74.
ChanTFGolubGHMuletP. A nonlinear primal-dual method for total variation-based image restoration. SIAM Journal on Scientific Computing.1999; 20(6): 1964-77.
75.
ChambolleA. An algorithm for total variation minimization and applications. Journal of Mathematical Imaging and Vision.2004; 20(1-2): 89-97.
76.
ChanSHKhoshabehRGibsonKBGillPENguyenTQ, editors. An augmented Lagrangian method for video restoration. 2011 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2011: IEEE.
77.
ChengboL. An efficient algorithm for total variation regularization with applications to the single pixel camera and compressive sensing. Houston: Department of Computational and Applied Mathematics, 2009.
78.
RockafellarRT. Augmented Lagrangians and applications of the proximal point algorithm in convex programming. Mathematics of Operations Research.1976; 1(2): 97-116.
79.
BertsekasDP. Multiplier methods: a survey. Automatica.1976; 12(2): 133-45.
80.
BremerR. Outliers in statistical data. Taylor & Francis; 1995.
81.
BakerSGrossRMatthewsI. Lucas-Kanade 20 years on: A unifying framework: Part 3. Citeseer, 2003.
82.
EvangelidisGDPsarakisEZ. Parametric image alignment using enhanced correlation coefficient maximization. IEEE Transactions on Pattern Analysis and Machine Intelligence.2008; 30(10): 1858-65.
