A realistic evaluation of indoor positioning systems based on Wi-Fi fingerprinting: The 2015 EvAAL

Abstract

This paper presents results from comparing different Wi-Fi fingerprinting algorithms on the same private dataset. The algorithms where realized by independent teams in the frame of the off-site track of the EvAAL–ETRI Indoor Localization Competition which was part of the Sixth International Conference on Indoor Positioning and Indoor Navigation (IPIN 2015). Competitors designed and validated their algorithms against the publicly available UJIIndoorLoc database which contains a huge reference- and validation data set. All competing systems were evaluated using the mean error in positioning, with penalties, using a private test dataset. The authors believe that this is the first work in which Wi-Fi fingerprinting algorithm results delivered by several independent and competing teams are fairly compared under the same evaluation conditions. The analysis also comprises a combined approach: Results indicate that the competing systems where complementary, since an ensemble that combines three competing methods reported the overall best results.

Keywords

Wi-Fi fingerprinting off-site competition indoor positioning

Get full access to this article

View all access options for this article.

References

Adler,

Schmitt,

Wolter and

Kyas, A survey of experimental evaluation in indoor localization research, in: Indoor Positioning and Indoor Navigation (IPIN), 2015 International Conference on, Banff, AB, Canada, 2015, p. 10.

J.A.

Álvarez-García,

Barsocchi,

Chessa and

Salvi, Evaluation of localization and activity recognition systems for ambient assisted living: The experience of the 2012 evaal competition, Journal of Ambient Intelligence and Smart Environments 5(1) (2013), 119–132.

Bahl and

V.N.

Padmanabhan, Radar: An in-building rf-based user location and tracking system, in: Proceedings of the Nineteenth Annual Joint Conference of the IEEE Computer and Communications Societies, Vol. 2, 2000, pp. 775–784.

Barsocchi,

Chessa,

Furfari and

Potortì, Evaluating AAL solutions through competitive benchmarking: The localization competition, IEEE Pervasive Computing Magazine 12(4) (2013), 72–79. doi:10.1109/MPRV.2013.23.

Barsocchi,

Potortì,

Furfari and

A.M.M.

Gil, Comparing AAL indoor localization systems, in: Evaluating AAL Systems Through Competitive Benchmarking. Indoor Localization and Tracking, Communications in Computer and Information Science, Vol. 309, Springer, Berlin Heidelberg, 2012, pp. 1–13. doi:10.1007/978-3-642-33533-4_1.

Berkvens,

Peremans and

Weyn, Localization performance quantification by information theory, Technical report, University of Antwerp, 2015.

Berkvens,

Weyn and

Peremans, Localization performance quantification by conditional entropy, in: Proceedings of the Sixth International Conference on Indoor Positioning and Indoor Navigation (IPIN), 2015.

J.A.

Botia,

J.A.

Alvarez-Garcia,

Fujinami,

Barsocchi and

Riedel (eds), Evaluating AAL Systems Through Competitive Benchmarking: International Competitions and Final Workshop, EvAAL 2013, Communications in Computer and Information Science, Vol. 366, Springer, 2013.

Calderoni,

Ferrara,

Franco and

Maio, Indoor localization in a hospital environment using random forest classifiers, Expert Systems with Applications 42(1) (2015), 125–134. doi:10.1016/j.eswa.2014.07.042.

10.

R.S.

Campos,

Lovisolo and

M.L.R.

de Campos, Wi-fi multi-floor indoor positioning considering architectural aspects and controlled computational complexity, Expert Systems with Applications 41(14) (2014), 6211–6223. doi:10.1016/j.eswa.2014.04.011.

11.

Chen,

Lymberopoulos,

Liu and

Priyantha, Indoor localization using fm signals, IEEE Transactions on Mobile Computing 12(8) (2013), 1502–1517. doi:10.1109/TMC.2013.58.

12.

Chessa,

Furfari,

Potortì,

Barsocchi,

M.-R.

Tazari and

Wichert, Evaluating AAL systems through competitive benchmarking (EVAAL) – technical aspects of the first competition, in: Partnerships for Social Innovation in Europe. Proceedings of the AAL Forum, Lecce, Italy, September 26–28,

Bierhoff,

H.H.

Nap,

Rijnen and

Wichert, eds, 2012, pp. 617–623. ISBN 978-90-819709-0-7.

13.

Choi,

Yoo and

H.J.

Kim, Machine learning for indoor localization: Deep learning and semi-supervised learning, in: USB On-Site Proceedings of the Sixth International Conference on Indoor Positioning and Indoor Navigation (IPIN), 2015.

14.

Chung,

Donahoe,

Schmandt,

I.J.

Kim,

Razavai and

Wiseman, Indoor location sensing using geo-magnetism, in: Proceedings of the 9th International Conference on Mobile Systems, Applications, and Services, MobiSys ’11, ACM, New York, NY, USA, 2011, pp. 141–154.

15.

Cover and

Hart, Nearest neighbor pattern classification, IEEE Trans. Inf. Theor. 13(1) (1967), 21–27. doi:10.1109/TIT.1967.1053964.

16.

Dammann,

Raulefs and

Zhang, On prospects of positioning in 5g, in: Communication Workshop (ICCW), 2015 IEEE International Conference on, 2015, pp. 1207–1213. doi:10.1109/ICCW.2015.7247342.

17.

A.R.

de M. Neves,

Á.M.G.

Carvalho and

C.G.

Ralha, Agent-based architecture for context-aware and personalized event recommendation, Expert Systems with Applications 41(2) (2014), 563–573. doi:10.1016/j.eswa.2013.07.081.

18.

del Corte-Valiente,

J.M.

Gómez-Pulido and

Gutiérrez-Blanco, Efficient techniques and algorithms for improving indoor localization precision on wlan networks applications, International Journal Communications, Networks and System Sciences 2(7) (2009), 645–651. doi:10.4236/ijcns.2009.27073.

19.

T.G.

Dietterich, Ensemble methods in machine learning, in: Multiple Classifier Systems. First International Workshop, MCS 2000,

Kittler, ed., Lecture Notes in Computer Science, Vol. 1857, Springer, Berlin/Heidelberg, 2000, pp. 1–15.

20.

A.G.

Estevez and

Carlsson, Geo-location-aware emulations for performance evaluation of mobile applications, in: Proceedings of the IEEE/IFIP Conference on Wireless On-Demand Network Systems and Services, 2014.

21.

Feldmann,

Kyamakya,

Zapater and

Lue, An indoor bluetooth-based positioning system: Concept, implementation and experimental evaluation, in: Proceedings of the International Conference on Wireless Networks, CSREA Press, 2003, pp. 109–113.

22.

H.T.

Friis, A note on a simple transmission formula, Proceedings of the IRE 34(5) (1946), 254–256. doi:10.1109/JRPROC.1946.234568.

23.

Gansemer,

Großmann and

Hakobyan, Rssi-based Euclidean distance algorithm for indoor positioning adapted for the use in dynamically changing wlan environments and multi-level buildings, in: Proceedings of the 1st the International Conference on Indoor Positioning and Indoor Navigation (IPIN’2010), 2010.

24.

García,

Poudereux,

Á.

Hernández,

J.J.

García and

Ureña, DS-UWB indoor positioning system implementation based on FPGAs, Sensors and Actuators A: Physical 201 (2013), 172–181. doi:10.1016/j.sna.2013.06.019.

25.

Garcia-Villalonga and

Perez-Navarro, Influence of human absorption of wi-fi signal in indoor positioning with wi-fi fingerprinting, in: Proceedings of the Sixth International Conference on Indoor Positioning and Indoor Navigation (IPIN), 2015.

26.

Gezici,

Tian,

G.B.

Giannakis,

Kobayashi,

A.F.

Molisch,

H.V.

Poor and

Sahinoglu, Localization via ultra-wideband radios: A look at positioning aspects for future sensor networks, Signal Processing Magazine, IEEE 22(4) (2005), 70–84. doi:10.1109/MSP.2005.1458289.

27.

Gualda,

Ureña,

J.C.

García and

Lindo, Locally-referenced ultrasonic – lps for localization and navigation, Sensors 14(11) (2014), 21750. doi:10.3390/s141121750.

28.

Guo,

Chen and

Liu, Indoor location estimation based on local magnetic field via hybrid learning, in: Extreme Learning Machines 2013: Algorithms and Applications,

Sun,

K.-A.

Toh,

M.G.

Romay and

Mao, eds, Adaptation, Learning, and Optimization, Vol. 16, Springer International Publishing, 2014, pp. 189–207. doi:10.1007/978-3-319-04741-6_14.

29.

Hashemi, The indoor radio propagation channel, Proceedings of the IEEE 81(7) (1993), 943–968. doi:10.1109/5.231342.

30.

Ijaz,

H.K.

Yang,

A.W.

Ahmad and

Lee, Indoor positioning: A review of indoor ultrasonic positioning systems, in: Proceedings of the Advanced Communication Technology, 2013, pp. 1146–1150.

31.

Jain, The Art of Computer Systems Performance Analysis. Techniques for Experimental Design, Measurement, Simulation, and Modeling, Wiley, New York, NY, 1991.

32.

A.R.

Jimenez Ruiz,

F.S.

Granja,

J.C.

Prieto Honorato and

J.I.G.

Rosas, Accurate pedestrian indoor navigation by tightly coupling foot-mounted imu and rfid measurements, Instrumentation and Measurement, IEEE Transactions on 61(1) (2012), 178–189. doi:10.1109/TIM.2011.2159317.

33.

G.Y.

Jin,

X.Y.

Lu and

M.S.

Park, An indoor localization mechanism using active rfid tag, in: Proceedings of the IEEE International Conference on Sensor Networks, Ubiquitous, and Trustworthy Computing, Vol. 1, 2006, 4 pp.

34.

Kavehrad,

M.I.S.

Chowdhury and

Zhou, Indoor Positioning Methods Using VLC LEDs, John Wiley & Sons, Ltd, 2015, pp. 225–262.

35.

Knauth,

Storz,

Dastageeri,

Koukofikis and

Mähser-Hipp, Fingerprint calibrated centroid and scalar product correlation rssi positioning in large environments, in: Proceedings of the Sixth International Conference on Indoor Positioning and Indoor Navigation (IPIN), 2015.

36.

R.J.

Kuo,

M.C.

Shieh,

J.W.

Zhang and

K.Y.

Chen, The application of an artificial immune system-based back-propagation neural network with feature selection to an RFID positioning system, Robotics and Computer-Integrated Manufacturing 29(6) (2013), 431–438. doi:10.1016/j.rcim.2013.04.002.

37.

Y.S.

Kuo,

Pannuto,

K.J.

Hsiao and

Dutta, Luxapose: Indoor positioning with mobile phones and visible light, in: Proceedings of the 20th Annual International Conference on Mobile Computing and Networking, MobiCom ’14, ACM, New York, NY, USA, 2014, pp. 447–458.

38.

K.C.

Lan and

W.Y.

Shih, On calibrating the sensor errors of a pdr-based indoor localization system, Sensors 13(4) (2013), 4781–4810. doi:10.3390/s130404781.

39.

Laoudias,

Piche and

C.G.

Panayiotou, Device self-calibration in location systems using signal strength histograms, Journal of Location Based Services 7(3) (2013), 165–181. doi:10.1080/17489725.2013.816792.

40.

E.E.L.

Lau and

W.Y.

Chung, Enhanced rssi-based real-time user location tracking system for indoor and outdoor environments, in: Proceedings of the 2007 International Conference on Convergence Information Technology, IEEE Computer Society, Washington, DC, USA, 2007, pp. 1213–1218.

41.

Le,

Wang,

Zhao and

Miao, A novel wifi indoor positioning method based on genetic algorithm and twin support vector regression, in: Control and Decision Conference (2014 CCDC), the 26th Chinese, 2014, pp. 4859–4862. doi:10.1109/CCDC.2014.6853043.

42.

Li,

Salter,

A.G.

Dempster and

Rizos, Indoor positioning techniques based on wireless lan, Technical report, 2006.

43.

Li, Low-cost 3d bluetooth indoor positioning with least square, Wireless Personal Communications (2014), 1–14. doi:10.1007/s11277-013-1491-3.

44.

Lymberopoulos,

R.R.

Choudhury,

Yang and

Sen, in: Microsoft Indoor Localization Competition – IPSN 2014, 2014, Available at: https://www.microsoft.com/en-us/research/event/microsoft-indoor-localization-competition-ipsn-2014/.

45.

Lymberopoulos,

Liu,

Yang,

Naguib,

Rowe,

Trigoni and

Moayeri, in: Microsoft Indoor Localization Competition – IPSN 2015, 2015, Available at: https://www.microsoft.com/en-us/research/event/microsoft-indoor-localization-competition-ipsn-2015/.

46.

Machaj,

Brida and

Piché, Rank based fingerprinting algorithm for indoor positioning, in: Proceedings of the 2nd the International Conference on Indoor Positioning and Indoor Navigation (IPIN’2011), 2011, pp. 1–6.

47.

Markets and Markets. Indoor location market by solution (tag-based, RF-based, sensor-based), by application (indoor maps & navigation, indoor location-based analytics, tracking & tracing, monitoring & emergency management, by service, by vertical, & by region – global forecast up to 2019.

48.

Marques,

Meneses and

Moreira, Combining similarity functions and majority rules for multi-building, multi-floor, wifi positioning, in: Proceedings of the 3th the International Conference on Indoor Positioning and Indoor Navigation (IPIN’2012), 2012.

49.

J.V.

Martí,

Sales,

Marín and

Jiménez-Ruiz, Localization of mobile sensors and actuators for intervention in low-visibility conditions: The zigbee fingerprinting approach, International Journal of Distributed Sensor Networks (2012), 10 pp.

50.

Montaser and

Moselhi, RFID indoor location identification for construction projects, Automation in Construction 39 (2014), 167–179. doi:10.1016/j.autcon.2013.06.012.

51.

Moreira,

M.J.

Nicolau,

Meneses and

Costa, Wi-fi fingerprinting in the real world – RTLS@UM at the EvAAL competition, in: Proceedings of the Sixth International Conference on Indoor Positioning and Indoor Navigation (IPIN), 2015.

52.

Nahrstedt and

Vu, CRAWDAD dataset uiuc/uim (v. 2012-01-24). Downloaded from http://crawdad.org/uiuc/uim/20120124, January 2012.

53.

J.A.

Paredes,

Aguilera,

F.J.

Álvarez,

Lozano and

Morera, Analysis of Doppler effect on the pulse compression of different codes emitted by an ultrasonic LPS, Sensors 11(11) (2011), 10765–10784. doi:10.3390/s111110765.

54.

Purohit,

Pan,

Chen,

Sun and

Zhang, CRAWDAD dataset cmu/supermarket (v. 2014-05-27). Downloaded from http://crawdad.org/cmu/supermarket/20140527, May 2014.

55.

Rhee,

Shin,

Hong,

Lee,

Kim and

Chong, CRAWDAD dataset ncsu/mobilitymodels (v. 2009-07-23). Downloaded from http://crawdad.org/ncsu/mobilitymodels/20090723, July 2009.

56.

Salvi,

Barsocchi,

M.T.

Arredondo and

J.P.L.

Ramos, Evaal, evaluating aal systems through competitive benchmarking, the experience of the 1st competition, in: Evaluating AAL Systems Through Competitive Benchmarking. Indoor Localization and Tracking, Communications in Computer and Information Science, Vol. 309, Springer, Berlin Heidelberg, 2012, pp. 14–25. doi:10.1007/978-3-642-33533-4_2.

57.

Savic and

E.G.

Larsson, Fingerprinting-based positioning in distributed massive MIMO systems, in: Proc. of IEEE 82nd Vehicular Technology Conference (VTC2015-Fall), Sept. 2015, 2015.

58.

O.E.

Segou,

S.A.

Mitilineos and

S.C.A.

Thomopoulos, Dale: A range-free, adaptive indoor localization method enhanced by limited fingerprinting, in: Proceedings of the 1st the International Conference on Indoor Positioning and Indoor Navigation (IPIN’2010), 2010, pp. 1–8.

59.

Torres-Sospedra, Ensembles of artificial neural networks: Analysis and development of design methods, PhD thesis, Department of Computer Science and Engineering, Universitat Jaume I, 2011.

60.

Torres-Sospedra,

Avariento,

Rambla,

Montoliu,

Casteleyn,

Benedito-Bordonau,

Gould and

Huerta, Enhancing integrated indoor/outdoor mobility in a smart campus, International Journal of Geographical Information Science 29(11) (2015), 1955–1968. doi:10.1080/13658816.2015.1049541.

61.

Torres-Sospedra,

Montoliu,

Martínez-Usó,

J.P.

Avariento,

T.J.

Arnau,

Benedito-Bordonau and

Huerta, UJIIndoorLoc: A new multi-building and multi-floor database for WLAN fingerprint-based indoor localization problems, in: Proceedings of the Fifth Conference on Indoor Positioning and Indoor Navigation, 2014, pp. 261–270.

62.

Torres-Sospedra,

Montoliu,

Trilles,

Ó.

Belmonte and

Huerta, Comprehensive analysis of distance and similarity measures for Wi-Fi fingerprinting indoor positioning systems, Expert Systems with Applications 42(23) (2015), 9263–9278. doi:10.1016/j.eswa.2015.08.013.

63.

Torres-Sospedra,

Rambla,

Montoliu,

Belmonte and

Huerta, UJIIndoorLoc-Mag: A new database for magnetic field-based localization problems, in: Proceedings of the Sixth Conference on Indoor Positioning and Indoor Navigation, 2015.

64.

Tumer and

Ghosh, Analysis of decision boundaries in linearly combined neural classifiers, Pattern Recognition 29(2) (1996), 341–348. doi:10.1016/0031-3203(95)00085-2.

65.

Tumer and

Ghosh, Error correlation and error reduction in ensemble classifiers, Connection Science 8(3–4) (1996), 385–403. doi:10.1080/095400996116839.

66.

Van Haute,

De Poorter,

Lemic,

Handziski,

Wirstrom,

Voigt,

Wolisz and

Moerman, Platform for benchmarking of RF-based indoor localization solutions, Communications Magazine, IEEE 53(9) (2015), 126–133. doi:10.1109/MCOM.2015.7263356.

67.

Weyn, Opportunistic seamless localization, PhD thesis, University of Antwerp, 2011.

68.

Yim, Introducing a decision tree-based indoor positioning technique, Expert Systems with Applications 34(2) (2008), 1296–1302. doi:10.1016/j.eswa.2006.12.028.

69.

Yu,

Jiang,

Liang,

Qin,

Hu,

Peng and

Hu, 5g wifi signal-based indoor localization system using cluster k-nearest neighbor algorithm, International Journal of Distributed Sensor Networks (2014).

70.

Zhou,

Tian,

Xu,

Yu,

Hong and

Wu, SCaNME: Location tracking system in large-scale campus wi-fi environment using unlabeled mobility map, Expert Systems with Applications 41(7) (2014), 3429–3443. doi:10.1016/j.eswa.2013.10.047.

71.

Zhou,

Zhang,

Tian,

Qiu and

Wu, Integrated location fingerprinting and physical neighborhood for wlan probabilistic localization, in: Computing, Communication and Networking Technologies (ICCCNT), 2014 International Conference on, 2014, pp. 1–5.

A realistic evaluation of indoor positioning systems based on Wi-Fi fingerprinting: The 2015 EvAAL–ETRI competition

Abstract

Keywords

Get full access to this article

References