Reframing in context: A systematic approach for model reuse in machine learning

Abstract

We describe a systematic approach called reframing, defined as the process of preparing a machine learning model (e.g., a classifier) to perform well over a range of operating contexts. One way to achieve this is by constructing a versatile model, which is not fitted to a particular context, and thus enables model reuse. We formally characterise reframing in terms of a taxonomy of context changes that may be encountered and distinguish it from model retraining and revision. We then identify three main kinds of reframing: input reframing, output reframing and structural reframing. We proceed by reviewing areas and problems where some notion of reframing has already been developed and shown useful, if under different names: re-optimising, adapting, tuning, thresholding, etc. This exploration of the landscape of reframing allows us to identify opportunities where reframing might be possible and useful. Finally, we describe related approaches in terms of the problems they address or the kind of solutions they obtain. The paper closes with a re-interpretation of the model development and deployment process with the use of reframing.

Keywords

Machine learning reframing model reuse operating context cost-sensitive evaluation

Get full access to this article

View all access options for this article.

References

G.D.

Abowd,

A.K.

Dey,

P.J.

Brown,

Davies,

Smith and

Steggles, Towards a better understanding of context and context-awareness, in: Handheld and Ubiquitous Computing, Springer, 1999, pp. 304–307. doi:10.1007/3-540-48157-5_29.

Adomavicius and

Tuzhilin, Context-aware recommender systems, in: Recommender Systems Handbook, Springer, 2011, pp. 217–253. doi:10.1007/978-0-387-85820-3_7.

C.F.

Ahmed,

Lachiche,

Charnay and

Braud, Reframing continuous input attributes, in: IEEE Intl. Conf. Tools with Artificial Intelligence (ICTAI-2014), 2014, pp. 31–38.

Angluin and

Laird, Learning from noisy examples, Machine Learning 2(4) (1988), 343–370.

Bansal,

Sinha and

Zhao, Tuning data mining methods for cost-sensitive regression: A study in loan charge-off forecasting, J. Management Inf. Systems 25 (2008), 315–336. doi:10.2753/MIS0742-1222250309.

Bareinboim and

Pearl, Transportability of causal effects: Completeness results, in: AAAI, 2012.

Ben-David,

Blitzer,

Crammer,

Pereira et al., Analysis of representations for domain adaptation, Advances in Neural Information Processing Systems 19 (2007), 137.

Bi and

K.P.

Bennett, Regression error characteristic curves, in: ICML, 2003.

Blanco-Vega,

Ferri,

Hernández-Orallo and

M.J.

Ramírez-Quintana, Estimating the class probability threshold without training data, in: ICML’06 Workshop on ROC Analysis in Machine Learning, 2006, pp. 9–16.

10.

Caruana, Multitask learning, Machine Learning 28(1) (1997), 41–75. doi:10.1023/A:1007379606734.

11.

Chow, On optimum recognition error and reject tradeoff, IEEE T. Information Theory 16(1) (1970), 41–46. doi:10.1109/TIT.1970.1054406.

12.

Daumé III, Frustratingly easy domain adaptation, in: ACL, 2007, pp. 256–263.

13.

Davis,

Smith,

Canny,

Good,

King and

Janakiraman, Towards context-aware face recognition, in: ACM International Conference on Multimedia, 2005, pp. 483–486.

14.

De Raedt, Interactive Theory Revision: An Inductive Logic Programming Approach, Academic Press, 1992.

15.

Dredze,

Kulesza and

Crammer, Multi-domain learning by confidence-weighted parameter combination, Machine Learning 79(1–2) (2010), 123–149. doi:10.1007/s10994-009-5148-0.

16.

Drummond and

R.C.

Holte, Cost curves: An improved method for visualizing classifier performance, Machine Learning 65 (2006), 95–130. doi:10.1007/s10994-006-8199-5.

17.

Eaton and

P.L.

Ruvolo, Ella: An efficient lifelong learning algorithm, in: ICML, 2013, pp. 507–515.

18.

C.F.

Ahmed,

Lachiche,

Charnay,

El Jelali and

Braud, Flexible propositionalization of continuous attributes in relational data mining, Expert Systems with Applications 42(21) (2015), 7698–7709. doi:10.1016/j.eswa.2015.05.053.

19.

Elkan, The foundations of cost-sensitive learning, in: IJCAI, 2001, pp. 973–978.

20.

Fawcett, An introduction to ROC analysis, Pattern Recognition Letters 27(8) (2006), 861–874. doi:10.1016/j.patrec.2005.10.010.

21.

Fernando,

Habrard,

Sebban and

Tuytelaars, Unsupervised visual domain adaptation using subspace alignment, in: ICCV, IEEE, 2013, pp. 2960–2967.

22.

Ferri,

Hernández-Orallo,

Martínez-Usó and

M.J.

Ramírez-Quintana, Identifying dominant models when the noise context is known, in: ECML’14 Workshop on Learning over Multiple Contexts, 2014.

23.

Ferri,

Hernández-Orallo and

Modroiu, An experimental comparison of performance measures for classification, Pattern Recognition Letters 30(1) (2009), 27–38. doi:10.1016/j.patrec.2008.08.010.

24.

Flach, ROC analysis, in: Encyclopedia of Machine Learning, Springer, 2010, pp. 869–875.

25.

Flach, Classification in context: Adapting to changes in class and cost distribution, in: ECML’14 Workshop on Learning over Multiple Contexts, 2014.

26.

Flach,

Blockeel,

Ferri,

Hernández-Orallo and

Struyf, Decision support for data mining, in: Data Mining and Decision Support, Springer, 2003, pp. 81–90. doi:10.1007/978-1-4615-0286-9_7.

27.

Flach,

Hernández-Orallo and

Ferri, A coherent interpretation of AUC as a measure of aggregated classification performance, in: ICML, 2011.

28.

Frenay and

Verleysen, Classification in the presence of label noise: A survey, IEEE T. Neural Networks and Learning Systems 25(5) (2014), 845–869. doi:10.1109/TNNLS.2013.2292894.

29.

Gama,

Žliobaitė,

Bifet,

Pechenizkiy and

Bouchachia, A survey on concept drift adaptation, ACM Comput. Surv. 46(4) (2014), 44:1–44:37. doi:10.1145/2523813.

30.

Giraud-Carrier,

Vilalta and

Brazdil, Introduction to the special issue on meta-learning, Machine Learning 54(3) (2004), 187–193. doi:10.1023/B:MACH.0000015878.60765.42.

31.

D.J.

Hand, Measuring classifier performance: A coherent alternative to the area under the ROC curve, Machine Learning 77(1) (2009), 103–123. doi:10.1007/s10994-009-5119-5.

32.

Hernández-Orallo, ROC curves for regression, Pattern Recognition 46(12) (2013), 3395–3411. doi:10.1016/j.patcog.2013.06.014.

33.

Hernández-Orallo, Probabilistic reframing for cost-sensitive regression, ACM T. Knowledge Discovery From Data 8(4) (2014), 17.

34.

Hernández-Orallo,

Ferri,

Lachiche,

Martínez-Usó and

M.J.

Ramírez-Quintana, Binarised regression tasks: Methods and evaluation metrics, Data Mining and Knowledge Discovery 30(4) (2016), 848–890. doi:10.1007/s10618-015-0443-9.

35.

Hernández-Orallo,

Flach and

Ferri, Brier curves: A new cost-based visualisation of classifier performance, in: Proceedings of the 28th International Conference on Machine Learning (ICML-11), ACM, New York, 2011, pp. 585–592.

36.

Hernández-Orallo,

Flach and

Ferri, A unified view of performance metrics: Translating threshold choice into expected classification loss, J. Machine Learning Research 13 (2012), 2813–2869.

37.

Hernández-Orallo,

Flach and

Ferri, ROC curves in cost space, Machine Learning 93(1) (2013), 71–91. doi:10.1007/s10994-013-5328-9.

38.

Jiang, A literature survey on domain adaptation of statistical classifiers, 2008, http://sifaka.cs.uiuc.edu/jiang4/domain_adaptation/survey/da_survey.pdf.

39.

Khreich,

Granger,

Miri and

Sabourin, A survey of techniques for incremental learning of HMM parameters, Information Sciences 197 (2012), 105–130. doi:10.1016/j.ins.2012.02.017.

40.

Kienzle and

Chellapilla, Personalized handwriting recognition via biased regularization, in: ICML, 2006, pp. 457–464. doi:10.1145/1143844.1143902.

41.

Kull and

Flach, Patterns of dataset shift, in: ECML’14 Workshop on Learning over Multiple Contexts, 2014.

42.

Kull and

Hernández-Orallo, Missing values on purpose: Model selection and reframing with attribute and prediction costs, Submitted, 2015.

43.

Kuzborskij and

Orabona, Stability and hypothesis transfer learning, in: ICML, 2013, pp. 942–950.

44.

Lindgren, Methods for rule conflict resolution, in: ECML, Springer, 2004, pp. 262–273.

45.

H.-Y.

Lo,

J.-C.

Wang,

H.-M.

Wang and

S.-D.

Lin, Cost-sensitive multi-label learning for audio tag annotation and retrieval, IEEE T. Multimedia 13(3) (2011), 518–529. doi:10.1109/TMM.2011.2129498.

46.

Martínez-Usó and

Hernández-Orallo, Multidimensional prediction models when the resolution context changes, in: ECML, Springer, 2015, pp. 509–524.

47.

C.E.

Metz, Basic principles of ROC analysis, Seminars in Nuclear Medicine 8(4) (1978), 283–298. doi:10.1016/S0001-2998(78)80014-2.

48.

Mihalkova,

Huynh and

R.J.

Mooney, Mapping and revising Markov logic networks for transfer learning, in: AAAI, 2007, pp. 608–614.

49.

J.G.

Moreno-Torres,

Raeder,

Alaiz-Rodríguez,

N.V.

Chawla and

Herrera, A unifying view on dataset shift in classification, Pattern Recognition 45(1) (2012), 521–530. doi:10.1016/j.patcog.2011.06.019.

50.

Muggleton and

De Raedt, Inductive logic programming: Theory and methods, J. Logic Programming 19 (1994), 629–679. doi:10.1016/0743-1066(94)90035-3.

51.

S.J.

Pan and

Yang, A survey on transfer learning, IEEE T. Knowledge and Data Engineering 22(10) (2010), 1345–1359. doi:10.1109/TKDE.2009.191.

52.

Pietraszek, On the use of ROC analysis for the optimization of abstaining classifiers, Machine Learning 68(2) (2007), 137–169. doi:10.1007/s10994-007-5013-y.

53.

Quiñonero-Candela,

Sugiyama,

Schwaighofer and

N.D.

Lawrence, Dataset Shift in Machine Learning, The MIT Press, 2009.

54.

B.L.

Richards and

R.J.

Mooney, First-order theory revision, in: ICML, 1991, pp. 447–451.

55.

Sannen,

Lughofer and

Van Brussel, Towards incremental classifier fusion, Intell. Data Anal. 14(1) (2010), 3–30.

56.

W.J.

Scheirer,

de Rezende-Rocha,

Sapkota and

T.E.

Boult, Toward open set recognition, IEEE T. Pattern Analysis and Machine Intelligence 35(7) (2013), 1757–1772. doi:10.1109/TPAMI.2012.256.

57.

Segev,

Harel,

Mannor,

Crammer and

El-Yaniv, Learn on source, refine on target: A model transfer learning framework with random forests, in: IEEE Transactions on Pattern Analysis and Machine Intelligence, to appear. doi:10.1109/TPAMI.2016.2618118.

58.

M.E.

Taylor and

Stone, Transfer learning for reinforcement learning domains: A survey, J. Machine Learning Research 10 (2009), 1633–1685.

59.

Thrun, Is learning the n-th thing any easier than learning the first?, in: Advances in Neural Information Processing Systems, 1996, pp. 640–646.

60.

Thrun and

Pratt, Learning to Learn, Springer Science & Business Media, 2012.

61.

Tommasi,

Orabona and

Caputo, Learning categories from few examples with multi model knowledge transfer, IEEE T. Pattern Analysis and Machine Intelligence 36(5) (2014), 928–941. doi:10.1109/TPAMI.2013.197.

62.

Torrey and

Shavlik, Transfer learning, Handbook of Research on Machine Learning Applications 3 (2009), 17–35.

63.

Tortorella, A ROC-based reject rule for dichotomizers, Pattern Recognition Letters 26(2) (2005), 167–180. doi:10.1016/j.patrec.2004.09.004.

64.

Turney, Types of cost in inductive concept learning, Canada National Research Council Publications Archive (2000).

65.

Vanderlooy,

Sprinkhuizen-Kuyper and

Smirnov, An analysis of reliable classifiers through ROC isometrics, in: ICML’06 Workshop on ROC Analysis in Machine Learning, 2006, pp. 55–62.

66.

D.H.

Wolpert, Stacked generalization, Neural Networks 5(2) (1992), 241–259. doi:10.1016/S0893-6080(05)80023-1.

67.

Xu,

M.J.

Kusner,

K.Q.

Weinberger,

Chen and

Chapelle, Classifier cascades and trees for minimizing feature evaluation cost, J. Machine Learning Research 15 (2014), 2113–2144.

68.

Zhao,

A.P.

Sinha and

Bansal, An extended tuning method for cost-sensitive regression and forecasting, Decision Support Systems 51(3) (2011), 372–383. doi:10.1016/j.dss.2011.01.003.