In this paper, we investigate MiniMax Entropy models, a class of neural symbolic models where symbolic and subsymbolic features are seamlessly integrated. We show how these models recover classical algorithms from both the deep learning and statistical relational learning scenarios. Novel hybrid settings are defined and experimentally explored, showing state-of-the-art performance in collective classification, knowledge base completion and graph (molecular) data generation.
KahnemanD., Thinking, fast and slow. Farrar, Straus and Giroux, (2017).
2.
LeCunY., BengioY. and HintonG., Deep learning, Nature521(7553) (2015), 436–444.
3.
De RaedtL.,
KerstingK.,
NatarajanS. and
PooleD., Statistical Relational Artificial Intelligence: Logic, Probability, and Computation. Synthesis Lectures on Artificial Intelligence and Machine Learning. Morgan & Claypool Publishers (2016). doi: 10.2200/S00692ED1V01Y201601AIM032. URL https://doi.org/10.2200/S00692ED1V01Y201601AIM032.
4.
d’Avila GarcezA.S.,
GoriM.,
LambL.C.,
SerafiniL.,
SprangerM. and
TranS.N., Neural-symbolic computing: An effective methodology for principled integration of machine learning and reasoning, FLAP6 (2019).
5.
De RaedtL.,
DumančićS.,
ManhaeveR. and
MarraG., From statistical relational to neural symbolic artificial intelligence, In IJCAI 2020, (2020).
6.
RocktäschelT. and RiedelS., End-to-end differentiable proving. In Advances in Neural Information Processing Systems (2017), 3788–3800.
7.
ŠourekG.,
AschenbrennerV.,
ZeleznýF.,
SchockaertS. and
KuželkaO., Lifted relational neural networks: Efficient learning of latent relational structures, JAIR62 (2018).
8.
DiligentiM., GoriM. and SaccàC., Semantic-based regularization for learning and inference, Artif. Intell.244 (2017).
9.
DonadelloI., SerafiniL. and d’Avila GarcezA.S., Logic tensor networks for semantic image interpretation, In IJCAI, (2017).
10.
XuJ., ZhangZ., FriedmanT., LiangY. and Van den BroeckG., A semantic loss function for deep learning with symbolic knowledge, In ICML (2018).
11.
HájekP., Metamathematics of fuzzy logic, volume 4, Springer Science & Business Media, (2013).
12.
MarraG., GianniniF., DiligentiM., MagginiM. and GoriM., Learning and t-norms theory, CoRR (2019).
13.
RichardsonM. and DomingosP., Markov logic networks, Machine learning62(1-2) (2006), 107–136.
14.
KuželkaO. and
DavisJ., Markov logic networks for knowledge base completion: A theoretical analysis under the MCAR assumption, In Proceedings of the Thirty-Fifth Conference on Uncertainty in Artificial Intelligence, UAI, (2019).
15.
BoydS. and VandenbergheL., Convex optimization, Cambridge university press, (2004).
16.
WainwrightM.J., JordanM.I., et al., Graphical models, exponential families, and variational inference, Foundations and Trends® in Machine Learning1(1–2) (2008), 1–305.
17.
KuželkaO.,
WangY.,
DavisJ.
and
SchockaertS., Relational marginal problems: Theory and estimation, In Thirty-Second AAAI Conference on Artificial Intelligence, (2018).
18.
ZhuS.C., WuY.N. and MumfordD., Minimax entropy principle and its application to texture modeling, Neural Computation9(8) (1997), 1627–1660.
19.
BachS.H., BroechelerM., HuangB. and GetoorL., Hinge-loss markov random fields and probabilistic soft logic, J. Mach. Learn. Res.,18 (2017).
20.
GianniniF., DiligentiM., GoriM. and MagginiM., On a convex logic fragment for learning and reasoning, IEEE TFS27 (2018).
ChoiA., ChaviraM. and DarwicheA., Node splitting: A scheme for generating upper bounds in bayesian networks, arXiv preprint arXiv:1206.5251 (2012).
23.
RobertC. and CasellaG., Monte Carlo statistical methods, Springer Science & Business Media (2013).
24.
BishopC.M., Training with noise is equivalent to tikhonov regularization, Neural Computation7(1) (1995), 108–116.
25.
KokS. and DomingosP., Statistical predicate invention, In Proceedings of the 24th international conference on Machine learning, pages 433–440. ACM, (2007).
26.
DiligentiM., GoriM. and SaccaC., Semantic-based regularization for learning and inference, Artificial Intelligence244 (2017), 143–165.
27.
MarraG., GianniniF., DiligentiM. and GoriM., Lyrics: a general interface layer to integrate ai and deep learning, arXiv preprint arXiv:1903.07534 (2019).
28.
MarraG., GianniniF., DiligentiM. and GoriM., Integrating learning and reasoning with deep logic models, In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pages 517–532. Springer, (2019).
29.
MarraG., DiligentiM., GianniniF., GoriM. and MagginiM., Relational neural machines. In ECAI, (2020).
30.
MarraG. and KuželkaO., Neural markov logic networks. In UAI, (2021).
31.
PoonH. and DomingosP., Sound and efficient inference with probabilistic and deterministic dependencies, In AAAI6 (2006), 458–463.
32.
KhotT., NatarajanS., KerstingK. and ShavlikJ., Gradient-based boosting for statistical relational learning: the markov logic network and missing data cases, Machine Learning100(1) (2015), 75–100.
33.
BesoldT.R., d’Avila GarcezA.,
BaderS.,
BowmanH.,
DomingosP.,
HitzlerP.,
KühnbergerK.-U.,
LambL.C.,
LowdD.,
LimaP.M.V., et al,. Neural-symbolic learning and reasoning: A survey and interpretation, arXiv preprint arXiv:1711.03902, (2017).
34.
LippiM. and FrasconiP., Prediction of protein β-residue contacts by markov logic networks with grounding-specific weights, Bioinformatics25(18) (2009), 2326–2333.
35.
ManhaeveR., DumančićS.,
KimmigA.,
DemeesterT. and
De RaedtL., Deepproblog: Neural probabilistic logic programming, In NeurIPS, (2018).
36.
De RaedtL.,
KimmigA. and
ToivonenH., Problog: A probabilistic prolog and its application in link discovery, In IJCAI, volume 7, pages 2462–2467. Hyderabad, (2007).
37.
EllisK., MoralesL., Sablé-MeyerM., Solar-LezamaA. and TenenbaumJ., Learning libraries of subroutines for neurally-guided bayesian program induction, In NeurIPS (2018).
38.
MinerviniP., BošnjakM.,
RocktäschelT.,
RiedelS. and
GrefenstetteE., Differentiable reasoning on large knowledge bases and natural language, In AAAI, (2020).
39.
SourekG., AschenbrennerV., ZeleznýF., SchockaertS. and KuzelkaO., Lifted relational neural networks: Efficient learning of latent relational structures, J. Artif. Intell. Res.62 (2018), 69–100.
40.
ScarselliF., GoriM., TsoiA.C., HagenbuchnerM. and MonfardiniG., The graph neural network model, IEEE Transactions on Neural Networks20(1) (2009), 61–80.
41.
XuK., HuW., LeskovecJ. and JegelkaS., How powerful are graph neural networks? In International Conference on Learning Representations (2018).
42.
WangQ., MaoZ., WangB. and GuoL., Knowledge graph embedding: A survey of approaches and applications, IEEE Transactions on Knowledge and Data Engineering29(12), 2724–2743.
43.
KazemiS.M. and PooleD., Relnn: A deep neural model for relational learning, In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI-18), pages 6367–6375, (2018).
44.
KingmaD.P. and BaJ., Adam: A method for stochastic optimization, arXiv preprint arXiv:1412.6980 (2014).
45.
LuQ. and GetoorL., Link-based classification. In Proceedings of the 20th International Conference on Machine Learning (ICML-03), (2003), 496–503.
46.
GammermanA., VovkV. and VapnikV., Learning by transduction. In Proceedings of the Fourteenth conference on Uncertainty in artificial intelligence, pages 148–155. Morgan Kaufmann Publishers Inc., (1998).
47.
TrouillonT., WelblJ., RiedelS., GaussierÉ. and BouchardG., Complex embeddings for simple link prediction, In ICML, (2016).
48.
MinerviniP., RiedelS., StenetorpP., GrefenstetteE. and RocktäschelT., Learning reasoning strategies in end-to-end differentiable proving, arXiv preprint arXiv:2007.06477 (2020).
49.
YouJ., YingR., RenX., HamiltonW.L. and LeskovecJ., Graphrnn: Generating realistic graphs with deep auto-regressive models, arXiv preprint arXiv:1802.08773, (2018).
50.
LiY., VinyalsO., DyerC., PascanuR. and BattagliaP., Learning deep generative models of graphs, arXiv preprint arXiv:1803.03324 (2018).
51.
GaultonA., HerseyA., NowotkaM. and Patrícia BentoA.,
ChambersJ.MendezD.MutowoP.AtkinsonF.BellisL.J.,
Cibrián-UhalteE., et al., The chembl database in 2017, Nucleic Acids Research45(D1) (2016), D945–D954.
