This paper describes how computational generative models can describe aspects of the artistic process, and how these generative models can provide tools for formulating and testing psychological theories of art. The term “generative models” here refers to algorithms that can generate artistic imagery, video, text, or other artistic media, including techniques developed in both computer graphics and AI research. Generative models can both describe artistic processes and offer useful experimental tools. This paper first outlines different ways to understand the types of research in generative models. It then surveys several recent examples of using generative models to develop theories and to perform experiments. The paper then discusses misleading uses of the concept of “AI-generated art” in psychological studies, and the need for study of our relationship with new artistic technologies. Finally, the paper offers a few remarks on pursuing interdisciplinary research across psychology and computer graphics.
AgarwalaA.AgrawalaM.CohenM.SalesinD.SzeliskiR. (2006). Photographing long scenes with multi-viewpoint panoramas. ACM Transactions on Graphics, 25(3), 853–861. https://doi.org/10.1145/1141911.1141966
2.
AgrawalaM.LiW.BerthouzozF. (2011). Design principles for visual communication. In Communications of the ACM (pp. 60–69).
3.
AgrawalaM.ZorinD.MunznerT. (2000). Artistic multiprojection rendering. In Proceedings of the Eurographics workshop on rendering techniques (pp. 125–136). Springer-Verlag.
4.
AigermanN.GroueixT. (2024). Generative Escher meshes. In Proc. SIGGRAPH.
5.
BadkiA.GalloO.KautzJ.SenP. (2017). Computational zoom: A framework for post-capture image composition. ACM Transactions on Graphics, 36(4), 1–14. https://doi.org/10.1145/3072959.3073687
6.
Bar-TalO.YarivL.LipmanY.DekelT. (2023). Multidiffusion: Fusing diffusion paths for controlled image generation. In Proc. ICML.
7.
BarnesC.ShechtmanE.FinkelsteinA.GoldmanD. B. (2009). Patchmatch: A randomized correspondence algorithm for structural image editing. ACM Transactions on Graphics, 28(3), 1–11. https://doi.org/10.1145/1531326.1531330
8.
BattagliaP. W.HamrickJ. B.TenenbaumJ. B. (2013). Simulation as an engine of physical scene understanding. Proceedings of the National Academy of Sciences, 110(45), 18327–18332. https://doi.org/10.1073/pnas.1306572110
9.
BenedettiL.WinnemöllerH.CorsiniM.ScopignoR. (2014). Painting with bob: Assisted creativity for novices. In Proceedings of the 27th Annual ACM symposium on user interface software and technology, UIST ’14, New York, NY, USA (pp. 419–428). Association for Computing Machinery.
10.
BergerJ. (1972). Ways of seeing. Penguin.
11.
BodenM. (2005). Creativity. In GautB.LopesD. M. (Eds.), The Routledge companion to aesthetics (2nd ed.). Routledge.
CatmullE.WallaceA. (2023). Creativity, Inc. (The Expanded Edition): Overcoming the unseen forces that stand in the way of true inspiration. Penguin Random House.
ChamberlainR.BerioD.MayerV.ChanaK.LeymarieF. F.OrgsG. (2022). A dot that went for a walk: People prefer lines drawn with human-like kinematics. British Journal of Psychology, 113(1), 105–130. https://doi.org/10.1111/bjop.12527
16.
ChanC.DurandF.IsolaP. (2022). Learning to generate line drawings that convey geometry and semantics. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR) (pp. 7915–7925).
17.
ChandraK.LiT.-M.TenenbaumJ.Ragan-KelleyJ. (2022). Designing perceptual puzzles by differentiating probabilistic programs. In ACM SIGGRAPH 2022 Conference Proceedings, SIGGRAPH ‘22, New York, NY, USA. Association for Computing Machinery.
18.
ChandraK.LiT.-M.TenenbaumJ.Ragan-KelleyJ. (2023). Acting as inverse inverse planning. In ACM SIGGRAPH 2023 Conference Proceedings, SIGGRAPH ‘23, New York, NY, USA. Association for Computing Machinery.
19.
ChenZ.KimV. G.FisherM.AigermanN.ZhangH.ChaudhuriS. (2021). DECOR-GAN: 3D shape detailization by conditional refinement. In Proc. CVPR.
20.
ChristensenP.FongJ.ShadeJ.WootenW.SchubertB.KenslerA., … LianiM. (2018). Renderman: An advanced path-tracing architecture for movie rendering. ACM Transactions on Graphics, 37(3), 1–21. https://doi.org/10.1145/3182162
21.
CohenH. (1995). The further exploits of AARON, painter. Stanford Humanities Review, 4(2).
22.
CohenM. F. (2000). Imagination amplification. IEEE Computer Graphics and Applications, 20(1), 54–55. https://doi.org/10.1109/38.814562
23.
ColeF.GolovinskiyA.LimpaecherA.BarrosH. S.FinkelsteinA.FunkhouserT.RusinkiewiczS. (2008). Where do people draw lines?ACM Transactions on Graphics, 27(3), 1–11. https://doi.org/10.1145/1360612.1360687
24.
ColeF.SanikK.DeCarloD.FinkelsteinA.FunkhouserT.RusinkiewiczS.SinghM. (2009). How well do line drawings depict shape?ACM Transactions on Graphics, 28(3), 1–9. https://doi.org/10.1145/1531326.1531334
CostelloD. (2017). On photography: A philosophical inquiry. Routledge.
27.
CriminisiA.PérezP.ToyamaK. (2004). Region filling and object removal by exemplar-based image inpainting. IEEE Transactions on Image Processing, 13(9), 1200–1212. https://doi.org/10.1109/TIP.2004.833105
28.
DavisN.WinnemöllerH.DontchevaM.DoE. Y.-L. (2013). Toward a cognitive theory of creativity support. In Proceedings of the 9th ACM conference on creativity & cognition, New York, NY, USA (pp. 13–22). Association for Computing Machinery.
29.
DeCarloD. (2012). Depicting 3d shape using lines. In Proc. SPIE Human Vision and Electronic Imaging XVII, Vol. 8291.
30.
DeCarloD.FinkelsteinA.RusinkiewiczS.SantellaA. (2003). Suggestive contours for conveying shape. ACM Transactions on Graphics, 22(3), 848–855. https://doi.org/10.1145/882262.882354
31.
DeCarloD.RusinkiewiczS. (2007). Highlight lines for conveying shape. In Proceedings of the 5th international symposium on non-photorealistic animation and rendering, NPAR ‘07 (pp. 63–70).
32.
DeregowskiJ. B. (1989). Real space and represented space: Cross-cultural perspectives. Behavioral and Brain Sciences, 12(1), 51–74. https://doi.org/10.1017/S0140525X00024286
33.
Di DioC.ArdizziM.SchieppatiS. V.MassaroD.GilliG.GalleseV.MarchettiA. (2023). Art made by artificial intelligence: The effect of authorship on aesthetic judgments. Psychology of Aesthetics, Creativity, and the Arts.
34.
DurandF. (2002). An invitation to discuss computer depiction. In Proceedings of the 2nd international symposium on non-photorealistic animation and rendering, NPAR ‘02, New York, NY, USA (pp. 111–124). ACM.
EfrosA. A.LeungT. (1999). Texture synthesis by non-parametric sampling. In Proc. ICCV.
37.
EpsteinZ.GrohM.DubeyA.PentlandA. (2021). Social influence leads to the formation of diverse local trends. Proceedings of the ACM on Human-Computer Interaction, 5(CSCW), 1–18. https://doi.org/10.1145/3479553
38.
EpsteinZ.HertzmannA.AktenM.FaridH.FjeldJ.FrankM. R., … SmithA. (2023). Art and the science of generative AI. Science, 380(6650), 1110–1111. https://doi.org/10.1126/science.adh4451
FanJ. E.BainbridgeW. A.ChamberlainR.WammesJ. D. (2023). Drawing as a versatile cognitive tool. Nature Reviews Psychology, 2(9), 556–568. https://doi.org/10.1038/s44159-023-00212-w
41.
FlemingR. W.DrorR. O.AdelsonE. H. (2003). Real-world illumination and the perception of surface reflectance properties. Journal of Vision, 3(5), 3–3. https://doi.org/10.1167/3.5.3
42.
GamboniD. (2004). Potential images: Ambiguity and indeterminacy in modern art. Reaktion Books.
43.
GangadharbatlaH. (2022). The role of ai attribution knowledge in the evaluation of artwork. Empirical Studies of the Arts, 40(2), 125–142. https://doi.org/10.1177/0276237421994697
44.
GatysL.EckerA. S.BethgeM. (2015). Texture synthesis using convolutional neural networks. Advances in Neural Information Processing Systems, 28.
45.
GatysL. A.EckerA. S.BethgeM. (2016). Image style transfer using convolutional neural networks. In Proc. CVPR.
46.
GautB. (2018). The value of creativity. In GautB.KieranM. (Eds.), Creativity and philosophy. Routledge.
47.
GengD.ParkI.OwensA. (2024a). Visual anagrams: Generating multi-view optical illusions with diffusion models. In Proc. CVPR.
48.
GengD.ParkI.OwensA. (2024b). Factorized diffusion: Perceptual illusions by noise decomposition. In Proc. ECCV.
49.
GlennersterA. (2023). Understanding 3d vision as a policy network. Philosophical Transactions of the Royal Society B: Biological Sciences, 378(1869). https://doi.org/10.1098/rstb.2021.0448
50.
GombrichE. H. (1961). Art and illusion: A study in the psychology of pictorial representation (2nd ed.). Princeton University Press.
51.
GoodfellowI. J.Pouget-AbadieJ.MirzaM.XuB.Warde-FarleyD.OzairS., … BengioY. (2014). Generative adversarial nets. In Proc. NIPS.
52.
GoodwinT.VollickI.HertzmannA. (2007). Isophote distance: A shading approach to artistic stroke thickness. In Proceedings of the 5th international symposium on non-photorealistic animation and rendering, NPAR ‘07 (pp. 53–62). ACM.
53.
GrablerF.AgrawalaM.SumnerR. W.PaulyM. (2008). Automatic generation of tourist maps. ACM Transactions on Graphics, 27(3), 1–11. https://doi.org/10.1145/1360612.1360699
54.
GrabliS.TurquinE.DurandF.SillionF. X. (2010). Programmable rendering of line drawing from 3d scenes. ACM Transactions on Graphics, 29(2), 1–20. https://doi.org/10.1145/1731047.1731056
55.
GreenbergD. P.TorranceK. E.ShirleyP.ArvoJ.LafortuneE.FerwerdaJ. A., … FooS.-C. (1997). A framework for realistic image synthesis. In Proceedings of the 24th annual conference on computer graphics and interactive techniques, SIGGRAPH ‘97, USA (pp. 477–494). ACM Press/Addison-Wesley Publishing Co.
56.
Hawley-DolanA.YoungL. (2011). Whose mind matters more: The moral agent or the artist? the role of intent in ethics and aesthetics. In Proceedings of the annual meeting of the cognitive science society (Vol. 33).
57.
HeL.-W.CohenM. F.SalesinD. H. (1996). The virtual cinematographer: A paradigm for automatic real-time camera control and directing. In Proceedings of the 23rd annual conference on Computer graphics and interactive techniques (pp. 217–224).
58.
HeegerD. J.BergenJ. R. (1995). Pyramid-based texture analysis/synthesis. In Proceedings of the 22nd annual conference on Computer graphics and interactive techniques (pp. 229–238).
59.
HermanL. M.HwangA. H.-C. (2022). In the eye of the beholder: A viewer-defined conception of online visual creativity. New Media & Society.
60.
HertzmannA. (1998). Painterly rendering with curved brush strokes of multiple sizes. In Proc. SIGGRAPH.
HertzmannA. (2010). Non-photorealistic rendering and the science of art. In Proceedings of the 8th international symposium on non-photorealistic animation and rendering, NPAR ‘10 (pp. 147–157).
HertzmannA.JacobsC. E.OliverN.CurlessB.SalesinD. H. (2001). Image analogies. In Proc. SIGGRAPH.
70.
HertzmannA.ZorinD. (2000). Illustrating smooth surfaces. In Proc. SIGGRAPH (pp. 517–526).
71.
HoJ.JainA.AbbeelP. (2020). Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33, 6840–6851.
72.
JoubertN.RobertsM.TruongA.BerthouzozF.HanrahanP. (2015). An interactive tool for designing quadrotor camera shots. ACM Transactions on Graphics (SIGGRAPH Asia 2015).
KomarV.MelamidA. (1999). Painting by numbers: Komar and Melamid's scientific guide to art. Univ. of California Press.
80.
KosinskiD. (Ed.). (1999). The artist and the camera. Dallas Museum of Art.
81.
KumariN.ZhangB.ZhangR.ShechtmanE.ZhuJ.-Y. (2023). Multi-concept customization of text-to-image diffusion. In CVPR.
82.
LasseterJ. (1987). Principles of traditional animation applied to 3d computer animation. In Proc. SIGGRAPH (Vol. 21).
83.
LeeY.MarkosianL.LeeS.HughesJ. F. (2007). Line drawings via abstracted shading. ACM Transactions on Graphics, 26(3).
84.
LevinG.BrainT. (2021). Code as creative medium: A handbook for computational art and design. The MIT Press.
85.
LiH.ZhangH.WangY.CaoJ.ShamirA.Cohen-OrD. (2013). Curve style analysis in a set of shapes. Computer Graphics Forum, 32(6), 77–88. https://doi.org/10.1111/cgf.12015
86.
LintonP.MorganM. J.ReadJ. C. A.VishwanathD.Creem-RegehrS. H.FulvioD. (2022). New approaches to 3d vision. Philosophical Transactions of the Royal Society B, 378.
87.
LiuY.-T.ZhangZ.GuoY.-C.FisherM.WangZ.ZhangS.-H. (2023). Dualvector: Unsupervised vector font synthesis with dual-part representation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR) (pp. 14193–14202).
88.
MilletK.BuehlerF.DuG.KokkorisM. D. (2023). Defending humankind: Anthropocentric bias in the appreciation of AI art. Computers in Human Behavior, 143, 107707. https://doi.org/10.1016/j.chb.2023.107707
MukherjeeK.HueyH.LuX.VinkerY.Aguina-KangR.ShamirA.FanJ. (2023). Evaluating machine comprehension of sketch meaning at different levels of abstraction. In Proceedings of the 45th annual meeting of the cognitive science society.
92.
MüllerP.WonkaP.HaeglerS.UlmerA.Van GoolL. (2006). Procedural modeling of buildings. In ACM Trans. Graphics.
PaikK.IwerksL. (2007). To infinity and beyond! The story of pixar animation studios. Virgin.
97.
PearsonD. E.RobinsonJ. A. (1985). Visual communication at very low data rates. Proceedings of the IEEE, 73(4), 795–812. https://doi.org/10.1109/PROC.1985.13202
98.
PellaciniF.FerwerdaJ. A.GreenbergD. P. (2000). Toward a psychophysically-based light reflection model for image synthesis. In Proceedings of the 27th annual conference on Computer graphics and interactive techniques (pp. 55–64).
99.
PepperellR. (2011). Connecting art and the brain: An artist’s perspective on visual indeterminacy. Frontiers in Human Neuroscience, 5. https://doi.org/10.3389/fnhum.2011.00084
PortillaJ.SimoncelliE. P. (2000). A parametric texture model based on joint statistics of complex wavelet coefficients. International Journal of Computer Vision, 40(1), 49–70. https://doi.org/10.1023/A:1026553619983
102.
RamanarayananG.FerwerdaJ.WalterB.BalaK. (2007). Visual equivalence: Towards a new standard for image fidelity. ACM Transactions on Graphics, 26(3). https://doi.org/10.1145/1276377.1276472
ReddyP.GharbiM.LukacM.MitraN. J. (2021). Im2vec: Synthesizing vector graphics without vector supervision. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 7342–7351).
105.
RichardsB. A.LillicrapT. P.BeaudoinP.BengioY.BogaczR.ChristensenA., … KordingK. P. (2019). A deep learning frame-work for neuroscience. Nature Neuroscience, 22(11), 1761–1770. https://doi.org/10.1038/s41593-019-0520-2
106.
RobertsL. (1963). Machine perception of three-dimensional solids [PhD thesis]. Massachusetts Institute of Technology, Dept. of Electrical Engineering.
107.
RomanA.GargG.LevoyM. (2004). Interactive design of multi-perspective images for visualizing urban landscapes. In IEEE visualization (pp. 537–544). IEEE.
108.
RosenholtzR. (2020). Demystifying visual awareness: Peripheral encoding plus limited decision complexity resolve the paradox of rich visual experience and curious perceptual failures. Attention, Perception, & Psychophysics, 82(3), 901–925. https://doi.org/10.3758/s13414-019-01968-1
109.
SalganikM. J.DoddsP. S.WattsD. J. (2006). Experimental study of inequality and unpredictability in an artificial cultural market. Science, 311(5762), 854–856. https://doi.org/10.1126/science.1121066
110.
SamoA.HighhouseS. (2023). Artificial intelligence and art: Identifying the aesthetic judgment factors that distinguish human- and machine-generated artwork. Psychology of Aesthetics, Creativity, and the Arts.
SecretanJ.BeatoN.AmbrosioD. B.RodriguezA.CampbellA.StanleyK. O. (2008). Picbreeder: Evolving pictures collaboratively online. In Proceedings of the SIGCHI conference on human factors in computing systems, CHI ‘08, New York, NY, USA (pp. 1759–1768). Association for Computing Machinery.
114.
ShihY.LaiW. S.LiangC. K. (2019). Distortion-free wide-angle portraits on camera phones. ACM Transactions on Graphics, 38(4), 1–12. https://doi.org/10.1145/3306346.3322948
115.
SnapperL.OrancC.Hawley-DolanA.NisselJ.WinnerE. (2015). Your kid could not have done that: Even untutored observers can discern intentionality and structure in abstract expressionist art. Cognition, 137, 154–165. https://doi.org/10.1016/j.cognition.2014.12.009
116.
StamJ. (1998). Stable fluids. In Proc. SIGGRAPH.
117.
StanleyK. O.LehmanJ. (2015). Why greatness cannot be planned: The myth of the objective. Springer Cham.
118.
SutherlandI. E.SproullR. F.SchumackerR. A. (1974). A characterization of ten hidden-surface algorithms. ACM Computing Surveys, 6(1), 1–55. https://doi.org/10.1145/356625.356626
119.
TenenbaumJ. B.KempC.GriffithsT. L.GoodmanN. D. (2011). How to grow a mind: Statistics, structure, and abstraction. Science, 331(6022), 1279–1285. https://doi.org/10.1126/science.1192788
120.
Van de CruysS.WagemansJ. (2011). Putting reward in art: A tentative prediction error account of visual art. i-Perception, 2(9), 1035–1062. https://doi.org/10.1068/i0466aap
121.
VinkerY.AlalufY.Cohen-OrD.ShamirA. (2023). Clipascene: Scene sketching with different types and levels of abstraction. In Proceedings of the IEEE/CVF international conference on computer vision (ICCV) (pp. 4146–4156).
VishwanathD.GirshickA. R.BanksM. S. (2005). Why pictures look right when viewed from the wrong place. Nature Neuroscience, 8(10), 1401–1410. https://doi.org/10.1038/nn1553
124.
WangX.BylinskiiZ.HertzmannA.PepperellR. (2020). Toward quantifying ambiguities in artistic images. ACM Transactions on Applied Perception, 17(4), 1–10. https://doi.org/10.1145/3418054
125.
WangX.BylinskiiZ.HertzmannA.PepperellR. (2023). A computational approach to studying aesthetic judgments of ambiguous artworks. Psychology of Aesthetics, Creativity, and the Arts, Advance online publication.
126.
WarburtonN. (2003). The art question. Routledge.
127.
WeissR. A. (1966). Be VISION, a package of IBM 7090 Fortran programs to draw orthographic views of combinations of plane and quadric surfaces. Journal of the ACM, 13(2), 194–204. https://doi.org/10.1145/321328.321330
128.
WillatsJ. (1997). Art and representation: New principles in the analysis of pictures. Princeton University Press.
129.
WinkenbachG.SalesinD. H. (1994). Computer-generated pen-and-ink illustration. In Proceedings of the 21st annual conference on computer graphics and interactive techniques, SIGGRAPH ‘94 (pp. 91–100).
130.
WinkenbachG.SalesinD. H. (1996). Rendering parametric surfaces in pen and ink. In Proceedings of the 23rd annual conference on computer graphics and interactive techniques, SIGGRAPH ‘96 (pp. 469–476).
131.
WongM. T.ZongkerD. E.SalesinD. H. (1998). Computer-generated floral ornament. In Proceedings of the 25th annual conference on Computer graphics and interactive techniques (pp. 423–434).
132.
XieJ.HertzmannA.LiW.WinnemöllerH. (2014). Portraitsketch: Face sketching assistance for novices. In Proceedings of the 27th annual ACM symposium on user interface software and technology, UIST ‘14 (pp. 407–417).
133.
XuK.LiH.ZhangH.Cohen-OrD.XiongY.ChengZ.-Q. (2010). Style-content separation by anisotropic part scales. ACM Transactions on Graphics, 29(5), 1–10.
134.
YaminsD.DiCarloJ. (2016). Using goal-driven deep learning models to understand sensory cortex. Nature Neuroscience, 19(3), 356–365. https://doi.org/10.1038/nn.4244
135.
YiuE.KosoyE.GopnikA. (2023). Transmission versus truth, imitation versus innovation: What children can do that large language and language-and-vision models cannot (yet). Perspectives on Psychological Science.
136.
Zelnik-ManorL.PetersG.PeronaP. (2005). Squaring the circle in panoramas. In Tenth IEEE international conference on computer vision (ICCV’05) (Vol. 1, Vol. 2, pp. 1292–1299). IEEE.
137.
ZhangL.RaoA.AgrawalaM. (2023). Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF international conference on computer vision (pp. 3836–3847).
138.
ZhengX.QiaoX.CaoY.LauR. W. H. (2019). Content-aware generative modeling of graphic design layouts. ACM Transactions on Graphics, 38(4), 1–15. https://doi.org/10.1145/3306346.3322971
