Bridging data and adaptation in agent-based modelling: A comparative reinforcement learning approach to urban food retail choice

Abstract

Agent-based models (ABMs) are powerful tools for simulating complex socioeconomic systems. Reinforcement learning (RL), especially online RL, is increasingly used to model agent behaviour, allowing agents to explore and learn optimal actions from the model environment. However, standard online RL approaches learn from scratch and often fail to incorporate valuable real-world data. Further, while purely offline RL can use this data, it does not adapt to new dynamics. To address this limitation, this study develops a hybrid offline-to-online (O2O) RL model and compares it with online and offline learning in an ABM of customer store choice. This allows us to address a critical gap, as most research on customer grocery shopping behaviour is static and fails to capture dynamic interactions between customers and food retailers. The result shows that the hybrid mode achieves the highest performance in customer rewards, successfully leveraging historical data for a strong start, while using online learning to adapt and optimize. The hybrid method also produced a distinct spatial pattern of competition, retaining a realistic baseline from its offline training while also showing emergent areas of intensified competition. Furthermore, the study assesses how well each model replicates observed customer behaviours, finding that pre-training with offline data is crucial for capturing specific shopping patterns, for example, specific temporal shopping patterns. Notably, store agents converged on a stable pricing strategy regardless of the customer learning model. This research highlights that the choice of RL training approach critically shapes the outcomes of ABM simulation. The O2O approach represents a powerful and adaptable framework that balances empirical data with learning ability, allowing its application to diverse socioeconomic systems by defining custom state, action, and reward structures.

Keywords

agent-based model reinforcement learning offline-to-online customer store choice individual decision-making urban grocery retail

Introduction

Agent-based modelling (ABM) is a powerful technique widely used to simulate the emergence of complex systems, particularly within socioeconomic frameworks such as urban systems (Crooks, 2010). This modelling approach simulates an entire system by decomposing it into autonomous agents, each possessing the ability to interact with other agents and their environment. Through these interactions and processes of self-organization, agents’ behaviours give rise to emergent, system-level patterns, making ABM particularly suitable for simulating complexity (Sengupta et al., 2005). However, a key challenge in ABM remains the design of individual agent behaviour (Heppenstall et al., 2021). For example, important drivers and factors can be overlooked, and identifying appropriate spatiotemporal resolutions is often difficult. Furthermore, although a variety of decision-making frameworks exist, the rationale for selecting one over others often remains ambiguous (An, 2012). To address this gap, recent studies have begun to integrate machine learning (ML), including supervised, unsupervised, and reinforcement learning (RL) into ABMs as a means to decipher agent behaviour (Dehkordi et al., 2023).

RL is a prominent subfield of artificial intelligence (AI) that develops algorithms enabling agents to learn optimal behaviours through trial-and-error interaction with a real or virtual environment to achieve specific goals (Sutton and Barto, 2018). This methodology has been applied to ABMs across various disciplines, including studies on social cooperation, traffic light control, and health science (Harati et al., 2021; Jalalimanesh et al., 2017; Wu et al., 2022). In geographical contexts, while spatially-explicit applications are less common (Heppenstall et al., 2021), notable applications exist in fields such as forest management and criminology (Bone and Dragićević, 2010; Joubert et al., 2022; Olmez et al., 2024).

Classic RL algorithms aim to maximize cumulative rewards and operate independently of direct supervision from real-world datasets (Zhang et al., 2023). Consequently, most existing RL-ABMs employ an online RL approach, where the agent’s learning model is updated dynamically through direct interactions with the simulated environment (Song et al., 2022). Online RL is highly preferred in scenarios where the environment can be accurately simulated, and when discovering novel, unconstrained strategies is the primary goal, such as in traffic light optimization (Wu et al., 2022) or robotic control. Moreover, it is also preferred when real-world behavioural trajectories are scarce or difficult to collect. However, because it learns purely through trial-and-error without prior knowledge, it suffers from the “cold start” problem and, without real world data, agents may converge on unrealistic behaviours that deviate from empirical human patterns. In contrast, offline RL utilizes pre-collected datasets of trajectories or real-world data to train a policy without requiring further interaction with its environment (He, 2023). Notably, the offline approach is crucial for safety-critical or heavily data-rich environments where trial-and-error exploration is dangerous, unethical, or prohibitively expensive, such as in personalized healthcare recommendations or autonomous driving models. However, because it cannot interact with the environment, an offline policy often fails to adapt when faced with novel dynamics not present in its training data. Unlike supervised or imitation learning, which often involve direct regression or classification, offline RL still employs core RL principles to derive a policy that maximizes a defined objective from static real-world data (Rashidinejad et al., 2021).

Bridging these two paradigms is offline-to-online (O2O) RL, a hybrid approach where a model is first pre-trained on a real-world dataset and subsequently fine-tuned through online interactions with its environment (Nair et al., 2021). This method combines the real-world grounding of offline data with the adaptability of online learning, a feature of particular significance for developing geographically explicit ABMs. It is a well-researched topic in the field of RL and machine learning (Lee et al., 2022; Nakamoto et al., 2023; Song et al., 2022). Despite its potential, this hybrid method has seen limited application within spatial ABMs. Consequently, there is a notable research gap concerning the comparative performance of online, offline, and O2O RL frameworks in a real-world geographical context. This study aims to address this gap by comparing the performance of these three RL frameworks within an ABM designed to simulate customer store choice in the food retail sector.

Due to intensifying competition in the retail sector, understanding customer store choice has become critically important for store managers. Investigating the factors and rationale behind these choices can reveal detailed profiles of consumer decision-making across different typologies, and help in generating a more equitable and healthier urban food landscape. A variety of studies have analyzed attributes influencing customer store selection, including store characteristics and customer demographics. While the specific attributes, such as different prices, assortment levels, distance and product qualities (Choi, 2014), vary across studies, they often align with the classic “4Ps” of marketing: place, price, product, and promotion (Armstrong et al., 2014).

However, prior research on customer store choice has predominantly relied on static data or surveys, often overlooking the dynamic interactions between stores and customers. While agent-based models have been applied to the retail sector, existing approaches rely on traditional, non-adaptive decision-making frameworks. These range from fixed rule-based heuristics, where agent behaviour is governed by deterministic “if-then” conditionals tied to demographic attributes, to classical utility maximization and random utility theory, where agents evaluate all available stores and distribute their spending budget proportionately based on the evaluated utility of each option. Schenk et al. (2007) simulated regional grocery shopping with consumer agents governed by fixed probabilistic choice rules based on store attractiveness and travel distance, Heppenstall et al. (2007) introduced a hybrid approach where petrol retailer agent pricing rules were optimized using genetic algorithms and consumer demand remained governed by aggregate spatial interaction models, while Sturley et al. (2018) advanced this approach by parameterizing a grocery retail ABM directly from loyalty card, reproducing aggregate market share patterns. As Ligmann-Zielinska et al. (2025) summarized, however, existing retail simulation models either lack a spatial context (Armendariz et al., 2015), or employ static agent rules that cannot adapt to changing conditions, such as dynamic promotional pricing strategies.

To address these limitations, this study proposes an RL-ABM that integrates a reward function in the form of a modified Huff model that accounts for product assortment and distance to store, with dynamic price incentives determined by the promotion of autonomous store agents. The RL algorithm equips consumer agents with the ability to adapt to these unobserved, shifting retail landscapes rather than simply repeating historical trajectories collected before the simulation. This allows us to simulate not just what consumers did in the past, but how they might learn and change their habits in response to changes in the environment. With this reward function, the offline and online RL modes can be unified. That is, the RL maximizes its reward to match real-world data using offline mode, and then uses an online mode to make its store choice with the same reward function. This creates a more comprehensive simulation where both customer and store behaviours are adaptive.

Therefore, we designed our study to answer one central research question, supported by three specific sub-questions:

• How do online, offline, and hybrid O2O RL frameworks compare in their ability to generate data-driven and robust outcomes within a spatially-explicit ABM of food retail choices?

To address this, we investigate the following sub-questions:

• Learning Performance: How do the frameworks differ in terms of agent learning efficiency and their capacity to maximize rewards over time?

• Behavioural Replication: To what extent can each framework replicate known, real-world behavioural patterns observed in the empirical data?

• Emergent Spatial Dynamics: What distinct patterns of spatial competition, market dominance, and competitive intensity emerge from each learning framework?

By answering these questions, this paper seeks to provide a methodological roadmap for applying RL in future geospatial ABMs. While grounded in real-world retail data, the primary contribution of this study is methodological: demonstrating how to balance empirical records with agent adaptability, using the urban food retail sector as an illustrative environment.

Methodology

It is first necessary to provide an introduction to the structure of the agents within the ABM, as well as the RL methods used. This includes individual decision-making framework for each of the agents that simulate both the grocery store and customer dynamics. Further, through analysis of a database of purchases made by holders of a loyalty card, one can generate the typology of grocery store customers, and obtain empirical data to train the offline RL model. Additionally, this structure provides further possibilities for O2O RL implementation and comparison. In addition, since previous studies are mainly data-driven and static, this paper goes beyond store selection of customers without feedback from stores, by creating a learning mechanism that accounts for dynamic pricing implemented by stores.

Data source

This study builds upon the results and data sources in Zhang et al. (2025)’s clustering work. We utilize two primary datasets: a synthetic population and customer loyalty card data. The first is a synthetic population from the Syntheco project at the McGill Centre for the Convergence of Health and Economics (MCCHE), which provides geolocated household and sociodemographic information derived from census data (Gieschen et al., 2025). The second dataset contains 32 months of loyalty card transactions (February 2015–September 2017) from a major Quebec food retailer. These records, which include details like timestamps, spending amounts, and store IDs, are used to analyze customer behaviours and provide the transition data for offline training. A list of the sociodemographic, behavioural, and typology attributes used in this study is provided in Table S1 in the Supplementary Materials.

ABM design

Model environment

The study area of this model is the City of Montreal. Montreal is the largest city in the province of Quebec and the second largest in Canada. Its status as Canada’s second-largest city means its residents’ shopping patterns offer a wealth of data for this research. In our model environment, we import the geographical space as the modelling background, and hence, the measurement of proximity is based on Euclidean distance. It is understood that temporal measurements based on actual transportation options (e.g., time taken to travel by car versus public transit) can provide a more precise simulation. However, these metrics depend on specific transportation modes, which are not available in our dataset. A regression analysis across four transport modes confirms a strong rank-preserving relationship between Euclidean distance and travel time in Montreal (see Appendix 2), supporting the use of this metric for the ordinal store-accessibility comparisons required by the Huff-based reward function.

This model is implemented on Mesa and Mesa-Geo (Masad and Kazil, 2015; Wang et al., 2022), a Python library for ABM simulation. Compared with other ABM platforms, a Python-based library provides a better interface for the application of RL and DL algorithms. Additionally, the timestep of this computational model is one “shopping” time slot, which is morning, afternoon, evening and night. These time slots cover 8am −12pm, 12pm–17pm, 17pm–23pm, and a constrained “night” period that prevents the customer agent from shopping at this time. Therefore, customers can only shop during time slots 1–3. To reduce the impacts of stochasticity, the simulation runs 10 times with different random seeds. Figure S1 in the Supplementary Material illustrates the visualization of the model environment, in which customers and stores are represented as yellow and green points.

Agents

In the simulated urban retail ABM, two principal autonomous agent types are instantiated: store and customer agents. Each fulfils a distinct behavioural role while participating in a shared spatial–temporal environment, and utilizes distinct RL-based decision-making networks, as shown in Figure S2 in the Supplementary Material. There are 37 store agents and 1,500 customer agents in the simulated environment. This sample size was determined through an analysis, as shown in Appendix A.1.1. Agents are drawn via stratified sampling to preserve the proportional representation of all seven behavioural clusters, while the nonlinear scaling of agent-environment interactions makes larger populations computationally prohibitive for the multi-framework comparison required by this study. The stores are the actual locations of an established grocery chain, and the customer agents are initialized from a stratified sample of 295,631 customers in the synthetic dataset created by combining loyalty card program data from a food retail chain and sociodemographic data (Zhang et al., 2025). In this paper, customers are classified into seven typologies with income, shopping frequency, physical distance to the nearest store, and weekend & evening shopping proportion, coded from 0 to 6, as described in Table 1. The rationale for choosing these attributes inherits from Sturley et al. (2018), where customers are also clustered with these attributes from loyalty card data, and these classified groups are used to represent the heterogeneity among customer behaviours. Both types of agents perceive their shared environment independent of each other, and take individual actions to gain and maximize rewards.

Table 1.

Customer typology and descriptive names.

Cluster	Name
0	The frequent store explorers
1	High-spending weekend shoppers
2	Near evening shoppers
3	High income shoppers
4	Weekday day shoppers
5	Far infrequent shoppers
6	Minimal engagement shoppers

Store agents correspond one-to-one with the physical grocery stores contained in the loyalty dataset. Lacking detailed data on promotional strategies, this study limits store attributes to location and area. Therefore, the store agents are trained via a pure online learning framework, without relying on real-world trajectories. Their objective is not to replicate complex real-world decisions, such as centralized chain-level promotional strategies, but solely to maximize their own rewards. Conceptually, a store agent embodies the decision-making of a retail manager who revises pricing policy at every 4 weeks, which is approximately a month. The agent maintains an internal state vector comprising: (i) sales during the current segment, (ii) a rolling 7-day aggregate of sales, (iii) the magnitude of any active price discount, and (iv) a categorical flag representing the prevailing promotion.

At each timestep, the agent selects an action from a discrete set of four choices: normal pricing, every day low pricing (EDLP), high–low promo of a monthly duration, or inaction to continue the current strategy. Specifically, EDLP strategy allows the store agent implement a continuous lower price than the normal pricing strategy, while Hi-low promo strategy have a higher promotion rate at specific duration, and then return to the normal price. In this study, the promotion rates of EDLP and Hi-low are set to 10% and 30% based on the estimation of previous studies (Hoch et al., 1994). Both pricing strategies incorporate a 4-week cooldown period. In the EDLP scenario, the store maintains a 10 % discount throughout all 4 weeks, whereas under the Hi-Low strategy it runs a 1-week sale followed by 3 promotion-free weeks. In the code, store agents are evaluated at each timestep; however, when they are in a cooldown phase, the agents are forced to inaction.

Customer agents represent individual loyalty-card holders derived from the empirical data. Each agent is initialized with data-driven profiles, represented in Table S1 in the Supplementary Materials, average income, intended shopping frequency, distance to the nearest store, weekend propensity, evening propensity, and placed in one of seven behavioural clusters, together with extrinsic context variables capturing day-of-week and intra-day time segment.

At every time segment, a customer observes its eight-dimensional state vector and takes a decision to shop (or not), and also decide where to shop. If a shopping trip is undertaken, the agent bootstraps an expenditure amount from a calibrated distribution conditioned on its cluster identity and the current time slot, thereby injecting empirically grounded heterogeneity into shopping behaviour. After the transaction, the agent updates its visitation log, registers the spending with the store, and returns home at the end of the time segment.

Rewards

For store agents, sales generated by visiting customers accrue continuously and form the sole reinforcement signal used to evaluate the action as R_storei(t)

R_{s t o r e i (t)} = \sum_{k = m a x (1, t - 27)}^{t} s_{i} (k)

(1)

In equation (1), R_storei(t) refers to the reward of store i at timestep t, while $s_{i} (k)$ denotes the total spending of all customers who visit store i at timestep k. Hence, the reward is calculated through a floating window of weekly sales. Therefore, the store agent therefore enacts an adaptive pricing heuristic whose long-run profit is assessed against a short-term profit-oriented objective.

The immediate reward that a customer receives in the simulation is built on top of the classic Huff retail gravity model so that learning signals remain interpretable in terms of spatial-interaction theory. Huff (1964) proposed a simple way to estimate the share of shoppers that any store captures from each residential area. In its most common form the utility of store i for customer (or zone) n is

U = \frac{A_{i}}{{d_{n i}}^{λ}}

(2)

where A_i is an attractiveness measure (usually floor area to represent the variety of available items), d_ni is the distance or travel time from the customer n to the store i; λ > 0 controls how fast the utility function decays with distance. While the Huff model was proposed in 1964, and modern consumers exhibit complex micro-level shopping behaviours, such as multi-purpose trip-chaining and infrequent long-distance travel to large stores, traditional gravity-based spatial models like the Huff model remain highly robust and valid at the macroscopic and aggregate levels (Zhong et al., 2025). Specifically, the macro-level validity of the Huff model has been recently empirically validated using large-scale, real-world transactional datasets. For instance, Suhara et al. (2021) demonstrated through millions of credit card transactions that gravity-based market share models remain highly viable assumptions for retail sales estimation. Furthermore, standard Huff models have also demonstrated high accuracy in capturing macro-level distributions utilizing massive smartphone mobility data across major cities (Arhipova et al., 2020; Liang et al., 2020; Tan et al., 2023). Crucially, within the context of multi-agent systems and simulation environments, the Huff model is actively employed as a mathematically tractable and highly interpretable mechanism-driven engine for destination choice (Gong et al., 2023; Zhong et al., 2025). With this spatial interaction value, we provide a controlled simulation environment to isolate and evaluate the performance of our RL algorithms. In our simulation, the core element of the Huff model (i.e., store size and distance decay) are retained and then extended to reflect behavioural details found in loyalty-card data:

R_{customer n} (t) = \{\begin{cases} (α \frac{\log (1 + S_{i})}{d_{n i}^{β}} + σ_{i} (t)) m_{c, t} - κ_{1} 1 \{Δ τ_{n} (t) < τ_{cool}\}, & if n s h o p s a t i, \\ 0, & if n d o e s n^{'} t s h o p . \end{cases}

(3)

In the reward equation (3), S_i captures a store’s selling-floor area and thus plays the same “attractiveness” role that A_i holds in the classic Huff model. α is a parameter that controls the importance of the classic Huff’s value, and its magnitude. The log transformation is used to standardize the area of store. σ_i(t) injects a time-varying price incentive, which is calculated by multiplying the discount rate and the spending, thereby extending Huff’s static attractiveness term. From the customer side, m_c,t represents a normalized multiplier based on specific timeslots and clusters that controls the variance among the typology of customers in their shopping behaviours, which is generated from the empirical dataset. Spatial interaction is governed by the Euclidean distance d_in between the customer’s home and the store, which is raised to the distance-decay exponent β, directly paralleling Huff’s decay parameter λ. Beyond the gravity core, κ imposes a penalty whenever the customer shops again within a cool-down window of τ_cool = 28 slots (7 days); the window is detected via Δτ_n(t), the number of slots that have been past since the customer’s previous purchase.

RL designs

Three different RL modes: online, offline, and O2O options were implemented for customer agents. For store agents, since no real-world data for their promotional strategies are available to us, all stores only used the online RL learning mode. The online option refers to a “pure RL” mode, in which agents interact directly with the model environment, and the RL model can be trained directly with the online sampled transitions. In this study, customer agents only use the interactions in the simulated ABM to update their behaviour models. In contrast, in the offline mode, agents learn and update the RL model from empirical, real-world data. In our case, this refers to the customer behaviours in the loyalty card data. O2O RL, hence, is an alternative that attempts to combine the advantages of online and offline RL together by pre-training the agent with offline transitions first, and then fine-tuning it with online experiences through interactions. Specifically, customer agents are pre-trained with the transitions in loyalty card data first, and update their strategies with the interactions in the online simulated ABM. For comparison, all these options adopt the advantage-weighted actor-critic (AWAC) algorithm, which is designed for O2O RL, but can also be updated in online or offline settings (Nair et al., 2021). The AWAC approaches are an updated version of advantaged actor-critic (A2C), where agents are trained in two networks: the actor network for policy and the critic network for value estimation. Compared with A2C, AWAC use a buffer β to store all offline transitions (s, a, s’, r) (state, action, next state, reward) at the beginning, and then train the weighted actor and critic network, π_θ, Q_ϕ, through the following steps:

\begin{gathered} y = r (s, a) + γ E_{a^{'} \sim π_{θ_{k - 1}}} [Q_{ϕ_{k - 1}} (s^{'}, a^{'})] \\ ϕ \leftarrow \arg \min_{s, a \sim β} {(Q_{ϕ} (s, a) - y)}^{2} \\ θ \leftarrow \arg \max_{s, a \sim β} [\log π_{θ} (a ∣ s) \exp (\frac{1}{λ} A_{ϕ} (s, a))] \\ A_{ϕ} (s, a) = Q_{ϕ} (s, a) - V_{π} (s) \end{gathered}

(4)

In each training iteration of AWAC, we first sample a transition (s, a, r, s’) from the replay buffer β and form the one-step temporal-difference target y. Parameters ϕ of the critic network are then updated by minimizing the MSE between Q_ϕ(s, a) and y. Once the critic is fitted, we estimate the advantage A_ϕ(s, a). Finally, parameters θ of the actor network are updated with a weighted maximum-likelihood objective that treats the replay actions as supervised labels but scales each log-likelihood by exp(A_ϕ(s, a)/λ). Actions with positive advantage therefore exert exponentially greater influence, while disadvantageous actions contribute negligibly. The temperature λ governs this trade-off, with smaller values yielding greedier, more exploitative updates. By iterating these critic and actor steps and periodically refreshing target networks, AWAC smoothly combines offline and online data with advantage-weighted policy improvement in the replay buffer β.

To clarify how agents map their perceived inputs to decisions, it is important to note that both the actor policy π_θ and the critic value function Q_ϕ in our AWAC framework are parameterized by artificial neural networks (ANNs). Specifically, the 8-dimensional state vector of a customer agent (e.g., income, distance, specific timeslots) serves as the input layer to the network. The hidden layers of the ANN process these inputs to capture complex relationships between customer demographics and spatial-temporal contexts. The output layer of the actor network produces a probability distribution over the available discrete actions (e.g., choosing a specific store or opting not to shop), while the critic network outputs a scalar estimating the expected cumulative reward. Through the AWAC loss functions described above, the neural networks continuously adjust their weights via backpropagation, learning to output higher probabilities for actions that yield higher spatial-interaction rewards. Over the course of training, this process causes agents to gradually shift from random action selection toward context-specific strategies.

AWAC provides an opportunity for online, offline, and online-to-offline options by adjusting the replay buffer β. Figure S3 in the Supplementary Material illustrates the differences of processes in online, offline, and online-to-offline modes. In the online mode: the ABM rolls forward, the current policy drives every action, and the resulting transitions are streamed into an online buffer that is immediately replayed to update the policy, so learning and simulation proceed in lock-step. While in the offline mode, the policy is trained once on a fixed offline buffer derived from real-world datasets, and the ABM then queries this frozen policy for decisions. O2O training starts from the same offline trained model yet allows the running ABM to append fresh online transitions to the buffer, coupling the sample efficiency and real-world information of offline RL with the continual adaptation of online RL. Crucially, in all three modes, each training step samples an identical fixed-size batch from the replay buffer. The three frameworks therefore differ not in the quantity of data processed per update, but in the composition of each batch: purely simulated transitions, purely empirical transitions, or a decaying mixture of both.

Results

Comparison of online, offline, O2O modes

Rewards comparison

Figure 1 delineates the performance trajectories of customer agents, measured by average reward, within an ABM simulated over 8,000 timesteps. The value of the reward is therefore a composite score that reflects how well a customer’s store choice aligns with the spatial-interaction logic of the Huff model, given the customer’s typology, the time of day, and any active promotions. A higher reward indicates that the agent is consistently selecting stores that offer a favourable combination of proximity, size, and promotional incentives relative to the agent’s profile. The analysis contrasts three distinct learning paradigms: pure online, pure offline, and a O2O modes. Each subplot displays the performance of seven discrete customer clusters, which is inherent from the typology in Zhang et al. (2025), and the aggregate average. To mitigate high-frequency oscillations caused by the agents’ epsilon-greedy exploration strategy, the visualized metric represents the 95th-percentile upper envelope of the reward, computed over a 100-timestep rolling window.

Figure 1.

The upper envelope of average customer reward per timestep through simulation.

A consistent trend across all three modes is the stable ranking of customer clusters based on their average rewards. Cluster 2 consistently achieves the highest rewards, significantly outperforming all other groups. Clusters 0, 3, and 4 exhibit average rewards, maintaining a consistent grouping. At the lower end, clusters 1, 5, and 6 show markedly lower rewards, with cluster 6 remaining close to zero throughout all simulations. This persistent stratification suggests that the inherent characteristics of each cluster are a primary determinant of their reward potential, regardless of the learning mode.

The different learning modes, however, result in clearly different performance dynamics. As expected, the pure offline model (based on real-world data) shows a static performance, with the overall average reward remaining on a flat plateau around 400 for all clusters. This is because its decision-making strategy is pre-trained on a fixed dataset and is not updated during the simulation, preventing any new learning. In contrast, the pure online model demonstrates a distinct learning curve. Starting from a cold start value of approximately 50, the average reward shows a continuous increase, reaching to about 200 by the end of the simulation. This upward trend confirms that the agents are successfully improving their strategies through ongoing interaction with the environment.

The O2O mode combines features of both approaches and delivers the most complex, yet ultimately the best, performance. This model starts with a strong initial average reward of around 300, thanks to its pre-trained policy obtained from real world transactional data. However, it then experiences a temporary drop in performance, hitting its lowest point of about 100 around the 1,500th timestep. This initial dip is caused by a “policy-environment mismatch,” as the policy trained on historical data adjusts to the live, unpredictable simulation environment. After this adaptation period, the agent begins to learn effectively, and the average reward rises steeply, eventually stabilizing at approximately 550. This final performance level is considerably higher than those achieved by either the pure online or pure offline modes individually. This result highlights the significant advantage of the O2O approach, which uses historical data to build a strong foundation while retaining the flexibility of online learning to achieve a more optimal outcome.

Calibrating the ABM to actual customer behaviour

This section evaluates the behavioural calibration of the ABM by comparing key emergent attributes of the simulated customer agents against the patterns observed in the real-world source data. The comparison, detailed in Table 2, focuses on three attributes that are significant components of the agent reward function: average shopping distance, percentage of evening shopping, and percentage of weekend shopping. Shopping frequency, though also a factor in the reward multiplier, was excluded from this comparative analysis. The RL framework inherently encourages exploration to maximize long-term rewards, which is fundamentally at odds with preserving a pre-defined shopping frequency observed in historical data. Furthermore, replicating the exact numerical values of behaviours is challenging. Therefore, instead of pursuing a direct numerical match, our validation focuses on the relative behavioural patterns across different customer clusters. The replicated rate represents the proportion of the seven clusters for which the simulated behavioural category (low, medium, or high) matches that observed in the empirical source data, following the relative-pattern validation approach. This approach, which assesses whether the model captures the relative differences between groups, is similar to the ABM calibration practices in Sturley et al. (2018)’s work.

Table 2.

Comparison of customer behavioural attributes by cluster.

Cluster	Average shopping distance				Percentage of evening shopping				Percentage of weekend shopping
Cluster	Data	Online	Hybrid	Offline	Data	Online	Hybrid	Offline	Data	Online	Hybrid	Offline
0	Medium	Low	Low	Medium	Medium	High	Medium	Medium	Medium	Medium	Medium	Medium
1	Medium	Medium	Medium	Medium	Low	Medium	Low	Low	High	Medium	Medium	High
2	Low	Low	Low	Low	High	Medium	High	High	Medium	Medium	Medium	Low
3	Medium	Medium	Medium	Medium	Medium	Medium	Medium	Medium	Medium	Medium	Medium	Medium
4	Low	High	Medium	Low	Low	Low	Low	Low	Low	Medium	Medium	Medium
5	High	High	High	High	Low	Low	High	Low	Low	Medium	Medium	Medium
6	High	High	High	High	Medium	Low	Medium	High	Medium	Medium	Medium	Medium
Replicated rate	-	71.43%	71.43%	100%	-	42.86%	85.71%	85.71%	-	57.14%	57.14%	57.14%

The analysis reveals varying levels of success in replicating real-world behaviours across the three models. The average shopping distance is the attribute most accurately reproduced. The pure offline model achieves a perfect 100% replication accuracy, directly mirroring the source data it was trained on. The online and O2O models also demonstrate strong fidelity, matching the real-world patterns with an accuracy of 71.43%.

For percentage of evening shopping, a notable divergence appears between the learning-enabled models. The pure online model, learning from scratch, struggles to capture every real-world tendency, showing a low replication rate of 42.86%. In contrast, both the O2O and pure offline models replicate this behaviour with high accuracy (85.71%). This success is likely because the training data provided to these models contains clear temporal patterns associated with evening shopping activity, which they are able to learn and reproduce.

A significant discrepancy arises in replicating the percentage of weekend shopping percentage. None of the three models successfully reproduce the patterns from the source data, with all simulations showing a uniform, medium-level tendency for this attribute. An explanation lies in the model’s temporal structure. The “evening” period is represented as a single, discrete timestep (the third of every four timesteps), providing a clear and immediate signal for the learning algorithm. Conversely, the “weekend” pattern is more complex and distributed over time (e.g., occurring every two out of 7 days, which has to be further translated into continuous timesteps). This less distinct signal may be too subtle for the agents to effectively identify and base their strategy upon, even when it is present in the initial training data. To improve performance in this area, several enhancements could be explored in future work:

(1) Enriching the state space with explicit temporal features such as a binary weekend flag, making the weekly periodicity directly observable to the agent rather than requiring it to be inferred;

(2) Incorporating recurrent or memory-augmented neural network architectures in the actor-critic networks, enabling agents to capture longer temporal dependencies;

(3) Extending the reward function to include an explicit temporal-preference component that rewards agents for shopping during time periods consistent with their cluster profile.

Spatial popularity and competition of stores

Figure 2 provides a geospatial visualization of the competitive landscape, illustrating the dominant store within different areas for each of the three learning modes. To map this spatial market share, the region was divided into 500-m hexagonal cells. For each cell, the colour corresponds to the store that captured the highest proportion of shopping trips originating from customers within that hexagon.

Figure 2.

Spatial dominance of stores across the Montreal city under three learning frameworks: (a) online, (b) offline, and (c) O2O. The study area is partitioned into 500 m hexagonal cells. Each cell is coloured according to the store that captured the highest proportion of customer shopping trips originating from within that cell; each unique colour represents one of the store agents.

A common pattern is observable across all three models: stores naturally establish localized catchment areas, dominating the hexagons in their immediate vicinity. This reflects a fundamental principle of spatial economics, where proximity is a key driver of consumer choice. However, the models reveal differences in the scale and nature of this dominance, directly reflecting their underlying learning mechanisms.

The online model results in a landscape characterized by large, contiguous zones of influence, where a few stores achieve dominance over vast geographic areas. Only 8 out of 37 stores win at least one hexagon cell. This “winner-take-all” pattern is the spatial manifestation of unconstrained online learning. Customer agents, through exploration, identify a locally optimal store and their collective behaviour converges on this choice, creating exaggerated monopolies that may not reflect a balanced, real-world competitive market. This pattern is not an artefact of the spatial metric, but rather reflects unconstrained online exploration on locally optimal choices without the moderating influence of empirical behavioural heterogeneity that pre-trained models inherit, since the same Euclidean distance is used as the spatial metric across all three frameworks.

In contrast, the offline model produces a highly fragmented market structure. The catchment areas are significantly smaller and more tightly constrained around each store. This is demonstrated by the fact that 34 of the 37 stores emerge as the dominant choice in at least one cell. This pattern is a direct reflection of the real-world data it was trained on, which captures the complex and diverse preferences of an existing customer base in a mature market, resulting in a more balanced spatial distribution that mirrors the empirical dataset.

The O2O model presents a hybrid spatial outcome that bridges the other two outcomes. 23 out of 37 stores are winners on the map. It largely inherits the fragmented market structure from its offline training, but shows clear evidence of online adaptation. For instance, certain stores (e.g., the circled mint-coloured store on the right) have visibly expanded their catchment areas compared to the offline model. This indicates that while the model retains a realistic baseline from its offline training, the online learning component has allowed customer agents to identify and increasingly favour this particular store, resulting in an emergent expansion of its catchment area at the expense of neighbouring stores’ market share. This spatial reorganization is consistent with the reward trajectories shown in Figure 1(c), where the O2O model’s average customer reward rises from its offline baseline of approximately 300 to a final value of approximately 550, confirming that online adaptation is actively reshaping agent store choices and, consequently, the spatial allocation of market share.

Figure 3 provides a deeper analysis of the competitive landscape by mapping the intensity of market competition across the study region. The metric used is the margin of victory, defined as the difference between the market share of the winning store and the second-place store within each hexagonal cell. The resulting heatmap visualizes the competitive tension: hot zones indicate areas where a single store is highly dominant, while cold zones represent areas of intense competition where market share is closely contested between several stores.

Figure 3.

Margin of victory in market share across the Montreal metropolitan area under three learning frameworks: (a) online, (b) offline, and (c) O2O. Each 500 m hexagonal cell is coloured by the difference in market share between the first place and second place store within that cell. Red cells indicate areas of strong single-store dominance; blue cells indicate intense competition where market share is closely contested.

The three models present profoundly different portraits of emergent spatial patterns dictated by their respective training modes. The online model displays a highly polarized landscape. It is characterized by large, deep-red zones of absolute dominance directly bordering deep-blue zones of intense competition. This spatial pattern reflects the model’s “winner-take-all” nature: in areas where one store provides a clearly superior reward, it captures the entire market; at the precise boundaries between the catchment areas of two strong stores, the competition becomes a near statistical tie.

The offline model, in contrast, shows a much more balanced competitive environment. The map is predominantly composed of neutral and low-intensity cells. This suggests a market where outright dominance is rare and most areas are characterized by moderate competition. This model setting prevents any single store from achieving total dominance, even in its core territory. The O2O model once again illustrates a dynamic, hybrid scenario. While retaining the generally balanced structure of the offline model, it shows the emergence of distinct competitive hotspots. Compared to the offline map, several areas have intensified to become hotter red or colder blue. The appearance of new red zones indicates that some stores, through online adaptation, have successfully solidified their dominance in their core territories, increasing their margin of victory. This map effectively visualizes a simulated environment in transition, where online optimization is actively reshaping the baseline spatial distribution.

Store strategy

Regardless of the customer learning framework, the store agents’ strategies are consistently updated using an online RL mode. This approach was adopted due to the low quality of calibrated data on store promotion. Although the different customer learning frameworks generate distinct market environments, the resulting strategic evolution of the stores demonstrates a remarkably consistent pattern across all three settings, as illustrated in Figure 4.

Figure 4.

The ratio of promotion strategy selected by stores.

Figure 4 displays the proportion of stores adopting each of the three available strategies, normal pricing (“none”), EDLP, high–low promo of a duration and no promotion, over the simulation’s 8000 timesteps. The three subplots correspond to the environments created by customers trained under (a) online, (b) offline, and (c) O2O modes.

Across all scenarios, the system exhibits an initial transient phase of high volatility in approximately the first 1,000 timesteps, which reflects the agents’ initial exploration of the strategy space. Subsequently, the strategy distribution converges to a dynamic equilibrium. Notably, the EDLP strategy emerges as the predominant choice, consistently adopted by approximately 50%–60% of the stores. Normal pricing strategy maintains a secondary position, while the high–low strategy is the least frequently selected strategy. The striking similarity in the final distribution and fluctuation of strategies suggests that the stores’ online learning process identifies a robust optimal equilibrium within the constrained rules of the simulation, largely independent of the specific learning model governing customer behaviour.

Model robustness testing

Our model employs an O2O reinforcement learning framework, where agent behaviour representing customers is guided by an offline pre-trained model that undergoes continuous online updating. Since the offline model’s parameters are directly learned from a fixed dataset, its components are not subjected to sensitivity analysis. Instead, our analysis focuses on the key parameters of the reward function governing the online learning stage.

Specifically, we investigate three parameters: the Huff model magnitude parameter, α; the distance decay exponent, β; and a “cool-down” period, τ_cool, which imposes a constraint to mitigate excessive purchasing behaviours. A comprehensive sensitivity analysis, such as a full factorial exploration of the parameter space, presents a significant computational burden. However, even with this reduced sample size, a single simulation run requires approximately 16 hours. As detailed in the computational profiling analysis in Appendix A.1.1, this cost is dominated by three bottlenecks: the neural network training step, the forward pass through the actor network for all 1,500 agents at each timestep, and the spatial agent-environment interactions that scale nonlinearly with population size. Conducting 30 runs (3 frameworks × 10 random seeds) therefore required approximately 480 hours of total compute time, which makes the full exploration of parameter space computationally prohibitive. Given these constraints, we adopted the one-factor-at-a-time (OFAT) methodology. While OFAT cannot enable the examination of interaction effects between parameters, it allows for an effective assessment of the model’s robustness to variations in individual parameters. For this analysis, we established a baseline parameter set of α = 60, β = 2, and τ_cool = 28 (1 week). The analysis proceeds by systematically varying one parameter while holding the others constant at their baseline values.

The results of the OFAT analysis are presented in Table 3. The analysis evaluates the model’s response by observing three metrics: the coefficient of variation (CV) of cells per winning stores, the mean margin of victory, and the Moran’s I of margin.

Table 3.

Results of OFAT analysis.

Type	CV of cells per winning stores	Mean of margin	Moran’s I of margin
Base	0.5738	0.3045	0.5577
α = 10	0.6019	0.2351	0.6203
α = 30	0.5039	0.2791	0.5813
α = 90	0.5706	0.3170	0.6295
α = 100	0.5734	0.3085	0.6067
β = 0.5	0.9408	0.1501	0.7617
β = 1	0.5722	0.2336	0.6488
β = 3	0.5216	0.2704	0.5703
β = 3.5	0.4782	0.2498	0.6061
τ_cool = 4 (1 day)	0.6376	0.3106	0.5994
τ_cool = 120 (1 month)	0.5777	0.2953	0.5749

The model is quite robust to the Huff magnitude parameter, α. When α is increased from the baseline, the mean margin of victory and other metrics remain very stable. When α is lowered, the margin of victory decreases. This indicates that when the huff’s value are less important to store promotion, their choices are more evenly split among the top competitors within a cell, leading to tighter local competition and less decisive wins.

The model is extremely sensitive to the distance decay exponent, β. This parameter fundamentally alters the competitive landscape. Decreasing β to 0.5 reveals a fascinating dynamic. The CV nearly doubles (0.9408), indicating a “winner-take-all” market at the global level, with a few super-stores capturing most of the customers. However, the mean margin of victory plummets to 0.1501. This signifies intense hyper-competition at the local level. When distance is not a barrier, any given cell is contested by multiple, equally attractive distant super-stores, heavily distributing the market share of any single winner within that cell. In addition, a high Moran’s I indicates the patterns of local competition are highly clustered in space. Increasing β to 3 and 3.5 results in a slightly lower margin of victory compared to the baseline. This suggests that in a highly localized market, cells are often situated between several nearby stores, leading to consistently tighter local competition than the baseline.

The model remains highly robust to the purchasing cool-down period, τ_cool. Both shortening the period to 1 day and lengthening it to a month result in metrics that are very close to the baseline. This implies that the frequency of agent purchases (within this range) does not significantly disrupt the stable patterns of local competition and overall market structure.

Thus, sensitivity analysis confirms that the model’s stability is not uniform across its parameters. The simulation is robust concerning Huff’s value magnitude and limitation of purchasing frequency. However, the model’s outcomes are critically dependent on the distance decay exponent. This parameter governs the interplay between global market concentration and local competitive intensity. The key insight is that a low friction of distance creates a market that is simultaneously globally monopolistic and locally hyper-competitive. Therefore, in this study, the baseline setting is applied in the comparison of three learning modes, since β = 2 is widely accepted in classic Huff’s value and gravity model.

Discussion and conclusion

This study compared the O2O RL framework with pure online and offline learning models to determine the most effective approach for spatial agent-based simulations. We answered our central research question by successfully completing three sub-objectives, with the findings consistently highlighting the superiority of the hybrid model.

Our first objective was to evaluate the learning performance of the three frameworks by comparing agent rewards. Because the training step samples are composed of a decaying mixture, it allows the hybrid offline-to-online mode to take the best of both of the other two modes (i.e., purely offline and purely online) and offer the highest performance. The results show that the hybrid model is the most effective at maximizing long-term rewards. While the pure online model learned slowly from a “cold start,” and the offline model’s performance was static, the hybrid model leveraged its pre-training for a strong start. After a brief adaptation period, its online learning component allowed it to optimize its strategy and achieve the highest reward levels, demonstrating a superior capacity for efficient learning.

The second objective was to investigate each model’s capacity to replicate real-world behaviours. Here, our findings underscore the critical importance of pre-training with empirical data. Both the offline and hybrid models successfully reproduced known shopping patterns from available real-world data, such as specific evening shopping tendencies. The pure online model, lacking this foundation, failed to capture these patterns. This confirms that to build behaviourally realistic agents, grounding them in empirical data is essential. This is because the simulation environment’s reward signal, while capturing key spatial-interaction drivers, does not encode every real-world behavioural mechanism, making offline data a complement that injects information the model’s mechanics alone cannot generate.

Our third objective was to examine the emergent spatial competition produced by each framework. The three models generated profoundly different market structures. The online model resulted in “winner-take-all” monopolies, while the offline model produced a highly fragmented landscape. The hybrid model presented a balanced pattern between the online and offline modes.

Furthermore, we examined the stores’ policy choices under each customer learning framework. Although the store strategies were learned purely online, the results show that EDLP and normal pricing policies were overwhelmingly preferred by store agents over Hi-Low strategy, regardless of how the customer agents learned. This finding may suggest that customer agents are more responsive to immediate rewards and are less sensitive to the long-term trade-offs presented by a Hi-Low strategy. This aligns with the model’s difficulty in replicating weekly shopping frequencies, as agents may not have effectively learned indirect and longer-term temporal patterns.

However, it is noteworthy that this algorithmic short-sightedness effectively parallels empirical findings regarding how human shoppers process price uncertainty and cognitive load. Modern retail literature demonstrates that human shoppers frequently exhibit a strong preference for the immediate certainty of EDLP to avoid the cognitive friction of temporal tracking demanded by cyclical discounts. Drawing upon loss aversion theory (Kahneman and Tversky, 1979), Hydock and Wathieu (2023) show that consumers weigh the pain of purchasing a non-discounted item more heavily than the pleasure of obtaining a discount, which drives a systematic preference for EDLP among consumers with rigid product preferences. Consequently, to minimize cognitive load when directly comparing competing stores, consumers often bypass the complex reference price tracking required by Hi-Low environments in favour of the heuristic ease provided by EDLP (Sheehan et al., 2022). Therefore, while the agents’ reliance on immediate rewards is a limitation of the current learning setup, it organically mirrors the behavioural friction that drives actual store managers to adopt EDLP to capture certainty-seeking consumers. Nevertheless, enhancing the model’s sensitivity to long-term attributes to fully capture complex promotional strategies remains an important area for future investigation.

By addressing these research objectives, this study confirms the value of the hybrid O2O framework. It successfully balances the adaptability gained from online interactions with the realism provided by offline pre-training. Crucially, after a short adaptation period, the hybrid framework can achieve a higher reward within the same number of timesteps than the pure online model, signifying a distinct advantage in learning efficiency. Therefore, this hybrid framework offers modellers a powerful new method that simultaneously accommodates both the learning capacity of agents and the integration of historical real-world information.

This framework also possesses high transferability and can be widely applied to other spatially-explicit ABMs. Modellers can retain the core learning architecture while adjusting the state, action spaces, and reward functions to suit specific applications. Within this structure, agents can autonomously develop behavioural strategies from both offline data and online interactions. Consequently, it provides an effective solution for real-world applications such as digital twins, where both real-time data and dynamic adaptation are critical.

However, the framework faces several limitations that affect its generalizability. The primary challenge is computational cost. To mitigate the computational burden in this study, we sampled 1,500 customers from a population of 295,631 loyalty card members. Even with this reduced sample size, a single simulation run requires 16 hours or more. This extensive runtime severely restricts our ability to conduct more complex sensitivity analyses or perform large batch simulations. A second limitation arises from the model’s strong dependency on the attributes present in the real-world dataset. The agent-based model’s state space is necessarily derived from the available offline data, which makes it difficult to incorporate other relevant states and actions. For instance, including transportation choices could enhance model realism. However, due to the absence of this information in the source data, we used Euclidean distance instead of the more representative commuting time. Similarly, the lack of data on store-level marketing strategies constrained us to the current modelling framework, while in the real world, the chain of stores adopts a more centralized promotional strategy, potentially allowing individual stores to operate as loss leaders to maximize overall corporate profit. In our simulation, store agents act as independent entities maximizing local profit, and their pricing rules act as theoretical approximations designed primarily to create a dynamic environment for customer adaptation. While this simplifies the store-side mechanics, it fulfils the methodological purpose of testing customer learning algorithms against a shifting landscape. Future research with access to data from multiple chains could replace these theoretical rules with highly calibrated, chain-level decision networks. Finally, while this approach simplifies the design of agent behaviours, the definitions of the state space, action space, and reward function warrant further investigation. In this study, we incorporated a modified Huff’s model and store discounts to represent factors of the 4Ps marketing mix. In other contexts, such as modelling the adoption of health foods, the reward function would need to be redesigned to focus on specific food categories and nutritional compositions. This adaptability provides modellers with design flexibility, but it also requires them to provide a clear and robust justification for their specific design choices.

In conclusion, this research offers more than a model of retail choice; it presents a case for a more integrated approach to ABM. By successfully merging the empirical grounding of real-world data with the adaptive power of RL, the O2O framework offers a promising pathway toward developing more robust and balanced simulations of complex socioeconomic systems.

Supplemental material

Supplemental Material - Bridging data and adaptation in agent-based modelling: A comparative reinforcement learning approach to urban food retail choice

Supplemental Material for Bridging data and adaptation in agent-based modelling: A comparative reinforcement learning approach to urban food retail choice by Duo Zhang, Catherine Paquet, Laurette Dubé, and Raja Sengupta in Environment and Planning B: Urban Analytics and City Science.

Footnotes

ORCID iD

Duo Zhang

Funding

The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This research was funded by the Natural Science and Engineering Research Council (NSERC) of Canada Discovery Grant number RGPIN-2022-04342 awarded to R.S., by the Canadian Institute of Health Research (CIHR) Grant number 02083-000, 2021-2027 awarded to L.D., and Computational and Data Systems Institute, Faculty of Science, McGill Collaborative for AI and Society (McCAIS) Grant to R.S.

Declaration of conflicting interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Data Availability Statement

The source data is subject to Non-Disclosure Agreement, while the processed data and codes are available on request.

Supplemental material

Supplemental material for this article is available online.

Author biographies

Duo Zhang is a PhD candidate in Geography at McGill University, working under the supervision of Prof. Raja Sengupta. His research explores the intersection of geographical information science, complex systems, artificial intelligence, and human decision-making. He is particularly interested in integrating individual-level intelligence into interactive environments to simulate dynamic socioeconomic systems. His work leverages an interdisciplinary toolkit that includes geospatial agent-based modelling, reinforcement learning, and large language models, with applications in fields such as marketing and epidemiology. Prior to his doctoral studies, he worked as a researcher specializing in natural resources economics and public policy.

Catherine Paquet is associate professor and department chair in Marketing at Université Laval. Her research investigates the role of community and organisational factors in shaping health related behaviours and how such factors can be used to promote optimal health across the lifespan. She brings to her research her unique multidisciplinary training and research experience in physiology, consumer psychology, health services marketing, socio-spatial epidemiology, and population health.

Laurette Dubé is Emerita Professor and James McGill Chair of Consumer and Lifestyle Psychology and Marketing at McGill University. She is the Founding Chair and Scientific Director of the McGill Centre for the Convergence of Health and Economics (MCCHE). Her disciplinary research examines the behavioural, economic, and social drivers of consumption, lifestyle, and health. In her transdisciplinary work, she focuses on translating this knowledge into scalable solutions for behavioural change and societal transformation at scale.

Raja Sengupta is an Associate Professor in the Department of Geography & Bieler School of Environment at McGill University. His research work falls broadly in the area of Environment and GIS, with specialization in complexity theory and the development of Agent Based Models (ABMs) of both human and primates to represent their behaviours and environmental outcomes. He has worked both on Agent-Based Models as well as dynamic visualization and has therefore developed ABMs for a variety of purposes, including those relating to historical movement. He is also currently researching the use of low-cost sensors for identifying spatial patterns of intra-city Urban Heat Islands.

References

(2012) Modeling human decisions in coupled human and natural systems: review of agent-based models. Ecological Modelling 229: 25–36. https://doi.org/10.1016/j.ecolmodel.2011.07.010

Arhipova

Berzins

Erglis

, et al. (2020) Huff model for shopping centre assessment using aggregated mobile phone data. In: Proceedings of the 2nd International Conference on Finance, Economics, Management and IT Business, May 5, 2020, Prague, Czech, pp. 91–97. SCITEPRESS - Science and Technology Publications. https://doi.org/10.5220/0009361400910097

Armendariz

Armenia

Atzori

, et al. (2015) Analyzing food supply and distribution systems using complex systems methodologies. In: Proceedings in Food System Dynamics, February 9, 2015, Innsbruck-Igls, Austria, pp. 36–58.

Armstrong

Adam

Denize

, et al. (2014) Principles of Marketing. Pearson Australia.

Bone

Dragićević

(2010) Simulation and validation of a reinforcement learning agent-based model for multi-stakeholder forest management. Computers, Environment and Urban Systems 34(2): 162–174. https://doi.org/10.1016/j.compenvurbsys.2009.10.001

Choi

(2014) Lead products and retail store choice decisions. In: Revolution in Marketing: Market Driving Changes: Proceedings of the 2006 Academy of Marketing Science (AMS) Annual Conference, May 24, 2006; First Online: 01 January 2014, San Antonio, USA, p. 53. Springer.

Crooks

(2010) Constructing and implementing an agent-based model of residential segregation through vector gis. International Journal of Geographical Information Science 24(5): 661–675. https://doi.org/10.1080/13658810903569572

Dehkordi

MAE

Lechner

Ghorbani

, et al. (2023) Using machine learning for agent specifications in agent-based models and simulations: a critical review and guidelines. The Journal of Artificial Societies and Social Simulation 26(1): 9.

Gieschen

Sengupta

Zhang

, et al. (2025) Syntheco: an open-source python application for the generation of a multi-scale digital ecosystem anchored in synthetic populations. In: Practice and Experience in Advanced Research Computing 2025: The Power of Collaboration, July 20, 2025, Columbus, USA, pp. 1–5.

10.

Gong

Dong

Wang

, et al. (2023) Agent-based modelling with geographically weighted calibration for intra-urban activities simulation using taxi gps trajectories. International Journal of Applied Earth Observation and Geoinformation 122: 103368. https://doi.org/10.1016/j.jag.2023.103368

11.

Harati

Perez

Molowny-Horas

(2021) Promoting the emergence of behavior norms in a principal–agent problem—an agent-based modeling approach using reinforcement learning. Applied Sciences 11(18): 8368. https://doi.org/10.3390/app11188368

12.

(2023) A survey on offline model-based reinforcement learning. arXiv preprint arXiv:2305.03360.

13.

Heppenstall

Evans

Birkin

(2007) Genetic algorithm optimisation of an agent-based model for simulating a retail market. Environment and Planning B: Planning and Design 34(6): 1051–1070. https://doi.org/10.1068/b32068

14.

Heppenstall

Crooks

Malleson

, et al. (2021) Future developments in geographical agent-based models: challenges and opportunities. Geographical Analysis 53(1): 76–91. https://doi.org/10.1111/gean.12267

15.

Hoch

Drèze

Purk

(1994) Edlp, hi-lo, and margin arithmetic. Journal of Marketing 58(4): 16–27. https://doi.org/10.1177/002224299405800402

16.

Huff

(1964) Defining and estimating a trading area. Journal of Marketing 28: 34–38. https://doi.org/10.1177/002224296402800307

17.

Hydock

Wathieu

(2023) Not just about price: how benefit focus determines consumers’ retailer pricing strategy preference. Journal of Consumer Research 50(3): 447–467. https://doi.org/10.1093/jcr/ucad043

18.

Jalalimanesh

Haghighi

Ahmadi

, et al. (2017) Simulation-based optimization of radiotherapy: Agent-based modeling and reinforcement learning. Mathematics and Computers in Simulation 133: 235–248. https://doi.org/10.1016/j.matcom.2016.05.008

19.

Joubert

Saprykin

Chokani

, et al. (2022) Large-scale agent-based modelling of street robbery using graphical processing units and reinforcement learning. Computers, Environment and Urban Systems 94: 101757. https://doi.org/10.1016/j.compenvurbsys.2022.101757

20.

Kahneman

Tversky

(1979) Prospect theory: an analysis of decision under risk. Econometrica 47(2): 263–292. https://doi.org/10.2307/1914185

21.

Lee

Seo

Lee

et al.(2022) Offline-to-online reinforcement learning via balanced replay and pessimistic q-ensemble. In Conference on Robot Learning, November 8, 2021, London, UK, pp. 1702–1712. PMLR.

22.

Liang

Gao

Cai

, et al. (2020) Calibrating the dynamic huff model for business analysis using location big data. Transactions in GIS 24(3): 681–703. https://doi.org/10.1111/tgis.12624

23.

Ligmann-Zielinska

Vojnovic

LeDoux

(2025) Grocery shopping behavior in detroit neighborhoods experiencing disinvestment and decline: an empirically grounded agent-based model of data-scarce communities. Journal of Transport and Land Use 18(1): 559–592. https://doi.org/10.5198/jtlu.2025.2381

24.

Masad

Kazil

(2015) Mesa: an agent-based modeling framework. In: SciPy , July 6, 2015, Austin, USA, pp. 51–58.

25.

Nair

Gupta

Dalal

, et al. (2021) Awac: Accelerating online reinforcement learning with offline datasets. arXiv preprint. arXiv:2006.09359.

26.

Nakamoto

Zhai

Singh

, et al. (2023) Cal-ql: calibrated offline rl pre-training for efficient online fine-tuning. Advances in Neural Information Processing Systems 36: 62244–62269.

27.

Olmez

Birks

Heppenstall

, et al. (2024) Learning the rational choice perspective: a reinforcement learning approach to simulating offender behaviours in criminological agent-based models. Computers, Environment and Urban Systems 112: 102141. https://doi.org/10.1016/j.compenvurbsys.2024.102141

28.

Rashidinejad

Zhu

, et al. (2021) Bridging offline reinforcement learning and imitation learning: a tale of pessimism. In: Ranzato

Beygelzimer

Dauphin

, et al. (eds) Advances in Neural Information Processing Systems. Curran Associates, Inc., Vol. 34, pp. 11702–11716.

29.

Schenk

Löffler

Rauh

(2007) Agent-based simulation of consumer behavior in grocery shopping on a regional level. Journal of Business Research 60(8): 894–903. https://doi.org/10.1016/j.jbusres.2007.02.005

30.

Sengupta

Lant

Kraft

, et al. (2005) Modeling enrollment in the conservation reserve program by using agents within spatial decision support systems: an example from southern illinois. Environment and Planning B: Planning and Design 32(6): 821–834. https://doi.org/10.1068/b31193

31.

Sheehan

Hamilton

Chellappa

(2022) When is hilo low? Price image formation based on frequency versus depth pricing strategies. Journal of Consumer Research 49(4): 543–560. https://doi.org/10.1093/jcr/ucac005

32.

Song

Zhou

Sekhari

, et al. (2022) Hybrid rl: Using both offline and online data can make rl efficient. arXiv preprint. arXiv:2210.06718.

33.

Sturley

Newing

Heppenstall

(2018) Evaluating the potential of agent-based modelling to capture consumer grocery retail store choice behaviours. The International Review of Retail, Distribution and Consumer Research 28(1): 27–46. https://doi.org/10.1080/09593969.2017.1397046

34.

Suhara

Bahrami

Bozkaya

, et al. (2021) Validating gravity-based market share models using large-scale transactional data. Big Data 9(3): 188–202. https://doi.org/10.1089/big.2020.0161

35.

Sutton

Barto

(2018) Reinforcement Learning: An Introduction. Adaptive Computation and Machine Learning Series. 2nd edition edition. The MIT Press.

36.

Tan

Lau

(2023) A big data approach to augmenting the huff model with road network and mobility data for store footfall prediction. In: 2023 IEEE International Conference on Big Data (BigData), December 15, 2023, Sorrento, Italy, pp. 1533–1538. IEEE. https://ieeexplore.ieee.org/document/10386152/

37.

Wang

Hess

, and Crooks A (2022) Mesa-geo: a gis extension for the mesa agent-based modeling framework in python. In: Proceedings of the 5th ACM SIGSPATIAL International Workshop on GeoSpatial Simulation, November 1, 2022, Seattle, USA, pp. 1–10.

38.

Shen

, et al. (2022) Distributed agent-based deep reinforcement learning for large scale traffic signal control. Knowledge-Based Systems 241: 108304. https://doi.org/10.1016/j.knosys.2022.108304

39.

Zhang

Valencia

Chang

(2023) Synergistic integration between machine learning and agent-based modeling: a multidisciplinary review. IEEE Transactions on Neural Networks and Learning Systems 34(5): 2170–2190. https://doi.org/10.1109/TNNLS.2021.3106777

40.

Zhang

Dubé

Gieschen

, et al. (2025) Quantitative and spatially explicit clustering of urban grocery shoppers in montreal: integrating loyalty data with synthetic population. ISPRS International Journal of Geo-Information 14(4): 159. https://doi.org/10.3390/ijgi14040159

41.

Zhong

Zhang

Ozaki

, et al. (2025) A comprehensive methodological review of human mobility simulation and modelling: current trends, challenges, and future directions. Physica A: Statistical Mechanics and Its Applications 674: 130791. https://doi.org/10.1016/j.physa.2025.130791

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

4.44 MB