Optimizing wind power integration: Increasing profitability and market efficiency through machine learning based wind speed forecasting in Japan's Electricity Grid

Abstract

Wind energy has become a key constituent in Japan's developing electricity sector. However, the fluctuation of wind speed in its deployment poses operational and economic risk, which puts the generators at risk for energy price imbalances (EP_IM). The forecast errors between predicted and actual wind output, which can result in imbalance penalties, negatively impact renewable producers’ profitability, obstruct deeper integration of renewables, and stagnate economic growth. Herein, a comprehensive optimization mechanism has been introduced in this work by integrating state-of-the-art machine-learning (ML) models, that is, Random Forest (RF) and Long Short-Term Memory (LSTM), with numerous metaheuristic optimization techniques, that is, Whale Optimization Algorithm, American Zebra Optimization (AZO) Algorithm, along with Sequential Quadratic Programing to maximize the system economy. The framework forecasts wind power and load demand concurrently through 24 h by embedding these into an optimal power flow framework: Maximizing profit for the system, and minimizing imbalance costs. It makes use of the real wind speed data of Tokyo for validation of the proposed method. The LSTM provides better forecasts than RF, and AZO continuously yields the best economic results after the implementation of the presented approach. The ML-metaheuristic approach facilitates dispatch efficiency, while simultaneously decreasing imbalance penalties and increasing the market profitability significantly. The proposed method is a technically feasible and operationally relevant approach to implementing a higher wind integration scale in Japan's deregulated market structure.

Introduction

Electrical energy is essential for both the social and economic growth of any nation. Due to its enormous importance, state agencies have traditionally owned and run the power sector. The deregulation of the power supply business over the past few decades has been one of the most significant international energy developments. Following the introduction of deregulation, the conventional monopolistic structure of the electrical industry was replaced by competition between generation companies (GENCOs), transmission companies, and distribution companies (DISCOs) to improve the social benefit and maximize the efficiency of the power generation (Das et al., 2022a). Renewable energy has the potential to play a significant role in the world's energy supply because it can optimize energy structures, close the gap between supply and demand, and, most importantly, protect the environment (Malik et al., 2023). Many power plant developers are keen to invest their money in setting up wind farms (WFs) due to the lack of requirements for fuel and other advantages. As a result, wind energy has risen to the top spot over the past few decades as the source of electrical energy with the highest global annual growth rate. An accurate control mechanism has to be created for the excess capacity because, in a deregulated system, the generation and transmission of resources are employed more efficiently (Biswal et al., 2024). Energy storage can be crucial in this situation by allowing the utility to exploit the use of resources. Renewable energy sources (RESs) are becoming progressively prevalent as substitute energy sources. However, there are numerous issues with RES, including their propensity to generate irregular power, their dependence on weather circumstances, and their inability to compete with non-renewable energy sources. In recent times, noteworthy advances have been made in overcoming some of these problems, with the application of energy storage systems and hybrid energy systems.

Wind power is now among the world's top renewable electricity sources to generate electricity because of its clean nature, abundant availability, scalability, and flexibility (Das et al., 2022b). Solar also has wide utility value, lighting and heating, industrial use, and a growing attraction for renewables (Ranjan et al., 2021). With the deregulatory process over the power sector, it had to change the model of monopolistically dominated grids into competitive marketplaces; however, the integration of renewables adds new problems. Wind and solar production are inconsistent, dependent on the weather, leading to uncertainty over the supply. In a deregulated environment, both the DISCOs and GENCOs are equally responsible for generating and distributing electricity to promote the efficient generation and supply of electricity where it is needed, and for the generation/distribution of power in response to the energy supply. Such coordination helps to provide a reliable supply to consumers. Users are also shielded from being charged more for their energy use through government oversight of electricity prices. Imbalance penalties, also regulated by the independent system operator (ISO), make generators and consumers liable for differences between scheduled and actual power. These penalties incentivize both accurate forecasting and contribute to the stabilization of the service system. The ISO leads to safe operation and continuous supply and demand balance in the power system.

RES remain favored among more sustainable energy sources than non-renewables, due to their environmental benefits. Their utilization is constrained with increasing frequency, their production is varied due to seasonally dependent inputs, and they have a lower competitive position in comparison to the traditional sources (Dawn et al., 2016; Dawn et al., 2019). Research on RES is continuing to enhance the efficiency, reliability, and economy, & make RES more efficient as their potential future energy system players. Solar energy has emerged as a major RES due to its clean production capability, availability, scalability, and applicability in a range of applications, including residential systems and large-scale industrial applications (Xia et al., 2021). Deregulation has also turned electricity markets from monopolistic to competitive, stimulating innovation and favoring consumers as much as producers. GENCOs and DISCOs ensure a reliable and consistent supply of electricity in these market situations and are assisted by public supervision by the government to prevent excessive pricing. The day-ahead market, which necessitates production forecasts being submitted to the ISO, further increases efficiency. In contrast to thermal-based units, renewable generators often experience forecast uncertainty for weather variability, putting them at risk for penalties or incentives due to imbalance pricing. This will address the impact of imbalanced prices on systems’ economics and machine-learning (ML)-based forecasting methods that have been adopted to mitigate these impacts, as previously studied (Gaboitaolelwe et al., 2023; Queen et al., 2021).

The primary aim of renewable energy forecasting is to enhance the stability and reliability of the grid by allowing operators to balance supply and demand more effectively. Superior forecasting supports greater integration of RES and provides a basis for more resilient and sustainable power systems. Several techniques with advanced forecasting and energy management have been developed in the recent past. The IMLEA-STLF approach (Mehedi et al., 2021) further improves short-term predictability of load using feature selection via Wavelet Transform (WVT), prediction with Elman Neural Networks, and optimization and convergence through Oppositional Artificial Fish Swarm Optimization and Oppositional-Based Learning. The Combined Bootstrap and Cumulant approach (Wan et al., 2023) also enhances the application of probabilistic wind forecasting by including higher orders of statistics to better model the uncertainty. Research by Faraji et al. (2020) offers Energy Management Systems for Prosumer-Microgrids in which weather and load predictions are employed to reduce operational uncertainties. Del Ser et al. (2022) developed randomization-based models for solar radiation estimation, applicable to solar power generation as well as other environmental settings. Moreover, Mostafa et al. (2022) emphasize that ML enhances the effectiveness of energy management in smart grids, which includes renewable integration and safe cloud management of data.

Several recent works work on other ML applications in renewable energy. Roy et al. (2024) analyse hybrid RES configurations for remote Indian islands and develop ML models to predict technical, economic, and environmental performance. ML algorithms for predicting the solar irradiance have been thoroughly reviewed by Allal et al. (2024a), and the analysis is useful in the optimization of designing and operating the solar system. Tziolis et al. (2023) report on the efficacy of Bayesian neural networks for short-term net load prediction in renewable-integrated microgrids. Alazemi et al. (2024) give an overview of ML-based power output forecasting methods, which contribute to increased accuracy, power allocation, and better grid stability. Lin et al. (2022) prove the predictive power of ML for revenue prediction in energy generation and storage systems, supporting revenue-maximizing operational decisions. Karthikeyan et al. (2024) present a hybrid optimization model that considers the fluctuations of the fuel price and the battery cost to improve the economic performance of remote photovoltaic (PV)–wind–storage systems. El-Aziz (2022) proposes a hybrid ML framework with Multilayer Perceptron, Support Vector Regression, and CatBoost integrated to predict renewable energy consumption accurately. Likewise, Dimd et al. (2022) investigated the effect of meteorological characteristics on PV forecast accuracy within Nordic climates and recommended some ensemble and optimization methods to improve forecasts. In addressing energy trading problems, Jia et al. (2022) present a decentralized solution for peer-to-peer energy trading by improving energy trades in grids containing renewable energy. Shamsi and Cuffe (2021) sets out a binary prediction market for simulating weather power derivatives on renewable energy, hence enhancing their financial stability. Roy et al. (2023) showcased a different application where long short-term memory (LSTM) is combined with feed-forward neural network and a back-propagation algorithm for better fault detection and secure operation. Nabavi et al. (2021) set forth the DWT-LSTM method, fusing deep learning to forecast with energy storage management for maximizing building energy demand and supply, cutting the need to access the grid and electricity prices.

The wide-ranging research outlined in the present study (Roy et al., 2024) thoroughly reveals that hybrid renewable energy systems (HRESs), that seamlessly integrate PV generation technology, wind turbines (WT), battery storage (BAT), converters (CONV), and distributed generation (DG) technologies, accomplish an extremely low levelized cost of energy, which was set at an astonishingly low rate of 0.319/kWh, along with net present cost of 6.81 million per kilowatt-hour, demonstrating significant economic efficiency among all selected hybrid renewable configurations evaluated in this study. In addition, the simultaneous deployment of PV systems and WTs in partially fossil-fuel-fired HRES will also substantially reduce diesel fuel consumption, bringing the renewable energy proportion to 86.7% and a staggering improvement in sustainability indicators. Moreover, the study (Roy et al., 2025) confirms that diverse combinations of HRES configuration can provide the power supply to remote villages, and deliver detailed investigations of different combinations of PV panels, WT, biogas generators, batteries, and converters, to assess their operation performance and efficiencies in those same areas. The paper (Parvathareddy et al., 2025) concludes that the results of the proposed hybrid framework present a notable increase in prediction accuracy and operational performance, both for the field of industrial energy management, with a 48% improvement over conventional energy management models, confirming its significant effect in the field. Moreover, an optimization part of the framework is a key aspect in performing its purposefully, as peak demand variation reduces peaks by 27%, further evidence of the advantages of the multi-objective optimization methodologies emphasized in energy management practices. According to the work (Pérez-Rodríguez et al., 2024), metaheuristic algorithms show good results, and Particle Swarm Optimization (PSO) is the most utilized algorithm due to its accuracy in solar radiation prediction of the PSO algorithm from current literature. The importance of employing hybrid models is emphasized by authors, such as the Adaptive Differential Swarm Optimization Algorithm integrated with networks, given its remarkably low root mean square error (RMSE) level and the remarkable performance achieved under the system, which implies the existence of a high-predictive capacity by combining optimization algorithms with ML methods. Additionally, incorporating HRES with the primary electrical grid is becoming more popular for applications that are associated with urban energy efficiency, where there is also balancing in the use of renewable and non-renewable energy sources for energy consumption at relatively less cost (Abdullah et al., 2023). Finally, Allal et al. (2024b) undertake a wide review of the merits and weaknesses of the modern RES, while a detailed survey of different ML algorithms and how they can be used in the framework of RES gives context to this important topic.

The demand for a clean and sustainable energy supply has increased worldwide; hence, conventional fossil-fuel-based power generation has been gradually replaced with RES generation. Of these, wind power is particularly attractive as it is fast-growing, has low environmental impact, and is potentially applicable at a significant scale. Nonetheless, the variability and unpredictability of wind power create tremendous difficulties associated with stabilizing a power system, particularly in deregulated markets where imbalance costs are essential considering overall profit. A major reason for this work is to investigate how power from wind affects system stability and its economic cost when the imbalance penalty is considered, to get acquainted with the real-world dynamics of operation of modern power systems. In addition to that, with artificial intelligence research in development, ML technology holds promise in forecasting the accuracy of wind power generation and electricity demand. These prediction techniques play an essential role in reducing uncertainty and imbalance costs by enhancing the profitability and reliability of real-time systems integrated with renewables. The main highlights of the work are as follows:

WFs cannot always be integrated with the grid due to the variability of wind speed, and this leads to the need for optimization methods that promote social welfare, consumer surplus, and producer revenue.

Fluctuations of wind also present considerable risk to electricity market contracts, and precise forecasting has become part of effective strategic planning.

In deregulated markets, profit asymmetries due to disparities between predicted and real wind generation are particularly consequential, meaning they should be minimized to still make a profit.

While energy storage systems compensate for these imbalances, their high costs diminish overall margins, leading to an even greater need for high-accuracy forecasting.

In this work, an optimization framework has been recommended to overcome some of these disadvantages by optimizing the system for wind speed forecasting of ML-based random forest (RF) and LSTM models to maximize profitability and reduce imbalance losses.

The effectiveness of the system is shown through simulations on the IEEE 14-bus system.

The optimization problem is solved by using two metaheuristic algorithms, that is, Whale Optimization Algorithm (WOA) and American Zebra Optimization (AZO), along with Sequential Quadratic Programing (SQP).

Imbalanced prices represent a significant economic challenge associated with the renewable energy sector, which results in the imposition of either financial penalties or economic adjustment when the level of wind generation differs from its predicted quantity. Such imbalances can have severe economic implications for energy producers and complicate system operation for grid operators. To address this issue, accurate wind generation forecasts are necessary. Improved predictions allow better scheduling and dispatching and, thus, lower variance and less exposure to imbalanced prices. Here, ML methodologies look promising due to their ability to learn complex nonlinear patterns from historical data, as well as their ability to adapt to dynamic system conditions. The present research deals with the utilization of two ML-based predictive models, that is, RF and LSTM networks, for improved wind power generation forecasting. To establish a realistic operational model, an in-depth study environment was created, replicating real-time scenarios in this work. Historical wind speed and power generation data, along with load demand profiles, were utilized to train and validate the forecasting models for 24 h. Both load demand and wind power generation were forecasted simultaneously to allow system-wide coordination and optimization. The findings of the simulation show that ML-based forecasting substantially decreases the negative financial effect of imbalanced prices. With more accurate forecasts, the system operator can better control generation resources, schedule reserve margins, and balance supply and demand. Therefore, the profitability of the power system is increased, and grid stability is improved.

Although there were many works in the context of integration, forecasting, and optimization of RES, there have been very few comprehensive research works focused on a comprehensive analysis of the potential effects of wind power forecasting accuracy and imbalance cost mitigation in deregulated electricity markets, especially in the case of Japan. Existing research has concentrated on augmenting the forecasting model or optimizing power flow individually, ignoring the relationship between uncertainty and market profitability. Additionally, there is limited evidence from prior studies considering simultaneous load and generation forecasting with profit optimization. In this study, that gap has been removed by constructing an integrated ML–metaheuristic framework using LSTM-based forecasting alongside different optimization to reduce the imbalance penalties and improve the profitability of the system. To compare the technical, economic, and operational performance of the proposed framework, an IEEE 14-bus system was chosen. Although the Japanese power grid is much larger and more complicated by comparison, the IEEE 14-bus framework detects critical market and operational forces such as locational marginal pricing (LMP), generator dispatch, and imbalance settlement processes. This helps enable the study to remain focused on validating the methodological framework by combining wind forecast, imbalance pricing, and economic optimization, without computational hurdles. This system offers a scalable and directly transferable approach to larger grids such as Japan's deregulated market, as the key optimization and forecasting principles are unchanged from one grid level to the next.

Mathematical modeling

This chapter provides an extensive review of various risk techniques, as well as a mathematical framework applicable to wind power generation. There is also a discussion regarding optimization techniques for improving the efficiency and effectiveness of these systems.

Wind speed data

The actual and forecasted wind speed data are collected for Tokyo, Japan, to conduct the analysis. The collected data were for a height of 10 m above sea level. In Japan, the typical hub height of a wind turbine is 100 m. As such, the wind speed at the desired height needs to be determined. The recorded wind speed data for the selected places are shown in Table 1. The wind speed at 100 m is calculated using the following power law equation:

$${\displaystyle \frac{W_h}{W_0}={\left({\displaystyle \frac{h}{h_0}}\right)}^{\propto }}$$

(1)

Table 1.

Forecasted and actual wind speed data at 10 m and 100 m height.

Hour	WSF @10 m (m/s)	WSA @10 m (m/s)	WSF @100 m (m/s)	WSA @100 m (m/s)
0	3	3	4.17	4.17
1	2	1	2.78	1.39
2	2	2	2.78	2.78
3	1	3	1.39	4.17
4	3	3	4.17	4.17
5	6	5	8.34	6.95
6	8	6	11.12	8.34
7	8	7	11.12	9.37
8	8	5	11.12	6.95
9	8	6	11.12	8.34
10	6	6	8.34	8.34
11	5	4	6.95	5.56
12	4	4	5.56	5.56
13	4	3	5.56	4.17
14	3	3	4.17	4.17
15	3	2	4.17	2.78
16	1	1	1.39	1.39
17	4	2	5.56	2.78
18	8	7	11.12	9.73
19	7	7	9.73	9.73
20	7	6	9.73	8.34
21	6	6	8.34	8.34
22	6	5	8.34	6.95
23	5	4	6.95	5.56

where “W_h” is the wind speed at height “h”; “W_o” is the wind speed at height “h_o”; “α” is the Hellman coefficient. The considerations for this work are as follows: h = 100 m and h_o = 10 m. Table 1 displays the calculated WS_A and WS_F at the desired height of 100 m (Hourly forecast for Tokyo, Japan, 2024, 25.10.2024 [22]).

The wind power available in a wind turbine depends on the swept area of the turbine (a) that is perpendicular to a wind stream moving at speed (v) with an air density (ρ) and having a turbine efficiency of η.

$$P_W={\displaystyle \frac{1}{2}*\rho a\eta v^3}$$

(2)

For this work, the considered parameters were as follows: ρ = 1.225 kg/m³; a radius of the turbine rotor of 40 m, and η = 0.49. The investment cost for the wind generation was considered to be 3.75 $/MWh, and 30 wind turbines were considered for the parallel operation.

Locational marginal pricing

LMP is formulated by taking into account important factors like transmission constraints, line losses, and load demand. It is different across the power grid due to varying transmission constraints and demand patterns across the whole system. In general, LMP is calculated using Equation (3).

$$LM{P}_i=\lambda +LM{P}_i^{cong}+LM{P}_i^{loss}$$

(3)

where $\lambda$ is the marginal cost of energy produced from the cheapest generator available, $LM{P}_i^{cong}$ is the marginal cost due to transmission line congestion, $LM{P}_i^{loss}$ is the marginal cost due to transmission losses.

Random forest (RF)

Within the realm of statistical learning and predictive modeling, the RF algorithm has been a well-accepted ensemble learning method. As opposed to classical models that are based on one decision tree, which tend to be highly sensitive to noise and subject to overfitting, RF utilizes the strength of a large ensemble of decision trees to provide stable and precise predictions. The essence of RF is in the creation of an ensemble of separate decision trees during training. Each tree is trained on a bootstrap sample, a randomly selected subset of the original set with replacement, thus creating data-level variability. Moreover, when building each decision tree, a random subset of features is chosen at each node to find the best split. Through this method for feature randomness, each tree in the model investigates different features of the feature space, reducing the correlation between trees and generalization capabilities. RF uses majority voting to fuse the output from single trees for classification problems, and average prediction computation for regression problems. This ensemble method has a much better variance reduction than a single decision tree, which minimizes the chance of overfitting, one of the most common problems of a complex model. Not just with respect to predictive performance, the power of the RF model stems from its non-parametric quality. It is free from any assumptions concerning the distribution of the underlying data and so extremely flexible and good for many areas ranging from bioinformatics through remote sensing, finance, healthcare, and natural language processing. It is especially valuable for processing high-dimensional datasets, where the size of features can be much larger than the number of observations, and is known for its robustness to outliers and noise. RF also has its intrinsic feature importance measures, which represent another important benefit. These statistics give good information about how the contribution of input variables to the model's predictions is, which aids in feature selection and explanation.

Long short-term memory

Recurrent Neural Networks (RNNs) have been widely praised for their recursion-based nature that automatically represents temporal dependencies and thus have provided the best sequential behavior for data handling and model building. Traditional RNNs, however, are usually severely hindered by the problem that they cannot learn long-distance dependencies in an efficient manner, mostly due to the vanishing and exploding gradient challenges introduced by training with backpropagation through time. These limitations prevent the network from retaining significant information on long sequences and, therefore, can hardly be used efficiently for the performance of more complex time-sensitive tasks. In order to overcome these constraints, an efficient extension to the standard RNN, that is, LSTM network, has been introduced. LSTMs are designed to model long-term dependencies in sequential data by incorporating a memory component. At the heart of the LSTM, a stable storage device with an ability to hold data for arbitrary periods, is a memory cell. This memory cell is actively operated by 3 gating mechanisms known as the input, forget, and output gates, which gate what information moves into, inside the cell, and out of the cell. The forget gate determines what parts of the prior cell state should be forgotten, allowing the network to discard irrelevant or stale information. The input gate controls the contribution of new information to the memory cell. Then, the output gate allows the refreshed cell state to be visible to the next layer or time step and, thus, determines how much of the internal state is outputted. As a result, such a gate structure can provide selective information storing, updating, and retrieval for LSTMs for storing contextual information over long sequences. Hence, LSTMs have shown superior performance in several applications with depth of understanding, as in the fields of natural language processing, time series forecasting, and automatic speech recognition.

Whale optimization algorithm

The WOA is a nature-based metaheuristic algorithm inspired by the peculiar hunting activity of humpback whales. In particular, it imitates the bubble-net foraging tactic wherein whales circle a target and approach in a narrowing circle. WOA has been popular because of its ease of implementation, rapid convergence, and power in escaping local optima and has been known to be a viable method to handle intricate, nonlinear optimization challenges like the optimal power flow (OPF) solutions in power grids (Kar et al., 2024). WOA is used to address the OPF problem by modeling the search agent (whale) as a candidate solution vector incorporating control variables such as generator output powers, bus voltages, transformer tap settings, reactive power compensation, etc. The variables are updated iteratively in an optimization attempt to maximize the objective function to satisfy all system constraints. The main steps of the WOA in OPF are as follows:

Initialization: Create a first population of whales inside the feasible solution space of the OPF problem. Then calculate the fitness for each whale based on the OPF criteria.

Encircling prey: Consider the current best solution to be the prey. Update each whale's position according to the best-known position.

Bubble-net attacking method: Move towards the best solution with a spiral-shaped movement following a logarithmic spiral model to simulate the bubble-net attack method.

Search for prey: Randomly pick a whale and push other whales away from the chosen one to stimulate exploration in new regions within the solution space.

Constraint handling: Use penalty functions or repair mechanisms to enforce that all candidate solutions satisfy the system's physical and operational constraints.

Termination criteria: The iteration proceeds until a stopping condition is reached, for example, a maximum number of iterations or convergence of the objective function.

Output: The best-found solution from the algorithm, that is, the optimal values for the settings of the power system that produce the minimum objective while adhering to all constraints.

The trade-off in exploration and exploitation between WOA and its escaping capabilities from local optima favors its application most on the non-convex, nonlinear problem that OPF possesses. The WOA convergence parameters for this work are explicitly stipulated here for full reproducibility. The population size was determined to be 30 search agents, and the maximum number of iterations is 150. A convergence tolerance of 10^-6 was chosen as the termination condition when there was no further improvement in the objective function. Such settings made the exploration–exploitation trade-off balanced and brought steady convergence to the OPF.

American Zebra optimization algorithm

The AZO is a nature-based optimization method based on the social organization and adaptive strategies of American zebras in their natural environment. Taking a cue from their survival mechanisms in the grasslands, that is, cohesive movement, cooperative foraging, and evading predators, the AZO algorithm provides a novel method for addressing challenging optimization problems (Mohapatra and Mohapatra, 2023). Zebras are both adaptable and collectively intelligent. They travel in tightly coordinated herds wherein individuals balance leader following with independent investigation of new areas in pursuit of better resources. This behavior model is strongly associated with the primary optimization principles: exploration and exploitation. AZO illustrates this balance by allowing the algorithm to dynamically adapt to fluctuating search conditions, similar to zebras moving around in uncertain environments. The main steps of the AZO algorithm are:

Initialization: A random population of zebras is produced. Every zebra holds a separate solution within the multidimensional search space.

Exploration: Agents traverse the environment to recognize regions of potential solutions of good quality. During this phase, global search and diversity within the population are enhanced.

Exploitation: Once the promising areas are discovered, the agents refine their local positions to enhance the solution precision and shift progressively towards higher quality.

Leader Election: The top-performing agents are appointed as leaders and determine the directions for group movements. The leadership switches regularly to avoid early convergence and promote persistent exploration.

Migration: To prevent trapping in local optima, agents occasionally move to new areas of the search space, promoting wider exploration.

Termination Criteria: The iterative procedure runs until a termination condition is satisfied, for example, achieving a given maximum number of iterations, a convergence level, or a satisfactory solution.

Global Optimum Output: The optimal solution determined in the process is taken to be the world optimum of the optimization problem provided.

The AZO algorithm was set to have a population size of 30 agents and up to 150 iterations in order to remain consistent with WOA for the comparative analysis. A convergence tolerance threshold of 10^-6 was imposed; the search terminated when no further improvement between consecutive iterations was found. The use of these parameters led to good convergence behavior, which ensured that the optimization results could be reproducible.

Sequential quadratic programing

SQP is a robust and popular gradient-based optimization method for solving nonlinear constrained optimization problems. For OPF, SQP is utilized to optimize an objective function like generation cost, power losses, or environmental emissions under system constraints concerning power balance, voltage limits, generator capabilities, and transmission limits. The OPF problem is itself nonlinear and non-convex, owing to the intricate relationship between power system variables. It includes both equality constraints and inequality constraints. SQP is suited to solve such problems, particularly when high accuracy of convergence and precision in the solution are desired. The main features of SQP in the OPF solution are as follows:

Problem formulation: The OPF problem is stated as a nonlinear optimization problem that includes an objective function, equality constraints, and inequality constraints.

Construction of quadratic subproblem: At each iteration, SQP solves a Quadratic Programing (QP) subproblem that approximates the nonlinear OPF problem. The QP subproblem is formed by a quadratic approximation of the Lagrangian function and a linearization of the nonlinear constraints by a first-order Taylor series approximation.

Solution of QP subproblem: The QP subproblem is resolved by applying conventional QP methods. The solution gives a search direction for the control variables to be updated.

Line search and update: A line search approach is applied to guarantee enough improvement of the objective function. The variables are updated, and the procedure is repeated.

Convergence criteria: The iteration stops when convergence criteria are fulfilled, usually by gradient norm, step size, or objective function change.

Objectives of the work

The study in this analysis aims for an in-depth analysis of an electric system that consists of “Nb” buses, “Nd” loads, ${}^{\mathrm{\prime }\mathrm{\prime }}{\mathrm{N}}_{\mathrm{TL}}{}^{\mathrm{\prime }\mathrm{\prime }}$ transmission lines and “Ng” generators. This study is implemented in a wind-integrated day-ahead power system, where power generation farms offer their power supply to the ISO at least one day before the planned power operation. The ISO uses this data to schedule and operate thermal power plants on the given day. The wind power plants bid based on wind forecast speeds, estimating the quantity of power available for generation. But, where actual wind speeds and forecasted wind speeds might deviate, generating and injecting a balance of power will be tilted. In these cases, an incentive or cost-benefit ratio for the farms in the ISO will be created for each one based on its deviation from the predicted output for a certain power generation. This pricing of WFs through the mechanism of imbalance pricing (EP_IM) is used in the wind utility industry, in which the market is balanced with WFs that can forecast better levels of power generation, and system disturbances are controlled. A competitive power network provides the system goal of providing consumers with an economic system while ensuring the reliability of quality power.

The ISO's fines or bonuses have an immediate impact on WFs’ bottom line, resulting in a fall in profitability. Wind power generation issues and the notions of wind must, therefore, be solved. One of the ways to minimize such challenges is to have erudite forecasting mechanisms that eliminate the potential disconnections. Power storage devices are yet another solution to match demand to supply instantaneously by using excess power during high wind generation hours in conjunction with discharging excess in the event of a gap in power availability. This paper seeks to address such challenges and find ways to improve system performance. This work focuses on maximizing profit for the whole system with a focus on an integrated thermal power system with wind energy for efficiency. This includes maximizing the economic returns generated from both wind and thermal power plants, in view of wind power generation asymmetry, revenues/costs. This increases the smoothness of variations of wind power generation, reducing its susceptibility to operating risk from unexpected power cuts or surpluses. The overall profit of the power system is calculated by obtaining a detailed formula that specifies various variables such as revenue from the wind energy, charges paid for imbalances, cost of investment, and rate charges under surplus and deficit wind power conditions. These are the factors that should be addressed to a very good extent for the economic efficiency and feasibility of such a system. Since the study mainly targets balancing the imbalances’ costs, it aims for a substantial reduction in the overall generation costs and also the risks of the system. This way, the study aims at improving both the financial gains realized from the power system operation and the social advantage obtained from incorporating RESs such as wind power. Any thorough analysis of the performance of a power system integrated with renewables needs to factor in the cost of imbalance pricing. Although crucial, this consideration has been analyzed by very few researchers in terms of its implications and effects in renewable-integrated systems. System operators are instrumental in rewarding and punishing generating companies, such that positive imbalance costs (when wind generation is above forecast levels) can add to profits, and negative imbalance costs (when wind generation is below forecasts) can reduce profits. The main objective function, the maximization of system profit, can be mathematically expressed by taking into account the different cost components involved in wind power generation and the balancing mechanisms involved.

$$P(t)=TR(t)-TGC(t)+E{P}_{IM}(t)$$

(4)

where $TR(t)$ is the total revenue and $TGC(t)$ is the total generation cost, and these are calculated as shown in (5)–(8). Profits can be increased by minimizing the generating costs $TGC(t)$ which is the objective function.

$$TR(t)=\sum _{i=1}^{N_G}{P}_{Gi}(t).LM{P}_i(t)$$

(5)

$$E{P}_{IM}(t)=\sum _{i=1}^{NG}\begin{array}{c}\left(SR(t)+DR(t).{\left({\displaystyle \frac{P_{Fi}(t)}{P_{Ai}(t)}}\right)}^2\right).(P_{Ai}(t)-P_{Fi}(t))\\ \end{array}$$

(6)

$$TGC(t)=\sum _{i=1}^{N_G}{C}_i(P_{Gi})+WG{C}_n(t)$$

(7)

$$a_i{P}_{Gi}^2(t)+b_i{P}_{Gi}(t)+c_i=C_i(P_{Gi})$$

(8)

where $P_{Gi}(t)$ is the generated power at the $i$ th bus and the $LM{P}_i(t)$ is the locational marginal price at the $i$ th bus at the time “t”. In equation (8) ${}^{\mathrm{\prime }\mathrm{\prime }}{a}_i,{}^{\mathrm{\prime }\mathrm{\prime }}$ ${}^{\mathrm{\prime }\mathrm{\prime }}{b}_i,{}^{\mathrm{\prime }\mathrm{\prime }}$ and ${}^{\mathrm{\prime }\mathrm{\prime }}{c}_i{}^{\mathrm{\prime }\mathrm{\prime }}$ are the cost coefficients of the generator at the $i$ th bus. $\lambda$ in (3) depends on the generator cost coefficients. However, the discrepancy between the predicted and actual wind speeds can result in imbalanced costs. Equation (6) represents the expression of the imbalance cost. The deficit charge rate (DCR) and excess charge rate are calculated as follows in (9)–(11).

$$DR(t)=(1+\lambda ).LM{P}_i(t),SR(t)=0when{P}_{Fi}(t)>P_{Ai}(t)$$

(9)

$$SR(t)=(1-\lambda ).LM{P}_i(t),DR(t)=0when{P}_{Fi}(t)<P_{Ai}(t)$$

(10)

$$SR(t)=DR(t)=0when{P}_{Fi}(t)=P_{Ai}(t)$$

(11)

Equation (4) shows the total profit calculated by the system as the difference between total revenue ( $TR$ ) and total generation cost ( $TGC$ ). Here, $TR$ , refers to the total revenues that arise through the sale of generated power at the LMP, while $TGC$ accounts for all generation-related expenses such as fuel costs, maintenance, and investment costs related to wind energy generation. Equations (5)–(7) elaborate these components further by describing total revenue as the sum of the products of power generation ( $P_{Gi}(t)$ ) at each bus i and its respective LMP ( $LM{P}_i(t)$ ) during the operation time series. The generation cost is expressed as an estimated loss rate as a quadratic costing function that includes coefficients, $a_i$ , $b_i$ and $c_i$ which are given by cost attributes for each generator. Equations (6)–(8) address the imbalance cost mechanism, which is an important factor in deregulated systems since the discrepancy in the forecasts generated and the actualizations generates financial penalties to be disbursed or credits to be taken. The quantity of the imbalance price coefficient, λ, in different values (0–1), is added to such equations to measure the penalties for imbalance. A λ value up to 1 indicates a more severe penalty system that will provide more reason for better prediction. The imbalance cost, therefore, can be decomposed into deficits and surpluses. When the power generation is well below the estimated value, the penalty to the market for deficit electricity is known as the DCR. Conversely, the surplus charge rate (SCR) determines payment quantities for the electricity generated in excess of expected levels. These rates are derived from the LMP at the bus and corrected by the imbalance coefficient (λ) to reflect market principles that govern on-board energy quality. Combined, they facilitate operators of the systems and sources of power to accurately calculate profit margins, as they are capable of modeling efficiency of operation, market risk, and various forms of renewable energy. This formulation paves the way for the optimization framework applied at a later point in the study, using metaheuristic algorithms such as WOA and AZO to maximize system profit by minimizing the cost of generation and the consequences of imbalance.

The value of “λ” lies between zero and one. For this work, the value was considered as 0.9 due to the relatively more strict structure of imbalance settlement typically adopted in deregulated electricity markets. Higher λ imposes stronger punishment for variances and rewards more precise forecasting by market participants. Previous works on imbalance cost modeling for IR have also employed values of λ between 0.85 and 1.0, rendering 0.9 a viable and grounded option. This choice is for realistically modeling the economic effects of forecast errors, with model stability considering all 24-h test scenarios. From equations (4–11), it can be observed that the economy of the system is based on an imbalanced price in the case of a wind-related power system. The influence of an imbalanced price on the system can be controlled by proper prediction of wind power generation. In this scenario, ML-based prediction is made available by accessing RF and LSTM techniques to reduce the adverse effect of imbalanced prices so that the system's profit can be improved. For the development of an actual study environment in real-time, load demand and wind power forecasting have been done using an ML-based model for 24 h. OPF is a fundamental optimization problem in power system operation and planning. It determines the optimal values for the generator power outputs, bus voltages, voltage angles, and the power flow in the branches to minimize the objective function, which is the generation cost, while satisfying the power system constraints. Its mathematical formulation is presented as follows. The considered objective function is the minimization problem, which is expressed as:

$$min(TGC(t))$$

(12)

This minimization problem is subject to the following constraints from (13) to (21). Here (13)–(16) are the equality constraints to ensure power balance, whereas (17)–(21) are the inequality constraints to maintain bus voltages and phase angles, transmission line power capacity, and the generator active and reactive power generation within specified limits.

$$\sum _{i=1}^{N_G}{P}_{Gi}-P_{loss}-P_D=0$$

(13)

where $P_{Gi}$ is the generated power at the ith generator and $P_{loss}$ , $P_D$ are the total transmission power loss and the total load demand on the system. $P_{loss}$ can be calculated as shown below.

$$P_{loss}=\sum _{j=1}^{N_{TL}}{G}_j[{|V_i|}^2+{|V_j|}^2-2|V_i||V_j|\cos ({\delta }_i-{\delta }_j)]$$

(14)

where $G_j$ is the conductance of the line connected between $i$ th and $j$ th buses; $|V_i|$ and $|V_j|$ are the magnitudes of the voltage at $i$ th and $j$ th buses; ${\delta }_i$ , ${\delta }_j$ are the phase angles of the voltages at the $i$ th and $j$ th buses.

$$P_i-\sum _{k=1}^{N_B}|V_i{V}_k{Y}_{ik}|\cos ({\theta }_{ik}-{\delta }_i+{\delta }_k)=0$$

(15)

$$Q_i+\sum _{k=1}^{N_B}|V_i{V}_k{Y}_{ik}|\sin ({\theta }_{ik}-{\delta }_i+{\delta }_k)=0$$

(16)

where $P_i$ , Q are the real and reactive power injected into the ith bus and | $V_i$ |, | $V_k$ | are the magnitudes of the voltages at ith and kth buses and ${\delta }_i$ , ${\delta }_k$ are the voltage angles at the ith and kth buses, $|Y_{ik}|$ , ${\theta }_{ik}$ are the magnitude and angle of the admittance at the ith row and k th column of the bus admittance matrix.

$$V_i^{MIN}\le V_i\le V_i^{MAX}i=1,2,3\dots \dots N_B$$

(17)

$${\varphi }_i^{MIN}\le {\varphi }_i\le {\varphi }_i^{MAX}i=1,2,3\dots \dots N_{TL}$$

(18)

$$T{L}_i\le T{L}_i^{MAX}i=1,2,3\dots \dots N_{TL}$$

(19)

$$P_{Gi}^{MIN}\le P_{Gi}\le P_{Gi}^{MAX}i=1,2,3\dots \dots N_B$$

(20)

$$Q_{Gi}^{MIN}\le Q_{Gi}\le Q_{Gi}^{MAX}i=1,2,3\dots \dots N_B$$

(21)

where $V_i^{MIN}$ , $V_i^{MAX}$ are the maximum and minimum voltage limits for the ith bus. ${\varphi }_i^{MIN}$ , ${\varphi }_i^{MAX}$ are the maximum and minimum load angles for the ith transmission line. $T{L}_i$ , $T{L}_i^{MAX}$ are the actual and maximum power flow in the ith transmission line. $P_{Gi}^{MIN}$ , $P_{Gi}^{MAX}$ are the minimum and maximum active power limits at the ith bus, $Q_{Gi}^{MIN}$ , $Q_{Gi}^{MAX}$ are the minimum and the maximum reactive power limits at the ith bus.

Proposed method

The presented methodology is the first to introduce a structured and strategic framework for enforcing the RES aggregator policy with the optimal deployment of DGs and including sustainable incentive systems. The methodology is particularly notable for Japan due to the country's energy diversification and transformation towards renewable sources being imperative because of the limited indigenous fossil fuel resources, the old age of energy infrastructure, and the legacy of decarbonization targets under international frameworks. Within the Japanese power system, the electricity market and higher penetrations of renewable sources require refined coordination methods to control a multitude of distributed generators. These necessities are addressed in the proposed approach via:

The model accounts for economic and technical objectives for the best DG placement. These are reducing system losses, maximizing social welfare, and promoting renewable integration with a key focus on Japan's electricity sector.

The aggregator-based policy is integrated into the method, allowing centralized control of dispersed energy producers to facilitate better coordination and market participation efficiency. In a place like Japan, where balancing variable renewable output on isolated regional grids is a longstanding puzzle, this becomes more critical.

This approach evaluates incentive policies that have been developed regarding fair compensation to DG owners and investment in clean technologies. The schemes are consistent with the Japanese policy path towards promoting decentralized, low-carbon energy systems and establishing grid stability and cost competitiveness.

Various operating conditions are also simulated to evaluate the system performance under different renewable penetration levels and load profiles. It is important to analyse Japan's seismically active and weather-sensitive areas, where energy resilience has become a government priority.

Employing the methodology described above, the research aims to enrich a smarter, cleaner, and more resilient Japanese power grid to fulfill the nation's ambition for carbon neutrality and sustainable energy security. This methodology is applied to the 24-h scenarios to consider the impact of the difference between the anticipated and the actual wind speed. Deficit and SCRs for each case were determined, and the overall imbalance cost was based on these results. Figure 1 displays a flowchart of the proposed approach. With the implementation of generators rescheduled using the constraints (Equations 13–21), the OPF problem has been solved while minimizing the cost of generation for the system. Furthermore, upon installation of the wind generators, the imbalance cost and profit for each wind speed were calculated. On the flowchart, “H_r” is an hour number, whereas “TP” is the total profit, and “P” is the profit for each hour. The flow chart outlines the calculation of the total profit within the energy category, including the costs of imbalances. And it starts from initializing the hour counter (H_r = 1) and total profit (TP = 0). The system read input data for bus, branch, generator parameters, and generator coefficients. This baseline data is used to run power flow calculations, along with an optimization function. After this, both actual wind speed and predicted wind speed are collected, which is very important for estimating renewable energy generation and analyzing the data.

Figure 1.

Flow chart representation for the proposed methodology.

Wind speed at the desired height is used to calculate wind power investment costs. This step is essential before determining the economic feasibility of including wind energy in the system. An OPF analysis is subsequently performed, accounting for the placement of WFs. This helps to ensure that energy dispatch meets grid needs and drives energy efficiency and cost reduction. The value of power generation and LMP which indicates the cost of the electricity supply to particular points on the grid. This includes generating electricity for that electricity point in various locations. Revenue and total generation expenses, including the costs of wind power, along with revenues, are found. In addition, the SCR and DCR calculated according to the actual wind speed vs predicted wind speed are calculated. These rates evaluate the financial penalties or adjustments for energy generation that exceeds or falls short of forecasts. These surplus or deficit rates lead to imbalanced costs and the calculation of profit per hour. This hourly profit is then added to the total profit. The overall profit (TP) is then written out at a 24-h time stage, indicating the total financial result after balancing the costs of the unbalance. The total strategy fuses renewable energy with forecasting and cost management; it delivers economic savings and grid stability and sustainability. This flowchart shows a clear path to enhance profits in addition to avoiding unbalanced spending.

System specifications

The IEEE 14-bus system has been chosen to examine the effectiveness of the presented hybrid approach. The single-line diagram of the IEEE 14 bus system is depicted in Figure 2. It has 14 buses with 5 generators and 11 loads. The transmission network of this system consists of 20 branches interconnecting the buses. Out of 14 buses, bus 1 is considered a slack bus, and buses 2, 3, 6, and 8 are the PV buses; the remaining are the PQ buses (Basu et al., 2022; Singh et al., 2022). The base MVA considered for the system is 100 MVA, and the maximum & minimum voltage constraints are 1.06 and 0.94 pu, respectively. Table 2 shows generator bus numbers for the 14-bus system used in this work and the cost coefficients of each generator with its maximum and minimum active power generation limits. Figure 3 shows the active power demand distributed across all the buses under peak load conditions on the system. It shows that bus 3 has the highest load, and buses 1, 7, and 8 are not loaded.

Figure 2.

Considered IEEE 14- bus system.

Figure 3.

Load demand at all buses in the IEEE 14-bus system during peak load time.

Table 2.

Generator cost coefficients and power limits for IEEE 14- bus system.

Sl. No.	Generator bus	$a$	$b$	$c$	$P_{MAX}$	$P_{MIN}$
1	1	0.043	20	0	332.4	0
2	2	0.25	20	0	140	0
3	3	0.01	40	0	100	0
4	6	0.01	40	0	100	0
5	8	0.01	40	0	100	0

This work is performed by taking into consideration that the load varies every hour, and the corresponding deviation from the peak load is evaluated by taking a percentage of the peak load that is present at a particular hour. To simulate the load variations for an entire day, a general load curve is considered and the load demand at every hour is scaled between 1 and 0 where 1 represents the peak load on the system and 0 represents the no-load condition as shown in Figure 4. Note that the hour of the day is represented from 1 to 24 where 1st hour refers to 1 AM and 24 refers to midnight.

Figure 4.

Load scaling factor to simulate load variations for every hour.

The load scaling factors at every hour are used for the 14-bus systems. The base case from the MATPOWER file that is, normal IEEE 14-bus system data is considered as peak load and for other hours, load variations are simulated by multiplying the base case with a suitable multiplication factor as shown in Figure 4. By analyzing the load scaling factors; peak load condition occurred at the 18th hour and the off-peak load occurred at the 1st hour and intermediate loading conditions occurred at the 13, 21 h respectively.

Wind power and load forecasting using a ML model

Wind power forecasting using ML

The dataset used for this analysis is taken from (AccuWeather, 2025). It contains six major features that are essential for understanding and forecasting wind power generation, as shown in Figure 5. The “Time stamp” column provides the date and time for each record, which allows tracking changes over the seasons. “System power generated” is the output of the wind turbine in kilowatts, meaning the amount of power at every interval. “Wind speed” and “Wind direction” contain information about the wind speed and direction. The “Pressure” column provides atmospheric pressure measurements, and “Air temperature” contains temperature measurements, both of which affect air density and, therefore, turbine performance. Figure 6 shows the changes in wind power for one month, depicting the variability of wind power.

Figure 5.

Sample dataset used for wind power predicting.

Figure 6.

Disparities in generated wind power over one month with the overall mean.

In this work, a dataset of meteorological factors influencing wind power generation, and its performance was evaluated by metrics such as Mean Squared Error (MSE), RMSE, Mean Absolute Error (MAE), and the coefficient of determination (R²). The models achieved very high accuracy, but LSTM surpassed RF in all the metrics, and their results are shown in Table 3.

Mean Squared Error

Table 3.

Results from the implementation of random forest and LSTM for wind power forecasting.

Performance metrics	Random forest	LSTM
MSE	24.292469	9.586891
RMSE	4.928739	3.096270
MAE	2.752152	2.207783
R 2	0.999967	0.999987

LSTM: long short-term memory; RMSE: root mean square error; MSE: mean squared error; MAE: mean absolute error.

It is defined as the average squared difference between actual and forecasted wind power generation values. A low MSE indicates that the forecasted wind power is closer to the actual wind power.

$$\mathrm{MSE}={\displaystyle \frac{1}{n}\sum _{i=1}^n(y_i-{\hat{y}}_i{)}^2}$$

(22)

where $n$ is the number of samples present in the dataset, $y_i$ is the actual wind power generation at the $i$ th data point and ${\hat{y}}_i$ is the forecasted wind power generation for the $i$ th data point.

Root Mean Square Error

It is obtained by the square root of MSE. This brings the error value back to the original units of the data, which makes it easy to interpret and evaluate.

$$\mathrm{RMSE}=\sqrt{{\displaystyle \frac{1}{n}\sum _{i=1}^n{(y_i-{\hat{y}}_i)}^2}}$$

(23)

• Mean Absolute Error

It measures the average absolute difference between actual and forecasted wind power generation values. It treats all errors equally, as it only considers the actual values.

$$\mathrm{MAE}={\displaystyle \frac{1}{n}\sum _{i=1}^n|y_i-{\hat{y}}_i|}$$

(24)

• R-squared (R²)

It indicates the degree to which the model clarifies the erraticism in the data. An R² rate near “1” advises a good fit, while “0” indicates no descriptive power.

$$R^2=1-{\displaystyle \frac{{\sum }_{i=1}^n{(y_i-{\hat{y}}_i)}^2}{{\sum }_{i=1}^n{(y_i-\overline{y})}^2}}$$

(25)

where $\overline{y}$ signifies the average of actual wind power generation, and it is measured as:

$$\overline{y}={\displaystyle \frac{1}{n}\sum _{i=1}^n{y}_i}$$

(26)

The correlation matrix in Figure 7 presents useful information concerning the correlation among different atmospheric and operational parameters for the performance of a wind turbine. The correlation, which is extremely positive, between power and wind speed serves as a reminder that higher wind speed leads to higher electrical power and therefore wind speed is the most important performance parameter on the turbine. That correlation is poor with wind direction means that while wind direction affects turbine direction, its direct influence on power output is negligible. The negative correlation between atmospheric pressure and power output shows that an increase in pressure, by acting on air density and wind behavior, will reduce the efficiency of a turbine a little. Moreover, the relationship between air temperature and pressure is also representative of normal atmospheric behavior, allowing temperature-related air density and pressure distribution. Overall interpretation of these correlations can benefit the maximization of turbine locations for efficiency and predictive maintenance of wind energy systems.

Figure 7.

Correlation matrix of the wind power dataset.

Table 3 shows the results of RF and LSTM-based models. The reliability of the LSTM model over RF for all error metrics in this case is observed, revealing its ability to better express the load demand data and the temporal dependencies. Compared to RF, LSTM obtains an MSE (9.59) and RMSE (3.10) of less than 24.29 and 4.93, respectively. Likewise, the MAE for LSTM was 2.21, which is less than RF's 2.75. Both models have similar near-perfect R² scores, suggesting their very good performance in predicting the target values, but LSTM did edge a little with the score of 0.999987 versus 0.999967 for the RF.

These findings prove that even if the models are very good, LSTM generates better forecasting accuracy, possibly due to the fact that LSTM is built for sequential data processing. This highlights the importance of choosing models aligned with the temporal nature of power system data for optimal forecasting performance. Figure 8 shows a set of data points for comparison of actual wind power from test data and the forecasted wind power from the trained RF model, where blue and red colors were used to indicate the actual and forecasted wind power, and Figure 9 shows the actual and forecasted power on X and Y axes, respectively, and their regression plot for RF model. Similarly, Figures 10 and 11 display the actual and forecasted wind power and the regression plot of them using the LSTM model.

Figure 8.

Actual vs forecasted wind power generation using random forest (RF).

Figure 9.

Regression plot of actual and forecasted wind power generation using random forest (RF).

Figure 10.

Actual vs forecasted wind power generation using long short-term memory (LSTM).

Figure 11.

Regression plot of actual and forecasted power generation using long short-term memory (LSTM).

The strong alignment in Figures 8 to 11 shows that forecasting models, like the LSTM architecture, are effective. The predictions are in good agreement with actual load trends, so the model has learned a lot about the underlying dynamics. This accuracy can be realized in simulation-based configurations, provided that the input data is clean, preprocessed, and free of noise. For model training, the model was assisted by optimizing its hyperparameter tuning and correctly stopping conditions to avoid overfitting the data, while convergence was achieved. This is in line with lower RMSE, MAE, and R² score closer to unity, indicating the accuracy of the underlying representation. Therefore, while this data might not be a general sense of well-defined reality due to environmental and measurement uncertainties, it is an indication of the model's reliability if precisely targeted data are included.

Load demand forecasting using ML

In a wind-integrated power system with 24-h forecasts on wind, accurate load predictions become very pertinent. Reliable load approximations allow operators to predict fluctuations in demand, which they can then adjust to meet the naturally variable wind output. Due to the dependence of wind generation on weather conditions, by estimating the load, the operators are better able to match the renewable and conventional sources.

Figure 12 indicates the data set adopted to perform load forecasts. It contains more than 2 million measurements of household electric power consumption at one-minute intervals, and spans the period from December 2006 to November 2024. It is a multivariate, time-series data set including six properties including date, time, global active power, global reactive power, voltage, and global intensity. Date and Time illustrate the time of taking, Global Active Power denotes the total active power used by the household every minute, measured as kW, Global Reactive Power denotes the reactive power used in kilowatts, and Voltage and Global Intensity denote the voltage in volts and current in amperes of the household's electricity system. These features together represent detailed electrical conductive information in a household, making the data set amenable to time series analysis as well as predictive analysis in energy consumption. Table 4 shows a comparison of RF models compared to LSTM models on load demand forecasting using standard regression metrics. LSTM outperforms RF on all error metrics (i.e., MSE, RMSE, MAE), so it better predicts the load demand data.

Figure 12.

Sample of the dataset used for load forecasting.

Table 4.

Results from implementation of random forest and LSTM for load demand forecasting.

Performance Metrics	Random Forest	LSTM
MSE	0.00177	1.2652e-5
RMSE	0.04209	0.00355
MAE	0.02356	0.00311
R 2	0.99841	0.99998

LSTM: long short-term memory; RMSE: root mean square error; MSE: mean squared error; MAE: mean absolute error.

The LSTM's MSE, showing it achieves a much lower average squared error than RF, indicating its finer precision. LSTM also shows the lowest RMSE and MAE, which suggests it continues to be robust in predicting near true targets. Both models have a high R² score, which indicates a good model fit; LSTM has a marginally better understanding of the data patterns. In general, LSTM is more appropriate for modeling sequential and time-sensitive behaviors for load forecasting, while RF, although powerful, might not be sufficient for these situations.

The correlation pattern in Figure 13 offers useful perspectives on the linear relationships of four prominent wind turbine parameters: Global active power, global reactive power, voltage, and global intensity. The matrix demonstrates that there is an ideal positive relationship between global active power and global intensity, and with changes in the active power, the intensity is proportionate. Similarly, global reactive power is moderately positively related to global active power and global intensity, implying a partial correlation. Voltage is negatively related to the other parameters, more so to global active power and global intensity. This means that more power (or higher intensity) means a small decrease in voltage. These correlations exhibit the interaction and behavior of both system and load demand patterns in a manner; therefore, higher power consumption can bring voltage drops to the system. Indeed, the matrix encapsulates key relationships that are instrumental to understanding turbine operation, ultimately determining the most efficient operational framework within the energy system.

Figure 13.

Correlation matrix of the load demand dataset.

Figure 14 shows a set of data points for comparison of actual wind load demand from test data and the forecasted load demand from the trained RF model, where blue and red colors were used to indicate the actual and forecasted load demand, and Figure 15 shows the actual and forecasted load demand on X and Y axes, respectively, and their regression plot for RF model. Similarly, Figures 16 and 17 display the actual and forecasted load demand and the regression plot of them using the LSTM model.

Figure 14.

Regression of actual and forecasted load demand by random forest (RF).

Figure 15.

Actual vs forecasted load demand using random forest (RF).

Figure 16.

Regression plot of actual and forecasted power demand using long short-term memory (LSTM).

Figure 17.

Actual vs forecasted load demand using long short-term memory (LSTM).

Visual similarity of the measured to the forecasted outputs in Figures 14 to 17 illustrates that the LSTM used prediction framework worked very well with excellent precision. And the forecasts are an accurate representation of real variances since the system is trained on well-curated data and multiple training epochs to model interdependencies in time. A feature of early stopping allowed model training to stop only at the most important time while maintaining generalization. The predictive power of this model is also confirmed by performance-based indices such as near-zero error levels and R² near 1.0. Indeed, while there is a small trade-off with precision on operational field usage, where there is no mitigation for changes in weather and sensor noise, these results illustrate the system's robustness and suggest that model holds promise for accurate and stable meteorological inputs.

The choice of hyperparameters for the LSTM and WOA was made by weighing prediction accuracy, convergence rate, and computational power. The number of layers, number of neurons per layer, learning rate, and batch size were optimized through cyclical experimentation based on validation loss. The selected configuration provides depth in the model to encapsulate long-term temporal dependencies in wind speed profiles, avoiding overfitting. Based upon pre-empirical experiments and previous experiments, the WOA hyperparameters, such as population size, maximum iterations, and convergence coefficients, were included in the exploration-exploitation balance parameters of the analysis WOA. This combination achieves stable convergence towards the global optimum while accounting for the nonlinear, nonconvex nature of the OPF problem with minimal perturbation. We therefore have set up settings to ensure a sound trade-off between computational cost and optimization efficiency.

Results and discussion

The main goal of this research is to optimize the total system profit of Japan's power grid when it is connected to the WF. The inclusion of WFs brings with it operational uncertainty because of the stochastic nature of wind generation, mainly from errors between actual and predicted wind speeds. The work is organized into two broad sections:

Imbalance cost evaluation using historical wind data: This section includes the estimation of imbalance costs using empirical wind data obtained from [22] and the evaluation of its effect on the economic performance and stability indicators of the grid.

Imbalance cost evaluation using ML-based forecasting: During this stage, ML algorithms are used to predict wind generation, and the resulting imbalance costs are analyzed to assess their impact on system economics and operational stability.

For both methods, three different optimization algorithms, that is, WOA, AZO, and SQP, are used to identify the most optimal strategy for reducing imbalance costs and maximizing grid profitability.

Imbalance cost evaluation using historical wind data

The OPF solution plays an important role in measuring the economic parameters as well as the system stability parameters while maintaining the considered objective functions. Table 5 shows the optimal generation at the generator buses for the peak, off-peak, and intermediate loading conditions. It is noted that the generator at bus 6 is not loaded, and the generators at bus 3 and 8 were loaded only during the peak load to meet the demand. The peak load on the system in this case is 259 MW, the off-peak load of 43.17 MW, and the intermediate loading conditions are 172.66 and 120.86 MW, respectively. This load situation has developed using the Load Scaling Factor, which is mentioned in Figure 4.

Without WF placement

Table 5.

Generation of generator (MW) buses during peak, off-peak, and intermediate loading.

Sl. No.	Bus No.	Hour 1	Hour 13	Hour 18	Hour 21
1	1	36.82	150.23	194.33	104.35
2	2	6.74	28.12	36.72	19.33
3	3	0	0	28.74	0
4	6	0	0	0	0
5	8	0	0	8.49	0

The LMP at all the generator buses for the above-mentioned loading conditions is shown in Figure 18. The LMP increases with increasing load on the system, with the highest LMP of 41.2 $/MWh at bus 14 during peak loading and the lowest of 23.2 $/MWh at bus 1 during off-peak loading. These variations are due to increasing congestion on the transmission line and line losses. The Profits of the system, which is the sum of profits at all the generators for an entire 24-h day, are shown in Figure 19. The hourly distribution of profit showcases significant variations influenced by factors such as energy demand. The data reveals a general trend where profits tend to be lower during the early hours of the day, gradually increasing during the morning, and peaking during the late afternoon and early evening. The graph for profits closely resembles the scaling factor curve considered in Figure 4, due to variations in LMP for varying load conditions. The highest profit in the system is 1970.8 $ at 18 th hour when the peak load occurred, and the lowest is 69.64 $ during off-peak load.

Figure 18.

Locational marginal price ($/h) for 14-bus systems using sequential quadratic programing (SQP) without wind power.

Figure 19.

Profit at every hour without wind power integration.

Figure 20 represents the voltage magnitudes at every bus during peak, off-peak, and intermediate loading in the 14-bus system. It can be observed that voltage levels are within the specified constraints and are relatively stable under all loading conditions, where a slight voltage drop is observed during the peak load. Also, generator buses are maintained at voltages close to 1 pu during no load and increased during loaded conditions to adjust for voltage drops and to maintain a stable voltage during loaded conditions.

With WF placement

Figure 20.

Voltage per unit for 14-bus systems without wind power integration.

Table 6 shows the forecasted and actual wind power (WPF, WPA) generation for 30 wind turbines (WT), and their generation costs are presented. Generation costs are derived by taking a general price for wind power, which is 3.75 $/MWh. After calculating the wind power generated for every hour, the OPF is calculated by adding the wind power at bus 4, which is a load bus and has higher connectivity with the rest of the system. The generation values for this scenario are shown in Table 7. It is noted that the generators reduced their capacity as compared to the case without wind power, and also, generators at buses 36,8 were not loaded at all. Figure 21 shows the LMP values at all 14 buses with wind power integration. LMP values were reduced notably as the generation cost of wind power is far less compared to that of the thermal system. Figure 22 shows the voltage levels at all buses, and the voltage regulation is also improved compared to the previous case. Table 8 shows the profit values for all 24 h, and Figure 23 is the bar chart representation of it.

Figure 21.

Locational marginal price ($/h) for 14 bus system.

Figure 22.

Voltage per unit for the 14 bus system.

Figure 23.

Profit from wind energy integration without imbalance costs at every hour.

Table 6.

Generation cost for forecasted and actual wind power for 30 turbines.

Hour	WPF for 30 WT in MW	GEN cost for WPF	WPA for 30 t in MW	GEN cost for WPA
1	3.34	12.54	3.34	12.54
2	0.99	3.71	0.12	0.46
3	0.99	3.71	0.99	3.71
4	0.12	0.46	3.34	12.54
5	3.34	12.54	3.34	12.54
6	26.75	100.30	15.48	58.04
7	63.40	237.75	26.75	100.30
8	63.40	237.75	42.47	159.27
9	63.40	237.75	15.48	58.04
10	63.40	237.75	26.75	100.30
11	26.75	100.30	26.75	100.30
12	15.48	58.04	7.92	29.72
13	7.62	29.72	7.29	29.72
14	7.92	29.72	3.34	12.24
15	3.34	12.54	3.34	12.54
16	3.34	12.54	0.99	3.71
17	0.12	0.46	0.12	0.46
18	7.92	29.72	0.99	3.71
19	63.40	237.75	42.47	159.27
20	42.47	159.27	42.47	159.27
21	42.47	1159.27	26.75	100.30
22	26.75	1200.30	26.75	100.30
23	26.75	100.30	15.48	58.04
24	15.48	58.04	7.92	29.72

WPF: wind power forecasted; WPA: wind power actual.

Table 7.

Generation of power in MW for 14-bus systems after wind power integration.

Sl. No.	Bus No.	Generation in MW
		Hour 1	Hour 4	Hour 8	Hour 12	Hour 16	Hour 20	Hour 24
1	1	36.73	71.21	136.51	97.42	142.44	96.08	37.65
2	2	6.73	13.12	25.48	18.03	26.61	17.79	6.9
3	3	0	0	0	0	0	0	0
4	6	0	0	0	0	0	0	0
5	8	0	0	0	0	0	0	0

Table 8.

Profit without considering imbalance costs after wind power integration using sequential quadratic programing (SQP).

Hour	Profit	Hour	Profit	Hour	Profit
1	71.9	9	1212.24	17	1800.31
2	117.09	10	829.52	18	3282.28
3	192.53	11	413.78	19	2678.14
4	338.68	12	694.58	20	1162.93
5	470.64	13	1227.36	21	983.65
6	1599.32	14	1109.39	22	532.33
7	2076.38	15	958.04	23	197.89
8	1427.61	16	1053.69	24	139.29

Profits were also increased significantly as wind power generation cost is less, and the generated power is sold at LMP, which results in more benefits and is also beneficiary for the consumers as the LMP is reduced compared to the previous case. All these results have been performed using SQP optimization techniques.

Imbalance cost's impact on the system

Imbalance cost refers to the financial penalties or charges incurred when there is a discrepancy between the forecasted and actual generation or consumption of electricity. This concept is especially relevant in power systems that integrate RES, such as wind power, due to their inherent variability and unpredictability. Imbalance costs arise from the need to maintain a balance between electricity supply and demand in real time to ensure the stability and reliability of the power grid. ISO or grid operator corrective measures must take actions to promote grid stability if the energy producers or consumers diverge from their planned electricity delivery or consumption. This could be done by positioning reserve generation, purchasing electricity from substitute sources, or regulating the load. Those interferences themselves come at a cost, and this is passed along to those responsible in the form of imbalance charges. They are charged according to both the greatness and direction of deviations, whether they are underproduction or overproduction. Imbalance costs are vital for RES producers because wind is intermittent. A WF, for example, might make certain forecasts of how much energy will be generated based on weather forecasts, but actual production could vary due to impulsive changes in wind speed. This not only reflects its own costs to the energy producer, but also the overall economic efficiency of the grid. The economic costs of imbalance can also create incentives for improved forecasting and operating practices.

A reliable prediction of energy production or consumption will reduce the deviations in energy consumption and, as a result, the financial penalties. The emergence of advanced forecasting features, including ML models along with real-time data integration, has served as the critical tools utilized in this segment to counteract the costs associated with imbalance. These approaches improve the predictability of renewable generation results and enable utilities to more effectively plan for potential reserve requirements. In addition, imbalance costs influence market strategies. Renewable energy producers must account for potential penalties when formulating bidding strategies in electricity markets. Similarly, grid operators use these costs as a mechanism to encourage participants to adhere to their schedules, thereby promoting grid reliability. Table 9 shows the profit considering the imbalance cost for all 24 h using SQP, and Figure 24 is the bar chart representation of it. By analyzing the results of profits with imbalance cost and comparing with the previous case where the imbalance costs were excluded with wind power integration, in most cases, the profits were less compared to profits without the imbalance cost. This is due to the slight mismatch between the actual and forecasted wind speeds, which is mostly overprediction, resulting in SCRs and causing a decrease in profits. But for hour 3, profit was increased as the actual power was more than the predicted power, and provided benefits to the generation side.

Figure 24.

Hourly profit with imbalance cost.

Table 9.

Profits with imbalance cost included using sequential quadratic programing (SQP).

Hour	Imbalance cost	Profit	Hour	Imbalance cost	Profit
1	−2.06	69.83	13	−15.97	1211.393
2	0	117.09	14	0	1109.39
3	135.64	328.16	15	−7.89	950.15
4	0	338.68	16	0	1053.69
5	−33.36	437.27	17	−52.09	1748.22
6	−118.33	1480.99	18	−81.21	3201.07
7	−68.41	2007.97	19	0	2678.14
8	−159.50	1268.11	20	−46.07	1116.86
9	−108.32	1103.92	21	0	983.65
10	0	829.52	22	−29.07	503.25
11	−20.09	393.69	23	−17.63	180.26
12	0	694.58	24	0	139.29

Table 10.

Profits with imbalance cost for the system with different optimization techniques.

Hour	Profit using SQP	Profit using WOA	Profit using AZO	Hour	Profit using SQP	Profit using WOA	Profit using AZO
1	69.83	70.66796	71.93998	13	1211.393	1225.93	1247.996
2	117.09	118.4951	120.628	14	1109.39	1122.703	1142.911
3	328.16	332.0979	338.0757	15	950.15	961.5518	978.8597
4	338.68	342.7442	348.9136	16	1053.69	1066.334	1085.528
5	437.27	442.5172	450.4826	17	1748.22	1769.199	1801.044
6	1480.99	1498.762	1525.74	18	3201.07	3239.483	3297.794
7	2007.97	2032.066	2068.643	19	2678.14	2710.278	2759.063
8	1268.11	1283.327	1306.427	20	1116.86	1130.262	1150.607
9	1103.92	1117.167	1137.276	21	983.65	995.4538	1013.372
10	829.52	839.4742	854.5848	22	503.25	509.289	518.4562
11	393.69	398.4143	405.5857	23	180.26	182.4231	185.7067
12	694.58	702.915	715.5674	24	139.29	140.9615	143.4988

WOA: whale optimization algorithm; AZO: American zebra optimization; SQP: sequential quadratic programing.

WOA and AZO have the following advantages compared to SQP when it comes to optimizing wind-integrated economic power system operations. First, both WOA and AZO are good global optimizers that minimize the potential for being trapped in local optimums, an area of limitation for SQP, which is a local search method that uses initial guesses. This power of searching the whole solution space makes it particularly useful in nonlinear, non-convex problems typical of wind power integrated systems. WOA and AZO are also easily adapted to tackle the uncertainty and stochasticity of wind generation since input data, such as wind power forecasts and speed, may vary. These methods are more suitable for handling uncertainty, in contrast to SQP, whose operation diminishes in the presence of uncertainty. Furthermore, WOA and AZO are more versatile since they neither need smoothness assumptions nor intricately detailed constraint statements and, therefore, are suitable for intricate, real-world systems. In contrast, SQP typically involves linearized approximations that could fail to capture the dynamic nature of wind power systems.

The convergence plot in Figure 25 sets up the performance of 4 metaheuristic algorithms (WOA, AZO, PSO, and GA) with the objective function values during 100 iterations. The results are given in Table 10. The best convergence with better convergence takes fewer iterations to get the best solution with a lesser frequency of iterations than any other algorithmic solution. Its early reduction of the objective function value in the first few iterations proves its good exploration, by its flattening at the end, where its efficient exploitation in the search space is shown without oscillations or an early convergence. In contrast, WOA and PSO converge on moderate levels with more iterations taken to achieve a goal. The small changes in the convergence curves of these algorithms imply the presence of local minima in highly complex search conditions. Because of their lower exploitation efficiency and the random crossover and mutation operators used, gradual convergence and the final objective value explain why GA always has the highest value. Overall, the success in the AZO algorithm also supports the fact that it possesses dynamic adjustment of parameters and the exploration–exploitation relationship with good performance and stability, and resolution accuracy. Therefore, this shows that AZO can be proposed as a great optimization tool in complex multi-objectives like optimal power dispatch or cost-optimization to balance in deregulated power systems.

Imbalance cost evaluation using ML-based forecasting

Figure 25.

Convergence characteristics of different optimization algorithms.

Applying LSTM and RF models to predict wind power has substantial benefits in terms of lowering imbalance prices over forecasted wind data that can be accessed from outside websites. LSTM, which is a form of RNN, is very good at extracting temporal dependencies and long-term trends in time-series data. In wind power prediction, LSTM can model sequential relationships in wind generation well and provide very accurate forecasting of power output in the future. By doing so, LSTM can keep deviations between predicted wind power and generated wind power low, directly saving the imbalance cost required to balance the demand and supply of a wind-integrated power grid. The more precise wind power predictions made by LSTM lead to less adjustment required in the system, hence reducing the economic costs of imbalance. RF, although not particularly designed for time-series prediction, nonetheless provides a robust alternative through the use of an ensemble of decision trees that can handle multiple input features, including past wind data, weather, and other pertinent variables. RF models improve prediction accuracy by integrating the prediction values of many decision trees, a capability that minimizes overfitting and facilitates capturing the intricate associations among the data. While RF may not represent temporal dependencies as well as LSTM, it has been shown to significantly improve on typical website forecasts to enable more accurate wind power forecasts that will, in turn, reduce the imbalance prices. The RF model is particularly applicable to many meteorological parameters, and it can be combined with a number of system inputs to produce accurate predictions.

Forecasted wind data available on websites usually depend on less complex statistical models or generic forecasting schemes, which usually do not capture the complex temporal and dynamic behavior of wind power generation. These forecasts are typically less precise and can result in greater forecast errors, making the difference between forecasted and actual wind output greater. Consequently, wind-integrated power systems based on website-based forecasts will have greater imbalances, leading to greater imbalance prices due to the necessity of more frequent balancing of supply and demand. In general, both LSTM and RF-based wind power forecasts are superior to conventional website-based forecasts in minimizing imbalance prices. LSTM provides the greatest improvement because of its sophisticated capacity to comprehend and forecast the time-varying behavior of wind power generation. Through more precise, real-time forecasts, both LSTM and RF enable a more economically efficient and stable operation of wind-integrated power systems, minimizing the monetary penalties of imbalances and improving overall system performance.

Table 11 and Figure 26 provide a graphical evaluation of system profitability and compare the outcomes after considering the imbalance price factor with multiple ML-based methods. The methods used to identify those results make it a highly accurate and comprehensive result that is both trustworthy and robust. It captures the financial gain reflecting both the positive revenue increases and the penalties suffered. This paper presents a comprehensive analysis of different operational results under various wind speed scenarios. The primary conclusion is that if forecasted wind speeds closely resemble real situations, then the effect of the imbalance pricing mechanism becomes virtually zero.

System economy assessment with optimization techniques

Figure 26.

System profit ($/h) with ML-based methods using SQP. ML: machine learning; SQP: sequential quadratic programing

Table 11.

System profit ($/h) with ML-based methods using SQP.

Hour	System profit with EPIM	System profit with EPIM using LSTM	System profit with EPIM using RF	Hour	System profit with EPIM	System profit with EPIM using LSTM	System profit with EPIM using RF
1	69.83	74.33426	73.83103	13	1211.393	1269.093	1260.502
2	117.09	121.0711	120.2514	14	1109.39	1147.109	1139.344
3	328.16	199.0657	197.718	15	950.15	990.6134	983.9071
4	338.68	350.1951	347.8244	16	1053.69	1089.515	1082.14
5	437.27	486.6314	483.337	17	1748.22	1861.521	1848.918
6	1480.99	1653.697	1642.502	18	3201.07	3393.878	3370.902
7	2007.97	2146.977	2132.442	19	2678.14	2769.197	2750.45
8	1268.11	1476.149	1466.155	20	1116.86	1202.47	1194.329
9	1103.92	1253.456	1244.97	21	983.65	1017.094	1010.209
10	829.52	857.7237	851.917	22	503.25	550.4189	546.6926
11	393.69	427.8485	424.9521	23	180.26	204.6183	203.233
12	694.58	718.1957	713.3337	24	139.29	144.0259	143.0508

LSTM: long short-term memory; ML: machine learning; RF: random forest; SQP: sequential quadratic programing.

It has been previously demonstrated that, in fact, system imbalance costs have negative impacts on economic performance and operational efficiency of generation units. Positive imbalance costs increase system profits, and negative imbalance costs decrease system profits. This study compares the SQP approach with the AZO method in both LSTM and RF-based forecasting frameworks, including those accounting for the effects imposed by imbalance pricing in the system. The study has proved that imbalance pricing can greatly improve system revenue by reducing financial risk from variations of wind power used as an adjustment to standard LSTM and RF methods. The second part is the AZO approach, which also compares well for all cases and exceeds SQP by optimizing the system as described in previous studies. AZO also achieves superior profit margins irrespective of forecasting method, indicating it is accurate. LSTM forecasts are often more profitable than RF on the whole and in all cases more profitable than RF forecasts, since LSTMs can better predict the sequential patterns in the wind speed dataset. The ultimate result is that together AZO and LSTM forecasting provide the superior competitive benefits in a balanced economy of costs, balance reduction, and profit increase. By providing such solutions, this demonstrates that sophisticated optimization and ML techniques can help to enhance the cost-effectiveness of renewables–integrated power systems.

Table 12 shows the profit outcomes of the system when imbalance costs are considered and assessed under various optimization techniques as well as ML-based forecasting techniques. The profitability is compared across different scenarios that include imbalance pricing, considering the optimization algorithms employed, and the forecasting models (like LSTM and RF). The table intends to indicate the effect of different ML-based methods on system profitability, especially when imbalance costs are included in the analysis.

Table 12.

Profits with imbalance cost for the system with different optimization techniques for different ML-based techniques.

Hour	Profit using SQP based on [22]	Profit using SQP based on LSTM	Profit using SQP based on RF	Profit using WOA based on [22]	Profit using WOA based on LSTM	Profit using WOA based on RF	Profit using AZO based on [22]	Profit using AZO based on LSTM	Profit using AZO based on RF
1	69.83	72.20	71.72	70.67	73.07	72.58	71.94	74.39	73.88
2	117.09	121.07	120.25	118.50	122.52	121.69	120.63	124.73	123.88
3	328.16	339.32	337.02	332.10	343.39	341.06	338.08	349.57	347.20
4	338.68	350.20	347.82	342.74	354.40	352.00	348.91	360.78	358.33
5	437.27	452.14	449.08	442.52	457.56	454.47	450.48	465.80	462.65
6	1480.99	1531.34	1520.98	1498.76	1549.72	1539.23	1525.74	1577.62	1566.93
7	2007.97	2076.24	2062.19	2032.07	2101.16	2086.93	2068.64	2138.98	2124.50
8	1268.11	1311.23	1302.35	1283.33	1326.96	1317.98	1306.43	1350.85	1341.70
9	1103.92	1141.45	1133.73	1117.17	1155.15	1147.33	1137.28	1175.94	1167.98
10	829.52	857.72	851.92	839.47	868.02	862.14	854.58	883.64	877.66
11	393.69	407.08	404.32	398.41	411.96	409.17	405.59	419.38	416.54
12	694.58	718.20	713.33	702.92	726.81	721.89	715.57	739.90	734.89
13	1211.39	1252.58	1244.10	1225.93	1267.61	1259.03	1248.00	1290.43	1281.69
14	1109.39	1147.11	1139.34	1122.70	1160.87	1153.02	1142.91	1181.77	1173.77
15	950.15	982.46	975.80	961.55	994.24	987.51	978.86	1012.14	1005.29
16	1053.69	1089.52	1082.14	1066.33	1102.59	1095.13	1085.53	1122.44	1114.84
17	1748.22	1807.66	1795.42	1769.20	1829.35	1816.97	1801.04	1862.28	1849.67
18	3201.07	3309.91	3287.50	3239.48	3349.63	3326.95	3297.79	3409.92	3386.83
19	2678.14	2769.20	2750.45	2710.28	2802.43	2783.46	2759.06	2852.87	2833.56
20	1116.86	1154.83	1147.02	1130.26	1168.69	1160.78	1150.61	1189.73	1181.67
21	983.65	1017.09	1010.21	995.45	1029.30	1022.33	1013.37	1047.83	1040.73
22	503.25	520.36	516.84	509.29	526.60	523.04	518.46	536.08	532.45
23	180.26	186.39	185.13	182.42	188.63	187.35	185.71	192.02	190.72
24	139.29	144.03	143.05	140.96	145.75	144.77	143.50	148.38	147.37

LSTM: long short-term memory; ML: machine learning; RF: random forest; WOA: whale optimization algorithm; AZO: American zebra optimization; SQP: sequential quadratic programing.

Although the proposed ML–metaheuristic optimization framework effectively achieves the performance of balancing the costs and benefits of balancing the imbalance and also maximizing the profits of the system, it has its drawbacks. The forecast performance of LSTM depends on the level, quality, and quantity of meteorological data, and whether the missing or noisy data result in a drop in the forecast performance. The trained and tested model was based on historical datasets under predetermined conditions, and may miss large-scale, unexpected real-world fluctuations, like abrupt environmental shifts or equipment failures. Furthermore, while the WOA and AZO have proven efficient for global search, the parameters of both need to be fine-tuned to avoid premature convergence and obtain computational efficiency in large-scale systems. Finally, the study is based on the IEEE 14-bus system, and therefore, its scalability to larger networks could be further investigated in order to validate robustness under more complex operational scenarios. While LSTM achieved better forecasting performance over the RF model in this study, there is still huge room for methodological scope for extending the forecast framework. More recent architectures, namely Gated Recurrent Units and hybrid CNN–LSTM models, will be used to further address this in the future, which have demonstrated the ability to capture both spatial and temporal dependencies in meteorological time-series data. Further, the current study employs a single location of wind data; hence, validation across geographic sites and other weather datasets will be conducted to gauge the generalization ability of the approach proposed. Such improvements will ultimately contribute to a broader, scalable, and geographically transferable forecasting and optimization framework.

Based on the simulation results, the operational and economic impacts on Japan's deregulated electricity market come out to be a significant concern. The decrease in imbalance costs and improvement of system profitability proved the practicality of using the proposed ML–metaheuristic framework to help Japan's ongoing transformation to a competitive low-carbon energy market. Thus, LSTM could improve wind power forecasting, promoting renewable producers to bid better in the day-ahead market, and thereby avoiding any monetary penalties introduced by the imbalance settlement mechanism. Also, WOA and AZO-powered optimization strategies can help industry members to achieve superior profitability without compromising system performance. These results closely represent the objectives of Japan's national renewable policy outlined under the Strategic Energy Plan to liberalize the market and transition to carbon neutrality by 2050, with more variable renewable energy generation. Thus, the results not only confirm the possible technical capability of our model but also enhance its applicability in policy-making, investment choice, and operational aspects of the sustainable Japanese power market. To evaluate the efficiency and convergence characteristics of the proposed AZO-based optimization framework, these simulations were performed on the IEEE 14-bus system. The IEEE 14-bus system is an established model, providing a benchmark that helps evaluate the algorithm's performance under controlled conditions. Larger systems (for example, IEEE 57-bus and 118-bus networks) have not been addressed in the current work; future work will extend this study to examine the scalability and computational robustness of larger systems. It is anticipated that, given the results presented in the present work and the stability shown by the algorithm, the proposed framework can provide comparable convergence behavior and similar optimization efficiency in larger and more complex systems. Along with the financial advantages derived from the simulation, real-world deployment of the developed ML metaheuristic mechanism under Japan's deregulated electricity market must be taken into account. Japan has a market design with zonal limitations, fragmented regional grids, and strict imbalance settlement rules that have made developing new forecasting methods that can function reliably under strict regulations very difficult. The LSTM-based forecasting and AZO/WOA optimization modules proposed to be implemented can be incorporated in existing operational structures in applications such as aggregators who already manage distributed renewable portfolios and are expected to make balanced day-ahead bids. Since imbalance penalties under Japan's Balancing Market can be partially mitigated by utilizing storage assets, such as BESS, pumped-hydro units, or demand-response resources, this framework can also be extended for co-optimizing the wind forecasts with storage dispatch, reserve procurement, and intra-hour correction strategies. In addition, the model corresponds to the current METI and OCCTO regulatory developments, which put great focus on flexible markets and the role of aggregators in balancing services. By exploiting system constraints, storage ramping restrictions, interconnector capacities between regional grids, and aggregator-level bidding activities, the approach is technically appropriate to Japan's actual market operation. Furthermore, in addition to theory-based performance in the IEEE 14-bus benchmark, our approach could be directly incorporated into Japan's continuously deregulated market system architecture and offer an alternative way for mitigating the exposure to imbalance, increasing participation in renewables, and enabling system long-term decarbonization.

Conclusion

Japan's effort to create a sustainable, low-carbon electricity system must be of high quality by integrating variable renewable energy, primarily wind power, with reliable integration. But it's a country's wildly uneven grid and inflexible price-for-imbalance rules that create major operational hurdles for renewable producers. This work demonstrates that accurate forecasting, along with appropriate optimization, can reduce these challenges to an important extent. With the LSTM forecasting, it can greatly reduce the prediction error, which would help make dispatch scheduling more accurate, reserve planning better, and the importance of traditional thermal units less. When combined with WOA, AZO, and SQP optimization strategies, the overall structure helps in minimizing the imbalance cost and increasing the profitability at the system level, indicating its major interest between the system owner and market players. The results also highlight that the model can support storage integration and flexible demand side resources—increasingly crucial as renewable penetration grows. Even though verified on the IEEE 14-bus system, the approach is generalizable to more extensive, more complex grid setups in Japan due to its modularity and the utilization of common OPF guidelines. The theoretical ML–optimization framework provides effective mitigations not only to reduce financial risk and improve operational stability, but also serves as a positive contribution to Japan's deregulated market environment as well as to Japan's national decarbonization and energy security goals. The article emphasizes the importance of combining advanced forecasting with economic optimization for resilient renewables integration.