In the past two decades,Precision Agriculture has received research attention since the development of robotics.Agricultural robotic equipment and drones,which can be operated by farmers,are appearing more frequently ...In the past two decades,Precision Agriculture has received research attention since the development of robotics.Agricultural robotic equipment and drones,which can be operated by farmers,are appearing more frequently and being used to make the process of farming easier and more productive.This paper attempts to develop a modified Q-learning algorithm.A reinforcement learning algorithm called Q-learning has Q-values that are updated in order to find the best routes for the robotic devices to follow while avoiding any obstacles.Different types of terrain and other factors that influence the development of good routes for the robotic devices are included in the experiments performed.Through an extensive set of experiments done with different types of terrain the researchers found that the modified Q-learning algorithm converges to the optimal path significantly quicker than the current benchmark Deep Q-Network(DQN)algorithms and that the average distance that the modified Q-learning algorithm travels to get to its destination over different terrane types was 28.7%shorter than the average distance traveled using the standard DQNs.The researchers also found that the modified Q-learning algorithm has been able to successfully avoid obstacles on 99.5%of all occasions tested.The shortest route to the destination is expected to take less time,and it demonstrates the benefit of using a robotic device that has the ability to detect and avoid obstacles in order to be effective on more difficult types of terrain.展开更多
In this paper,a novel opportunistic scheduling(OS)scheme with antenna selection(AS)for the energy harvesting(EH)cooperative communication system where the relay can harvest energy from the source transmission is propo...In this paper,a novel opportunistic scheduling(OS)scheme with antenna selection(AS)for the energy harvesting(EH)cooperative communication system where the relay can harvest energy from the source transmission is proposed.In this considered scheme,we take into both traditional mathematical analysis and reinforcement learning(RL)scenarios with the power splitting(PS)factor constraint.For the case of traditional mathematical analysis of a fixed-PS factor,we derive an exact closed-form expressions for the ergodic capacity and outage probability in general signal-to-noise ratio(SNR)regime.Then,we combine the optimal PS factor with performance metrics to achieve the optimal transmission performance.Subsequently,based on the optimized PS factor,a RL technique called as Q-learning(QL)algorithm is proposed to derive the optimal antenna selection strategy.To highlight the performance advantage of the proposed QL with training the received SNR at the destination,we also examine the scenario of QL scheme with training channel between the relay and the destination.The results illustrate that,the optimized scheme is always superior to the fixed-PS factor scheme.In addition,a better system parameter setting with QL significantly outperforms the traditional mathematical analysis scheme.展开更多
The major environmental hazard in this pandemic is the unhygienic dis-posal of medical waste.Medical wastage is not properly managed it will become a hazard to the environment and humans.Managing medical wastage is a ...The major environmental hazard in this pandemic is the unhygienic dis-posal of medical waste.Medical wastage is not properly managed it will become a hazard to the environment and humans.Managing medical wastage is a major issue in the city,municipalities in the aspects of the environment,and logistics.An efficient supply chain with edge computing technology is used in managing medical waste.The supply chain operations include processing of waste collec-tion,transportation,and disposal of waste.Many research works have been applied to improve the management of wastage.The main issues in the existing techniques are ineffective and expensive and centralized edge computing which leads to failure in providing security,trustworthiness,and transparency.To over-come these issues,in this paper we implement an efficient Naive Bayes classifier algorithm and Q-Learning algorithm in decentralized edge computing technology with a binary bat optimization algorithm(NBQ-BBOA).This proposed work is used to track,detect,and manage medical waste.To minimize the transferring cost of medical wastage from various nodes,the Q-Learning algorithm is used.The accuracy obtained for the Naïve Bayes algorithm is 88%,the Q-Learning algo-rithm is 82%and NBQ-BBOA is 98%.The error rate of Root Mean Square Error(RMSE)and Mean Error(MAE)for the proposed work NBQ-BBOA are 0.012 and 0.045.展开更多
Reinforcement learning is an excellent approach which is used in artificial intelligence,automatic control, etc. However, ordinary reinforcement learning algorithm, such as Q-learning with lookup table cannot cope wit...Reinforcement learning is an excellent approach which is used in artificial intelligence,automatic control, etc. However, ordinary reinforcement learning algorithm, such as Q-learning with lookup table cannot cope with extremely complex and dynamic environment due to the huge state space. To reduce the state space, modular neural network Q-learning algorithm is proposed, which combines Q-learning algorithm with neural network and module method. Forward feedback neural network, Elman neural network and radius-basis neural network are separately employed to construct such algorithm. It is revealed that Elman neural network Q-learning algorithm has the best performance under the condition that the same neural network training method, i.e. gradient descent error back-propagation algorithm is applied.展开更多
Aiming at the dimension disaster problem, poor model generalization ability and deadlock problem in special obstacles environment caused by the increase of state information in the local path planning process of mobil...Aiming at the dimension disaster problem, poor model generalization ability and deadlock problem in special obstacles environment caused by the increase of state information in the local path planning process of mobile robot, this paper proposed a Double BP Q-learning algorithm based on the fusion of Double Q-learning algorithm and BP neural network. In order to solve the dimensional disaster problem, two BP neural network fitting value functions with the same network structure were used to replace the two Q value tables in Double Q-Learning algorithm to solve the problem that the Q value table cannot store excessive state information. By adding the mechanism of priority experience replay and using the parameter transfer to initialize the model parameters in different environments, it could accelerate the convergence rate of the algorithm, improve the learning efficiency and the generalization ability of the model. By designing specific action selection strategy in special environment, the deadlock state could be avoided and the mobile robot could reach the target point. Finally, the designed Double BP Q-learning algorithm was simulated and verified, and the probability of mobile robot reaching the target point in the parameter update process was compared with the Double Q-learning algorithm under the same condition of the planned path length. The results showed that the model trained by the improved Double BP Q-learning algorithm had a higher success rate in finding the optimal or sub-optimal path in the dense discrete environment, besides, it had stronger model generalization ability, fewer redundant sections, and could reach the target point without entering the deadlock zone in the special obstacles environment.展开更多
The Distributed Flexible Job Shop Scheduling Problem(DFJSP)is critical in modern manufacturing;however,existing research has not sufficiently addressed dynamic disturbances,particularly unexpected worker absences.This...The Distributed Flexible Job Shop Scheduling Problem(DFJSP)is critical in modern manufacturing;however,existing research has not sufficiently addressed dynamic disturbances,particularly unexpected worker absences.This study extends the DFJSPW model to introduce an enhanced framework,DFJSPWA,which optimizes maximum makespan,worker workload,and total energy consumption by integrating worker load factors with random absenteeism.To solve this complex problem,we propose a Q-learning-based Hyper-heuristic Evolutionary Algorithm(QLHHEA).This algorithm features a segmented encoding scheme that implicitly captures absenteeism information,utilizing a decoding process tailored for both standard and rescheduling contexts.Additionally,we construct a pool of twelve efficient Low-Level Heuristics(LLHs)combined with Q-learning to enable the adaptive selection of operators.Furthermore,a Hybrid Rescheduling Method(HRM)is developed,employing three response strategies based on worker status and the urgency of the absenteeism.Comprehensive experiments on 58 benchmark instances demonstrate that QLHHEA significantly outperforms six established algorithms,including MOEA/D and NSGA-II.Statistical validation confirms the superiority of the proposed method.This research provides a robust theoretical and methodological framework for improving scheduling efficiency and resource utilization in distributed production systems facing worker absenteeism.展开更多
To address low learning efficiency and inadequate path safety in spraying robot navigation within complex obstacle-rich environments—with dense,dynamic,unpredictable obstacles challenging conventional methods—this p...To address low learning efficiency and inadequate path safety in spraying robot navigation within complex obstacle-rich environments—with dense,dynamic,unpredictable obstacles challenging conventional methods—this paper proposes a hybrid algorithm integrating Q-learning and improved A*-Artificial Potential Field(A-APF).Centered on theQ-learning framework,the algorithmleverages safety-oriented guidance generated byA-APF and employs a dynamic coordination mechanism that adaptively balances exploration and exploitation.The proposed system comprises four core modules:(1)an environment modeling module that constructs grid-based obstacle maps;(2)an A-APF module that combines heuristic search from A*algorithm with repulsive force strategies from APF to generate guidance;(3)a Q-learning module that learns optimal state-action values(Q-values)through spraying robot-environment interaction and a reward function emphasizing path optimality and safety;and(4)a dynamic optimization module that ensures adaptive cooperation between Q-learning and A-APF through exploration rate control and environment-aware constraints.Simulation results demonstrate that the proposed method significantly enhances path safety in complex underground mining environments.Quantitative results indicate that,compared to the traditional Q-learning algorithm,the proposed method shortens training time by 42.95% and achieves a reduction in training failures from 78 to just 3.Compared to the static fusion algorithm,it further reduces both training time(by 10.78%)and training failures(by 50%),thereby improving overall training efficiency.展开更多
Vertical roller mills are essential for energy-intensive grinding in cement,minerals,and metallurgy industries,consuming up to 50%of plant electricity and frequently experiencing operational instabilities(including ex...Vertical roller mills are essential for energy-intensive grinding in cement,minerals,and metallurgy industries,consuming up to 50%of plant electricity and frequently experiencing operational instabilities(including excessive vibration and main motor current fluctuations)that drive unplanned downtime,increased wear,and reduced throughput.Despite their importance,real-time autonomous optimization remains challenging due to the nonlinear interactions among grinding pressure,feed rate,separator speed,and aerodynamic factors,which limit traditional control strategies under varying loads.This paper presents a real-time operational optimization system for large-scale vertical roller mills using big industrial data and artificial intelligence(AI).From a 5400 kW Loesche LM56.4 mill,2,764,800 samples were collected at 1 Hz over 32 days of continuous production.A systematic pipeline was developed:quartile-based outlier-robust cleaning;domain-informed feature engineering including Total Current;Random Forest(RF)permutation importance selection of the top 15 parameters;and Extreme Gradient Boosting(XGBoost)regression models with hyperparameters tuned by Tree-structured Parzen Estimator(TPE)Bayesian optimization.The resulting models achieved strong predictive performance,Mean Absolute Percentage Error(MAPE)of 1.3%(95%CI:1.1%–1.5%)for main motor current(R2=0.9997)and 5.8%(95%CI:5.3%–6.3%)for shell vibration(R2=0.9717),representing reductions of 89%and 59%,respectively,relative to the Long Short-Term Memory(LSTM)baseline.These surrogates were embedded into a tabular Q-learning Reinforcement Learning(RL)agent that autonomously adjusts feed rate,grinding pressure,separator speed,and exhaust damper position via a discrete action space and multi-objective reward function,communicating with the Distributed Control System(DCS)via Open Platform Communications Unified Architecture(OPC-UA).Closed-loop evaluation yielded simultaneous reductions of 6.0%in peak current(181.92→170.04 A)and 9.4%in peak vibration(5.51→4.99 mm/s)while maintaining throughput.A PyQt5-based graphical interface enabling real-time monitoring,predictive alerts,and automatic DCS write-back was deployed and operated stably for two weeks.展开更多
Q-learning is a classical reinforcement learning method with broad applicability.It can respond effectively to environmental changes and provide flexible strategies,making it suitable for solving robot path-planning p...Q-learning is a classical reinforcement learning method with broad applicability.It can respond effectively to environmental changes and provide flexible strategies,making it suitable for solving robot path-planning problems.However,Q-learning faces challenges in search and update efficiency.To address these issues,we propose an improved Q-learning(IQL)algorithm.We use an enhanced Ant Colony Optimization(ACO)algorithmto optimizeQtable initialization.We also introduce the UCH mechanism to refine the reward function and overcome the exploration dilemma.The IQL algorithm is extensively tested in three grid environments of different scales.The results validate the accuracy of themethod and demonstrate superior path-planning performance compared to traditional approaches.The algorithm reduces the number of trials required for convergence,improves learning efficiency,and enables faster adaptation to environmental changes.It also enhances stability and accuracy by reducing the standard deviation of trials to zero.On grid maps of different sizes,IQL achieves higher expected returns.Compared with the original Q-learning algorithm,IQL improves performance by 12.95%,18.28%,and 7.98% on 10*10,20*20,and 30*30 maps,respectively.The proposed algorithm has promising applications in robotics,path planning,intelligent transportation,aerospace,and game development.展开更多
The integration of High-Altitude Platform Stations(HAPS)with Reconfigurable Intelligent Surfaces(RIS)represents a critical advancement for next-generation wireless networks,offering unprecedented opportunities for ubi...The integration of High-Altitude Platform Stations(HAPS)with Reconfigurable Intelligent Surfaces(RIS)represents a critical advancement for next-generation wireless networks,offering unprecedented opportunities for ubiquitous connectivity.However,existing research reveals significant gaps in dynamic resource allocation,joint optimization,and equitable service provisioning under varying channel conditions,limiting practical deployment of these technologies.This paper addresses these challenges by proposing a novel Fairness-Aware Deep Q-Learning(FAIRDQL)framework for joint resource management and phase configuration in HAPS-RIS systems.Our methodology employs a comprehensive three-tier algorithmic architecture integrating adaptive power control,priority-based user scheduling,and dynamic learning mechanisms.The FAIR-DQL approach utilizes advanced reinforcement learning with experience replay and fairness-aware reward functions to balance competing objectives while adapting to dynamic environments.Key findings demonstrate substantial improvements:9.15 dB SINR gain,12.5 bps/Hz capacity,78%power efficiency,and 0.82 fairness index.The framework achieves rapid 40-episode convergence with consistent delay performance.These contributions establish new benchmarks for fairness-aware resource allocation in aerial communications,enabling practical HAPS-RIS deployments in rural connectivity,emergency communications,and urban networks.展开更多
基金supported by LUT internationalization fundsupported by Createch Wake Up Etelä-Karjala!project,Co-funded by the European Union.
摘要In the past two decades,Precision Agriculture has received research attention since the development of robotics.Agricultural robotic equipment and drones,which can be operated by farmers,are appearing more frequently and being used to make the process of farming easier and more productive.This paper attempts to develop a modified Q-learning algorithm.A reinforcement learning algorithm called Q-learning has Q-values that are updated in order to find the best routes for the robotic devices to follow while avoiding any obstacles.Different types of terrain and other factors that influence the development of good routes for the robotic devices are included in the experiments performed.Through an extensive set of experiments done with different types of terrain the researchers found that the modified Q-learning algorithm converges to the optimal path significantly quicker than the current benchmark Deep Q-Network(DQN)algorithms and that the average distance that the modified Q-learning algorithm travels to get to its destination over different terrane types was 28.7%shorter than the average distance traveled using the standard DQNs.The researchers also found that the modified Q-learning algorithm has been able to successfully avoid obstacles on 99.5%of all occasions tested.The shortest route to the destination is expected to take less time,and it demonstrates the benefit of using a robotic device that has the ability to detect and avoid obstacles in order to be effective on more difficult types of terrain.
基金supported in part by the National Natural Science Foundation of China under Grant 61720106003,Grant 61401165,Grant 61379006,Grant 61671144,and Grant 61701538in part by the Natural Science Foundation of Fujian Province under Grants 2015J01262+3 种基金in part by Promotion Program for Young and Middle-aged Teacher in Science and Technology Research of Huaqiao University under Grant ZQN-PY407in part by Science and Technology Innovation Teams of Henan Province for Colleges and Universities(17IRTSTHN014)in part by the Scientific and Technological Key Project of Henan Province under Grant 172102210080 and Grant 182102210449in part by the Collaborative Innovation Center for Aviation Economy Development of Henan Province。
摘要In this paper,a novel opportunistic scheduling(OS)scheme with antenna selection(AS)for the energy harvesting(EH)cooperative communication system where the relay can harvest energy from the source transmission is proposed.In this considered scheme,we take into both traditional mathematical analysis and reinforcement learning(RL)scenarios with the power splitting(PS)factor constraint.For the case of traditional mathematical analysis of a fixed-PS factor,we derive an exact closed-form expressions for the ergodic capacity and outage probability in general signal-to-noise ratio(SNR)regime.Then,we combine the optimal PS factor with performance metrics to achieve the optimal transmission performance.Subsequently,based on the optimized PS factor,a RL technique called as Q-learning(QL)algorithm is proposed to derive the optimal antenna selection strategy.To highlight the performance advantage of the proposed QL with training the received SNR at the destination,we also examine the scenario of QL scheme with training channel between the relay and the destination.The results illustrate that,the optimized scheme is always superior to the fixed-PS factor scheme.In addition,a better system parameter setting with QL significantly outperforms the traditional mathematical analysis scheme.
摘要The major environmental hazard in this pandemic is the unhygienic dis-posal of medical waste.Medical wastage is not properly managed it will become a hazard to the environment and humans.Managing medical wastage is a major issue in the city,municipalities in the aspects of the environment,and logistics.An efficient supply chain with edge computing technology is used in managing medical waste.The supply chain operations include processing of waste collec-tion,transportation,and disposal of waste.Many research works have been applied to improve the management of wastage.The main issues in the existing techniques are ineffective and expensive and centralized edge computing which leads to failure in providing security,trustworthiness,and transparency.To over-come these issues,in this paper we implement an efficient Naive Bayes classifier algorithm and Q-Learning algorithm in decentralized edge computing technology with a binary bat optimization algorithm(NBQ-BBOA).This proposed work is used to track,detect,and manage medical waste.To minimize the transferring cost of medical wastage from various nodes,the Q-Learning algorithm is used.The accuracy obtained for the Naïve Bayes algorithm is 88%,the Q-Learning algo-rithm is 82%and NBQ-BBOA is 98%.The error rate of Root Mean Square Error(RMSE)and Mean Error(MAE)for the proposed work NBQ-BBOA are 0.012 and 0.045.
摘要Reinforcement learning is an excellent approach which is used in artificial intelligence,automatic control, etc. However, ordinary reinforcement learning algorithm, such as Q-learning with lookup table cannot cope with extremely complex and dynamic environment due to the huge state space. To reduce the state space, modular neural network Q-learning algorithm is proposed, which combines Q-learning algorithm with neural network and module method. Forward feedback neural network, Elman neural network and radius-basis neural network are separately employed to construct such algorithm. It is revealed that Elman neural network Q-learning algorithm has the best performance under the condition that the same neural network training method, i.e. gradient descent error back-propagation algorithm is applied.
摘要Aiming at the dimension disaster problem, poor model generalization ability and deadlock problem in special obstacles environment caused by the increase of state information in the local path planning process of mobile robot, this paper proposed a Double BP Q-learning algorithm based on the fusion of Double Q-learning algorithm and BP neural network. In order to solve the dimensional disaster problem, two BP neural network fitting value functions with the same network structure were used to replace the two Q value tables in Double Q-Learning algorithm to solve the problem that the Q value table cannot store excessive state information. By adding the mechanism of priority experience replay and using the parameter transfer to initialize the model parameters in different environments, it could accelerate the convergence rate of the algorithm, improve the learning efficiency and the generalization ability of the model. By designing specific action selection strategy in special environment, the deadlock state could be avoided and the mobile robot could reach the target point. Finally, the designed Double BP Q-learning algorithm was simulated and verified, and the probability of mobile robot reaching the target point in the parameter update process was compared with the Double Q-learning algorithm under the same condition of the planned path length. The results showed that the model trained by the improved Double BP Q-learning algorithm had a higher success rate in finding the optimal or sub-optimal path in the dense discrete environment, besides, it had stronger model generalization ability, fewer redundant sections, and could reach the target point without entering the deadlock zone in the special obstacles environment.
基金supported by National Natural Science Foundation of China under Grant U21A20464,62066005.
摘要The Distributed Flexible Job Shop Scheduling Problem(DFJSP)is critical in modern manufacturing;however,existing research has not sufficiently addressed dynamic disturbances,particularly unexpected worker absences.This study extends the DFJSPW model to introduce an enhanced framework,DFJSPWA,which optimizes maximum makespan,worker workload,and total energy consumption by integrating worker load factors with random absenteeism.To solve this complex problem,we propose a Q-learning-based Hyper-heuristic Evolutionary Algorithm(QLHHEA).This algorithm features a segmented encoding scheme that implicitly captures absenteeism information,utilizing a decoding process tailored for both standard and rescheduling contexts.Additionally,we construct a pool of twelve efficient Low-Level Heuristics(LLHs)combined with Q-learning to enable the adaptive selection of operators.Furthermore,a Hybrid Rescheduling Method(HRM)is developed,employing three response strategies based on worker status and the urgency of the absenteeism.Comprehensive experiments on 58 benchmark instances demonstrate that QLHHEA significantly outperforms six established algorithms,including MOEA/D and NSGA-II.Statistical validation confirms the superiority of the proposed method.This research provides a robust theoretical and methodological framework for improving scheduling efficiency and resource utilization in distributed production systems facing worker absenteeism.
基金supported by the National Natural Science Foundation of China(Grant No.52374156).
摘要To address low learning efficiency and inadequate path safety in spraying robot navigation within complex obstacle-rich environments—with dense,dynamic,unpredictable obstacles challenging conventional methods—this paper proposes a hybrid algorithm integrating Q-learning and improved A*-Artificial Potential Field(A-APF).Centered on theQ-learning framework,the algorithmleverages safety-oriented guidance generated byA-APF and employs a dynamic coordination mechanism that adaptively balances exploration and exploitation.The proposed system comprises four core modules:(1)an environment modeling module that constructs grid-based obstacle maps;(2)an A-APF module that combines heuristic search from A*algorithm with repulsive force strategies from APF to generate guidance;(3)a Q-learning module that learns optimal state-action values(Q-values)through spraying robot-environment interaction and a reward function emphasizing path optimality and safety;and(4)a dynamic optimization module that ensures adaptive cooperation between Q-learning and A-APF through exploration rate control and environment-aware constraints.Simulation results demonstrate that the proposed method significantly enhances path safety in complex underground mining environments.Quantitative results indicate that,compared to the traditional Q-learning algorithm,the proposed method shortens training time by 42.95% and achieves a reduction in training failures from 78 to just 3.Compared to the static fusion algorithm,it further reduces both training time(by 10.78%)and training failures(by 50%),thereby improving overall training efficiency.
基金funded by the Zhejiang ProvincialNatural Science Foundation of China(Baima Lake Laboratory Joint Fund),grant number LBMHZ25F030002the National Natural Science Foundation of China,grant number 52372420+3 种基金the Guangdong Basic and Applied Basic Research Foundation(Offshore Wind Power Joint Fund),grant number 2024A1515240073the Scientific Research Foundation of Hangzhou City University,grant number X-202404the Zhejiang Province Key Research Project,grant numbers 2025C02242 and 2024C01039Ningbo’s Key Technology Breakthrough Program of KeChuang Yongjiang 2035,grant number 2024Z177.
摘要Vertical roller mills are essential for energy-intensive grinding in cement,minerals,and metallurgy industries,consuming up to 50%of plant electricity and frequently experiencing operational instabilities(including excessive vibration and main motor current fluctuations)that drive unplanned downtime,increased wear,and reduced throughput.Despite their importance,real-time autonomous optimization remains challenging due to the nonlinear interactions among grinding pressure,feed rate,separator speed,and aerodynamic factors,which limit traditional control strategies under varying loads.This paper presents a real-time operational optimization system for large-scale vertical roller mills using big industrial data and artificial intelligence(AI).From a 5400 kW Loesche LM56.4 mill,2,764,800 samples were collected at 1 Hz over 32 days of continuous production.A systematic pipeline was developed:quartile-based outlier-robust cleaning;domain-informed feature engineering including Total Current;Random Forest(RF)permutation importance selection of the top 15 parameters;and Extreme Gradient Boosting(XGBoost)regression models with hyperparameters tuned by Tree-structured Parzen Estimator(TPE)Bayesian optimization.The resulting models achieved strong predictive performance,Mean Absolute Percentage Error(MAPE)of 1.3%(95%CI:1.1%–1.5%)for main motor current(R2=0.9997)and 5.8%(95%CI:5.3%–6.3%)for shell vibration(R2=0.9717),representing reductions of 89%and 59%,respectively,relative to the Long Short-Term Memory(LSTM)baseline.These surrogates were embedded into a tabular Q-learning Reinforcement Learning(RL)agent that autonomously adjusts feed rate,grinding pressure,separator speed,and exhaust damper position via a discrete action space and multi-objective reward function,communicating with the Distributed Control System(DCS)via Open Platform Communications Unified Architecture(OPC-UA).Closed-loop evaluation yielded simultaneous reductions of 6.0%in peak current(181.92→170.04 A)and 9.4%in peak vibration(5.51→4.99 mm/s)while maintaining throughput.A PyQt5-based graphical interface enabling real-time monitoring,predictive alerts,and automatic DCS write-back was deployed and operated stably for two weeks.
基金Financial supports from the National Natural Science Foundation of China(GrantNo.52374123&51974144)Project of Liaoning Provincial Department of Education(GrantNo.LJKZ0340)Liaoning Revitalization Talents Program(Grant No.XLYC2211085)are greatly acknowledged.
摘要Q-learning is a classical reinforcement learning method with broad applicability.It can respond effectively to environmental changes and provide flexible strategies,making it suitable for solving robot path-planning problems.However,Q-learning faces challenges in search and update efficiency.To address these issues,we propose an improved Q-learning(IQL)algorithm.We use an enhanced Ant Colony Optimization(ACO)algorithmto optimizeQtable initialization.We also introduce the UCH mechanism to refine the reward function and overcome the exploration dilemma.The IQL algorithm is extensively tested in three grid environments of different scales.The results validate the accuracy of themethod and demonstrate superior path-planning performance compared to traditional approaches.The algorithm reduces the number of trials required for convergence,improves learning efficiency,and enables faster adaptation to environmental changes.It also enhances stability and accuracy by reducing the standard deviation of trials to zero.On grid maps of different sizes,IQL achieves higher expected returns.Compared with the original Q-learning algorithm,IQL improves performance by 12.95%,18.28%,and 7.98% on 10*10,20*20,and 30*30 maps,respectively.The proposed algorithm has promising applications in robotics,path planning,intelligent transportation,aerospace,and game development.
基金supported by the Princess Nourah bint Abdulrahman University Researchers Supporting Project,number PNURSP2025R757Princess Nourah bint Abdulrahman University,Riyadh,Saudi Arabia.
摘要The integration of High-Altitude Platform Stations(HAPS)with Reconfigurable Intelligent Surfaces(RIS)represents a critical advancement for next-generation wireless networks,offering unprecedented opportunities for ubiquitous connectivity.However,existing research reveals significant gaps in dynamic resource allocation,joint optimization,and equitable service provisioning under varying channel conditions,limiting practical deployment of these technologies.This paper addresses these challenges by proposing a novel Fairness-Aware Deep Q-Learning(FAIRDQL)framework for joint resource management and phase configuration in HAPS-RIS systems.Our methodology employs a comprehensive three-tier algorithmic architecture integrating adaptive power control,priority-based user scheduling,and dynamic learning mechanisms.The FAIR-DQL approach utilizes advanced reinforcement learning with experience replay and fairness-aware reward functions to balance competing objectives while adapting to dynamic environments.Key findings demonstrate substantial improvements:9.15 dB SINR gain,12.5 bps/Hz capacity,78%power efficiency,and 0.82 fairness index.The framework achieves rapid 40-episode convergence with consistent delay performance.These contributions establish new benchmarks for fairness-aware resource allocation in aerial communications,enabling practical HAPS-RIS deployments in rural connectivity,emergency communications,and urban networks.