We present a framework for the automatic encoding and repair of high-level tasks. Given a set of skills a robot can perform, our approach first abstracts sensor data into symbols and then automatically encodes the robot’s capabilities in Linear Temporal Logic (LTL). Using this encoding, a user can specify reactive high-level tasks, for which we can automatically synthesize a strategy that executes on the robot, if the task is feasible. If a task is not feasible given the robot’s capabilities, we present two methods, one enumeration-based and one synthesis-based, for automatically suggesting additional skills for the robot or modifications to existing skills that would make the task feasible. We demonstrate our framework on a Baxter robot manipulating blocks on a table, a Baxter robot manipulating plates on a table, and a Kinova arm manipulating vials, with multiple sensor modalities, including raw images.
AhmetogluASekerMSayinA, et al. (2020) DeepSym: deep symbol generation and rule learning from unsupervised continuous robot interaction for planning. ArXiv preprint arXiv:2012.02532.
2.
AlurRMoarrefSTopcuU (2013) Counter-strategy guided refinement of gr (1) temporal logic specifications. In: Formal methods in computer-aided design, Portland, OR, 20–23 October 2013, pp. 26–33. IEEE.
3.
AmesBThackstonAKonidarisG (2018) Learning symbolic representations for planning with parameterized skills. In: IEEE/RSJ international conference on intelligent robots and systems, Madrid, Spain, 01–05 October 2018, pp. 526–533. IEEE.
4.
AsaiM (2021) Unsupervised grounding of plannable first-order logic representation from images. Proceedings of the International Conference on Automated Planning and Scheduling29: 583–591.
5.
AsaiMFukunagaA (2018) Classical planning in deep latent space: bridging the subsymbolic-symbolic boundaryProceedings of the AAAI Conference on Artificial Intelligence32.
6.
BloemRJobstmannBPitermanN, et al. (2012) Synthesis of reactive (1) designs. Journal of Computer and System Sciences78(3): 911–938.
7.
ChatterjeeKHenzingerTAJobstmannB (2008) Environment assumptions for synthesis. International Conference on Concurrency Theory. Berlin, Germany: Springer, 147–161.
DantamNTKingstonZKChaudhuriS, et al. (2018) An incremental constraint-based framework for task and motion planning. The International Journal of Robotics Research37(10): 1134–1151. DOI: 10.1177/0278364918761570
10.
DeCastroJAKress-GazitH (2016) Nonlinear controller synthesis and automatic workspace partitioning for reactive high-level behaviors. In: International conference on hybrid systems: computation and control, Pittsburgh, PA, April 2016, 225–234. ACM.
11.
EhlersRRamanV (2016) Slugs: extensible gr (1) synthesis. International Conference on Computer Aided Verification. Berlin, Germany: Springer, 333–339.
12.
EsterMKriegelHPSanderJ, et al. (1996) A density-based algorithm for discovering clusters in large spatial databases with noise. In: Proceedings of the second international conference on knowledge discovery and data mining, KDD’96, 2–4 August 1996, Portland, Oregon, pp. 226–231. AAAI Press.
13.
FainekosGE (2011) Revising temporal logic specifications for motion planning. In: International conference on robotics and automation, 09–13 May 2011, Shanghai, China, pp. 40–45. IEEE.
14.
FikesRENilssonNJ (1971) Strips: a new approach to the application of theorem proving to problem solving. Artificial Intelligence2(3–4): 189–208.
15.
FinucaneCJingGKress-GazitH (2010) Ltlmop: experimenting with language, temporal logic and robot control. In: IEEE/RSJ international conference on intelligent robots and systems, 18–22 October 2010, Taipei, Taiwan, pp. 1988–1993.
16.
FoxMLongD (2003) Pddl2. 1: an extension to pddl for expressing temporal planning domains. Journal of Artificial Intelligence Research20: 61–124.
17.
GarrettCRChitnisRHolladayR, et al. (2021) Integrated task and motion planning. Annual Review of Control, Robotics, and Autonomous Systems4: 265–293.
18.
GhallabMNauDTraversoP (2004) Automated Planning: Theory and Practice. Amsterdam, Netherlands: Elsevier.
19.
GhallabMNauDTraversoP (2016) Automated Planning and Acting. Cambridge, UK: Cambridge University Press.
20.
HeKLahijanianMKavrakiLE, et al. (2019) Automated abstraction of manipulation domains for cost-based reactive synthesis. IEEE Robotics and Automation Letters4: 285–292.
21.
HeKWellsAKavrakiLE, et al. (2019) Efficient symbolic reactive synthesis for finite-horizon tasks. In: International conference on robotics and automation, Montreal, QC, 20–24 May 2019, pp. 8993–8999. IEEE.
22.
HollanderMWolfeDAChickenE (2014) Nonparametric Statistical Methods. 3rd edition. New York, NY: John Wiley & Sons, Inc.
23.
HyvärinenAOjaE (2000) Independent component analysis: algorithms and applications. Neural Networks: The Official Journal of the International Neural Network Society13(4–5): 411–430.
24.
JamesSRosmanBKonidarisG (2020) Learning portable representations for high-level planning. In: International conference on machine learning, Baltimore, MD, 23–29 Jul 2023, pp. 4682–4691. PMLR.
25.
JetchevNLangTToussaintM (2013) Learning grounded relational symbols from continuous data for abstract reasoning. In: Proceedings of the 2013 ICRA workshop on autonomous learning, Karlsruhe, Germany, 6–10 May 2013.
26.
KimKFainekosGSankaranarayananS (2015) On the minimal revision problem of specification automata. The International Journal of Robotics Research34(12): 1515–1535.
27.
KonidarisGKaelblingLPLozano-PerezT (2018) From skills to symbols: Learning symbolic representations for abstract high-level planning. Journal of Artificial Intelligence Research61: 215–289.
28.
KönighoferRHofferekGBloemR (2009) Debugging formal specifications using simple counterstrategies. In: Formal methods in computer-aided design, Austin, TX, 15–18 November 2009, pp. 152–159. IEEE.
29.
Kress-GazitHFainekosGEPappasGJ (2009) Temporal-logic-based reactive mission and motion planning. IEEE Transactions on Robotics25(6): 1370–1381.
30.
Kress-GazitHLahijanianMRamanV (2018) Synthesis for robots: guarantees and feedback for robot behavior. Annual Review of Control, Robotics, and Autonomous Systems1: 211–236.
31.
LahijanianMAnderssonSBBeltaC (2012) Temporal logic motion planning and control with probabilistic satisfaction guarantees. IEEE Transactions on Robotics28(2): 396–409.
32.
LahijanianMMalyMRFriedD, et al. (2016) Iterative temporal planning in uncertain environments with partial satisfaction guarantees. IEEE Transactions on Robotics32(3): 583–599.
33.
LeeJNamCParkJ, et al. (2021) Tree search-based task and motion planning with prehensile and non-prehensile manipulation for obstacle rearrangement in clutter. In: IEEE international conference on robotics and automation, Xi’an, China, 30 May 2021–05 June 2021, pp. 8516–8522. IEEE.
34.
LiWDworkinLSeshiaSA (2011) Mining assumptions for synthesis. In: ACM/IEEE international conference on formal methods and models for codesign, Cambridge, UK, 11–13 July 2011, pp. 43–50. IEEE Computer Society.
35.
MazoMDavitianATabuadaP (2010) Pessoa: a tool for embedded controller synthesis. In: International conference on computer aided verification, Berlin, Heidelberg, 17–22 July 2010, pp. 566–569. Springer-Verlag.
36.
McDermottDGhallabMHoweA, et al. (1998) Pddl—the planning domain definition language. Yale Center for Computational Vision and Control, Technical report CVC TR98003/DCS TR1165.
37.
MuganJKuipersB (2009) Autonomously learning an action hierarchy using a learned qualitative state representation. In: Proceedings of the 21st international jont conference on artifical intelligence, Pasadena, CA, 11–17 July 2009, pp. 1175–1180.
38.
MuganJKuipersB (2012) Autonomous learning of high-level states and actions in continuous environments. IEEE Transactions on Autonomous Mental Development4(1): 70–86.
39.
MuhayyuddinMollMKavrakiL, et al. (2018) Randomized physics-based motion planning for grasping in cluttered and uncertain environments. IEEE Robotics and Automation Letters3(2): 712–719.
40.
PacheckAKonidarisGKress-GazitH (2019) Automatic encoding and repair of reactive high-level tasks with learned abstract representations. In: International symposium on robotics research, Hanoi, Vietnam, 6–10 October 2019.
41.
PacheckAMoarrefSKress-GazitH (2020) Finding missing skills for high-level behaviors. In: International conference on robotics and automation, Paris, France, 31 May 2020–31 August 2020, pp. 10335–10341. IEEE.
42.
ParzenE (1962) On estimation of a probability density function and mode. The Annals of Mathematical Statistics33(3): 1065–1076.
43.
PnueliA (1977) The temporal logic of programs. In: 18th annual symposium on foundations of computer science, Providence, RI, 31 October 1977–02 November 1977, pp. 46–57. IEEE.
RosenblattM (1956) Remarks on some nonparametric estimates of a density function. The Annals of Mathematical Statistics27: 832–837.
46.
SrivastavaSFangERianoL, et al. (2014) Combined task and motion planning through an extensible planner-independent interface layer. In: IEEE international conference on robotics and automation, Hong Kong, China, 31 May 2014–07 June 2014, pp. 639–646. IEEE.
47.
UgurEPiaterJ (2015a) Bottom-up learning of object categories, action effects and logical rules: from continuous manipulative exploration to symbolic planning. In: International conference on robotics and automation, Seattle, WA, 26–30 May 2015, pp. 2627–2633. IEEE.
48.
UgurEPiaterJ (2015b) Refining discovered symbols with multi-step interaction experience. In: 2015 IEEE-RAS 15th international conference on humanoid robots, Seoul, South Korea, 03–05 November 2015, pp. 1007–1012. IEEE.
49.
WangJOlsonE (2016) AprilTag 2: efficient and robust fiducial detection. In: International conference on intelligent robots and systems, Daejeon, South Korea, 09–14 October 2016, pp. 4193–4198. IEEE.
50.
WongpiromsarnTTopcuUMurrayRM (2010) Receding horizon control for temporal logic specifications. In: ACM international conference on hybrid systems: computation and control. ACM, Milan Italy, 4–6 May 2022, pp. 101–110.
51.
YoonSFernAGivanR (2007) Ff-replan: a baseline for probabilistic planning. In: Proceedings of the seventeenth international conference on automated planning and scheduling, ICAPS, Providence, RI, 22–26 September 2007, pp. 352–359.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
