PlanBench: An Extensible Benchmark for Evaluating Large Language Models on Planning and Reasoning about Change — part 5/5
Cleaned-text code points [53953, 65882). Bibliography omitted here; source.md includes it.
After re - planning from the new state , the plan is as follows : [ PLAN ] - - - - - - - - - GPT3 response - - - - - - - - - unstack the orange block from on top of the red block put down the orange block stack the orange block on top of the yellow block [ PLAN END ] - - - - - - - - - E xt ra ct ed plan - - - - - - - - - ( unstack orange red ) ( put - down orange ) ( stack orange yellow ) - - - - - - - - Ground truth plan - - - - - - - - - = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = FAILURE = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = Listing 5: Replanning when an unexpected event occurs and changes the state. A.2.4 Plan Generalization [ ST AT EM EN T ] As initial c o n d i t i o n s I have that , the blue block is clear , the yellow block is clear , the magenta block is clear , the black block is clear , the cyan block is clear , the green block is clear , the violet block is clear , the gold block is clear , the hand is empty , the blue block is on the table , the yellow block is on the table , the magenta block is on the table , the black block is on the table , the cyan block is on the table , the green block is on the table , the violet block is on the table and the gold block is on the table . My goal is to have that the blue block is on top of the cyan block , the yellow block is on top of the blue block , the magenta block is on top of the green block , the black block is on top of the yellow block , the cyan block is on top of the gold block , the green block is on top of the black block and the violet block is on top of the magenta block . My plan is as follows : [ PLAN ] pick up the cyan block stack the cyan block on top of the gold block pick up the blue block stack the blue block on top of the cyan block pick up the yellow block stack the yellow block on top of the blue block pick up the black block stack the black block on top of the yellow block pick up the green block stack the green block on top of the black block pick up the magenta block stack the magenta block on top of the green block pick up the violet block stack the violet block on top of the magenta block [ PLAN END ] [ ST AT EM EN T ] As initial c o n d i t i o n s I have that , the red block is clear , the blue block is clear , the black block is clear , the cyan block is clear , the green block is clear , the violet block is clear , the silver block is clear , the hand is empty , the red block is on the table , the blue block is on the table , the black block is on the table , the cyan block is on the table , the green block is on the table , 16 the violet block is on the table and the silver block is on the table . My goal is to have that the red block is on top of the blue block , the blue block is on top of the silver block , the black block is on top of the red block , the cyan block is on top of the green block , the violet block is on top of the black block and the silver block is on top of the cyan block . My plan is as follows : [ PLAN ] - - - - - - - - - GPT3 response - - - - - - - - - pick up the red block stack the red block on top of the blue block pick up the blue block stack the blue block on top of the silver block pick up the black block stack the black block on top of the red block pick up the cyan block stack the cyan block on top of the green block pick up the violet block stack the violet block on top of the black block pick up the silver block stack the silver block on top of the cyan block [ PLAN END ] - - - - - - - - - E xt ra ct ed plan - - - - - - - - - ( pick - up red ) ( stack red blue ) ( pick - up blue ) ( stack blue silver ) ( pick - up black ) ( stack black red ) ( pick - up cyan ) ( stack cyan green ) ( pick - up violet ) ( stack violet black ) ( pick - up silver ) ( stack silver cyan ) - - - - - - - - Ground truth plan - - - - - - - - - ( pick - up cyan ) ( stack cyan green ) ( pick - up silver ) ( stack silver cyan ) ( pick - up blue ) ( stack blue silver ) ( pick - up red ) ( stack red blue ) ( pick - up black ) ( stack black red ) ( pick - up violet ) ( stack violet black ) = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = FAILURE = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = Listing 6: Plan generalization A.2.5 Plan Reuse [ ST AT EM EN T ] As initial c o n d i t i o n s I have that , the red block is clear , the blue block is clear , the yellow block is clear , the hand is empty , the blue block is on top of the orange block , the red block is on the table , the orange block is on the table and the yellow block is on the table . 17 My goal is to have that the orange block is on top of the blue block . My plan is as follows : [ PLAN ] unstack the blue block from on top of the orange block put down the blue block pick up the orange block stack the orange block on top of the blue block [ PLAN END ] [ ST AT EM EN T ] As initial c o n d i t i o n s I have that , the red block is clear , the blue block is clear , the yellow block is clear , the hand is empty , the blue block is on top of the orange block , the red block is on the table , the orange block is on the table and the yellow block is on the table . My goal is to have that the red block is clear , the orange block is clear , the yellow block is clear , the hand is c ur re ntl y holding blue block , the red block is on the table , the orange block is on the table and the yellow block is on the table . My plan is as follows : [ PLAN ] - - - - - - - - - GPT3 response - - - - - - - - - unstack the blue block from on top of the red block put down the blue block pick up the red block stack the red block on top of the orange block [ PLAN END ] - - - - - - - - Ground truth plan - - - - - - - - - ( unstack blue orange ) = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = FAILURE = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = Listing 7: Using a certain prefix of the plan provided in the example A.2.6 Robustness to Goal Reformulation (Shuffling goal predicates) [ ST AT EM EN T ] As initial c o n d i t i o n s I have that , the red block is clear , the blue block is clear , the yellow block is clear , the hand is empty , the blue block is on top of the orange block , the red block is on the table , the orange block is on the table and the yellow block is on the table . My goal is to have that the orange block is on top of the blue block . My plan is as follows : [ PLAN ] unstack the blue block from on top of the orange block put down the blue block pick up the orange block stack the orange block on top of the blue block [ PLAN END ] [ ST AT EM EN T ] As initial c o n d i t i o n s I have that , the red block is clear , the blue block is clear , the yellow block is clear , the hand is empty , the blue block is on top of the orange block , the red block is on the table , the orange block is on the table and the yellow block is on the table . My goal is to have that the orange block is on top of the blue block . My plan is as follows : 18 [ PLAN ] - - - - - - - - - GPT3 response - - - - - - - - - unstack the blue block from on top of the orange block put down the blue block pick up the orange block stack the orange block on top of the blue block [ PLAN END ] - - - - - - - - - E xt ra ct ed plan - - - - - - - - - ( unstack blue orange ) ( put - down blue ) ( pick - up orange ) ( stack orange blue ) - - - - - - - - Ground truth plan - - - - - - - - - ( unstack blue orange ) ( put - down blue ) ( pick - up orange ) ( stack orange blue ) = = = = = = = = = = = = = = = = = = = ( GOAL ORDERING CHANGE ) SUCCESS = = = = = = = = = = = = = = = = = = = Listing 8: Changing the order of goal predicates A.2.7 Robustness to Goal Reformulation (Full →Partial) [ ST AT EM EN T ] As initial c o n d i t i o n s I have that , the red block is clear , the blue block is clear , the yellow block is clear , the hand is empty , the blue block is on top of the orange block , the red block is on the table , the orange block is on the table and the yellow block is on the table . My goal is to have that the orange block is on top of the blue block , the red block is clear , the red block is on the table , the blue block is on the table , the hand is empty , the yellow block is on the table , the orange block is clear and the yellow block is clear . My plan is as follows : [ PLAN ] unstack the blue block from on top of the orange block put down the blue block pick up the orange block stack the orange block on top of the blue block [ PLAN END ] [ ST AT EM EN T ] As initial c o n d i t i o n s I have that , the red block is clear , the blue block is clear , the yellow block is clear , the hand is empty , the blue block is on top of the orange block , the red block is on the table , the orange block is on the table and the yellow block is on the table . My goal is to have that the orange block is on top of the blue block . My plan is as follows : [ PLAN ] - - - - - - - - - GPT3 response - - - - - - - - - unstack the blue block from on top of the orange block put down the blue block pick up the orange block stack the orange block on top of the blue block [ PLAN END ] - - - - - - - - - E xt ra ct ed plan - - - - - - - - - 19 ( unstack blue orange ) ( put - down blue ) ( pick - up orange ) ( stack orange blue ) - - - - - - - - Ground truth plan - - - - - - - - - ( unstack blue orange ) ( put - down blue ) ( pick - up orange ) ( stack orange blue ) = = = = = = = = = = = = = = = = = = = = = = ( Full - > Partial ) SUCCESS = = = = = = = = = = = = = = = = = = = = = = Listing 9: Providing an example with fully specified goal and asking to complete for the same example with a partially specified goal A.2.8 Robustness to Goal Reformulation (Partial →Full) [ ST AT EM EN T ] As initial c o n d i t i o n s I have that , the red block is clear , the blue block is clear , the yellow block is clear , the hand is empty , the blue block is on top of the orange block , the red block is on the table , the orange block is on the table and the yellow block is on the table . My goal is to have that the orange block is on top of the blue block . My plan is as follows : [ PLAN ] unstack the blue block from on top of the orange block put down the blue block pick up the orange block stack the orange block on top of the blue block [ PLAN END ] [ ST AT EM EN T ] As initial c o n d i t i o n s I have that , the red block is clear , the blue block is clear , the yellow block is clear , the hand is empty , the blue block is on top of the orange block , the red block is on the table , the orange block is on the table and the yellow block is on the table . My goal is to have that the orange block is on top of the blue block , the red block is clear , the red block is on the table , the blue block is on the table , the hand is empty , the yellow block is on the table , the orange block is clear and the yellow block is clear . My plan is as follows : [ PLAN ] - - - - - - - - - GPT3 response - - - - - - - - - unstack the blue block from on top of the orange block put down the blue block pick up the orange block stack the orange block on top of the blue block unstack the red block from on top of the orange block put down the red block pick up the yellow block stack the yellow block on top of the red block [ PLAN END ] - - - - - - - - - E xt ra ct ed plan - - - - - - - - - ( unstack blue orange ) ( put - down blue ) ( pick - up orange ) ( stack orange blue ) 20 ( unstack red orange ) ( put - down red ) ( pick - up yellow ) ( stack yellow red ) - - - - - - - - Ground truth plan - - - - - - - - - ( unstack blue orange ) ( put - down blue ) ( pick - up orange ) ( stack orange blue ) = = = = = = = = = = = = = = = = = = = = = = ( Partial - > Full ) FAILURE = = = = = = = = = = = = = = = = = = = = = = Listing 10: Providing an example with partially specified goal and asking to complete for the same example with a fully specified goal 21