sample_run00
Select Configuration:
📊 Sample Metrics
🔍 Exploration
last_node_coverage: 1
last_edge_coverage: 0.488
n_exploration_steps: 11
action_counts:
move: 2
rotate: 7
return: 0
observe: 10
term: 1
forced_term: 0
query: 0
action_cost: 10
final_information_gain: 0.659
action_fail_ratio: 0
valid_action_ratio: 1
false_belief_steps: 6
false_belief_f1: 1
false_belief_f1_position: 1
false_belief_f1_facing: 1
false_belief_action_cost: 5
false_belief_action_cost_after_seen: 0
✅ Evaluation
overall:
n_total: 27
total_score: 20.372
avg_accuracy: 0.755
per_task:
DirectionEvaluationTask:
n_total: 3
task_score: 1.500
avg_accuracy: 0.500
PovEvaluationTask:
n_total: 3
task_score: 2.500
avg_accuracy: 0.833
BackwardPovTextEvaluationTask:
n_total: 3
task_score: 3
avg_accuracy: 1
View2ActionTextEvaluationTask:
n_total: 3
task_score: 2
avg_accuracy: 0.667
AlloMappingEvaluationTask:
n_total: 3
task_score: 1.917
avg_accuracy: 0.639
RotEvaluationTask:
n_total: 3
task_score: 2
avg_accuracy: 0.667
Location2ViewEvaluationTask:
n_total: 3
task_score: 3
avg_accuracy: 1
View2LocationTextEvaluationTask:
n_total: 3
task_score: 2.455
avg_accuracy: 0.818
Action2ViewEvaluationTask:
n_total: 3
task_score: 2
avg_accuracy: 0.667
✅ Evaluation (prompt_cogmap)
overall:
n_total: 27
total_score: 18.068
avg_accuracy: 0.669
per_task:
DirectionEvaluationTask:
n_total: 3
task_score: 2
avg_accuracy: 0.667
PovEvaluationTask:
n_total: 3
task_score: 1.500
avg_accuracy: 0.500
BackwardPovTextEvaluationTask:
n_total: 3
task_score: 2
avg_accuracy: 0.667
Action2ViewEvaluationTask:
n_total: 3
task_score: 1.500
avg_accuracy: 0.500
View2ActionTextEvaluationTask:
n_total: 3
task_score: 2
avg_accuracy: 0.667
AlloMappingEvaluationTask:
n_total: 3
task_score: 2.386
avg_accuracy: 0.795
RotEvaluationTask:
n_total: 3
task_score: 2
avg_accuracy: 0.667
Location2ViewEvaluationTask:
n_total: 3
task_score: 2.500
avg_accuracy: 0.833
View2LocationTextEvaluationTask:
n_total: 3
task_score: 2.182
avg_accuracy: 0.727
✅ Evaluation (use_gt_cogmap)
overall:
n_total: 27
total_score: 26
avg_accuracy: 0.963
per_task:
DirectionEvaluationTask:
n_total: 3
task_score: 3
avg_accuracy: 1
PovEvaluationTask:
n_total: 3
task_score: 3
avg_accuracy: 1
BackwardPovTextEvaluationTask:
n_total: 3
task_score: 3
avg_accuracy: 1
Action2ViewEvaluationTask:
n_total: 3
task_score: 3
avg_accuracy: 1
View2ActionTextEvaluationTask:
n_total: 3
task_score: 2
avg_accuracy: 0.667
AlloMappingEvaluationTask:
n_total: 3
task_score: 3
avg_accuracy: 1
RotEvaluationTask:
n_total: 3
task_score: 3
avg_accuracy: 1
Location2ViewEvaluationTask:
n_total: 3
task_score: 3
avg_accuracy: 1
View2LocationTextEvaluationTask:
n_total: 3
task_score: 3
avg_accuracy: 1
✅ Evaluation (use_model_cogmap)
overall:
n_total: 27
total_score: 18.251
avg_accuracy: 0.676
per_task:
DirectionEvaluationTask:
n_total: 3
task_score: 2
avg_accuracy: 0.667
PovEvaluationTask:
n_total: 3
task_score: 2
avg_accuracy: 0.667
BackwardPovTextEvaluationTask:
n_total: 3
task_score: 2
avg_accuracy: 0.667
Action2ViewEvaluationTask:
n_total: 3
task_score: 1
avg_accuracy: 0.333
View2ActionTextEvaluationTask:
n_total: 3
task_score: 2
avg_accuracy: 0.667
AlloMappingEvaluationTask:
n_total: 3
task_score: 2.446
avg_accuracy: 0.815
RotEvaluationTask:
n_total: 3
task_score: 2
avg_accuracy: 0.667
Location2ViewEvaluationTask:
n_total: 3
task_score: 2.500
avg_accuracy: 0.833
View2LocationTextEvaluationTask:
n_total: 3
task_score: 2.305
avg_accuracy: 0.768
🧠 Cognitive Map
exploration:
error:
local_vs_gt_local_avg:
dir: 0.856
facing: 1
pos: 0.861
overall: 0.905
global_vs_gt_global_avg:
dir: 0.782
facing: 0.945
pos: 0.821
overall: 0.850
agent_vs_gt_agent_avg:
dir: 1
facing: 1
pos: 0.876
overall: 0.959
newly_observed_vs_gt_local_avg:
dir: 0.806
facing: 1
pos: 0.858
overall: 0.888
correctness:
last_global_vs_gt_full:
dir: 0.842
facing: 1
pos: 0.831
overall: 0.891
consistency:
local_vs_global_avg:
dir: 0.689
facing: 0.889
pos: 0.868
overall: 0.815
position_update_avg: 0.684
facing_update_avg: 0.950
position_stability_avg: 0.684
facing_stability_avg: 0.950
fog_probe:
f1_avg: 0.496
precision_avg: 0.451
recall_avg: 0.583
cogmap_fb:
metrics:
changed:
dir: None
pos: 0.492
facing: 1
overall: None
retention:
dir: None
pos: 0.479
facing: 0
overall: None
unchanged:
dir: 0.358
facing: 0.657
pos: 0.514
overall: 0.510
inertia: 0.798
inertia_list: [0.7394947251633471, 0.857147147011493]
unchanged_retention:
dir: None
pos: 0.553
facing: 0.750
overall: None
unchanged_retention_minus_retention:
pos: 0.252
facing: 1
unchanged_exploration:
dir: 0.818
facing: 1
pos: 0.843
overall: 0.887
🧭 False Belief CogMap
inertia: 0.798
changed:
dir: None
pos: 0.492
facing: 1
overall: None
retention:
dir: None
pos: 0.479
facing: 0
overall: None
unchanged:
dir: 0.358
facing: 0.657
pos: 0.514
overall: 0.510
unchanged_retention:
dir: None
pos: 0.553
facing: 0.750
overall: None
unchanged_retention_minus_retention:
pos: 0.252
facing: 1
unchanged_exploration:
dir: 0.818
facing: 1
pos: 0.843
overall: 0.887
📊 Performance Charts
Cognitive Map (Update)
Cognitive Map (Full)
Cognitive Map (Self-Tracking)
FB CogMap (Unchanged)
Fog Probe F1
Fog Probe Precision
Fog Probe Recall
Position Update
Facing Update
Position Stability
Facing Stability
🌍 Exploration Phase
🔄 Turn 1
💬 Assistant Action
Actions: [Observe()]
Actions: [Observe()]
📊 Cognitive Map JSONs
🤖 Predicted
{
"agent": {
"position": [
0,
0
],
"facing": "north"
},
"bike": {
"position": [
2,
3
],
"facing": "west"
},
"lamp": {
"position": [
0,
3
]
},
"blue door": {
"position": [
3,
5
]
}
}
🎯 Ground Truth (Observed)
{
"bike": {
"position": [
2,
3
],
"facing": "west"
},
"lamp": {
"position": [
0,
4
],
"facing": "east"
},
"blue door": {
"position": [
5,
5
],
"facing": "north"
},
"agent": {
"position": [
0,
0
],
"facing": "north"
}
}
🎯 Ground Truth (Full)
{
"television": {
"position": [
2,
-1
],
"facing": "east"
},
"chair": {
"position": [
2,
7
],
"facing": "east"
},
"truck": {
"position": [
10,
1
],
"facing": "west"
},
"bike": {
"position": [
2,
3
],
"facing": "west"
},
"backpack": {
"position": [
0,
9
],
"facing": "south"
},
"pan": {
"position": [
8,
1
],
"facing": "east"
},
"cap": {
"position": [
4,
-1
],
"facing": "east"
},
"lamp": {
"position": [
0,
4
],
"facing": "east"
},
"vase": {
"position": [
0,
11
],
"facing": "east"
},
"shelf": {
"position": [
12,
-1
],
"facing": "east"
},
"rubberduck": {
"position": [
4,
11
],
"facing": "west"
},
"laptop": {
"position": [
8,
-1
],
"facing": "east"
},
"green door": {
"position": [
6,
3
],
"facing": "east"
},
"blue door": {
"position": [
5,
5
],
"facing": "north"
},
"agent": {
"position": [
0,
0
],
"facing": "north"
}
}
📊 Cognitive Map JSONs
🤖 Predicted
{
"origin": "agent",
"objects": {
"bike": {
"position": [
2,
2
],
"facing": "-x"
},
"lamp": {
"position": [
0,
3
]
},
"blue door": {
"position": [
4,
4
]
}
}
}
🎯 Ground Truth
{
"blue door": {
"position": [
5,
5
],
"facing": "north"
},
"bike": {
"position": [
2,
3
],
"facing": "west"
},
"lamp": {
"position": [
0,
4
],
"facing": "east"
}
}
📊 Cognitive Map JSONs
🤖 Predicted
{
"origin": "agent",
"objects": {
"bike": {
"position": [
2,
2
],
"facing": "-x"
},
"lamp": {
"position": [
0,
3
]
},
"blue door": {
"position": [
4,
4
]
}
}
}
🎯 Ground Truth
{
"blue door": {
"position": [
5,
5
],
"facing": "north"
},
"bike": {
"position": [
2,
3
],
"facing": "west"
},
"lamp": {
"position": [
0,
4
],
"facing": "east"
}
}
🗺️ Symbolic Fog Map
############### #......######## #......######## #......######## #......######## #......######## #......######## ######+######## #D....C######## #.A.F..+......# #......#......# #.....B#......# #*..E..#......# #......#......# ########......# ###############
🔍 Fog Probe JSONs
📍 Candidates: A, B, C, D, E, F
🤖 Predicted: B, C, E, F
🎯 Ground Truth: B, C, E
🧠 Cognitive Map Metrics
Global:
dir: 0.800
facing: 1
pos: 0.873
overall: 0.891
Global (Full):
dir: 0.067
facing: 0.200
pos: 0.233
overall: 0.167
Local:
dir: 1
facing: 1
pos: 0.870
overall: 0.957
Local (Newly):
dir: 1
facing: 1
pos: 0.870
overall: 0.957
Fog Probe:
precision: 0.750
recall: 1
overall: 0.857
📈 Turn Metrics
node_coverage: 0.214
edge_coverage: 0.073
is_action_fail: False
step: 1
action_counts:
move: 0
rotate: 0
return: 0
observe: 1
term: 0
forced_term: 0
query: 0
information_gain: 0.160
reward: -0.100
is_done: False
is_valid_action: True
🔄 Turn 2
💬 Assistant Action
Actions: [JumpTo(blue door), Observe()]
Actions: [JumpTo(blue door), Observe()]
📊 Cognitive Map JSONs
🤖 Predicted
{
"agent": {
"position": [
3,
5
],
"facing": "north"
},
"lamp": {
"position": [
0,
3
]
},
"bike": {
"position": [
2,
3
],
"facing": "west"
},
"blue door": {
"position": [
3,
5
]
},
"vase": {
"position": [
0,
10
]
},
"rubberduck": {
"position": [
2,
11
],
"facing": "west"
}
}
🎯 Ground Truth (Observed)
{
"bike": {
"position": [
2,
3
],
"facing": "west"
},
"lamp": {
"position": [
0,
4
],
"facing": "east"
},
"vase": {
"position": [
0,
11
],
"facing": "east"
},
"rubberduck": {
"position": [
4,
11
],
"facing": "west"
},
"blue door": {
"position": [
5,
5
],
"facing": "north"
},
"agent": {
"position": [
5,
5
],
"facing": "north"
}
}
🎯 Ground Truth (Full)
{
"television": {
"position": [
2,
-1
],
"facing": "east"
},
"chair": {
"position": [
2,
7
],
"facing": "east"
},
"truck": {
"position": [
10,
1
],
"facing": "west"
},
"bike": {
"position": [
2,
3
],
"facing": "west"
},
"backpack": {
"position": [
0,
9
],
"facing": "south"
},
"pan": {
"position": [
8,
1
],
"facing": "east"
},
"cap": {
"position": [
4,
-1
],
"facing": "east"
},
"lamp": {
"position": [
0,
4
],
"facing": "east"
},
"vase": {
"position": [
0,
11
],
"facing": "east"
},
"shelf": {
"position": [
12,
-1
],
"facing": "east"
},
"rubberduck": {
"position": [
4,
11
],
"facing": "west"
},
"laptop": {
"position": [
8,
-1
],
"facing": "east"
},
"green door": {
"position": [
6,
3
],
"facing": "east"
},
"blue door": {
"position": [
5,
5
],
"facing": "north"
},
"agent": {
"position": [
5,
5
],
"facing": "north"
}
}
📊 Cognitive Map JSONs
🤖 Predicted
{
"origin": "agent",
"objects": {
"vase": {
"position": [
-3,
5
]
},
"rubberduck": {
"position": [
-2,
6
],
"facing": "-x"
}
}
}
🎯 Ground Truth
{
"rubberduck": {
"position": [
-1,
6
],
"facing": "west"
},
"vase": {
"position": [
-5,
6
],
"facing": "east"
}
}
📊 Cognitive Map JSONs
🤖 Predicted
{
"origin": "agent",
"objects": {
"vase": {
"position": [
-3,
5
]
},
"rubberduck": {
"position": [
-2,
6
],
"facing": "-x"
}
}
}
🎯 Ground Truth
{
"rubberduck": {
"position": [
-1,
6
],
"facing": "west"
},
"vase": {
"position": [
-5,
6
],
"facing": "east"
}
}
🗺️ Symbolic Fog Map
############### #......######## #H.K...######## #..B...######## #......######## #..I...######## #E..D..######## ######*######## #.J....######## #....G.+......# #F.....#......# #...A..#......# #C.....#......# #.....L#......# ########......# ###############
🔍 Fog Probe JSONs
📍 Candidates: A, B, C, D, E, F, G, H, I, J, K, L
🤖 Predicted: A, B, C, D, E, F, G, H, I, J, K
🎯 Ground Truth: A, L, G, E, I, D
🧠 Cognitive Map Metrics
Global:
dir: 0.762
facing: 1
pos: 0.831
overall: 0.864
Global (Full):
dir: 0.133
facing: 0.300
pos: 0.332
overall: 0.255
Local:
dir: 0.667
facing: 1
pos: 0.811
overall: 0.826
Local (Newly):
dir: 0.667
facing: 1
pos: 0.811
overall: 0.826
Fog Probe:
precision: 0.455
recall: 0.833
overall: 0.588
📈 Turn Metrics
node_coverage: 0.357
edge_coverage: 0.122
is_action_fail: False
step: 2
action_counts:
move: 1
rotate: 0
return: 0
observe: 2
term: 0
forced_term: 0
query: 0
information_gain: 0.213
reward: -0.100
is_done: False
is_valid_action: True
🔄 Turn 3
💬 Assistant Action
Actions: [Rotate(90), Observe()]
Actions: [Rotate(90), Observe()]
📊 Cognitive Map JSONs
🤖 Predicted
{
"agent": {
"position": [
3,
6
],
"facing": "east"
},
"lamp": {
"position": [
0,
3
]
},
"bike": {
"position": [
2,
3
],
"facing": "west"
},
"blue door": {
"position": [
3,
6
]
},
"vase": {
"position": [
1,
10
]
},
"rubberduck": {
"position": [
2,
11
],
"facing": "west"
}
}
🎯 Ground Truth (Observed)
{
"bike": {
"position": [
2,
3
],
"facing": "west"
},
"lamp": {
"position": [
0,
4
],
"facing": "east"
},
"vase": {
"position": [
0,
11
],
"facing": "east"
},
"rubberduck": {
"position": [
4,
11
],
"facing": "west"
},
"blue door": {
"position": [
5,
5
],
"facing": "north"
},
"agent": {
"position": [
5,
5
],
"facing": "east"
}
}
🎯 Ground Truth (Full)
{
"television": {
"position": [
2,
-1
],
"facing": "east"
},
"chair": {
"position": [
2,
7
],
"facing": "east"
},
"truck": {
"position": [
10,
1
],
"facing": "west"
},
"bike": {
"position": [
2,
3
],
"facing": "west"
},
"backpack": {
"position": [
0,
9
],
"facing": "south"
},
"pan": {
"position": [
8,
1
],
"facing": "east"
},
"cap": {
"position": [
4,
-1
],
"facing": "east"
},
"lamp": {
"position": [
0,
4
],
"facing": "east"
},
"vase": {
"position": [
0,
11
],
"facing": "east"
},
"shelf": {
"position": [
12,
-1
],
"facing": "east"
},
"rubberduck": {
"position": [
4,
11
],
"facing": "west"
},
"laptop": {
"position": [
8,
-1
],
"facing": "east"
},
"green door": {
"position": [
6,
3
],
"facing": "east"
},
"blue door": {
"position": [
5,
5
],
"facing": "north"
},
"agent": {
"position": [
5,
5
],
"facing": "east"
}
}
📊 Cognitive Map JSONs
🤖 Predicted
{
"origin": "agent",
"objects": {}
}
🎯 Ground Truth
(no data)
📊 Cognitive Map JSONs
🤖 Predicted
{
"origin": "agent",
"objects": {}
}
🎯 Ground Truth
(no data)
🗺️ Symbolic Fog Map
############### #....J.######## #...D..######## #A.....######## #I.....######## #......######## #..F..L######## ######*######## #......######## #.H....+......# #......#......# #.C.G..#......# #B.E...#......# #..K...#......# ########......# ###############
🔍 Fog Probe JSONs
📍 Candidates: A, B, C, D, E, F, G, H, I, J, K, L
🤖 Predicted: A, B, C, D, E, F, G, H, K, L
🎯 Ground Truth: K, G, E, A, I, F
🧠 Cognitive Map Metrics
Global:
dir: 0.667
facing: 1
pos: 0.816
overall: 0.827
Global (Full):
dir: 0.117
facing: 0.300
pos: 0.326
overall: 0.248
Local:
(none)
Local (Newly):
(none)
Fog Probe:
precision: 0.500
recall: 0.833
overall: 0.625
📈 Turn Metrics
node_coverage: 0.357
edge_coverage: 0.122
is_action_fail: False
step: 3
action_counts:
move: 1
rotate: 1
return: 0
observe: 3
term: 0
forced_term: 0
query: 0
information_gain: 0.213
reward: -0.100
is_done: False
is_valid_action: True
🔄 Turn 4
💬 Assistant Action
Actions: [Rotate(180), Observe()]
Actions: [Rotate(180), Observe()]
📊 Cognitive Map JSONs
🤖 Predicted
{
"agent": {
"position": [
3,
5
],
"facing": "west"
},
"blue door": {
"position": [
3,
5
]
},
"bike": {
"position": [
2,
3
],
"facing": "north"
},
"lamp": {
"position": [
-1,
3
]
},
"vase": {
"position": [
1,
9
]
},
"rubberduck": {
"position": [
2,
10
],
"facing": "west"
},
"chair": {
"position": [
1,
6
],
"facing": "south"
},
"backpack": {
"position": [
-1,
7
],
"facing": "west"
}
}
🎯 Ground Truth (Observed)
{
"chair": {
"position": [
2,
7
],
"facing": "east"
},
"bike": {
"position": [
2,
3
],
"facing": "west"
},
"backpack": {
"position": [
0,
9
],
"facing": "south"
},
"lamp": {
"position": [
0,
4
],
"facing": "east"
},
"vase": {
"position": [
0,
11
],
"facing": "east"
},
"rubberduck": {
"position": [
4,
11
],
"facing": "west"
},
"blue door": {
"position": [
5,
5
],
"facing": "north"
},
"agent": {
"position": [
5,
5
],
"facing": "west"
}
}
🎯 Ground Truth (Full)
{
"television": {
"position": [
2,
-1
],
"facing": "east"
},
"chair": {
"position": [
2,
7
],
"facing": "east"
},
"truck": {
"position": [
10,
1
],
"facing": "west"
},
"bike": {
"position": [
2,
3
],
"facing": "west"
},
"backpack": {
"position": [
0,
9
],
"facing": "south"
},
"pan": {
"position": [
8,
1
],
"facing": "east"
},
"cap": {
"position": [
4,
-1
],
"facing": "east"
},
"lamp": {
"position": [
0,
4
],
"facing": "east"
},
"vase": {
"position": [
0,
11
],
"facing": "east"
},
"shelf": {
"position": [
12,
-1
],
"facing": "east"
},
"rubberduck": {
"position": [
4,
11
],
"facing": "west"
},
"laptop": {
"position": [
8,
-1
],
"facing": "east"
},
"green door": {
"position": [
6,
3
],
"facing": "east"
},
"blue door": {
"position": [
5,
5
],
"facing": "north"
},
"agent": {
"position": [
5,
5
],
"facing": "west"
}
}
📊 Cognitive Map JSONs
🤖 Predicted
{
"origin": "agent",
"objects": {
"chair": {
"position": [
2,
2
],
"facing": "-y"
},
"bike": {
"position": [
-2,
2
],
"facing": "+y"
},
"backpack": {
"position": [
4,
4
],
"facing": "-x"
},
"lamp": {
"position": [
-2,
5
]
}
}
}
🎯 Ground Truth
{
"bike": {
"position": [
-2,
3
],
"facing": "west"
},
"chair": {
"position": [
2,
3
],
"facing": "east"
},
"lamp": {
"position": [
-1,
5
],
"facing": "east"
},
"backpack": {
"position": [
4,
5
],
"facing": "south"
}
}
📊 Cognitive Map JSONs
🤖 Predicted
{
"origin": "agent",
"objects": {
"chair": {
"position": [
2,
2
],
"facing": "-y"
},
"bike": {
"position": [
-2,
2
],
"facing": "+y"
},
"backpack": {
"position": [
4,
4
],
"facing": "-x"
},
"lamp": {
"position": [
-2,
5
]
}
}
}
🎯 Ground Truth
{
"chair": {
"position": [
2,
3
],
"facing": "east"
},
"backpack": {
"position": [
4,
5
],
"facing": "south"
}
}
🗺️ Symbolic Fog Map
############### #......######## #......######## #......######## #......######## #......######## #......######## ######*######## #DE.A..######## #......+......# #...CF.#......# #......#......# #......#......# #.....B#......# ########......# ###############
🔍 Fog Probe JSONs
📍 Candidates: A, B, C, D, E, F
🤖 Predicted: A, C, D, E, F
🎯 Ground Truth: C, B, F
🧠 Cognitive Map Metrics
Global:
dir: 0.778
facing: 0.400
pos: 0.801
overall: 0.659
Global (Full):
dir: 0.233
facing: 0.200
pos: 0.427
overall: 0.287
Local:
dir: 0.900
facing: 1
pos: 0.886
overall: 0.929
Local (Newly):
dir: 1
facing: 1
pos: 0.886
overall: 0.962
Fog Probe:
precision: 0.400
recall: 0.667
overall: 0.500
📈 Turn Metrics
node_coverage: 0.500
edge_coverage: 0.220
is_action_fail: False
step: 4
action_counts:
move: 1
rotate: 2
return: 0
observe: 4
term: 0
forced_term: 0
query: 0
information_gain: 0.329
reward: -0.100
is_done: False
is_valid_action: True
🔄 Turn 5
💬 Assistant Action
Actions: [Rotate(90), Observe()]
Actions: [Rotate(90), Observe()]
📊 Cognitive Map JSONs
🤖 Predicted
{
"agent": {
"position": [
4,
4
],
"facing": "north"
},
"blue door": {
"position": [
4,
4
]
},
"lamp": {
"position": [
0,
3
]
},
"bike": {
"position": [
2,
2
],
"facing": "west"
},
"vase": {
"position": [
1,
9
]
},
"rubberduck": {
"position": [
3,
10
],
"facing": "west"
},
"chair": {
"position": [
2,
6
],
"facing": "east"
},
"backpack": {
"position": [
0,
8
],
"facing": "south"
}
}
🎯 Ground Truth (Observed)
{
"chair": {
"position": [
2,
7
],
"facing": "east"
},
"bike": {
"position": [
2,
3
],
"facing": "west"
},
"backpack": {
"position": [
0,
9
],
"facing": "south"
},
"lamp": {
"position": [
0,
4
],
"facing": "east"
},
"vase": {
"position": [
0,
11
],
"facing": "east"
},
"rubberduck": {
"position": [
4,
11
],
"facing": "west"
},
"blue door": {
"position": [
5,
5
],
"facing": "north"
},
"agent": {
"position": [
5,
5
],
"facing": "north"
}
}
🎯 Ground Truth (Full)
{
"television": {
"position": [
2,
-1
],
"facing": "east"
},
"chair": {
"position": [
2,
7
],
"facing": "east"
},
"truck": {
"position": [
10,
1
],
"facing": "west"
},
"bike": {
"position": [
2,
3
],
"facing": "west"
},
"backpack": {
"position": [
0,
9
],
"facing": "south"
},
"pan": {
"position": [
8,
1
],
"facing": "east"
},
"cap": {
"position": [
4,
-1
],
"facing": "east"
},
"lamp": {
"position": [
0,
4
],
"facing": "east"
},
"vase": {
"position": [
0,
11
],
"facing": "east"
},
"shelf": {
"position": [
12,
-1
],
"facing": "east"
},
"rubberduck": {
"position": [
4,
11
],
"facing": "west"
},
"laptop": {
"position": [
8,
-1
],
"facing": "east"
},
"green door": {
"position": [
6,
3
],
"facing": "east"
},
"blue door": {
"position": [
5,
5
],
"facing": "north"
},
"agent": {
"position": [
5,
5
],
"facing": "north"
}
}
📊 Cognitive Map JSONs
🤖 Predicted
{
"origin": "agent",
"objects": {
"vase": {
"position": [
-3,
4
]
},
"rubberduck": {
"position": [
-2,
5
],
"facing": "-x"
}
}
}
🎯 Ground Truth
{
"rubberduck": {
"position": [
-1,
6
],
"facing": "west"
},
"vase": {
"position": [
-5,
6
],
"facing": "east"
}
}
📊 Cognitive Map JSONs
🤖 Predicted
{
"origin": "agent",
"objects": {
"vase": {
"position": [
-3,
4
]
},
"rubberduck": {
"position": [
-2,
5
],
"facing": "-x"
}
}
}
🎯 Ground Truth
(no data)
🗺️ Symbolic Fog Map
############### #......######## #......######## #......######## #......######## #......######## #......######## ######*######## #....E.######## #.A..F.+......# #..C.B.#......# #......#......# #......#......# #.D....#......# ########......# ###############
🔍 Fog Probe JSONs
📍 Candidates: A, B, C, D, E, F
🤖 Predicted: A, C, D
🎯 Ground Truth: B, F, D
🧠 Cognitive Map Metrics
Global:
dir: 0.889
facing: 1
pos: 0.847
overall: 0.912
Global (Full):
dir: 0.267
facing: 0.500
pos: 0.452
overall: 0.406
Local:
dir: 0.667
facing: 1
pos: 0.763
overall: 0.810
Local (Newly):
(none)
Fog Probe:
precision: 0.333
recall: 0.333
overall: 0.333
📈 Turn Metrics
node_coverage: 0.500
edge_coverage: 0.220
is_action_fail: False
step: 5
action_counts:
move: 1
rotate: 3
return: 0
observe: 5
term: 0
forced_term: 0
query: 0
information_gain: 0.329
reward: -0.100
is_done: False
is_valid_action: True
🔄 Turn 6
💬 Assistant Action
Actions: [Rotate(180), Observe()]
Actions: [Rotate(180), Observe()]
📊 Cognitive Map JSONs
🤖 Predicted
{
"agent": {
"position": [
3,
5
],
"facing": "south"
},
"blue door": {
"position": [
3,
5
]
},
"bike": {
"position": [
2,
3
],
"facing": "west"
},
"lamp": {
"position": [
-2,
4
]
},
"vase": {
"position": [
0,
9
]
},
"rubberduck": {
"position": [
2,
10
],
"facing": "west"
},
"chair": {
"position": [
0,
7
],
"facing": "east"
},
"backpack": {
"position": [
-1,
7
],
"facing": "south"
},
"television": {
"position": [
1,
1
],
"facing": "east"
},
"cap": {
"position": [
2,
0
],
"facing": "east"
},
"green door": {
"position": [
5,
2
]
}
}
🎯 Ground Truth (Observed)
{
"television": {
"position": [
2,
-1
],
"facing": "east"
},
"chair": {
"position": [
2,
7
],
"facing": "east"
},
"bike": {
"position": [
2,
3
],
"facing": "west"
},
"backpack": {
"position": [
0,
9
],
"facing": "south"
},
"cap": {
"position": [
4,
-1
],
"facing": "east"
},
"lamp": {
"position": [
0,
4
],
"facing": "east"
},
"vase": {
"position": [
0,
11
],
"facing": "east"
},
"rubberduck": {
"position": [
4,
11
],
"facing": "west"
},
"green door": {
"position": [
6,
3
],
"facing": "east"
},
"blue door": {
"position": [
5,
5
],
"facing": "north"
},
"agent": {
"position": [
5,
5
],
"facing": "south"
}
}
🎯 Ground Truth (Full)
{
"television": {
"position": [
2,
-1
],
"facing": "east"
},
"chair": {
"position": [
2,
7
],
"facing": "east"
},
"truck": {
"position": [
10,
1
],
"facing": "west"
},
"bike": {
"position": [
2,
3
],
"facing": "west"
},
"backpack": {
"position": [
0,
9
],
"facing": "south"
},
"pan": {
"position": [
8,
1
],
"facing": "east"
},
"cap": {
"position": [
4,
-1
],
"facing": "east"
},
"lamp": {
"position": [
0,
4
],
"facing": "east"
},
"vase": {
"position": [
0,
11
],
"facing": "east"
},
"shelf": {
"position": [
12,
-1
],
"facing": "east"
},
"rubberduck": {
"position": [
4,
11
],
"facing": "west"
},
"laptop": {
"position": [
8,
-1
],
"facing": "east"
},
"green door": {
"position": [
6,
3
],
"facing": "east"
},
"blue door": {
"position": [
5,
5
],
"facing": "north"
},
"agent": {
"position": [
5,
5
],
"facing": "south"
}
}
📊 Cognitive Map JSONs
🤖 Predicted
{
"origin": "agent",
"objects": {
"television": {
"position": [
3,
5
],
"facing": "-x"
},
"cap": {
"position": [
2,
5
],
"facing": "-x"
},
"green door": {
"position": [
-2,
3
]
}
}
}
🎯 Ground Truth
{
"cap": {
"position": [
1,
6
],
"facing": "east"
},
"television": {
"position": [
3,
6
],
"facing": "east"
},
"green door": {
"position": [
-1,
2
],
"facing": "east"
}
}
📊 Cognitive Map JSONs
🤖 Predicted
{
"origin": "agent",
"objects": {
"television": {
"position": [
3,
5
],
"facing": "-x"
},
"cap": {
"position": [
2,
5
],
"facing": "-x"
},
"green door": {
"position": [
-2,
3
]
}
}
}
🎯 Ground Truth
{
"cap": {
"position": [
1,
6
],
"facing": "east"
},
"television": {
"position": [
3,
6
],
"facing": "east"
},
"green door": {
"position": [
-1,
2
],
"facing": "east"
}
}
🧠 Cognitive Map Metrics
Global:
dir: 0.758
facing: 1
pos: 0.789
overall: 0.849
Global (Full):
dir: 0.417
facing: 0.700
pos: 0.579
overall: 0.565
Local:
dir: 0.833
facing: 1
pos: 0.855
overall: 0.896
Local (Newly):
dir: 0.833
facing: 1
pos: 0.855
overall: 0.896
Fog Probe:
(none)
📈 Turn Metrics
node_coverage: 0.714
edge_coverage: 0.293
is_action_fail: False
step: 6
action_counts:
move: 1
rotate: 4
return: 0
observe: 6
term: 0
forced_term: 0
query: 0
information_gain: 0.452
reward: -0.100
is_done: False
is_valid_action: True
🔄 Turn 7
💬 Assistant Action
Actions: [JumpTo(green door), Observe()]
Actions: [JumpTo(green door), Observe()]
📊 Cognitive Map JSONs
🤖 Predicted
{
"agent": {
"position": [
5,
3
],
"facing": "south"
},
"lamp": {
"position": [
0,
4
]
},
"bike": {
"position": [
2,
3
],
"facing": "west"
},
"blue door": {
"position": [
4,
5
]
},
"vase": {
"position": [
0,
9
]
},
"rubberduck": {
"position": [
3,
11
],
"facing": "west"
},
"chair": {
"position": [
2,
7
],
"facing": "east"
},
"backpack": {
"position": [
0,
8
],
"facing": "south"
},
"television": {
"position": [
1,
-1
],
"facing": "east"
},
"cap": {
"position": [
2,
-1
],
"facing": "east"
},
"green door": {
"position": [
5,
3
]
},
"pan": {
"position": [
7,
0
],
"facing": "east"
},
"laptop": {
"position": [
7,
-1
],
"facing": "east"
}
}
🎯 Ground Truth (Observed)
{
"television": {
"position": [
2,
-1
],
"facing": "east"
},
"chair": {
"position": [
2,
7
],
"facing": "east"
},
"bike": {
"position": [
2,
3
],
"facing": "west"
},
"backpack": {
"position": [
0,
9
],
"facing": "south"
},
"pan": {
"position": [
8,
1
],
"facing": "east"
},
"cap": {
"position": [
4,
-1
],
"facing": "east"
},
"lamp": {
"position": [
0,
4
],
"facing": "east"
},
"vase": {
"position": [
0,
11
],
"facing": "east"
},
"rubberduck": {
"position": [
4,
11
],
"facing": "west"
},
"laptop": {
"position": [
8,
-1
],
"facing": "east"
},
"green door": {
"position": [
6,
3
],
"facing": "east"
},
"blue door": {
"position": [
5,
5
],
"facing": "north"
},
"agent": {
"position": [
6,
3
],
"facing": "south"
}
}
🎯 Ground Truth (Full)
{
"television": {
"position": [
2,
-1
],
"facing": "east"
},
"chair": {
"position": [
2,
7
],
"facing": "east"
},
"truck": {
"position": [
10,
1
],
"facing": "west"
},
"bike": {
"position": [
2,
3
],
"facing": "west"
},
"backpack": {
"position": [
0,
9
],
"facing": "south"
},
"pan": {
"position": [
8,
1
],
"facing": "east"
},
"cap": {
"position": [
4,
-1
],
"facing": "east"
},
"lamp": {
"position": [
0,
4
],
"facing": "east"
},
"vase": {
"position": [
0,
11
],
"facing": "east"
},
"shelf": {
"position": [
12,
-1
],
"facing": "east"
},
"rubberduck": {
"position": [
4,
11
],
"facing": "west"
},
"laptop": {
"position": [
8,
-1
],
"facing": "east"
},
"green door": {
"position": [
6,
3
],
"facing": "east"
},
"blue door": {
"position": [
5,
5
],
"facing": "north"
},
"agent": {
"position": [
6,
3
],
"facing": "south"
}
}
📊 Cognitive Map JSONs
🤖 Predicted
{
"origin": "agent",
"objects": {
"television": {
"position": [
4,
4
],
"facing": "-x"
},
"cap": {
"position": [
3,
4
],
"facing": "-x"
},
"pan": {
"position": [
-2,
3
],
"facing": "-x"
},
"laptop": {
"position": [
-3,
4
],
"facing": "-x"
}
}
}
🎯 Ground Truth
{
"laptop": {
"position": [
-2,
4
],
"facing": "east"
},
"pan": {
"position": [
-2,
2
],
"facing": "east"
},
"television": {
"position": [
4,
4
],
"facing": "east"
},
"cap": {
"position": [
2,
4
],
"facing": "east"
}
}
📊 Cognitive Map JSONs
🤖 Predicted
{
"origin": "agent",
"objects": {
"television": {
"position": [
4,
4
],
"facing": "-x"
},
"cap": {
"position": [
3,
4
],
"facing": "-x"
},
"pan": {
"position": [
-2,
3
],
"facing": "-x"
},
"laptop": {
"position": [
-3,
4
],
"facing": "-x"
}
}
}
🎯 Ground Truth
{
"laptop": {
"position": [
-2,
4
],
"facing": "east"
},
"pan": {
"position": [
-2,
2
],
"facing": "east"
}
}
🗺️ Symbolic Fog Map
############### #......######## #......######## #......######## #......######## #......######## #......######## ######+######## #......######## #......*......# #......#..A...# #......#......# #......#.DB..C# #......#......# ########E....F# ###############
🔍 Fog Probe JSONs
📍 Candidates: A, B, C, D, E, F
🤖 Predicted: C, D
🎯 Ground Truth: C, F, A
🧠 Cognitive Map Metrics
Global:
dir: 0.912
facing: 1
pos: 0.871
overall: 0.928
Global (Full):
dir: 0.692
facing: 0.900
pos: 0.755
overall: 0.782
Local:
dir: 0.800
facing: 1
pos: 0.900
overall: 0.900
Local (Newly):
dir: 0.667
facing: 1
pos: 0.886
overall: 0.851
Fog Probe:
precision: 0.500
recall: 0.333
overall: 0.400
📈 Turn Metrics
node_coverage: 0.857
edge_coverage: 0.390
is_action_fail: False
step: 7
action_counts:
move: 2
rotate: 4
return: 0
observe: 7
term: 0
forced_term: 0
query: 0
information_gain: 0.555
reward: -0.100
is_done: False
is_valid_action: True
🔄 Turn 8
💬 Assistant Action
Actions: [Rotate(90), Observe()]
Actions: [Rotate(90), Observe()]
📊 Cognitive Map JSONs
🤖 Predicted
{
"agent": {
"position": [
6,
3
],
"facing": "west"
},
"bike": {
"position": [
2,
3
],
"facing": "west"
},
"lamp": {
"position": [
0,
4
]
},
"blue door": {
"position": [
4,
5
]
},
"vase": {
"position": [
0,
10
]
},
"rubberduck": {
"position": [
1,
11
],
"facing": "west"
},
"chair": {
"position": [
1,
7
],
"facing": "east"
},
"backpack": {
"position": [
-1,
9
],
"facing": "south"
},
"television": {
"position": [
0,
1
],
"facing": "east"
},
"cap": {
"position": [
-1,
1
],
"facing": "east"
},
"green door": {
"position": [
6,
3
]
},
"pan": {
"position": [
9,
1
],
"facing": "east"
},
"laptop": {
"position": [
10,
0
],
"facing": "east"
}
}
🎯 Ground Truth (Observed)
{
"television": {
"position": [
2,
-1
],
"facing": "east"
},
"chair": {
"position": [
2,
7
],
"facing": "east"
},
"bike": {
"position": [
2,
3
],
"facing": "west"
},
"backpack": {
"position": [
0,
9
],
"facing": "south"
},
"pan": {
"position": [
8,
1
],
"facing": "east"
},
"cap": {
"position": [
4,
-1
],
"facing": "east"
},
"lamp": {
"position": [
0,
4
],
"facing": "east"
},
"vase": {
"position": [
0,
11
],
"facing": "east"
},
"rubberduck": {
"position": [
4,
11
],
"facing": "west"
},
"laptop": {
"position": [
8,
-1
],
"facing": "east"
},
"green door": {
"position": [
6,
3
],
"facing": "east"
},
"blue door": {
"position": [
5,
5
],
"facing": "north"
},
"agent": {
"position": [
6,
3
],
"facing": "west"
}
}
🎯 Ground Truth (Full)
{
"television": {
"position": [
2,
-1
],
"facing": "east"
},
"chair": {
"position": [
2,
7
],
"facing": "east"
},
"truck": {
"position": [
10,
1
],
"facing": "west"
},
"bike": {
"position": [
2,
3
],
"facing": "west"
},
"backpack": {
"position": [
0,
9
],
"facing": "south"
},
"pan": {
"position": [
8,
1
],
"facing": "east"
},
"cap": {
"position": [
4,
-1
],
"facing": "east"
},
"lamp": {
"position": [
0,
4
],
"facing": "east"
},
"vase": {
"position": [
0,
11
],
"facing": "east"
},
"shelf": {
"position": [
12,
-1
],
"facing": "east"
},
"rubberduck": {
"position": [
4,
11
],
"facing": "west"
},
"laptop": {
"position": [
8,
-1
],
"facing": "east"
},
"green door": {
"position": [
6,
3
],
"facing": "east"
},
"blue door": {
"position": [
5,
5
],
"facing": "north"
},
"agent": {
"position": [
6,
3
],
"facing": "west"
}
}
📊 Cognitive Map JSONs
🤖 Predicted
{
"origin": "agent",
"objects": {
"television": {
"position": [
-4,
4
],
"facing": "-y"
},
"bike": {
"position": [
0,
3
],
"facing": "+y"
},
"lamp": {
"position": [
1,
5
]
}
}
}
🎯 Ground Truth
{
"bike": {
"position": [
0,
4
],
"facing": "west"
},
"television": {
"position": [
-4,
4
],
"facing": "east"
},
"lamp": {
"position": [
1,
6
],
"facing": "east"
}
}
📊 Cognitive Map JSONs
🤖 Predicted
{
"origin": "agent",
"objects": {
"television": {
"position": [
-4,
4
],
"facing": "-y"
},
"bike": {
"position": [
0,
3
],
"facing": "+y"
},
"lamp": {
"position": [
1,
5
]
}
}
}
🎯 Ground Truth
(no data)
🗺️ Symbolic Fog Map
############### #......######## #......######## #......######## #......######## #......######## #......######## ######+######## #......######## #......*..F...# #......#D.....# #......#B.E...# #......#.....C# #......#A.....# ########......# ###############
🔍 Fog Probe JSONs
📍 Candidates: A, B, C, D, E, F
🤖 Predicted: C, D, F
🎯 Ground Truth: E, F, C
🧠 Cognitive Map Metrics
Global:
dir: 0.703
facing: 1
pos: 0.778
overall: 0.827
Global (Full):
dir: 0.533
facing: 0.900
pos: 0.674
overall: 0.702
Local:
dir: 1
facing: 1
pos: 0.906
overall: 0.969
Local (Newly):
(none)
Fog Probe:
precision: 0.667
recall: 0.667
overall: 0.667
📈 Turn Metrics
node_coverage: 0.857
edge_coverage: 0.439
is_action_fail: False
step: 8
action_counts:
move: 2
rotate: 5
return: 0
observe: 8
term: 0
forced_term: 0
query: 0
information_gain: 0.569
reward: -0.100
is_done: False
is_valid_action: True
🔄 Turn 9
💬 Assistant Action
Actions: [Rotate(90), Observe()]
Actions: [Rotate(90), Observe()]
📊 Cognitive Map JSONs
🤖 Predicted
{
"agent": {
"position": [
4,
3
],
"facing": "north"
},
"bike": {
"position": [
2,
3
],
"facing": "west"
},
"lamp": {
"position": [
0,
3
]
},
"blue door": {
"position": [
3,
6
]
},
"vase": {
"position": [
0,
11
]
},
"rubberduck": {
"position": [
2,
11
],
"facing": "west"
},
"chair": {
"position": [
1,
8
],
"facing": "east"
},
"backpack": {
"position": [
-1,
10
],
"facing": "south"
},
"television": {
"position": [
2,
0
],
"facing": "east"
},
"cap": {
"position": [
1,
0
],
"facing": "east"
},
"green door": {
"position": [
4,
3
]
},
"pan": {
"position": [
6,
0
],
"facing": "east"
},
"laptop": {
"position": [
6,
1
],
"facing": "east"
}
}
🎯 Ground Truth (Observed)
{
"television": {
"position": [
2,
-1
],
"facing": "east"
},
"chair": {
"position": [
2,
7
],
"facing": "east"
},
"bike": {
"position": [
2,
3
],
"facing": "west"
},
"backpack": {
"position": [
0,
9
],
"facing": "south"
},
"pan": {
"position": [
8,
1
],
"facing": "east"
},
"cap": {
"position": [
4,
-1
],
"facing": "east"
},
"lamp": {
"position": [
0,
4
],
"facing": "east"
},
"vase": {
"position": [
0,
11
],
"facing": "east"
},
"rubberduck": {
"position": [
4,
11
],
"facing": "west"
},
"laptop": {
"position": [
8,
-1
],
"facing": "east"
},
"green door": {
"position": [
6,
3
],
"facing": "east"
},
"blue door": {
"position": [
5,
5
],
"facing": "north"
},
"agent": {
"position": [
6,
3
],
"facing": "north"
}
}
🎯 Ground Truth (Full)
{
"television": {
"position": [
2,
-1
],
"facing": "east"
},
"chair": {
"position": [
2,
7
],
"facing": "east"
},
"truck": {
"position": [
10,
1
],
"facing": "west"
},
"bike": {
"position": [
2,
3
],
"facing": "west"
},
"backpack": {
"position": [
0,
9
],
"facing": "south"
},
"pan": {
"position": [
8,
1
],
"facing": "east"
},
"cap": {
"position": [
4,
-1
],
"facing": "east"
},
"lamp": {
"position": [
0,
4
],
"facing": "east"
},
"vase": {
"position": [
0,
11
],
"facing": "east"
},
"shelf": {
"position": [
12,
-1
],
"facing": "east"
},
"rubberduck": {
"position": [
4,
11
],
"facing": "west"
},
"laptop": {
"position": [
8,
-1
],
"facing": "east"
},
"green door": {
"position": [
6,
3
],
"facing": "east"
},
"blue door": {
"position": [
5,
5
],
"facing": "north"
},
"agent": {
"position": [
6,
3
],
"facing": "north"
}
}
📊 Cognitive Map JSONs
🤖 Predicted
{
"origin": "agent",
"objects": {
"blue door": {
"position": [
-2,
2
]
}
}
}
🎯 Ground Truth
{
"blue door": {
"position": [
-1,
2
],
"facing": "north"
}
}
📊 Cognitive Map JSONs
🤖 Predicted
{
"origin": "agent",
"objects": {
"blue door": {
"position": [
-2,
2
]
}
}
}
🎯 Ground Truth
(no data)
🗺️ Symbolic Fog Map
############### #......######## #......######## #......######## #......######## #......######## #......######## ######+######## #......######## #......*.A..BC# #......#D.....# #......#......# #......#.E....# #......#..F...# ########......# ###############
🔍 Fog Probe JSONs
📍 Candidates: A, B, C, D, E, F
🤖 Predicted: D, E, F
🎯 Ground Truth: A, B, C
🧠 Cognitive Map Metrics
Global:
dir: 0.747
facing: 1
pos: 0.796
overall: 0.848
Global (Full):
dir: 0.567
facing: 0.900
pos: 0.690
overall: 0.719
Local:
dir: 1
facing: 1
pos: 0.886
overall: 0.962
Local (Newly):
(none)
Fog Probe:
precision: 0
recall: 0
overall: 0
📈 Turn Metrics
node_coverage: 0.857
edge_coverage: 0.439
is_action_fail: False
step: 9
action_counts:
move: 2
rotate: 6
return: 0
observe: 9
term: 0
forced_term: 0
query: 0
information_gain: 0.569
reward: -0.100
is_done: False
is_valid_action: True
🔄 Turn 10
💬 Assistant Action
Actions: [Rotate(90), Observe()]
Actions: [Rotate(90), Observe()]
📊 Cognitive Map JSONs
🤖 Predicted
{
"agent": {
"position": [
6,
3
],
"facing": "east"
},
"blue door": {
"position": [
4,
6
]
},
"green door": {
"position": [
6,
3
]
},
"lamp": {
"position": [
0,
4
]
},
"bike": {
"position": [
2,
3
],
"facing": "west"
},
"chair": {
"position": [
2,
8
],
"facing": "east"
},
"backpack": {
"position": [
0,
10
],
"facing": "south"
},
"television": {
"position": [
1,
2
],
"facing": "east"
},
"cap": {
"position": [
0,
2
],
"facing": "east"
},
"vase": {
"position": [
1,
11
]
},
"rubberduck": {
"position": [
2,
11
],
"facing": "west"
},
"pan": {
"position": [
9,
2
],
"facing": "east"
},
"laptop": {
"position": [
9,
0
],
"facing": "east"
},
"truck": {
"position": [
10,
2
],
"facing": "west"
},
"shelf": {
"position": [
11,
1
]
}
}
🎯 Ground Truth (Observed)
{
"television": {
"position": [
2,
-1
],
"facing": "east"
},
"chair": {
"position": [
2,
7
],
"facing": "east"
},
"truck": {
"position": [
10,
1
],
"facing": "west"
},
"bike": {
"position": [
2,
3
],
"facing": "west"
},
"backpack": {
"position": [
0,
9
],
"facing": "south"
},
"pan": {
"position": [
8,
1
],
"facing": "east"
},
"cap": {
"position": [
4,
-1
],
"facing": "east"
},
"lamp": {
"position": [
0,
4
],
"facing": "east"
},
"vase": {
"position": [
0,
11
],
"facing": "east"
},
"shelf": {
"position": [
12,
-1
],
"facing": "east"
},
"rubberduck": {
"position": [
4,
11
],
"facing": "west"
},
"laptop": {
"position": [
8,
-1
],
"facing": "east"
},
"green door": {
"position": [
6,
3
],
"facing": "east"
},
"blue door": {
"position": [
5,
5
],
"facing": "north"
},
"agent": {
"position": [
6,
3
],
"facing": "east"
}
}
🎯 Ground Truth (Full)
{
"television": {
"position": [
2,
-1
],
"facing": "east"
},
"chair": {
"position": [
2,
7
],
"facing": "east"
},
"truck": {
"position": [
10,
1
],
"facing": "west"
},
"bike": {
"position": [
2,
3
],
"facing": "west"
},
"backpack": {
"position": [
0,
9
],
"facing": "south"
},
"pan": {
"position": [
8,
1
],
"facing": "east"
},
"cap": {
"position": [
4,
-1
],
"facing": "east"
},
"lamp": {
"position": [
0,
4
],
"facing": "east"
},
"vase": {
"position": [
0,
11
],
"facing": "east"
},
"shelf": {
"position": [
12,
-1
],
"facing": "east"
},
"rubberduck": {
"position": [
4,
11
],
"facing": "west"
},
"laptop": {
"position": [
8,
-1
],
"facing": "east"
},
"green door": {
"position": [
6,
3
],
"facing": "east"
},
"blue door": {
"position": [
5,
5
],
"facing": "north"
},
"agent": {
"position": [
6,
3
],
"facing": "east"
}
}
📊 Cognitive Map JSONs
🤖 Predicted
{
"origin": "agent",
"objects": {
"pan": {
"position": [
2,
2
],
"facing": "+y"
},
"truck": {
"position": [
2,
4
],
"facing": "-y"
},
"shelf": {
"position": [
4,
4
]
}
}
}
🎯 Ground Truth
{
"shelf": {
"position": [
4,
6
],
"facing": "east"
},
"pan": {
"position": [
2,
2
],
"facing": "east"
},
"truck": {
"position": [
2,
4
],
"facing": "west"
}
}
📊 Cognitive Map JSONs
🤖 Predicted
{
"origin": "agent",
"objects": {
"pan": {
"position": [
2,
2
],
"facing": "+y"
},
"truck": {
"position": [
2,
4
],
"facing": "-y"
},
"shelf": {
"position": [
4,
4
]
}
}
}
🎯 Ground Truth
{
"shelf": {
"position": [
4,
6
],
"facing": "east"
},
"truck": {
"position": [
2,
4
],
"facing": "west"
}
}
🧠 Cognitive Map Metrics
Global:
dir: 0.750
facing: 1
pos: 0.795
overall: 0.848
Global (Full):
dir: 0.750
facing: 1
pos: 0.795
overall: 0.848
Local:
dir: 0.833
facing: 1
pos: 0.870
overall: 0.901
Local (Newly):
dir: 0.667
facing: 1
pos: 0.843
overall: 0.836
Fog Probe:
(none)
📈 Turn Metrics
node_coverage: 1
edge_coverage: 0.488
is_action_fail: False
step: 10
action_counts:
move: 2
rotate: 7
return: 0
observe: 10
term: 0
forced_term: 0
query: 0
information_gain: 0.659
reward: -0.100
is_done: False
is_valid_action: True
🔄 Turn 11
💬 Assistant Action
Actions: [Term()]
Actions: [Term()]
📊 Cognitive Map JSONs
🤖 Predicted
{
"agent": {
"position": [
6,
3
],
"facing": "east"
},
"blue door": {
"position": [
4,
5
]
},
"green door": {
"position": [
6,
3
]
},
"lamp": {
"position": [
0,
4
]
},
"bike": {
"position": [
2,
3
],
"facing": "west"
},
"chair": {
"position": [
2,
7
],
"facing": "east"
},
"backpack": {
"position": [
-1,
9
],
"facing": "south"
},
"vase": {
"position": [
1,
9
]
},
"rubberduck": {
"position": [
2,
9
],
"facing": "west"
},
"television": {
"position": [
0,
0
],
"facing": "east"
},
"cap": {
"position": [
1,
0
],
"facing": "east"
},
"pan": {
"position": [
8,
1
],
"facing": "east"
},
"laptop": {
"position": [
8,
-1
],
"facing": "east"
},
"truck": {
"position": [
10,
1
],
"facing": "west"
},
"shelf": {
"position": [
10,
0
]
}
}
🎯 Ground Truth (Observed)
{
"television": {
"position": [
2,
-1
],
"facing": "east"
},
"chair": {
"position": [
2,
7
],
"facing": "east"
},
"truck": {
"position": [
10,
1
],
"facing": "west"
},
"bike": {
"position": [
2,
3
],
"facing": "west"
},
"backpack": {
"position": [
0,
9
],
"facing": "south"
},
"pan": {
"position": [
8,
1
],
"facing": "east"
},
"cap": {
"position": [
4,
-1
],
"facing": "east"
},
"lamp": {
"position": [
0,
4
],
"facing": "east"
},
"vase": {
"position": [
0,
11
],
"facing": "east"
},
"shelf": {
"position": [
12,
-1
],
"facing": "east"
},
"rubberduck": {
"position": [
4,
11
],
"facing": "west"
},
"laptop": {
"position": [
8,
-1
],
"facing": "east"
},
"green door": {
"position": [
6,
3
],
"facing": "east"
},
"blue door": {
"position": [
5,
5
],
"facing": "north"
},
"agent": {
"position": [
6,
3
],
"facing": "east"
}
}
🎯 Ground Truth (Full)
{
"television": {
"position": [
2,
-1
],
"facing": "east"
},
"chair": {
"position": [
2,
7
],
"facing": "east"
},
"truck": {
"position": [
10,
1
],
"facing": "west"
},
"bike": {
"position": [
2,
3
],
"facing": "west"
},
"backpack": {
"position": [
0,
9
],
"facing": "south"
},
"pan": {
"position": [
8,
1
],
"facing": "east"
},
"cap": {
"position": [
4,
-1
],
"facing": "east"
},
"lamp": {
"position": [
0,
4
],
"facing": "east"
},
"vase": {
"position": [
0,
11
],
"facing": "east"
},
"shelf": {
"position": [
12,
-1
],
"facing": "east"
},
"rubberduck": {
"position": [
4,
11
],
"facing": "west"
},
"laptop": {
"position": [
8,
-1
],
"facing": "east"
},
"green door": {
"position": [
6,
3
],
"facing": "east"
},
"blue door": {
"position": [
5,
5
],
"facing": "north"
},
"agent": {
"position": [
6,
3
],
"facing": "east"
}
}
🧠 Cognitive Map Metrics
Global:
dir: 0.842
facing: 1
pos: 0.831
overall: 0.891
Global (Full):
dir: 0.842
facing: 1
pos: 0.831
overall: 0.891
Local:
(none)
Local (Newly):
(none)
Fog Probe:
(none)
📈 Turn Metrics
node_coverage: 1
edge_coverage: 0.488
is_action_fail: False
step: 11
action_counts:
move: 2
rotate: 7
return: 0
observe: 10
term: 1
forced_term: 0
query: 0
information_gain: 0.659
reward: -0.100
is_done: True
is_valid_action: True
🧭 False Belief Exploration
🔄 FB Turn 12
💬 Assistant Action
Actions: [Observe()]
Actions: [Observe()]
📈 Turn Metrics
reward: -0.100
is_done: False
is_valid_action: True
🧭 False Belief Cogmap Metrics
Changed: laptop:
dir: None
pos: 0.843
facing: None
overall: None
Retention: laptop:
dir: None
pos: 0.418
facing: None
overall: None
Unchanged (all):
dir: 0.409
facing: 1
pos: 0.564
overall: 0.658
🧾 False Belief Object Sets
all_objects: ['backpack', 'bike', 'blue door', 'cap', 'chair', 'green door', 'lamp', 'laptop', 'pan', 'rubberduck', 'shelf', 'television', 'truck', 'vase']
all_changed_objects: ['cap', 'laptop', 'pan', 'vase']
newly_observed_changed (this turn): ['laptop']
unchanged_objects: ['backpack', 'bike', 'blue door', 'chair', 'green door', 'lamp', 'rubberduck', 'shelf', 'television', 'truck']
pred_changed_keys: ['laptop']
pred_unchanged_keys: ['backpack', 'bike', 'blue door', 'cap', 'chair', 'green door', 'lamp', 'laptop', 'pan', 'rubberduck', 'shelf', 'television', 'truck', 'vase']
🧭 False Belief Info
step: 1
correctly_identified_changes: None
f1_overall: None
f1_position: None
f1_facing: None
action_cost: None
action_cost_after_seen: None
ground_truth_changes: [{'name': 'cap', 'ori': True}, {'name': 'pan', 'ori': True}, {'name': 'laptop', 'pos': True}, {'name': 'vase', 'pos': True}]
reported_changes: []
newly_observed_changed_objects: ['laptop']
newly_observed_unchanged_objects: ['lamp', 'bike', 'blue door']
🔄 FB Turn 13
💬 Assistant Action
Actions: [JumpTo(blue door), Observe()]
Actions: [JumpTo(blue door), Observe()]
📈 Turn Metrics
reward: -0.100
is_done: False
is_valid_action: True
🧭 False Belief Cogmap Metrics
Unchanged (all):
dir: 0.227
facing: 0.429
pos: 0.401
overall: 0.352
🧾 False Belief Object Sets
all_objects: ['backpack', 'bike', 'blue door', 'cap', 'chair', 'green door', 'lamp', 'laptop', 'pan', 'rubberduck', 'shelf', 'television', 'truck', 'vase']
all_changed_objects: ['cap', 'laptop', 'pan', 'vase']
newly_observed_changed (this turn): []
unchanged_objects: ['backpack', 'bike', 'blue door', 'chair', 'green door', 'lamp', 'rubberduck', 'shelf', 'television', 'truck']
pred_changed_keys: []
pred_unchanged_keys: ['backpack', 'bike', 'blue door', 'cap', 'chair', 'green door', 'lamp', 'laptop', 'pan', 'rubberduck', 'shelf', 'television', 'truck', 'vase']
🧭 False Belief Info
step: 2
correctly_identified_changes: None
f1_overall: None
f1_position: None
f1_facing: None
action_cost: None
action_cost_after_seen: None
ground_truth_changes: [{'name': 'cap', 'ori': True}, {'name': 'pan', 'ori': True}, {'name': 'laptop', 'pos': True}, {'name': 'vase', 'pos': True}]
reported_changes: []
newly_observed_changed_objects: []
newly_observed_unchanged_objects: ['rubberduck']
🔄 FB Turn 14
💬 Assistant Action
Actions: [Rotate(180), Observe()]
Actions: [Rotate(180), Observe()]
📈 Turn Metrics
reward: -0.100
is_done: False
is_valid_action: True
🧭 False Belief Cogmap Metrics
Changed: cap:
dir: None
pos: None
facing: 1
overall: None
Retention: cap:
dir: None
pos: None
facing: 0
overall: None
Unchanged (all):
dir: 0.136
facing: 0.429
pos: 0.409
overall: 0.325
🧾 False Belief Object Sets
all_objects: ['backpack', 'bike', 'blue door', 'cap', 'chair', 'green door', 'lamp', 'laptop', 'pan', 'rubberduck', 'shelf', 'television', 'truck', 'vase']
all_changed_objects: ['cap', 'laptop', 'pan', 'vase']
newly_observed_changed (this turn): ['cap']
unchanged_objects: ['backpack', 'bike', 'blue door', 'chair', 'green door', 'lamp', 'rubberduck', 'shelf', 'television', 'truck']
pred_changed_keys: ['cap']
pred_unchanged_keys: ['backpack', 'bike', 'blue door', 'cap', 'chair', 'green door', 'lamp', 'laptop', 'pan', 'rubberduck', 'shelf', 'television', 'truck', 'vase']
🧭 False Belief Info
step: 3
correctly_identified_changes: None
f1_overall: None
f1_position: None
f1_facing: None
action_cost: None
action_cost_after_seen: None
ground_truth_changes: [{'name': 'cap', 'ori': True}, {'name': 'pan', 'ori': True}, {'name': 'laptop', 'pos': True}, {'name': 'vase', 'pos': True}]
reported_changes: []
newly_observed_changed_objects: ['cap']
newly_observed_unchanged_objects: ['green door', 'television']
🔄 FB Turn 15
💬 Assistant Action
Actions: [JumpTo(green door), Observe()]
Actions: [JumpTo(green door), Observe()]
📈 Turn Metrics
reward: -0.100
is_done: False
is_valid_action: True
🧭 False Belief Cogmap Metrics
Changed: pan:
dir: None
pos: None
facing: 1
overall: None
Retention: pan:
dir: None
pos: None
facing: 0
overall: None
Unchanged (all):
dir: 0.894
facing: 1
pos: 0.849
overall: 0.914
🧾 False Belief Object Sets
all_objects: ['backpack', 'bike', 'blue door', 'cap', 'chair', 'green door', 'lamp', 'laptop', 'pan', 'rubberduck', 'shelf', 'television', 'truck', 'vase']
all_changed_objects: ['cap', 'laptop', 'pan', 'vase']
newly_observed_changed (this turn): ['pan']
unchanged_objects: ['backpack', 'bike', 'blue door', 'chair', 'green door', 'lamp', 'rubberduck', 'shelf', 'television', 'truck']
pred_changed_keys: ['pan']
pred_unchanged_keys: ['backpack', 'bike', 'blue door', 'cap', 'chair', 'green door', 'lamp', 'laptop', 'pan', 'rubberduck', 'shelf', 'television', 'truck', 'vase']
🧭 False Belief Info
step: 4
correctly_identified_changes: None
f1_overall: None
f1_position: None
f1_facing: None
action_cost: None
action_cost_after_seen: None
ground_truth_changes: [{'name': 'cap', 'ori': True}, {'name': 'pan', 'ori': True}, {'name': 'laptop', 'pos': True}, {'name': 'vase', 'pos': True}]
reported_changes: []
newly_observed_changed_objects: ['pan']
newly_observed_unchanged_objects: []
🔄 FB Turn 16
💬 Assistant Action
Actions: [Rotate(-90), Observe()]
Actions: [Rotate(-90), Observe()]
📈 Turn Metrics
reward: -0.100
is_done: False
is_valid_action: True
🧭 False Belief Cogmap Metrics
Changed: vase:
dir: None
pos: 0.142
facing: None
overall: None
Retention: vase:
dir: None
pos: 0.539
facing: None
overall: None
Unchanged (all):
dir: 0.121
facing: 0.429
pos: 0.348
overall: 0.299
🧾 False Belief Object Sets
all_objects: ['backpack', 'bike', 'blue door', 'cap', 'chair', 'green door', 'lamp', 'laptop', 'pan', 'rubberduck', 'shelf', 'television', 'truck', 'vase']
all_changed_objects: ['cap', 'laptop', 'pan', 'vase']
newly_observed_changed (this turn): ['vase']
unchanged_objects: ['backpack', 'bike', 'blue door', 'chair', 'green door', 'lamp', 'rubberduck', 'shelf', 'television', 'truck']
pred_changed_keys: ['vase']
pred_unchanged_keys: ['backpack', 'bike', 'blue door', 'cap', 'chair', 'green door', 'lamp', 'laptop', 'pan', 'rubberduck', 'shelf', 'television', 'truck', 'vase']
🧭 False Belief Info
step: 5
correctly_identified_changes: None
f1_overall: None
f1_position: None
f1_facing: None
action_cost: None
action_cost_after_seen: None
ground_truth_changes: [{'name': 'cap', 'ori': True}, {'name': 'pan', 'ori': True}, {'name': 'laptop', 'pos': True}, {'name': 'vase', 'pos': True}]
reported_changes: []
newly_observed_changed_objects: ['vase']
newly_observed_unchanged_objects: ['shelf', 'truck']
🔄 FB Turn 17
💬 Assistant Action
Actions: [Term(changes="laptop: position, vase: position, cap: orientation, pan: orientation")]
Actions: [Term(changes="laptop: position, vase: position, cap: orientation, pan: orientation")]
📈 Turn Metrics
reward: 1
is_done: True
is_valid_action: True
success: 1
🧭 False Belief Info
step: 6
correctly_identified_changes: 1
f1_overall: 1
f1_position: 1
f1_facing: 1
action_cost: 5
action_cost_after_seen: 0
ground_truth_changes: [{'name': 'cap', 'ori': True}, {'name': 'pan', 'ori': True}, {'name': 'laptop', 'pos': True}, {'name': 'vase', 'pos': True}]
reported_changes: [{'name': 'laptop', 'pos': True}, {'name': 'vase', 'pos': True}, {'name': 'cap', 'ori': True}, {'name': 'pan', 'ori': True}]
newly_observed_changed_objects: []
newly_observed_unchanged_objects: []
📊 Evaluation (default)
📊 Task: DirectionEvaluationTask
Question 1 (ID: 88ffe47276e19372)
✅ Evaluation Results
id: 88ffe47276e19372
question: You return to your starting position and face north.
From a Top-Down map, describe where shelf is relative to truck.
Answer format: <cardinal direction>, <distance>
Example: north-west, near
action: None
answer: south east, mid distance
task_type: DirectionEvaluationTask
choices: []
kwargs:
(none)
score: 0
evaluation_mode: default
Question 2 (ID: 717cb6885cdcac87)
✅ Evaluation Results
id: 717cb6885cdcac87
question: You return to your starting position and face north.
From a Top-Down map, describe where rubberduck is relative to shelf.
Answer format: <cardinal direction>, <distance>
Example: north-west, near
action: None
answer: north west, far
task_type: DirectionEvaluationTask
choices: []
kwargs:
(none)
score: 1
evaluation_mode: default
Question 3 (ID: c7a5d595404c5cc8)
✅ Evaluation Results
id: c7a5d595404c5cc8
question: You return to your starting position and face north.
From a Top-Down map, describe where backpack is relative to pan.
Answer format: <cardinal direction>, <distance>
Example: north-west, near
action: None
answer: north west, far
task_type: DirectionEvaluationTask
choices: []
kwargs:
(none)
score: 0.500
evaluation_mode: default
📊 Task: PovEvaluationTask
Question 1 (ID: 14625b47c376defb)
✅ Evaluation Results
id: 14625b47c376defb
question: Now you jump to backpack's direction, facing its direction.
Describe where chair is relative to you.
Answer format: <ego direction>, <distance>
Example: front-left, near
action: None
answer: front-left, mid distance
task_type: PovEvaluationTask
choices: []
kwargs:
(none)
score: 1
evaluation_mode: default
Question 2 (ID: f49d98e66552e13e)
✅ Evaluation Results
id: f49d98e66552e13e
question: Now you jump to laptop's direction, facing its direction.
Describe where shelf is relative to you.
Answer format: <ego direction>, <distance>
Example: front-left, near
action: None
answer: front, mid distance
task_type: PovEvaluationTask
choices: []
kwargs:
(none)
score: 0.500
evaluation_mode: default
Question 3 (ID: a211c0863d2dc523)
✅ Evaluation Results
id: a211c0863d2dc523
question: Now you jump to television's direction, facing its direction.
Describe where cap is relative to you.
Answer format: <ego direction>, <distance>
Example: front-left, near
action: None
answer: front, near
task_type: PovEvaluationTask
choices: []
kwargs:
(none)
score: 1
evaluation_mode: default
📊 Task: BackwardPovTextEvaluationTask
Question 1 (ID: 6f2581c5107b62fa)
✅ Evaluation Results
id: 6f2581c5107b62fa
question: Now you jump to an object's position, facing its direction.
You observe that truck is front-left, mid distance, facing backward; shelf is front, mid distance.
Which object are you standing at?
Answer format: <object_name>
Example: lamp
action: None
answer:
answer: laptop
final_pos: [9, 2]
final_ori: [1, 0]
final_observation: [{'name': 'truck', 'direction': 'front-left', 'distance': 'mid distance', 'orientation': 'facing backward'}, {'name': 'shelf', 'direction': 'front', 'distance': 'mid distance', 'orientation': None}]
object_positions:
television: [3.0, 2.0]
chair: [3.0, 10.0]
truck: [11.0, 4.0]
bike: [3.0, 6.0]
backpack: [1.0, 12.0]
pan: [9.0, 4.0]
cap: [5.0, 2.0]
lamp: [1.0, 7.0]
vase: [1.0, 14.0]
shelf: [13.0, 2.0]
rubberduck: [5.0, 14.0]
laptop: [9.0, 2.0]
green door: [7.0, 6.0]
blue door: [6.0, 8.0]
object_orientations:
television: [1, 0]
chair: [1, 0]
truck: [-1, 0]
bike: [-1, 0]
backpack: [0, -1]
pan: [1, 0]
cap: [1, 0]
rubberduck: [-1, 0]
laptop: [1, 0]
green door: [1, 0]
blue door: [0, 1]
room_id: 3
gate_info:
green door:
room_ids: [1, 3]
ori_by_room:
1: [-1, 0]
3: [1, 0]
blue door:
room_ids: [1, 2]
ori_by_room:
1: [0, -1]
2: [0, 1]
object_rooms:
television: 1
chair: 2
truck: 3
bike: 1
backpack: 2
pan: 3
cap: 1
lamp: 1
vase: 2
shelf: 3
rubberduck: 2
laptop: 3
green door: [1, 3]
blue door: [1, 2]
task_type: BackwardPovTextEvaluationTask
choices: []
kwargs:
(none)
name_match: True
view_match: True
score: 1
evaluation_mode: default
Question 2 (ID: 534ed0980f85c17a)
✅ Evaluation Results
id: 534ed0980f85c17a
question: Now you jump to an object's position, facing its direction.
You observe that pan is front, near, facing backward; laptop is front-left, mid distance, facing backward; green door is front-right, slightly far, on front wall.
Which object are you standing at?
Answer format: <object_name>
Example: lamp
action: None
answer:
answer: truck
final_pos: [11, 4]
final_ori: [-1, 0]
final_observation: [{'name': 'pan', 'direction': 'front', 'distance': 'near', 'orientation': 'facing backward'}, {'name': 'laptop', 'direction': 'front-left', 'distance': 'mid distance', 'orientation': 'facing backward'}, {'name': 'green door', 'direction': 'front-right', 'distance': 'slightly far', 'orientation': 'on front wall'}]
object_positions:
television: [3.0, 2.0]
chair: [3.0, 10.0]
truck: [11.0, 4.0]
bike: [3.0, 6.0]
backpack: [1.0, 12.0]
pan: [9.0, 4.0]
cap: [5.0, 2.0]
lamp: [1.0, 7.0]
vase: [1.0, 14.0]
shelf: [13.0, 2.0]
rubberduck: [5.0, 14.0]
laptop: [9.0, 2.0]
green door: [7.0, 6.0]
blue door: [6.0, 8.0]
object_orientations:
television: [1, 0]
chair: [1, 0]
truck: [-1, 0]
bike: [-1, 0]
backpack: [0, -1]
pan: [1, 0]
cap: [1, 0]
rubberduck: [-1, 0]
laptop: [1, 0]
green door: [1, 0]
blue door: [0, 1]
room_id: 3
gate_info:
green door:
room_ids: [1, 3]
ori_by_room:
1: [-1, 0]
3: [1, 0]
blue door:
room_ids: [1, 2]
ori_by_room:
1: [0, -1]
2: [0, 1]
object_rooms:
television: 1
chair: 2
truck: 3
bike: 1
backpack: 2
pan: 3
cap: 1
lamp: 1
vase: 2
shelf: 3
rubberduck: 2
laptop: 3
green door: [1, 3]
blue door: [1, 2]
task_type: BackwardPovTextEvaluationTask
choices: []
kwargs:
(none)
name_match: True
view_match: True
score: 1
evaluation_mode: default
Question 3 (ID: 947e7c0517a89154)
✅ Evaluation Results
id: 947e7c0517a89154
question: Now you jump to an object's position, facing its direction.
You observe that truck is front, near, facing backward; shelf is front-right, slightly far.
Which object are you standing at?
Answer format: <object_name>
Example: lamp
action: None
answer:
answer: pan
final_pos: [9, 4]
final_ori: [1, 0]
final_observation: [{'name': 'truck', 'direction': 'front', 'distance': 'near', 'orientation': 'facing backward'}, {'name': 'shelf', 'direction': 'front-right', 'distance': 'slightly far', 'orientation': None}]
object_positions:
television: [3.0, 2.0]
chair: [3.0, 10.0]
truck: [11.0, 4.0]
bike: [3.0, 6.0]
backpack: [1.0, 12.0]
pan: [9.0, 4.0]
cap: [5.0, 2.0]
lamp: [1.0, 7.0]
vase: [1.0, 14.0]
shelf: [13.0, 2.0]
rubberduck: [5.0, 14.0]
laptop: [9.0, 2.0]
green door: [7.0, 6.0]
blue door: [6.0, 8.0]
object_orientations:
television: [1, 0]
chair: [1, 0]
truck: [-1, 0]
bike: [-1, 0]
backpack: [0, -1]
pan: [1, 0]
cap: [1, 0]
rubberduck: [-1, 0]
laptop: [1, 0]
green door: [1, 0]
blue door: [0, 1]
room_id: 3
gate_info:
green door:
room_ids: [1, 3]
ori_by_room:
1: [-1, 0]
3: [1, 0]
blue door:
room_ids: [1, 2]
ori_by_room:
1: [0, -1]
2: [0, 1]
object_rooms:
television: 1
chair: 2
truck: 3
bike: 1
backpack: 2
pan: 3
cap: 1
lamp: 1
vase: 2
shelf: 3
rubberduck: 2
laptop: 3
green door: [1, 3]
blue door: [1, 2]
task_type: BackwardPovTextEvaluationTask
choices: []
kwargs:
(none)
name_match: True
view_match: True
score: 1
evaluation_mode: default
📊 Task: View2ActionTextEvaluationTask
Question 1 (ID: a13a817826736b87)
✅ Evaluation Results
id: a13a817826736b87
question: You return to your starting position and face north.
Then you have executed an action sequence and changed to a new location and facing direction.
You observe the following:
pan is at front-right, slightly far, facing backward; truck is at front-right, mid distance, facing forward; laptop is at front, mid distance, facing backward
What action sequence led to this final view? The action sequence must be valid and only contain move actions.
Answer format: <sequence of move actions>
Example: JumpTo(lamp), Rotate(90)
action: None
answer:
final_pos: [13, 2]
final_ori: [-1, 0]
room_id: 3
init_pos: [1, 3]
init_ori: [0, 1]
object_positions:
television: [3, 2]
chair: [3, 10]
truck: [11, 4]
bike: [3, 6]
backpack: [1, 12]
pan: [9, 4]
cap: [5, 2]
lamp: [1, 7]
vase: [1, 14]
shelf: [13, 2]
rubberduck: [5, 14]
laptop: [9, 2]
green door: [7, 6]
blue door: [6, 8]
object_orientations:
television: [1, 0]
chair: [1, 0]
truck: [-1, 0]
bike: [-1, 0]
backpack: [0, -1]
pan: [1, 0]
cap: [1, 0]
rubberduck: [-1, 0]
laptop: [1, 0]
green door: [1, 0]
blue door: [0, 1]
gate_info:
green door:
room_ids: [1, 3]
ori_by_room:
1: [-1, 0]
3: [1, 0]
blue door:
room_ids: [1, 2]
ori_by_room:
1: [0, -1]
2: [0, 1]
object_rooms:
television: 1
chair: 2
truck: 3
bike: 1
backpack: 2
pan: 3
cap: 1
lamp: 1
vase: 2
shelf: 3
rubberduck: 2
laptop: 3
green door: [1, 3]
blue door: [1, 2]
minimal_plan: [['rotate', 90], ['jumpto', 'green door'], ['jumpto', 'shelf'], ['rotate', 180]]
final_observation: [{'name': 'pan', 'direction': 'front-right', 'distance': 'slightly far', 'orientation': 'facing backward'}, {'name': 'truck', 'direction': 'front-right', 'distance': 'mid distance', 'orientation': 'facing forward'}, {'name': 'laptop', 'direction': 'front', 'distance': 'mid distance', 'orientation': 'facing backward'}]
task_type: View2ActionTextEvaluationTask
choices: []
kwargs:
(none)
error: target_not_visible
score: 0
evaluation_mode: default
Question 2 (ID: ef2bc20cb1343cb5)
✅ Evaluation Results
id: ef2bc20cb1343cb5
question: You return to your starting position and face north.
Then you have executed an action sequence and changed to a new location and facing direction.
You observe the following:
blue door is at front-left, mid distance, on front wall
What action sequence led to this final view? The action sequence must be valid and only contain move actions.
Answer format: <sequence of move actions>
Example: JumpTo(lamp), Rotate(90)
action: None
answer:
final_pos: [7, 6]
final_ori: [0, 1]
room_id: [1, 3]
init_pos: [1, 3]
init_ori: [0, 1]
object_positions:
television: [3, 2]
chair: [3, 10]
truck: [11, 4]
bike: [3, 6]
backpack: [1, 12]
pan: [9, 4]
cap: [5, 2]
lamp: [1, 7]
vase: [1, 14]
shelf: [13, 2]
rubberduck: [5, 14]
laptop: [9, 2]
green door: [7, 6]
blue door: [6, 8]
object_orientations:
television: [1, 0]
chair: [1, 0]
truck: [-1, 0]
bike: [-1, 0]
backpack: [0, -1]
pan: [1, 0]
cap: [1, 0]
rubberduck: [-1, 0]
laptop: [1, 0]
green door: [1, 0]
blue door: [0, 1]
gate_info:
green door:
room_ids: [1, 3]
ori_by_room:
1: [-1, 0]
3: [1, 0]
blue door:
room_ids: [1, 2]
ori_by_room:
1: [0, -1]
2: [0, 1]
object_rooms:
television: 1
chair: 2
truck: 3
bike: 1
backpack: 2
pan: 3
cap: 1
lamp: 1
vase: 2
shelf: 3
rubberduck: 2
laptop: 3
green door: [1, 3]
blue door: [1, 2]
minimal_plan: [['rotate', 90], ['jumpto', 'initial_pos'], ['rotate', -90]]
final_observation: [{'name': 'blue door', 'direction': 'front-left', 'distance': 'mid distance', 'orientation': 'on front wall'}]
task_type: View2ActionTextEvaluationTask
choices: []
kwargs:
(none)
pos_match: True
ori_match: True
final_pos: [7.0, 6.0]
final_ori: [0, 1]
visible_match: True
score: 1
evaluation_mode: default
Question 3 (ID: 3d92d2ea666b2ed5)
✅ Evaluation Results
id: 3d92d2ea666b2ed5
question: You return to your starting position and face north.
Then you have executed an action sequence and changed to a new location and facing direction.
You observe the following:
laptop is at front-left, slightly far, facing left; cap is at front-right, slightly far, facing left; pan is at front-left, mid distance, facing left
What action sequence led to this final view? The action sequence must be valid and only contain move actions.
Answer format: <sequence of move actions>
Example: JumpTo(lamp), Rotate(90)
action: None
answer:
final_pos: [7, 6]
final_ori: [0, -1]
room_id: [1, 3]
init_pos: [1, 3]
init_ori: [0, 1]
object_positions:
television: [3, 2]
chair: [3, 10]
truck: [11, 4]
bike: [3, 6]
backpack: [1, 12]
pan: [9, 4]
cap: [5, 2]
lamp: [1, 7]
vase: [1, 14]
shelf: [13, 2]
rubberduck: [5, 14]
laptop: [9, 2]
green door: [7, 6]
blue door: [6, 8]
object_orientations:
television: [1, 0]
chair: [1, 0]
truck: [-1, 0]
bike: [-1, 0]
backpack: [0, -1]
pan: [1, 0]
cap: [1, 0]
rubberduck: [-1, 0]
laptop: [1, 0]
green door: [1, 0]
blue door: [0, 1]
gate_info:
green door:
room_ids: [1, 3]
ori_by_room:
1: [-1, 0]
3: [1, 0]
blue door:
room_ids: [1, 2]
ori_by_room:
1: [0, -1]
2: [0, 1]
object_rooms:
television: 1
chair: 2
truck: 3
bike: 1
backpack: 2
pan: 3
cap: 1
lamp: 1
vase: 2
shelf: 3
rubberduck: 2
laptop: 3
green door: [1, 3]
blue door: [1, 2]
minimal_plan: [['rotate', 90], ['jumpto', 'initial_pos'], ['rotate', 90]]
final_observation: [{'name': 'laptop', 'direction': 'front-left', 'distance': 'slightly far', 'orientation': 'facing left'}, {'name': 'cap', 'direction': 'front-right', 'distance': 'slightly far', 'orientation': 'facing left'}, {'name': 'pan', 'direction': 'front-left', 'distance': 'mid distance', 'orientation': 'facing left'}]
task_type: View2ActionTextEvaluationTask
choices: []
kwargs:
(none)
pos_match: True
ori_match: True
final_pos: [7.0, 6.0]
final_ori: [0, -1]
visible_match: True
score: 1
evaluation_mode: default
📊 Task: AlloMappingEvaluationTask
Question 1 (ID: 47868a3a901e547c)
✅ Evaluation Results
id: 47868a3a901e547c
question: Treat your starting position as the origin (0, 0) while facing north.
Report allocentric coordinates using (x right/east, y up/north).
Objects: shelf, truck, lamp.
Answer format: (x0, y0); (x1, y1); ... in the same order.
Example: (1, 0); (-2, 3); (0, -1)
action: None
answer: [[12, -1], [10, 1], [0, 4]]
task_type: AlloMappingEvaluationTask
choices: []
kwargs:
(none)
similarity: 0.741
threshold: 0.900
score: 0.741
evaluation_mode: default
Question 2 (ID: adbcc906c717830a)
✅ Evaluation Results
id: adbcc906c717830a
question: Treat your starting position as the origin (0, 0) while facing north.
Report allocentric coordinates using (x right/east, y up/north).
Objects: rubberduck, shelf, pan.
Answer format: (x0, y0); (x1, y1); ... in the same order.
Example: (1, 0); (-2, 3); (0, -1)
action: None
answer: [[4, 11], [12, -1], [8, 1]]
task_type: AlloMappingEvaluationTask
choices: []
kwargs:
(none)
similarity: 0.297
threshold: 0.900
score: 0.297
evaluation_mode: default
Question 3 (ID: 9cdfd701876daa0f)
✅ Evaluation Results
id: 9cdfd701876daa0f
question: Treat your starting position as the origin (0, 0) while facing north.
Report allocentric coordinates using (x right/east, y up/north).
Objects: cap, pan, backpack, shelf.
Answer format: (x0, y0); (x1, y1); ... in the same order.
Example: (1, 0); (-2, 3); (0, -1)
action: None
answer: [[4, -1], [8, 1], [0, 9], [12, -1]]
task_type: AlloMappingEvaluationTask
choices: []
kwargs:
(none)
similarity: 0.880
threshold: 0.900
score: 0.880
evaluation_mode: default
📊 Task: RotEvaluationTask
Question 1 (ID: c7b0da8750e06c67)
✅ Evaluation Results
id: c7b0da8750e06c67
question: You return to your starting position and face north.
You will perform a full 360-degree rotation by continuously turning counterclockwise in place.
Assume all walls are removed (you can see through walls), so every object is visible.
Focus on this set of objects: bike, pan, television.
List them in the exact order they appear directly ahead while you rotate.
If two objects share a bearing, place the nearer one first.
Answer format: <object_name1>, <object_name2>, ...
Example: mug, sofa, plant
action: None
answer: ['television', 'pan', 'bike']
task_type: RotEvaluationTask
choices: []
kwargs:
(none)
score: 1
evaluation_mode: default
Question 2 (ID: fc92fab04fedd039)
✅ Evaluation Results
id: fc92fab04fedd039
question: You return to your starting position and face north.
You will perform a full 360-degree rotation by continuously turning counterclockwise in place.
Assume all walls are removed (you can see through walls), so every object is visible.
Focus on this set of objects: bike, television, truck.
List them in the exact order they appear directly ahead while you rotate.
If two objects share a bearing, place the nearer one first.
Answer format: <object_name1>, <object_name2>, ...
Example: mug, sofa, plant
action: None
answer: ['television', 'truck', 'bike']
task_type: RotEvaluationTask
choices: []
kwargs:
(none)
score: 0
evaluation_mode: default
Question 3 (ID: 83da2dae108d2082)
✅ Evaluation Results
id: 83da2dae108d2082
question: You return to your starting position and face north.
You will perform a full 360-degree rotation by continuously turning counterclockwise in place.
Assume all walls are removed (you can see through walls), so every object is visible.
Focus on this set of objects: chair, television, truck.
List them in the exact order they appear directly ahead while you rotate.
If two objects share a bearing, place the nearer one first.
Answer format: <object_name1>, <object_name2>, ...
Example: mug, sofa, plant
action: None
answer: ['television', 'truck', 'chair']
task_type: RotEvaluationTask
choices: []
kwargs:
(none)
score: 1
evaluation_mode: default
📊 Task: RotDualEvaluationTask
Question 1 (ID: cded270d537256ef)
✅ Evaluation Results
id: cded270d537256ef
question: You return to your starting position and face north.
You performed a complete 360° rotation in place.
Assume all walls are removed (you can see through walls), so every object is visible.
During the rotation, these objects appeared directly in front of you in this order:
television, pan, bike
Based on this sequence, in which direction did you rotate?
Answer format: clockwise or counterclockwise
Example: clockwise
action: None
answer: counterclockwise
task_type: RotDualEvaluationTask
choices: []
kwargs:
(none)
score: 1
evaluation_mode: default
Question 2 (ID: 646028d7ae8a4b2f)
✅ Evaluation Results
id: 646028d7ae8a4b2f
question: You return to your starting position and face north.
You performed a complete 360° rotation in place.
Assume all walls are removed (you can see through walls), so every object is visible.
During the rotation, these objects appeared directly in front of you in this order:
television, truck, bike
Based on this sequence, in which direction did you rotate?
Answer format: clockwise or counterclockwise
Example: clockwise
action: None
answer: counterclockwise
task_type: RotDualEvaluationTask
choices: []
kwargs:
(none)
score: 1
evaluation_mode: default
Question 3 (ID: 16cddaf912fd8d1e)
✅ Evaluation Results
id: 16cddaf912fd8d1e
question: You return to your starting position and face north.
You performed a complete 360° rotation in place.
Assume all walls are removed (you can see through walls), so every object is visible.
During the rotation, these objects appeared directly in front of you in this order:
television, truck, chair
Based on this sequence, in which direction did you rotate?
Answer format: clockwise or counterclockwise
Example: clockwise
action: None
answer: counterclockwise
task_type: RotDualEvaluationTask
choices: []
kwargs:
(none)
score: 1
evaluation_mode: default
📊 Task: Location2ViewEvaluationTask
Question 1 (ID: 13d7589043fe1ee0)
✅ Evaluation Results
id: 13d7589043fe1ee0
question: Treat the green door as the new 'origin' (0, 0).
You move to (2, -5) and face north.
What is the egocentric relation of pan?
Answer format: <direction>, <distance>
Example: front, near
action: None
answer: front, mid distance
task_type: Action2LocationEvaluationTask
choices: []
kwargs:
(none)
score: 1
evaluation_mode: default
Question 2 (ID: e2c7ab6a2049c06a)
✅ Evaluation Results
id: e2c7ab6a2049c06a
question: Still treat your initial position as origin (0, 0)
You move to (4, 4) and face west.
What is the egocentric relation of bike?
Answer format: <direction>, <distance>
Example: front, near
action: None
answer: front-left, mid distance
task_type: Action2LocationEvaluationTask
choices: []
kwargs:
(none)
score: 1
evaluation_mode: default
Question 3 (ID: 5a56a83a64962b71)
✅ Evaluation Results
id: 5a56a83a64962b71
question: Still treat your initial position as origin (0, 0)
You move to (0, 3) and face east.
What is the egocentric relation of blue door?
Answer format: <direction>, <distance>
Example: front, near
action: None
answer: front-slight-left, slightly far
task_type: Action2LocationEvaluationTask
choices: []
kwargs:
(none)
score: 1
evaluation_mode: default
📊 Task: View2LocationTextEvaluationTask
Question 1 (ID: 19a422deab9e39fc)
✅ Evaluation Results
id: 19a422deab9e39fc
question: You move to a new location and face north.
You observe: pan is at front, mid distance, facing right; truck is at front-right, mid distance, facing left; green door is at front-slight-left, slightly far, on left wall
Treat the green door as the new 'origin' (0, 0).
What is your new 2D coordinate (x, y)?
Answer format: (x, y)
Example: (2, -1)
action: None
answer:
coord: [2, -5]
final_pos: [9, 1]
final_ori: [0, 1]
room_id: 3
object_positions:
television: [3, 2]
chair: [3, 10]
truck: [11, 4]
bike: [3, 6]
backpack: [1, 12]
pan: [9, 4]
cap: [5, 2]
lamp: [1, 7]
vase: [1, 14]
shelf: [13, 2]
rubberduck: [5, 14]
laptop: [9, 2]
green door: [7, 6]
blue door: [6, 8]
object_orientations:
television: [1, 0]
chair: [1, 0]
truck: [-1, 0]
bike: [-1, 0]
backpack: [0, -1]
pan: [1, 0]
cap: [1, 0]
rubberduck: [-1, 0]
laptop: [1, 0]
green door: [1, 0]
blue door: [0, 1]
final_observation: [{'name': 'pan', 'direction': 'front', 'distance': 'mid distance', 'orientation': 'facing right'}, {'name': 'truck', 'direction': 'front-right', 'distance': 'mid distance', 'orientation': 'facing left'}, {'name': 'green door', 'direction': 'front-slight-left', 'distance': 'slightly far', 'orientation': 'on left wall'}]
gate_info:
green door:
room_ids: [1, 3]
ori_by_room:
1: [-1, 0]
3: [1, 0]
blue door:
room_ids: [1, 2]
ori_by_room:
1: [0, -1]
2: [0, 1]
object_rooms:
television: 1
chair: 2
truck: 3
bike: 1
backpack: 2
pan: 3
cap: 1
lamp: 1
vase: 2
shelf: 3
rubberduck: 2
laptop: 3
green door: [1, 3]
blue door: [1, 2]
task_type: Location2ActionTextEvaluationTask
choices: []
kwargs:
(none)
similarity: 0.831
raw_score: 0.831
score: 0.831
evaluation_mode: default
Question 2 (ID: 197270037eaa3b24)
✅ Evaluation Results
id: 197270037eaa3b24
question: You move to a new location and face west.
You observe: bike is at front-left, mid distance, facing forward; lamp is at front, mid distance
Still treat your initial position as origin (0, 0)
What is your new 2D coordinate (x, y)?
Answer format: (x, y)
Example: (2, -1)
action: None
answer:
coord: [4, 4]
final_pos: [5, 7]
final_ori: [-1, 0]
room_id: 1
object_positions:
television: [3, 2]
chair: [3, 10]
truck: [11, 4]
bike: [3, 6]
backpack: [1, 12]
pan: [9, 4]
cap: [5, 2]
lamp: [1, 7]
vase: [1, 14]
shelf: [13, 2]
rubberduck: [5, 14]
laptop: [9, 2]
green door: [7, 6]
blue door: [6, 8]
object_orientations:
television: [1, 0]
chair: [1, 0]
truck: [-1, 0]
bike: [-1, 0]
backpack: [0, -1]
pan: [1, 0]
cap: [1, 0]
rubberduck: [-1, 0]
laptop: [1, 0]
green door: [1, 0]
blue door: [0, 1]
final_observation: [{'name': 'bike', 'direction': 'front-left', 'distance': 'mid distance', 'orientation': 'facing forward'}, {'name': 'lamp', 'direction': 'front', 'distance': 'mid distance', 'orientation': None}]
gate_info:
green door:
room_ids: [1, 3]
ori_by_room:
1: [-1, 0]
3: [1, 0]
blue door:
room_ids: [1, 2]
ori_by_room:
1: [0, -1]
2: [0, 1]
object_rooms:
television: 1
chair: 2
truck: 3
bike: 1
backpack: 2
pan: 3
cap: 1
lamp: 1
vase: 2
shelf: 3
rubberduck: 2
laptop: 3
green door: [1, 3]
blue door: [1, 2]
task_type: Location2ActionTextEvaluationTask
choices: []
kwargs:
(none)
similarity: 1
raw_score: 1
score: 1
evaluation_mode: default
Question 3 (ID: 7d2a51a882153983)
✅ Evaluation Results
id: 7d2a51a882153983
question: You move to a new location and face east.
You observe: blue door is at front-slight-left, slightly far, on left wall; cap is at front-right, slightly far, facing forward; bike is at front, near, facing backward
Still treat your initial position as origin (0, 0)
What is your new 2D coordinate (x, y)?
Answer format: (x, y)
Example: (2, -1)
action: None
answer:
coord: [0, 3]
final_pos: [1, 6]
final_ori: [1, 0]
room_id: 1
object_positions:
television: [3, 2]
chair: [3, 10]
truck: [11, 4]
bike: [3, 6]
backpack: [1, 12]
pan: [9, 4]
cap: [5, 2]
lamp: [1, 7]
vase: [1, 14]
shelf: [13, 2]
rubberduck: [5, 14]
laptop: [9, 2]
green door: [7, 6]
blue door: [6, 8]
object_orientations:
television: [1, 0]
chair: [1, 0]
truck: [-1, 0]
bike: [-1, 0]
backpack: [0, -1]
pan: [1, 0]
cap: [1, 0]
rubberduck: [-1, 0]
laptop: [1, 0]
green door: [1, 0]
blue door: [0, 1]
final_observation: [{'name': 'blue door', 'direction': 'front-slight-left', 'distance': 'slightly far', 'orientation': 'on left wall'}, {'name': 'cap', 'direction': 'front-right', 'distance': 'slightly far', 'orientation': 'facing forward'}, {'name': 'bike', 'direction': 'front', 'distance': 'near', 'orientation': 'facing backward'}]
gate_info:
green door:
room_ids: [1, 3]
ori_by_room:
1: [-1, 0]
3: [1, 0]
blue door:
room_ids: [1, 2]
ori_by_room:
1: [0, -1]
2: [0, 1]
object_rooms:
television: 1
chair: 2
truck: 3
bike: 1
backpack: 2
pan: 3
cap: 1
lamp: 1
vase: 2
shelf: 3
rubberduck: 2
laptop: 3
green door: [1, 3]
blue door: [1, 2]
task_type: Location2ActionTextEvaluationTask
choices: []
kwargs:
(none)
similarity: 0.624
raw_score: 0.624
score: 0.624
evaluation_mode: default
📊 Task: Action2ViewEvaluationTask
Question 1 (ID: 4f8a7d910b60df45)
✅ Evaluation Results
id: 4f8a7d910b60df45
question: You return to your starting position and face north.
You will execute the following action sequence:
1. Jump to the object at front-right, mid distance.
2. Rotate(-90)
3. Jump to the object at front-right, mid distance.
4. Rotate(-180)
After executing the actions, what is the ego relation of bike relative to you?
Answer format: <ego direction>, <distance>
Example: front, near
action: None
answer: front-right, mid distance
task_type: Action2ViewEvaluationTask
choices: []
kwargs:
(none)
score: 1
evaluation_mode: default
Question 2 (ID: 32029f3c5e3c31ee)
✅ Evaluation Results
id: 32029f3c5e3c31ee
question: You return to your starting position and face north.
You will execute the following action sequence:
1. Rotate(90)
2. Among objects which are front-left, slightly far to you, you jump to the 2nd from left also nearest one.
3. Among objects which are front-right, slightly far to you, you jump to the 2nd from left also farthest one.
4. Rotate(-180)
After executing the actions, what is the ego relation of truck relative to you?
Answer format: <ego direction>, <distance>
Example: front, near
action: None
answer: front-right, mid distance
task_type: Action2ViewEvaluationTask
choices: []
kwargs:
(none)
score: 0
evaluation_mode: default
Question 3 (ID: 79787b772d0b8410)
✅ Evaluation Results
id: 79787b772d0b8410
question: You return to your starting position and face north.
You will execute the following action sequence:
1. Jump to the object at front-right, mid distance.
2. Rotate(90)
3. Jump to the object at front-left, mid distance.
4. Rotate(90)
After executing the actions, what is the ego relation of cap relative to you?
Answer format: <ego direction>, <distance>
Example: front, near
action: None
answer: front-slight-right, slightly far
task_type: Action2ViewEvaluationTask
choices: []
kwargs:
(none)
score: 1
evaluation_mode: default
📊 Evaluation (prompt_cogmap)
📊 Task: DirectionEvaluationTask
Question 1 (ID: 88ffe47276e19372)
✅ Evaluation Results
id: 88ffe47276e19372
question: You return to your starting position and face north.
From a Top-Down map, describe where shelf is relative to truck.
Answer format: <cardinal direction>, <distance>
Example: north-west, near
action: None
answer: south east, mid distance
task_type: DirectionEvaluationTask
choices: []
kwargs:
(none)
score: 0
evaluation_mode: prompt_cogmap
Question 2 (ID: 717cb6885cdcac87)
✅ Evaluation Results
id: 717cb6885cdcac87
question: You return to your starting position and face north.
From a Top-Down map, describe where rubberduck is relative to shelf.
Answer format: <cardinal direction>, <distance>
Example: north-west, near
action: None
answer: north west, far
task_type: DirectionEvaluationTask
choices: []
kwargs:
(none)
score: 1
evaluation_mode: prompt_cogmap
Question 3 (ID: c7a5d595404c5cc8)
✅ Evaluation Results
id: c7a5d595404c5cc8
question: You return to your starting position and face north.
From a Top-Down map, describe where backpack is relative to pan.
Answer format: <cardinal direction>, <distance>
Example: north-west, near
action: None
answer: north west, far
task_type: DirectionEvaluationTask
choices: []
kwargs:
(none)
score: 1
evaluation_mode: prompt_cogmap
📊 Task: PovEvaluationTask
Question 1 (ID: 14625b47c376defb)
✅ Evaluation Results
id: 14625b47c376defb
question: Now you jump to backpack's direction, facing its direction.
Describe where chair is relative to you.
Answer format: <ego direction>, <distance>
Example: front-left, near
action: None
answer: front-left, mid distance
task_type: PovEvaluationTask
choices: []
kwargs:
(none)
score: 0.500
evaluation_mode: prompt_cogmap
Question 2 (ID: f49d98e66552e13e)
✅ Evaluation Results
id: f49d98e66552e13e
question: Now you jump to laptop's direction, facing its direction.
Describe where shelf is relative to you.
Answer format: <ego direction>, <distance>
Example: front-left, near
action: None
answer: front, mid distance
task_type: PovEvaluationTask
choices: []
kwargs:
(none)
score: 0
evaluation_mode: prompt_cogmap
Question 3 (ID: a211c0863d2dc523)
✅ Evaluation Results
id: a211c0863d2dc523
question: Now you jump to television's direction, facing its direction.
Describe where cap is relative to you.
Answer format: <ego direction>, <distance>
Example: front-left, near
action: None
answer: front, near
task_type: PovEvaluationTask
choices: []
kwargs:
(none)
score: 1
evaluation_mode: prompt_cogmap
📊 Task: BackwardPovTextEvaluationTask
Question 1 (ID: 6f2581c5107b62fa)
✅ Evaluation Results
id: 6f2581c5107b62fa
question: Now you jump to an object's position, facing its direction.
You observe that truck is front-left, mid distance, facing backward; shelf is front, mid distance.
Which object are you standing at?
Answer format: <object_name>
Example: lamp
action: None
answer:
answer: laptop
final_pos: [9, 2]
final_ori: [1, 0]
final_observation: [{'name': 'truck', 'direction': 'front-left', 'distance': 'mid distance', 'orientation': 'facing backward'}, {'name': 'shelf', 'direction': 'front', 'distance': 'mid distance', 'orientation': None}]
object_positions:
television: [3.0, 2.0]
chair: [3.0, 10.0]
truck: [11.0, 4.0]
bike: [3.0, 6.0]
backpack: [1.0, 12.0]
pan: [9.0, 4.0]
cap: [5.0, 2.0]
lamp: [1.0, 7.0]
vase: [1.0, 14.0]
shelf: [13.0, 2.0]
rubberduck: [5.0, 14.0]
laptop: [9.0, 2.0]
green door: [7.0, 6.0]
blue door: [6.0, 8.0]
object_orientations:
television: [1, 0]
chair: [1, 0]
truck: [-1, 0]
bike: [-1, 0]
backpack: [0, -1]
pan: [1, 0]
cap: [1, 0]
rubberduck: [-1, 0]
laptop: [1, 0]
green door: [1, 0]
blue door: [0, 1]
room_id: 3
gate_info:
green door:
room_ids: [1, 3]
ori_by_room:
1: [-1, 0]
3: [1, 0]
blue door:
room_ids: [1, 2]
ori_by_room:
1: [0, -1]
2: [0, 1]
object_rooms:
television: 1
chair: 2
truck: 3
bike: 1
backpack: 2
pan: 3
cap: 1
lamp: 1
vase: 2
shelf: 3
rubberduck: 2
laptop: 3
green door: [1, 3]
blue door: [1, 2]
task_type: BackwardPovTextEvaluationTask
choices: []
kwargs:
(none)
name_match: False
view_match: False
score: 0
evaluation_mode: prompt_cogmap
Question 2 (ID: 534ed0980f85c17a)
✅ Evaluation Results
id: 534ed0980f85c17a
question: Now you jump to an object's position, facing its direction.
You observe that pan is front, near, facing backward; laptop is front-left, mid distance, facing backward; green door is front-right, slightly far, on front wall.
Which object are you standing at?
Answer format: <object_name>
Example: lamp
action: None
answer:
answer: truck
final_pos: [11, 4]
final_ori: [-1, 0]
final_observation: [{'name': 'pan', 'direction': 'front', 'distance': 'near', 'orientation': 'facing backward'}, {'name': 'laptop', 'direction': 'front-left', 'distance': 'mid distance', 'orientation': 'facing backward'}, {'name': 'green door', 'direction': 'front-right', 'distance': 'slightly far', 'orientation': 'on front wall'}]
object_positions:
television: [3.0, 2.0]
chair: [3.0, 10.0]
truck: [11.0, 4.0]
bike: [3.0, 6.0]
backpack: [1.0, 12.0]
pan: [9.0, 4.0]
cap: [5.0, 2.0]
lamp: [1.0, 7.0]
vase: [1.0, 14.0]
shelf: [13.0, 2.0]
rubberduck: [5.0, 14.0]
laptop: [9.0, 2.0]
green door: [7.0, 6.0]
blue door: [6.0, 8.0]
object_orientations:
television: [1, 0]
chair: [1, 0]
truck: [-1, 0]
bike: [-1, 0]
backpack: [0, -1]
pan: [1, 0]
cap: [1, 0]
rubberduck: [-1, 0]
laptop: [1, 0]
green door: [1, 0]
blue door: [0, 1]
room_id: 3
gate_info:
green door:
room_ids: [1, 3]
ori_by_room:
1: [-1, 0]
3: [1, 0]
blue door:
room_ids: [1, 2]
ori_by_room:
1: [0, -1]
2: [0, 1]
object_rooms:
television: 1
chair: 2
truck: 3
bike: 1
backpack: 2
pan: 3
cap: 1
lamp: 1
vase: 2
shelf: 3
rubberduck: 2
laptop: 3
green door: [1, 3]
blue door: [1, 2]
task_type: BackwardPovTextEvaluationTask
choices: []
kwargs:
(none)
name_match: True
view_match: True
score: 1
evaluation_mode: prompt_cogmap
Question 3 (ID: 947e7c0517a89154)
✅ Evaluation Results
id: 947e7c0517a89154
question: Now you jump to an object's position, facing its direction.
You observe that truck is front, near, facing backward; shelf is front-right, slightly far.
Which object are you standing at?
Answer format: <object_name>
Example: lamp
action: None
answer:
answer: pan
final_pos: [9, 4]
final_ori: [1, 0]
final_observation: [{'name': 'truck', 'direction': 'front', 'distance': 'near', 'orientation': 'facing backward'}, {'name': 'shelf', 'direction': 'front-right', 'distance': 'slightly far', 'orientation': None}]
object_positions:
television: [3.0, 2.0]
chair: [3.0, 10.0]
truck: [11.0, 4.0]
bike: [3.0, 6.0]
backpack: [1.0, 12.0]
pan: [9.0, 4.0]
cap: [5.0, 2.0]
lamp: [1.0, 7.0]
vase: [1.0, 14.0]
shelf: [13.0, 2.0]
rubberduck: [5.0, 14.0]
laptop: [9.0, 2.0]
green door: [7.0, 6.0]
blue door: [6.0, 8.0]
object_orientations:
television: [1, 0]
chair: [1, 0]
truck: [-1, 0]
bike: [-1, 0]
backpack: [0, -1]
pan: [1, 0]
cap: [1, 0]
rubberduck: [-1, 0]
laptop: [1, 0]
green door: [1, 0]
blue door: [0, 1]
room_id: 3
gate_info:
green door:
room_ids: [1, 3]
ori_by_room:
1: [-1, 0]
3: [1, 0]
blue door:
room_ids: [1, 2]
ori_by_room:
1: [0, -1]
2: [0, 1]
object_rooms:
television: 1
chair: 2
truck: 3
bike: 1
backpack: 2
pan: 3
cap: 1
lamp: 1
vase: 2
shelf: 3
rubberduck: 2
laptop: 3
green door: [1, 3]
blue door: [1, 2]
task_type: BackwardPovTextEvaluationTask
choices: []
kwargs:
(none)
name_match: True
view_match: True
score: 1
evaluation_mode: prompt_cogmap
📊 Task: Action2ViewEvaluationTask
Question 1 (ID: 4f8a7d910b60df45)
✅ Evaluation Results
id: 4f8a7d910b60df45
question: You return to your starting position and face north.
You will execute the following action sequence:
1. Jump to the object at front-right, mid distance.
2. Rotate(-90)
3. Jump to the object at front-right, mid distance.
4. Rotate(-180)
After executing the actions, what is the ego relation of bike relative to you?
Answer format: <ego direction>, <distance>
Example: front, near
action: None
answer: front-right, mid distance
task_type: Action2ViewEvaluationTask
choices: []
kwargs:
(none)
score: 1
evaluation_mode: prompt_cogmap
Question 2 (ID: 32029f3c5e3c31ee)
✅ Evaluation Results
id: 32029f3c5e3c31ee
question: You return to your starting position and face north.
You will execute the following action sequence:
1. Rotate(90)
2. Among objects which are front-left, slightly far to you, you jump to the 2nd from left also nearest one.
3. Among objects which are front-right, slightly far to you, you jump to the 2nd from left also farthest one.
4. Rotate(-180)
After executing the actions, what is the ego relation of truck relative to you?
Answer format: <ego direction>, <distance>
Example: front, near
action: None
answer: front-right, mid distance
task_type: Action2ViewEvaluationTask
choices: []
kwargs:
(none)
score: 0
evaluation_mode: prompt_cogmap
Question 3 (ID: 79787b772d0b8410)
✅ Evaluation Results
id: 79787b772d0b8410
question: You return to your starting position and face north.
You will execute the following action sequence:
1. Jump to the object at front-right, mid distance.
2. Rotate(90)
3. Jump to the object at front-left, mid distance.
4. Rotate(90)
After executing the actions, what is the ego relation of cap relative to you?
Answer format: <ego direction>, <distance>
Example: front, near
action: None
answer: front-slight-right, slightly far
task_type: Action2ViewEvaluationTask
choices: []
kwargs:
(none)
score: 0.500
evaluation_mode: prompt_cogmap
📊 Task: View2ActionTextEvaluationTask
Question 1 (ID: a13a817826736b87)
✅ Evaluation Results
id: a13a817826736b87
question: You return to your starting position and face north.
Then you have executed an action sequence and changed to a new location and facing direction.
You observe the following:
pan is at front-right, slightly far, facing backward; truck is at front-right, mid distance, facing forward; laptop is at front, mid distance, facing backward
What action sequence led to this final view? The action sequence must be valid and only contain move actions.
Answer format: <sequence of move actions>
Example: JumpTo(lamp), Rotate(90)
action: None
answer:
final_pos: [13, 2]
final_ori: [-1, 0]
room_id: 3
init_pos: [1, 3]
init_ori: [0, 1]
object_positions:
television: [3, 2]
chair: [3, 10]
truck: [11, 4]
bike: [3, 6]
backpack: [1, 12]
pan: [9, 4]
cap: [5, 2]
lamp: [1, 7]
vase: [1, 14]
shelf: [13, 2]
rubberduck: [5, 14]
laptop: [9, 2]
green door: [7, 6]
blue door: [6, 8]
object_orientations:
television: [1, 0]
chair: [1, 0]
truck: [-1, 0]
bike: [-1, 0]
backpack: [0, -1]
pan: [1, 0]
cap: [1, 0]
rubberduck: [-1, 0]
laptop: [1, 0]
green door: [1, 0]
blue door: [0, 1]
gate_info:
green door:
room_ids: [1, 3]
ori_by_room:
1: [-1, 0]
3: [1, 0]
blue door:
room_ids: [1, 2]
ori_by_room:
1: [0, -1]
2: [0, 1]
object_rooms:
television: 1
chair: 2
truck: 3
bike: 1
backpack: 2
pan: 3
cap: 1
lamp: 1
vase: 2
shelf: 3
rubberduck: 2
laptop: 3
green door: [1, 3]
blue door: [1, 2]
minimal_plan: [['rotate', 90], ['jumpto', 'green door'], ['jumpto', 'shelf'], ['rotate', 180]]
final_observation: [{'name': 'pan', 'direction': 'front-right', 'distance': 'slightly far', 'orientation': 'facing backward'}, {'name': 'truck', 'direction': 'front-right', 'distance': 'mid distance', 'orientation': 'facing forward'}, {'name': 'laptop', 'direction': 'front', 'distance': 'mid distance', 'orientation': 'facing backward'}]
task_type: View2ActionTextEvaluationTask
choices: []
kwargs:
(none)
error: target_not_visible
score: 0
evaluation_mode: prompt_cogmap
Question 2 (ID: ef2bc20cb1343cb5)
✅ Evaluation Results
id: ef2bc20cb1343cb5
question: You return to your starting position and face north.
Then you have executed an action sequence and changed to a new location and facing direction.
You observe the following:
blue door is at front-left, mid distance, on front wall
What action sequence led to this final view? The action sequence must be valid and only contain move actions.
Answer format: <sequence of move actions>
Example: JumpTo(lamp), Rotate(90)
action: None
answer:
final_pos: [7, 6]
final_ori: [0, 1]
room_id: [1, 3]
init_pos: [1, 3]
init_ori: [0, 1]
object_positions:
television: [3, 2]
chair: [3, 10]
truck: [11, 4]
bike: [3, 6]
backpack: [1, 12]
pan: [9, 4]
cap: [5, 2]
lamp: [1, 7]
vase: [1, 14]
shelf: [13, 2]
rubberduck: [5, 14]
laptop: [9, 2]
green door: [7, 6]
blue door: [6, 8]
object_orientations:
television: [1, 0]
chair: [1, 0]
truck: [-1, 0]
bike: [-1, 0]
backpack: [0, -1]
pan: [1, 0]
cap: [1, 0]
rubberduck: [-1, 0]
laptop: [1, 0]
green door: [1, 0]
blue door: [0, 1]
gate_info:
green door:
room_ids: [1, 3]
ori_by_room:
1: [-1, 0]
3: [1, 0]
blue door:
room_ids: [1, 2]
ori_by_room:
1: [0, -1]
2: [0, 1]
object_rooms:
television: 1
chair: 2
truck: 3
bike: 1
backpack: 2
pan: 3
cap: 1
lamp: 1
vase: 2
shelf: 3
rubberduck: 2
laptop: 3
green door: [1, 3]
blue door: [1, 2]
minimal_plan: [['rotate', 90], ['jumpto', 'initial_pos'], ['rotate', -90]]
final_observation: [{'name': 'blue door', 'direction': 'front-left', 'distance': 'mid distance', 'orientation': 'on front wall'}]
task_type: View2ActionTextEvaluationTask
choices: []
kwargs:
(none)
pos_match: True
ori_match: True
final_pos: [7.0, 6.0]
final_ori: [0, 1]
visible_match: True
score: 1
evaluation_mode: prompt_cogmap
Question 3 (ID: 3d92d2ea666b2ed5)
✅ Evaluation Results
id: 3d92d2ea666b2ed5
question: You return to your starting position and face north.
Then you have executed an action sequence and changed to a new location and facing direction.
You observe the following:
laptop is at front-left, slightly far, facing left; cap is at front-right, slightly far, facing left; pan is at front-left, mid distance, facing left
What action sequence led to this final view? The action sequence must be valid and only contain move actions.
Answer format: <sequence of move actions>
Example: JumpTo(lamp), Rotate(90)
action: None
answer:
final_pos: [7, 6]
final_ori: [0, -1]
room_id: [1, 3]
init_pos: [1, 3]
init_ori: [0, 1]
object_positions:
television: [3, 2]
chair: [3, 10]
truck: [11, 4]
bike: [3, 6]
backpack: [1, 12]
pan: [9, 4]
cap: [5, 2]
lamp: [1, 7]
vase: [1, 14]
shelf: [13, 2]
rubberduck: [5, 14]
laptop: [9, 2]
green door: [7, 6]
blue door: [6, 8]
object_orientations:
television: [1, 0]
chair: [1, 0]
truck: [-1, 0]
bike: [-1, 0]
backpack: [0, -1]
pan: [1, 0]
cap: [1, 0]
rubberduck: [-1, 0]
laptop: [1, 0]
green door: [1, 0]
blue door: [0, 1]
gate_info:
green door:
room_ids: [1, 3]
ori_by_room:
1: [-1, 0]
3: [1, 0]
blue door:
room_ids: [1, 2]
ori_by_room:
1: [0, -1]
2: [0, 1]
object_rooms:
television: 1
chair: 2
truck: 3
bike: 1
backpack: 2
pan: 3
cap: 1
lamp: 1
vase: 2
shelf: 3
rubberduck: 2
laptop: 3
green door: [1, 3]
blue door: [1, 2]
minimal_plan: [['rotate', 90], ['jumpto', 'initial_pos'], ['rotate', 90]]
final_observation: [{'name': 'laptop', 'direction': 'front-left', 'distance': 'slightly far', 'orientation': 'facing left'}, {'name': 'cap', 'direction': 'front-right', 'distance': 'slightly far', 'orientation': 'facing left'}, {'name': 'pan', 'direction': 'front-left', 'distance': 'mid distance', 'orientation': 'facing left'}]
task_type: View2ActionTextEvaluationTask
choices: []
kwargs:
(none)
pos_match: True
ori_match: True
final_pos: [7.0, 6.0]
final_ori: [0, -1]
visible_match: True
score: 1
evaluation_mode: prompt_cogmap
📊 Task: AlloMappingEvaluationTask
Question 1 (ID: 47868a3a901e547c)
✅ Evaluation Results
id: 47868a3a901e547c
question: Treat your starting position as the origin (0, 0) while facing north.
Report allocentric coordinates using (x right/east, y up/north).
Objects: shelf, truck, lamp.
Answer format: (x0, y0); (x1, y1); ... in the same order.
Example: (1, 0); (-2, 3); (0, -1)
action: None
answer: [[12, -1], [10, 1], [0, 4]]
task_type: AlloMappingEvaluationTask
choices: []
kwargs:
(none)
similarity: 0.763
threshold: 0.900
score: 0.763
evaluation_mode: prompt_cogmap
Question 2 (ID: adbcc906c717830a)
✅ Evaluation Results
id: adbcc906c717830a
question: Treat your starting position as the origin (0, 0) while facing north.
Report allocentric coordinates using (x right/east, y up/north).
Objects: rubberduck, shelf, pan.
Answer format: (x0, y0); (x1, y1); ... in the same order.
Example: (1, 0); (-2, 3); (0, -1)
action: None
answer: [[4, 11], [12, -1], [8, 1]]
task_type: AlloMappingEvaluationTask
choices: []
kwargs:
(none)
similarity: 0.812
threshold: 0.900
score: 0.812
evaluation_mode: prompt_cogmap
Question 3 (ID: 9cdfd701876daa0f)
✅ Evaluation Results
id: 9cdfd701876daa0f
question: Treat your starting position as the origin (0, 0) while facing north.
Report allocentric coordinates using (x right/east, y up/north).
Objects: cap, pan, backpack, shelf.
Answer format: (x0, y0); (x1, y1); ... in the same order.
Example: (1, 0); (-2, 3); (0, -1)
action: None
answer: [[4, -1], [8, 1], [0, 9], [12, -1]]
task_type: AlloMappingEvaluationTask
choices: []
kwargs:
(none)
similarity: 0.810
threshold: 0.900
score: 0.810
evaluation_mode: prompt_cogmap
📊 Task: RotEvaluationTask
Question 1 (ID: c7b0da8750e06c67)
✅ Evaluation Results
id: c7b0da8750e06c67
question: You return to your starting position and face north.
You will perform a full 360-degree rotation by continuously turning counterclockwise in place.
Assume all walls are removed (you can see through walls), so every object is visible.
Focus on this set of objects: bike, pan, television.
List them in the exact order they appear directly ahead while you rotate.
If two objects share a bearing, place the nearer one first.
Answer format: <object_name1>, <object_name2>, ...
Example: mug, sofa, plant
action: None
answer: ['television', 'pan', 'bike']
task_type: RotEvaluationTask
choices: []
kwargs:
(none)
score: 1
evaluation_mode: prompt_cogmap
Question 2 (ID: fc92fab04fedd039)
✅ Evaluation Results
id: fc92fab04fedd039
question: You return to your starting position and face north.
You will perform a full 360-degree rotation by continuously turning counterclockwise in place.
Assume all walls are removed (you can see through walls), so every object is visible.
Focus on this set of objects: bike, television, truck.
List them in the exact order they appear directly ahead while you rotate.
If two objects share a bearing, place the nearer one first.
Answer format: <object_name1>, <object_name2>, ...
Example: mug, sofa, plant
action: None
answer: ['television', 'truck', 'bike']
task_type: RotEvaluationTask
choices: []
kwargs:
(none)
score: 0
evaluation_mode: prompt_cogmap
Question 3 (ID: 83da2dae108d2082)
✅ Evaluation Results
id: 83da2dae108d2082
question: You return to your starting position and face north.
You will perform a full 360-degree rotation by continuously turning counterclockwise in place.
Assume all walls are removed (you can see through walls), so every object is visible.
Focus on this set of objects: chair, television, truck.
List them in the exact order they appear directly ahead while you rotate.
If two objects share a bearing, place the nearer one first.
Answer format: <object_name1>, <object_name2>, ...
Example: mug, sofa, plant
action: None
answer: ['television', 'truck', 'chair']
task_type: RotEvaluationTask
choices: []
kwargs:
(none)
score: 1
evaluation_mode: prompt_cogmap
📊 Task: Location2ViewEvaluationTask
Question 1 (ID: 13d7589043fe1ee0)
✅ Evaluation Results
id: 13d7589043fe1ee0
question: Treat the green door as the new 'origin' (0, 0).
You move to (2, -5) and face north.
What is the egocentric relation of pan?
Answer format: <direction>, <distance>
Example: front, near
action: None
answer: front, mid distance
task_type: Location2ViewEvaluationTask
choices: []
kwargs:
(none)
score: 1
evaluation_mode: prompt_cogmap
Question 2 (ID: e2c7ab6a2049c06a)
✅ Evaluation Results
id: e2c7ab6a2049c06a
question: Still treat your initial position as origin (0, 0)
You move to (4, 4) and face west.
What is the egocentric relation of bike?
Answer format: <direction>, <distance>
Example: front, near
action: None
answer: front-left, mid distance
task_type: Location2ViewEvaluationTask
choices: []
kwargs:
(none)
score: 1
evaluation_mode: prompt_cogmap
Question 3 (ID: 5a56a83a64962b71)
✅ Evaluation Results
id: 5a56a83a64962b71
question: Still treat your initial position as origin (0, 0)
You move to (0, 3) and face east.
What is the egocentric relation of blue door?
Answer format: <direction>, <distance>
Example: front, near
action: None
answer: front-slight-left, slightly far
task_type: Location2ViewEvaluationTask
choices: []
kwargs:
(none)
score: 0.500
evaluation_mode: prompt_cogmap
📊 Task: View2LocationTextEvaluationTask
Question 1 (ID: 19a422deab9e39fc)
✅ Evaluation Results
id: 19a422deab9e39fc
question: You move to a new location and face north.
You observe: pan is at front, mid distance, facing right; truck is at front-right, mid distance, facing left; green door is at front-slight-left, slightly far, on left wall
Treat the green door as the new 'origin' (0, 0).
What is your new 2D coordinate (x, y)?
Answer format: (x, y)
Example: (2, -1)
action: None
answer:
coord: [2, -5]
final_pos: [9, 1]
final_ori: [0, 1]
room_id: 3
object_positions:
television: [3, 2]
chair: [3, 10]
truck: [11, 4]
bike: [3, 6]
backpack: [1, 12]
pan: [9, 4]
cap: [5, 2]
lamp: [1, 7]
vase: [1, 14]
shelf: [13, 2]
rubberduck: [5, 14]
laptop: [9, 2]
green door: [7, 6]
blue door: [6, 8]
object_orientations:
television: [1, 0]
chair: [1, 0]
truck: [-1, 0]
bike: [-1, 0]
backpack: [0, -1]
pan: [1, 0]
cap: [1, 0]
rubberduck: [-1, 0]
laptop: [1, 0]
green door: [1, 0]
blue door: [0, 1]
final_observation: [{'name': 'pan', 'direction': 'front', 'distance': 'mid distance', 'orientation': 'facing right'}, {'name': 'truck', 'direction': 'front-right', 'distance': 'mid distance', 'orientation': 'facing left'}, {'name': 'green door', 'direction': 'front-slight-left', 'distance': 'slightly far', 'orientation': 'on left wall'}]
gate_info:
green door:
room_ids: [1, 3]
ori_by_room:
1: [-1, 0]
3: [1, 0]
blue door:
room_ids: [1, 2]
ori_by_room:
1: [0, -1]
2: [0, 1]
object_rooms:
television: 1
chair: 2
truck: 3
bike: 1
backpack: 2
pan: 3
cap: 1
lamp: 1
vase: 2
shelf: 3
rubberduck: 2
laptop: 3
green door: [1, 3]
blue door: [1, 2]
task_type: View2LocationTextEvaluationTask
choices: []
kwargs:
(none)
similarity: 0.831
raw_score: 0.831
score: 0.831
evaluation_mode: prompt_cogmap
Question 2 (ID: 197270037eaa3b24)
✅ Evaluation Results
id: 197270037eaa3b24
question: You move to a new location and face west.
You observe: bike is at front-left, mid distance, facing forward; lamp is at front, mid distance
Still treat your initial position as origin (0, 0)
What is your new 2D coordinate (x, y)?
Answer format: (x, y)
Example: (2, -1)
action: None
answer:
coord: [4, 4]
final_pos: [5, 7]
final_ori: [-1, 0]
room_id: 1
object_positions:
television: [3, 2]
chair: [3, 10]
truck: [11, 4]
bike: [3, 6]
backpack: [1, 12]
pan: [9, 4]
cap: [5, 2]
lamp: [1, 7]
vase: [1, 14]
shelf: [13, 2]
rubberduck: [5, 14]
laptop: [9, 2]
green door: [7, 6]
blue door: [6, 8]
object_orientations:
television: [1, 0]
chair: [1, 0]
truck: [-1, 0]
bike: [-1, 0]
backpack: [0, -1]
pan: [1, 0]
cap: [1, 0]
rubberduck: [-1, 0]
laptop: [1, 0]
green door: [1, 0]
blue door: [0, 1]
final_observation: [{'name': 'bike', 'direction': 'front-left', 'distance': 'mid distance', 'orientation': 'facing forward'}, {'name': 'lamp', 'direction': 'front', 'distance': 'mid distance', 'orientation': None}]
gate_info:
green door:
room_ids: [1, 3]
ori_by_room:
1: [-1, 0]
3: [1, 0]
blue door:
room_ids: [1, 2]
ori_by_room:
1: [0, -1]
2: [0, 1]
object_rooms:
television: 1
chair: 2
truck: 3
bike: 1
backpack: 2
pan: 3
cap: 1
lamp: 1
vase: 2
shelf: 3
rubberduck: 2
laptop: 3
green door: [1, 3]
blue door: [1, 2]
task_type: View2LocationTextEvaluationTask
choices: []
kwargs:
(none)
similarity: 0.838
raw_score: 0.838
score: 0.838
evaluation_mode: prompt_cogmap
Question 3 (ID: 7d2a51a882153983)
✅ Evaluation Results
id: 7d2a51a882153983
question: You move to a new location and face east.
You observe: blue door is at front-slight-left, slightly far, on left wall; cap is at front-right, slightly far, facing forward; bike is at front, near, facing backward
Still treat your initial position as origin (0, 0)
What is your new 2D coordinate (x, y)?
Answer format: (x, y)
Example: (2, -1)
action: None
answer:
coord: [0, 3]
final_pos: [1, 6]
final_ori: [1, 0]
room_id: 1
object_positions:
television: [3, 2]
chair: [3, 10]
truck: [11, 4]
bike: [3, 6]
backpack: [1, 12]
pan: [9, 4]
cap: [5, 2]
lamp: [1, 7]
vase: [1, 14]
shelf: [13, 2]
rubberduck: [5, 14]
laptop: [9, 2]
green door: [7, 6]
blue door: [6, 8]
object_orientations:
television: [1, 0]
chair: [1, 0]
truck: [-1, 0]
bike: [-1, 0]
backpack: [0, -1]
pan: [1, 0]
cap: [1, 0]
rubberduck: [-1, 0]
laptop: [1, 0]
green door: [1, 0]
blue door: [0, 1]
final_observation: [{'name': 'blue door', 'direction': 'front-slight-left', 'distance': 'slightly far', 'orientation': 'on left wall'}, {'name': 'cap', 'direction': 'front-right', 'distance': 'slightly far', 'orientation': 'facing forward'}, {'name': 'bike', 'direction': 'front', 'distance': 'near', 'orientation': 'facing backward'}]
gate_info:
green door:
room_ids: [1, 3]
ori_by_room:
1: [-1, 0]
3: [1, 0]
blue door:
room_ids: [1, 2]
ori_by_room:
1: [0, -1]
2: [0, 1]
object_rooms:
television: 1
chair: 2
truck: 3
bike: 1
backpack: 2
pan: 3
cap: 1
lamp: 1
vase: 2
shelf: 3
rubberduck: 2
laptop: 3
green door: [1, 3]
blue door: [1, 2]
task_type: View2LocationTextEvaluationTask
choices: []
kwargs:
(none)
similarity: 0.513
raw_score: 0.513
score: 0.513
evaluation_mode: prompt_cogmap
📊 Evaluation (use_gt_cogmap)
📊 Task: DirectionEvaluationTask
Question 1 (ID: 88ffe47276e19372)
✅ Evaluation Results
id: 88ffe47276e19372
question: You return to your starting position and face north.
From a Top-Down map, describe where shelf is relative to truck.
Answer format: <cardinal direction>, <distance>
Example: north-west, near
action: None
answer: south east, mid distance
task_type: DirectionEvaluationTask
choices: []
kwargs:
(none)
score: 1
evaluation_mode: use_gt_cogmap
Question 2 (ID: 717cb6885cdcac87)
✅ Evaluation Results
id: 717cb6885cdcac87
question: You return to your starting position and face north.
From a Top-Down map, describe where rubberduck is relative to shelf.
Answer format: <cardinal direction>, <distance>
Example: north-west, near
action: None
answer: north west, far
task_type: DirectionEvaluationTask
choices: []
kwargs:
(none)
score: 1
evaluation_mode: use_gt_cogmap
Question 3 (ID: c7a5d595404c5cc8)
✅ Evaluation Results
id: c7a5d595404c5cc8
question: You return to your starting position and face north.
From a Top-Down map, describe where backpack is relative to pan.
Answer format: <cardinal direction>, <distance>
Example: north-west, near
action: None
answer: north west, far
task_type: DirectionEvaluationTask
choices: []
kwargs:
(none)
score: 1
evaluation_mode: use_gt_cogmap
📊 Task: PovEvaluationTask
Question 1 (ID: 14625b47c376defb)
✅ Evaluation Results
id: 14625b47c376defb
question: Now you jump to backpack's direction, facing its direction.
Describe where chair is relative to you.
Answer format: <ego direction>, <distance>
Example: front-left, near
action: None
answer: front-left, mid distance
task_type: PovEvaluationTask
choices: []
kwargs:
(none)
score: 1
evaluation_mode: use_gt_cogmap
Question 2 (ID: f49d98e66552e13e)
✅ Evaluation Results
id: f49d98e66552e13e
question: Now you jump to laptop's direction, facing its direction.
Describe where shelf is relative to you.
Answer format: <ego direction>, <distance>
Example: front-left, near
action: None
answer: front, mid distance
task_type: PovEvaluationTask
choices: []
kwargs:
(none)
score: 1
evaluation_mode: use_gt_cogmap
Question 3 (ID: a211c0863d2dc523)
✅ Evaluation Results
id: a211c0863d2dc523
question: Now you jump to television's direction, facing its direction.
Describe where cap is relative to you.
Answer format: <ego direction>, <distance>
Example: front-left, near
action: None
answer: front, near
task_type: PovEvaluationTask
choices: []
kwargs:
(none)
score: 1
evaluation_mode: use_gt_cogmap
📊 Task: BackwardPovTextEvaluationTask
Question 1 (ID: 6f2581c5107b62fa)
✅ Evaluation Results
id: 6f2581c5107b62fa
question: Now you jump to an object's position, facing its direction.
You observe that truck is front-left, mid distance, facing backward; shelf is front, mid distance.
Which object are you standing at?
Answer format: <object_name>
Example: lamp
action: None
answer:
answer: laptop
final_pos: [9, 2]
final_ori: [1, 0]
final_observation: [{'name': 'truck', 'direction': 'front-left', 'distance': 'mid distance', 'orientation': 'facing backward'}, {'name': 'shelf', 'direction': 'front', 'distance': 'mid distance', 'orientation': None}]
object_positions:
television: [3.0, 2.0]
chair: [3.0, 10.0]
truck: [11.0, 4.0]
bike: [3.0, 6.0]
backpack: [1.0, 12.0]
pan: [9.0, 4.0]
cap: [5.0, 2.0]
lamp: [1.0, 7.0]
vase: [1.0, 14.0]
shelf: [13.0, 2.0]
rubberduck: [5.0, 14.0]
laptop: [9.0, 2.0]
green door: [7.0, 6.0]
blue door: [6.0, 8.0]
object_orientations:
television: [1, 0]
chair: [1, 0]
truck: [-1, 0]
bike: [-1, 0]
backpack: [0, -1]
pan: [1, 0]
cap: [1, 0]
rubberduck: [-1, 0]
laptop: [1, 0]
green door: [1, 0]
blue door: [0, 1]
room_id: 3
gate_info:
green door:
room_ids: [1, 3]
ori_by_room:
1: [-1, 0]
3: [1, 0]
blue door:
room_ids: [1, 2]
ori_by_room:
1: [0, -1]
2: [0, 1]
object_rooms:
television: 1
chair: 2
truck: 3
bike: 1
backpack: 2
pan: 3
cap: 1
lamp: 1
vase: 2
shelf: 3
rubberduck: 2
laptop: 3
green door: [1, 3]
blue door: [1, 2]
task_type: BackwardPovTextEvaluationTask
choices: []
kwargs:
(none)
name_match: True
view_match: True
score: 1
evaluation_mode: use_gt_cogmap
Question 2 (ID: 534ed0980f85c17a)
✅ Evaluation Results
id: 534ed0980f85c17a
question: Now you jump to an object's position, facing its direction.
You observe that pan is front, near, facing backward; laptop is front-left, mid distance, facing backward; green door is front-right, slightly far, on front wall.
Which object are you standing at?
Answer format: <object_name>
Example: lamp
action: None
answer:
answer: truck
final_pos: [11, 4]
final_ori: [-1, 0]
final_observation: [{'name': 'pan', 'direction': 'front', 'distance': 'near', 'orientation': 'facing backward'}, {'name': 'laptop', 'direction': 'front-left', 'distance': 'mid distance', 'orientation': 'facing backward'}, {'name': 'green door', 'direction': 'front-right', 'distance': 'slightly far', 'orientation': 'on front wall'}]
object_positions:
television: [3.0, 2.0]
chair: [3.0, 10.0]
truck: [11.0, 4.0]
bike: [3.0, 6.0]
backpack: [1.0, 12.0]
pan: [9.0, 4.0]
cap: [5.0, 2.0]
lamp: [1.0, 7.0]
vase: [1.0, 14.0]
shelf: [13.0, 2.0]
rubberduck: [5.0, 14.0]
laptop: [9.0, 2.0]
green door: [7.0, 6.0]
blue door: [6.0, 8.0]
object_orientations:
television: [1, 0]
chair: [1, 0]
truck: [-1, 0]
bike: [-1, 0]
backpack: [0, -1]
pan: [1, 0]
cap: [1, 0]
rubberduck: [-1, 0]
laptop: [1, 0]
green door: [1, 0]
blue door: [0, 1]
room_id: 3
gate_info:
green door:
room_ids: [1, 3]
ori_by_room:
1: [-1, 0]
3: [1, 0]
blue door:
room_ids: [1, 2]
ori_by_room:
1: [0, -1]
2: [0, 1]
object_rooms:
television: 1
chair: 2
truck: 3
bike: 1
backpack: 2
pan: 3
cap: 1
lamp: 1
vase: 2
shelf: 3
rubberduck: 2
laptop: 3
green door: [1, 3]
blue door: [1, 2]
task_type: BackwardPovTextEvaluationTask
choices: []
kwargs:
(none)
name_match: True
view_match: True
score: 1
evaluation_mode: use_gt_cogmap
Question 3 (ID: 947e7c0517a89154)
✅ Evaluation Results
id: 947e7c0517a89154
question: Now you jump to an object's position, facing its direction.
You observe that truck is front, near, facing backward; shelf is front-right, slightly far.
Which object are you standing at?
Answer format: <object_name>
Example: lamp
action: None
answer:
answer: pan
final_pos: [9, 4]
final_ori: [1, 0]
final_observation: [{'name': 'truck', 'direction': 'front', 'distance': 'near', 'orientation': 'facing backward'}, {'name': 'shelf', 'direction': 'front-right', 'distance': 'slightly far', 'orientation': None}]
object_positions:
television: [3.0, 2.0]
chair: [3.0, 10.0]
truck: [11.0, 4.0]
bike: [3.0, 6.0]
backpack: [1.0, 12.0]
pan: [9.0, 4.0]
cap: [5.0, 2.0]
lamp: [1.0, 7.0]
vase: [1.0, 14.0]
shelf: [13.0, 2.0]
rubberduck: [5.0, 14.0]
laptop: [9.0, 2.0]
green door: [7.0, 6.0]
blue door: [6.0, 8.0]
object_orientations:
television: [1, 0]
chair: [1, 0]
truck: [-1, 0]
bike: [-1, 0]
backpack: [0, -1]
pan: [1, 0]
cap: [1, 0]
rubberduck: [-1, 0]
laptop: [1, 0]
green door: [1, 0]
blue door: [0, 1]
room_id: 3
gate_info:
green door:
room_ids: [1, 3]
ori_by_room:
1: [-1, 0]
3: [1, 0]
blue door:
room_ids: [1, 2]
ori_by_room:
1: [0, -1]
2: [0, 1]
object_rooms:
television: 1
chair: 2
truck: 3
bike: 1
backpack: 2
pan: 3
cap: 1
lamp: 1
vase: 2
shelf: 3
rubberduck: 2
laptop: 3
green door: [1, 3]
blue door: [1, 2]
task_type: BackwardPovTextEvaluationTask
choices: []
kwargs:
(none)
name_match: True
view_match: True
score: 1
evaluation_mode: use_gt_cogmap
📊 Task: Action2ViewEvaluationTask
Question 1 (ID: 4f8a7d910b60df45)
✅ Evaluation Results
id: 4f8a7d910b60df45
question: You return to your starting position and face north.
You will execute the following action sequence:
1. Jump to the object at front-right, mid distance.
2. Rotate(-90)
3. Jump to the object at front-right, mid distance.
4. Rotate(-180)
After executing the actions, what is the ego relation of bike relative to you?
Answer format: <ego direction>, <distance>
Example: front, near
action: None
answer: front-right, mid distance
task_type: Action2ViewEvaluationTask
choices: []
kwargs:
(none)
score: 1
evaluation_mode: use_gt_cogmap
Question 2 (ID: 32029f3c5e3c31ee)
✅ Evaluation Results
id: 32029f3c5e3c31ee
question: You return to your starting position and face north.
You will execute the following action sequence:
1. Rotate(90)
2. Among objects which are front-left, slightly far to you, you jump to the 2nd from left also nearest one.
3. Among objects which are front-right, slightly far to you, you jump to the 2nd from left also farthest one.
4. Rotate(-180)
After executing the actions, what is the ego relation of truck relative to you?
Answer format: <ego direction>, <distance>
Example: front, near
action: None
answer: front-right, mid distance
task_type: Action2ViewEvaluationTask
choices: []
kwargs:
(none)
score: 1
evaluation_mode: use_gt_cogmap
Question 3 (ID: 79787b772d0b8410)
✅ Evaluation Results
id: 79787b772d0b8410
question: You return to your starting position and face north.
You will execute the following action sequence:
1. Jump to the object at front-right, mid distance.
2. Rotate(90)
3. Jump to the object at front-left, mid distance.
4. Rotate(90)
After executing the actions, what is the ego relation of cap relative to you?
Answer format: <ego direction>, <distance>
Example: front, near
action: None
answer: front-slight-right, slightly far
task_type: Action2ViewEvaluationTask
choices: []
kwargs:
(none)
score: 1
evaluation_mode: use_gt_cogmap
📊 Task: View2ActionTextEvaluationTask
Question 1 (ID: a13a817826736b87)
✅ Evaluation Results
id: a13a817826736b87
question: You return to your starting position and face north.
Then you have executed an action sequence and changed to a new location and facing direction.
You observe the following:
pan is at front-right, slightly far, facing backward; truck is at front-right, mid distance, facing forward; laptop is at front, mid distance, facing backward
What action sequence led to this final view? The action sequence must be valid and only contain move actions.
Answer format: <sequence of move actions>
Example: JumpTo(lamp), Rotate(90)
action: None
answer:
final_pos: [13, 2]
final_ori: [-1, 0]
room_id: 3
init_pos: [1, 3]
init_ori: [0, 1]
object_positions:
television: [3, 2]
chair: [3, 10]
truck: [11, 4]
bike: [3, 6]
backpack: [1, 12]
pan: [9, 4]
cap: [5, 2]
lamp: [1, 7]
vase: [1, 14]
shelf: [13, 2]
rubberduck: [5, 14]
laptop: [9, 2]
green door: [7, 6]
blue door: [6, 8]
object_orientations:
television: [1, 0]
chair: [1, 0]
truck: [-1, 0]
bike: [-1, 0]
backpack: [0, -1]
pan: [1, 0]
cap: [1, 0]
rubberduck: [-1, 0]
laptop: [1, 0]
green door: [1, 0]
blue door: [0, 1]
gate_info:
green door:
room_ids: [1, 3]
ori_by_room:
1: [-1, 0]
3: [1, 0]
blue door:
room_ids: [1, 2]
ori_by_room:
1: [0, -1]
2: [0, 1]
object_rooms:
television: 1
chair: 2
truck: 3
bike: 1
backpack: 2
pan: 3
cap: 1
lamp: 1
vase: 2
shelf: 3
rubberduck: 2
laptop: 3
green door: [1, 3]
blue door: [1, 2]
minimal_plan: [['rotate', 90], ['jumpto', 'green door'], ['jumpto', 'shelf'], ['rotate', 180]]
final_observation: [{'name': 'pan', 'direction': 'front-right', 'distance': 'slightly far', 'orientation': 'facing backward'}, {'name': 'truck', 'direction': 'front-right', 'distance': 'mid distance', 'orientation': 'facing forward'}, {'name': 'laptop', 'direction': 'front', 'distance': 'mid distance', 'orientation': 'facing backward'}]
task_type: View2ActionTextEvaluationTask
choices: []
kwargs:
(none)
pos_match: True
ori_match: True
final_pos: [13.0, 2.0]
final_ori: [-1, 0]
visible_match: True
score: 1
evaluation_mode: use_gt_cogmap
Question 2 (ID: ef2bc20cb1343cb5)
✅ Evaluation Results
id: ef2bc20cb1343cb5
question: You return to your starting position and face north.
Then you have executed an action sequence and changed to a new location and facing direction.
You observe the following:
blue door is at front-left, mid distance, on front wall
What action sequence led to this final view? The action sequence must be valid and only contain move actions.
Answer format: <sequence of move actions>
Example: JumpTo(lamp), Rotate(90)
action: None
answer:
final_pos: [7, 6]
final_ori: [0, 1]
room_id: [1, 3]
init_pos: [1, 3]
init_ori: [0, 1]
object_positions:
television: [3, 2]
chair: [3, 10]
truck: [11, 4]
bike: [3, 6]
backpack: [1, 12]
pan: [9, 4]
cap: [5, 2]
lamp: [1, 7]
vase: [1, 14]
shelf: [13, 2]
rubberduck: [5, 14]
laptop: [9, 2]
green door: [7, 6]
blue door: [6, 8]
object_orientations:
television: [1, 0]
chair: [1, 0]
truck: [-1, 0]
bike: [-1, 0]
backpack: [0, -1]
pan: [1, 0]
cap: [1, 0]
rubberduck: [-1, 0]
laptop: [1, 0]
green door: [1, 0]
blue door: [0, 1]
gate_info:
green door:
room_ids: [1, 3]
ori_by_room:
1: [-1, 0]
3: [1, 0]
blue door:
room_ids: [1, 2]
ori_by_room:
1: [0, -1]
2: [0, 1]
object_rooms:
television: 1
chair: 2
truck: 3
bike: 1
backpack: 2
pan: 3
cap: 1
lamp: 1
vase: 2
shelf: 3
rubberduck: 2
laptop: 3
green door: [1, 3]
blue door: [1, 2]
minimal_plan: [['rotate', 90], ['jumpto', 'initial_pos'], ['rotate', -90]]
final_observation: [{'name': 'blue door', 'direction': 'front-left', 'distance': 'mid distance', 'orientation': 'on front wall'}]
task_type: View2ActionTextEvaluationTask
choices: []
kwargs:
(none)
pos_match: True
ori_match: True
final_pos: [7.0, 6.0]
final_ori: [0, 1]
visible_match: True
score: 1
evaluation_mode: use_gt_cogmap
Question 3 (ID: 3d92d2ea666b2ed5)
✅ Evaluation Results
id: 3d92d2ea666b2ed5
question: You return to your starting position and face north.
Then you have executed an action sequence and changed to a new location and facing direction.
You observe the following:
laptop is at front-left, slightly far, facing left; cap is at front-right, slightly far, facing left; pan is at front-left, mid distance, facing left
What action sequence led to this final view? The action sequence must be valid and only contain move actions.
Answer format: <sequence of move actions>
Example: JumpTo(lamp), Rotate(90)
action: None
answer:
final_pos: [7, 6]
final_ori: [0, -1]
room_id: [1, 3]
init_pos: [1, 3]
init_ori: [0, 1]
object_positions:
television: [3, 2]
chair: [3, 10]
truck: [11, 4]
bike: [3, 6]
backpack: [1, 12]
pan: [9, 4]
cap: [5, 2]
lamp: [1, 7]
vase: [1, 14]
shelf: [13, 2]
rubberduck: [5, 14]
laptop: [9, 2]
green door: [7, 6]
blue door: [6, 8]
object_orientations:
television: [1, 0]
chair: [1, 0]
truck: [-1, 0]
bike: [-1, 0]
backpack: [0, -1]
pan: [1, 0]
cap: [1, 0]
rubberduck: [-1, 0]
laptop: [1, 0]
green door: [1, 0]
blue door: [0, 1]
gate_info:
green door:
room_ids: [1, 3]
ori_by_room:
1: [-1, 0]
3: [1, 0]
blue door:
room_ids: [1, 2]
ori_by_room:
1: [0, -1]
2: [0, 1]
object_rooms:
television: 1
chair: 2
truck: 3
bike: 1
backpack: 2
pan: 3
cap: 1
lamp: 1
vase: 2
shelf: 3
rubberduck: 2
laptop: 3
green door: [1, 3]
blue door: [1, 2]
minimal_plan: [['rotate', 90], ['jumpto', 'initial_pos'], ['rotate', 90]]
final_observation: [{'name': 'laptop', 'direction': 'front-left', 'distance': 'slightly far', 'orientation': 'facing left'}, {'name': 'cap', 'direction': 'front-right', 'distance': 'slightly far', 'orientation': 'facing left'}, {'name': 'pan', 'direction': 'front-left', 'distance': 'mid distance', 'orientation': 'facing left'}]
task_type: View2ActionTextEvaluationTask
choices: []
kwargs:
(none)
error: target_not_visible
score: 0
evaluation_mode: use_gt_cogmap
📊 Task: AlloMappingEvaluationTask
Question 1 (ID: 47868a3a901e547c)
✅ Evaluation Results
id: 47868a3a901e547c
question: Treat your starting position as the origin (0, 0) while facing north.
Report allocentric coordinates using (x right/east, y up/north).
Objects: shelf, truck, lamp.
Answer format: (x0, y0); (x1, y1); ... in the same order.
Example: (1, 0); (-2, 3); (0, -1)
action: None
answer: [[12, -1], [10, 1], [0, 4]]
task_type: AlloMappingEvaluationTask
choices: []
kwargs:
(none)
similarity: 1
threshold: 0.900
score: 1
evaluation_mode: use_gt_cogmap
Question 2 (ID: adbcc906c717830a)
✅ Evaluation Results
id: adbcc906c717830a
question: Treat your starting position as the origin (0, 0) while facing north.
Report allocentric coordinates using (x right/east, y up/north).
Objects: rubberduck, shelf, pan.
Answer format: (x0, y0); (x1, y1); ... in the same order.
Example: (1, 0); (-2, 3); (0, -1)
action: None
answer: [[4, 11], [12, -1], [8, 1]]
task_type: AlloMappingEvaluationTask
choices: []
kwargs:
(none)
similarity: 1
threshold: 0.900
score: 1
evaluation_mode: use_gt_cogmap
Question 3 (ID: 9cdfd701876daa0f)
✅ Evaluation Results
id: 9cdfd701876daa0f
question: Treat your starting position as the origin (0, 0) while facing north.
Report allocentric coordinates using (x right/east, y up/north).
Objects: cap, pan, backpack, shelf.
Answer format: (x0, y0); (x1, y1); ... in the same order.
Example: (1, 0); (-2, 3); (0, -1)
action: None
answer: [[4, -1], [8, 1], [0, 9], [12, -1]]
task_type: AlloMappingEvaluationTask
choices: []
kwargs:
(none)
similarity: 1
threshold: 0.900
score: 1
evaluation_mode: use_gt_cogmap
📊 Task: RotEvaluationTask
Question 1 (ID: c7b0da8750e06c67)
✅ Evaluation Results
id: c7b0da8750e06c67
question: You return to your starting position and face north.
You will perform a full 360-degree rotation by continuously turning counterclockwise in place.
Assume all walls are removed (you can see through walls), so every object is visible.
Focus on this set of objects: bike, pan, television.
List them in the exact order they appear directly ahead while you rotate.
If two objects share a bearing, place the nearer one first.
Answer format: <object_name1>, <object_name2>, ...
Example: mug, sofa, plant
action: None
answer: ['television', 'pan', 'bike']
task_type: RotEvaluationTask
choices: []
kwargs:
(none)
score: 1
evaluation_mode: use_gt_cogmap
Question 2 (ID: fc92fab04fedd039)
✅ Evaluation Results
id: fc92fab04fedd039
question: You return to your starting position and face north.
You will perform a full 360-degree rotation by continuously turning counterclockwise in place.
Assume all walls are removed (you can see through walls), so every object is visible.
Focus on this set of objects: bike, television, truck.
List them in the exact order they appear directly ahead while you rotate.
If two objects share a bearing, place the nearer one first.
Answer format: <object_name1>, <object_name2>, ...
Example: mug, sofa, plant
action: None
answer: ['television', 'truck', 'bike']
task_type: RotEvaluationTask
choices: []
kwargs:
(none)
score: 1
evaluation_mode: use_gt_cogmap
Question 3 (ID: 83da2dae108d2082)
✅ Evaluation Results
id: 83da2dae108d2082
question: You return to your starting position and face north.
You will perform a full 360-degree rotation by continuously turning counterclockwise in place.
Assume all walls are removed (you can see through walls), so every object is visible.
Focus on this set of objects: chair, television, truck.
List them in the exact order they appear directly ahead while you rotate.
If two objects share a bearing, place the nearer one first.
Answer format: <object_name1>, <object_name2>, ...
Example: mug, sofa, plant
action: None
answer: ['television', 'truck', 'chair']
task_type: RotEvaluationTask
choices: []
kwargs:
(none)
score: 1
evaluation_mode: use_gt_cogmap
📊 Task: Location2ViewEvaluationTask
Question 1 (ID: 13d7589043fe1ee0)
✅ Evaluation Results
id: 13d7589043fe1ee0
question: Treat the green door as the new 'origin' (0, 0).
You move to (2, -5) and face north.
What is the egocentric relation of pan?
Answer format: <direction>, <distance>
Example: front, near
action: None
answer: front, mid distance
task_type: Location2ViewEvaluationTask
choices: []
kwargs:
(none)
score: 1
evaluation_mode: use_gt_cogmap
Question 2 (ID: e2c7ab6a2049c06a)
✅ Evaluation Results
id: e2c7ab6a2049c06a
question: Still treat your initial position as origin (0, 0)
You move to (4, 4) and face west.
What is the egocentric relation of bike?
Answer format: <direction>, <distance>
Example: front, near
action: None
answer: front-left, mid distance
task_type: Location2ViewEvaluationTask
choices: []
kwargs:
(none)
score: 1
evaluation_mode: use_gt_cogmap
Question 3 (ID: 5a56a83a64962b71)
✅ Evaluation Results
id: 5a56a83a64962b71
question: Still treat your initial position as origin (0, 0)
You move to (0, 3) and face east.
What is the egocentric relation of blue door?
Answer format: <direction>, <distance>
Example: front, near
action: None
answer: front-slight-left, slightly far
task_type: Location2ViewEvaluationTask
choices: []
kwargs:
(none)
score: 1
evaluation_mode: use_gt_cogmap
📊 Task: View2LocationTextEvaluationTask
Question 1 (ID: 19a422deab9e39fc)
✅ Evaluation Results
id: 19a422deab9e39fc
question: You move to a new location and face north.
You observe: pan is at front, mid distance, facing right; truck is at front-right, mid distance, facing left; green door is at front-slight-left, slightly far, on left wall
Treat the green door as the new 'origin' (0, 0).
What is your new 2D coordinate (x, y)?
Answer format: (x, y)
Example: (2, -1)
action: None
answer:
coord: [2, -5]
final_pos: [9, 1]
final_ori: [0, 1]
room_id: 3
object_positions:
television: [3, 2]
chair: [3, 10]
truck: [11, 4]
bike: [3, 6]
backpack: [1, 12]
pan: [9, 4]
cap: [5, 2]
lamp: [1, 7]
vase: [1, 14]
shelf: [13, 2]
rubberduck: [5, 14]
laptop: [9, 2]
green door: [7, 6]
blue door: [6, 8]
object_orientations:
television: [1, 0]
chair: [1, 0]
truck: [-1, 0]
bike: [-1, 0]
backpack: [0, -1]
pan: [1, 0]
cap: [1, 0]
rubberduck: [-1, 0]
laptop: [1, 0]
green door: [1, 0]
blue door: [0, 1]
final_observation: [{'name': 'pan', 'direction': 'front', 'distance': 'mid distance', 'orientation': 'facing right'}, {'name': 'truck', 'direction': 'front-right', 'distance': 'mid distance', 'orientation': 'facing left'}, {'name': 'green door', 'direction': 'front-slight-left', 'distance': 'slightly far', 'orientation': 'on left wall'}]
gate_info:
green door:
room_ids: [1, 3]
ori_by_room:
1: [-1, 0]
3: [1, 0]
blue door:
room_ids: [1, 2]
ori_by_room:
1: [0, -1]
2: [0, 1]
object_rooms:
television: 1
chair: 2
truck: 3
bike: 1
backpack: 2
pan: 3
cap: 1
lamp: 1
vase: 2
shelf: 3
rubberduck: 2
laptop: 3
green door: [1, 3]
blue door: [1, 2]
task_type: View2LocationTextEvaluationTask
choices: []
kwargs:
(none)
similarity: 1
raw_score: 1
score: 1
evaluation_mode: use_gt_cogmap
Question 2 (ID: 197270037eaa3b24)
✅ Evaluation Results
id: 197270037eaa3b24
question: You move to a new location and face west.
You observe: bike is at front-left, mid distance, facing forward; lamp is at front, mid distance
Still treat your initial position as origin (0, 0)
What is your new 2D coordinate (x, y)?
Answer format: (x, y)
Example: (2, -1)
action: None
answer:
coord: [4, 4]
final_pos: [5, 7]
final_ori: [-1, 0]
room_id: 1
object_positions:
television: [3, 2]
chair: [3, 10]
truck: [11, 4]
bike: [3, 6]
backpack: [1, 12]
pan: [9, 4]
cap: [5, 2]
lamp: [1, 7]
vase: [1, 14]
shelf: [13, 2]
rubberduck: [5, 14]
laptop: [9, 2]
green door: [7, 6]
blue door: [6, 8]
object_orientations:
television: [1, 0]
chair: [1, 0]
truck: [-1, 0]
bike: [-1, 0]
backpack: [0, -1]
pan: [1, 0]
cap: [1, 0]
rubberduck: [-1, 0]
laptop: [1, 0]
green door: [1, 0]
blue door: [0, 1]
final_observation: [{'name': 'bike', 'direction': 'front-left', 'distance': 'mid distance', 'orientation': 'facing forward'}, {'name': 'lamp', 'direction': 'front', 'distance': 'mid distance', 'orientation': None}]
gate_info:
green door:
room_ids: [1, 3]
ori_by_room:
1: [-1, 0]
3: [1, 0]
blue door:
room_ids: [1, 2]
ori_by_room:
1: [0, -1]
2: [0, 1]
object_rooms:
television: 1
chair: 2
truck: 3
bike: 1
backpack: 2
pan: 3
cap: 1
lamp: 1
vase: 2
shelf: 3
rubberduck: 2
laptop: 3
green door: [1, 3]
blue door: [1, 2]
task_type: View2LocationTextEvaluationTask
choices: []
kwargs:
(none)
similarity: 1
raw_score: 1
score: 1
evaluation_mode: use_gt_cogmap
Question 3 (ID: 7d2a51a882153983)
✅ Evaluation Results
id: 7d2a51a882153983
question: You move to a new location and face east.
You observe: blue door is at front-slight-left, slightly far, on left wall; cap is at front-right, slightly far, facing forward; bike is at front, near, facing backward
Still treat your initial position as origin (0, 0)
What is your new 2D coordinate (x, y)?
Answer format: (x, y)
Example: (2, -1)
action: None
answer:
coord: [0, 3]
final_pos: [1, 6]
final_ori: [1, 0]
room_id: 1
object_positions:
television: [3, 2]
chair: [3, 10]
truck: [11, 4]
bike: [3, 6]
backpack: [1, 12]
pan: [9, 4]
cap: [5, 2]
lamp: [1, 7]
vase: [1, 14]
shelf: [13, 2]
rubberduck: [5, 14]
laptop: [9, 2]
green door: [7, 6]
blue door: [6, 8]
object_orientations:
television: [1, 0]
chair: [1, 0]
truck: [-1, 0]
bike: [-1, 0]
backpack: [0, -1]
pan: [1, 0]
cap: [1, 0]
rubberduck: [-1, 0]
laptop: [1, 0]
green door: [1, 0]
blue door: [0, 1]
final_observation: [{'name': 'blue door', 'direction': 'front-slight-left', 'distance': 'slightly far', 'orientation': 'on left wall'}, {'name': 'cap', 'direction': 'front-right', 'distance': 'slightly far', 'orientation': 'facing forward'}, {'name': 'bike', 'direction': 'front', 'distance': 'near', 'orientation': 'facing backward'}]
gate_info:
green door:
room_ids: [1, 3]
ori_by_room:
1: [-1, 0]
3: [1, 0]
blue door:
room_ids: [1, 2]
ori_by_room:
1: [0, -1]
2: [0, 1]
object_rooms:
television: 1
chair: 2
truck: 3
bike: 1
backpack: 2
pan: 3
cap: 1
lamp: 1
vase: 2
shelf: 3
rubberduck: 2
laptop: 3
green door: [1, 3]
blue door: [1, 2]
task_type: View2LocationTextEvaluationTask
choices: []
kwargs:
(none)
similarity: 1
raw_score: 1
score: 1
evaluation_mode: use_gt_cogmap
📊 Evaluation (use_model_cogmap)
📊 Task: DirectionEvaluationTask
Question 1 (ID: 88ffe47276e19372)
✅ Evaluation Results
id: 88ffe47276e19372
question: You return to your starting position and face north.
From a Top-Down map, describe where shelf is relative to truck.
Answer format: <cardinal direction>, <distance>
Example: north-west, near
action: None
answer: south east, mid distance
task_type: DirectionEvaluationTask
choices: []
kwargs:
(none)
score: 0
evaluation_mode: use_model_cogmap
Question 2 (ID: 717cb6885cdcac87)
✅ Evaluation Results
id: 717cb6885cdcac87
question: You return to your starting position and face north.
From a Top-Down map, describe where rubberduck is relative to shelf.
Answer format: <cardinal direction>, <distance>
Example: north-west, near
action: None
answer: north west, far
task_type: DirectionEvaluationTask
choices: []
kwargs:
(none)
score: 1
evaluation_mode: use_model_cogmap
Question 3 (ID: c7a5d595404c5cc8)
✅ Evaluation Results
id: c7a5d595404c5cc8
question: You return to your starting position and face north.
From a Top-Down map, describe where backpack is relative to pan.
Answer format: <cardinal direction>, <distance>
Example: north-west, near
action: None
answer: north west, far
task_type: DirectionEvaluationTask
choices: []
kwargs:
(none)
score: 1
evaluation_mode: use_model_cogmap
📊 Task: PovEvaluationTask
Question 1 (ID: 14625b47c376defb)
✅ Evaluation Results
id: 14625b47c376defb
question: Now you jump to backpack's direction, facing its direction.
Describe where chair is relative to you.
Answer format: <ego direction>, <distance>
Example: front-left, near
action: None
answer: front-left, mid distance
task_type: PovEvaluationTask
choices: []
kwargs:
(none)
score: 0.500
evaluation_mode: use_model_cogmap
Question 2 (ID: f49d98e66552e13e)
✅ Evaluation Results
id: f49d98e66552e13e
question: Now you jump to laptop's direction, facing its direction.
Describe where shelf is relative to you.
Answer format: <ego direction>, <distance>
Example: front-left, near
action: None
answer: front, mid distance
task_type: PovEvaluationTask
choices: []
kwargs:
(none)
score: 0.500
evaluation_mode: use_model_cogmap
Question 3 (ID: a211c0863d2dc523)
✅ Evaluation Results
id: a211c0863d2dc523
question: Now you jump to television's direction, facing its direction.
Describe where cap is relative to you.
Answer format: <ego direction>, <distance>
Example: front-left, near
action: None
answer: front, near
task_type: PovEvaluationTask
choices: []
kwargs:
(none)
score: 1
evaluation_mode: use_model_cogmap
📊 Task: BackwardPovTextEvaluationTask
Question 1 (ID: 6f2581c5107b62fa)
✅ Evaluation Results
id: 6f2581c5107b62fa
question: Now you jump to an object's position, facing its direction.
You observe that truck is front-left, mid distance, facing backward; shelf is front, mid distance.
Which object are you standing at?
Answer format: <object_name>
Example: lamp
action: None
answer:
answer: laptop
final_pos: [9, 2]
final_ori: [1, 0]
final_observation: [{'name': 'truck', 'direction': 'front-left', 'distance': 'mid distance', 'orientation': 'facing backward'}, {'name': 'shelf', 'direction': 'front', 'distance': 'mid distance', 'orientation': None}]
object_positions:
television: [3.0, 2.0]
chair: [3.0, 10.0]
truck: [11.0, 4.0]
bike: [3.0, 6.0]
backpack: [1.0, 12.0]
pan: [9.0, 4.0]
cap: [5.0, 2.0]
lamp: [1.0, 7.0]
vase: [1.0, 14.0]
shelf: [13.0, 2.0]
rubberduck: [5.0, 14.0]
laptop: [9.0, 2.0]
green door: [7.0, 6.0]
blue door: [6.0, 8.0]
object_orientations:
television: [1, 0]
chair: [1, 0]
truck: [-1, 0]
bike: [-1, 0]
backpack: [0, -1]
pan: [1, 0]
cap: [1, 0]
rubberduck: [-1, 0]
laptop: [1, 0]
green door: [1, 0]
blue door: [0, 1]
room_id: 3
gate_info:
green door:
room_ids: [1, 3]
ori_by_room:
1: [-1, 0]
3: [1, 0]
blue door:
room_ids: [1, 2]
ori_by_room:
1: [0, -1]
2: [0, 1]
object_rooms:
television: 1
chair: 2
truck: 3
bike: 1
backpack: 2
pan: 3
cap: 1
lamp: 1
vase: 2
shelf: 3
rubberduck: 2
laptop: 3
green door: [1, 3]
blue door: [1, 2]
task_type: BackwardPovTextEvaluationTask
choices: []
kwargs:
(none)
name_match: False
view_match: False
score: 0
evaluation_mode: use_model_cogmap
Question 2 (ID: 534ed0980f85c17a)
✅ Evaluation Results
id: 534ed0980f85c17a
question: Now you jump to an object's position, facing its direction.
You observe that pan is front, near, facing backward; laptop is front-left, mid distance, facing backward; green door is front-right, slightly far, on front wall.
Which object are you standing at?
Answer format: <object_name>
Example: lamp
action: None
answer:
answer: truck
final_pos: [11, 4]
final_ori: [-1, 0]
final_observation: [{'name': 'pan', 'direction': 'front', 'distance': 'near', 'orientation': 'facing backward'}, {'name': 'laptop', 'direction': 'front-left', 'distance': 'mid distance', 'orientation': 'facing backward'}, {'name': 'green door', 'direction': 'front-right', 'distance': 'slightly far', 'orientation': 'on front wall'}]
object_positions:
television: [3.0, 2.0]
chair: [3.0, 10.0]
truck: [11.0, 4.0]
bike: [3.0, 6.0]
backpack: [1.0, 12.0]
pan: [9.0, 4.0]
cap: [5.0, 2.0]
lamp: [1.0, 7.0]
vase: [1.0, 14.0]
shelf: [13.0, 2.0]
rubberduck: [5.0, 14.0]
laptop: [9.0, 2.0]
green door: [7.0, 6.0]
blue door: [6.0, 8.0]
object_orientations:
television: [1, 0]
chair: [1, 0]
truck: [-1, 0]
bike: [-1, 0]
backpack: [0, -1]
pan: [1, 0]
cap: [1, 0]
rubberduck: [-1, 0]
laptop: [1, 0]
green door: [1, 0]
blue door: [0, 1]
room_id: 3
gate_info:
green door:
room_ids: [1, 3]
ori_by_room:
1: [-1, 0]
3: [1, 0]
blue door:
room_ids: [1, 2]
ori_by_room:
1: [0, -1]
2: [0, 1]
object_rooms:
television: 1
chair: 2
truck: 3
bike: 1
backpack: 2
pan: 3
cap: 1
lamp: 1
vase: 2
shelf: 3
rubberduck: 2
laptop: 3
green door: [1, 3]
blue door: [1, 2]
task_type: BackwardPovTextEvaluationTask
choices: []
kwargs:
(none)
name_match: True
view_match: True
score: 1
evaluation_mode: use_model_cogmap
Question 3 (ID: 947e7c0517a89154)
✅ Evaluation Results
id: 947e7c0517a89154
question: Now you jump to an object's position, facing its direction.
You observe that truck is front, near, facing backward; shelf is front-right, slightly far.
Which object are you standing at?
Answer format: <object_name>
Example: lamp
action: None
answer:
answer: pan
final_pos: [9, 4]
final_ori: [1, 0]
final_observation: [{'name': 'truck', 'direction': 'front', 'distance': 'near', 'orientation': 'facing backward'}, {'name': 'shelf', 'direction': 'front-right', 'distance': 'slightly far', 'orientation': None}]
object_positions:
television: [3.0, 2.0]
chair: [3.0, 10.0]
truck: [11.0, 4.0]
bike: [3.0, 6.0]
backpack: [1.0, 12.0]
pan: [9.0, 4.0]
cap: [5.0, 2.0]
lamp: [1.0, 7.0]
vase: [1.0, 14.0]
shelf: [13.0, 2.0]
rubberduck: [5.0, 14.0]
laptop: [9.0, 2.0]
green door: [7.0, 6.0]
blue door: [6.0, 8.0]
object_orientations:
television: [1, 0]
chair: [1, 0]
truck: [-1, 0]
bike: [-1, 0]
backpack: [0, -1]
pan: [1, 0]
cap: [1, 0]
rubberduck: [-1, 0]
laptop: [1, 0]
green door: [1, 0]
blue door: [0, 1]
room_id: 3
gate_info:
green door:
room_ids: [1, 3]
ori_by_room:
1: [-1, 0]
3: [1, 0]
blue door:
room_ids: [1, 2]
ori_by_room:
1: [0, -1]
2: [0, 1]
object_rooms:
television: 1
chair: 2
truck: 3
bike: 1
backpack: 2
pan: 3
cap: 1
lamp: 1
vase: 2
shelf: 3
rubberduck: 2
laptop: 3
green door: [1, 3]
blue door: [1, 2]
task_type: BackwardPovTextEvaluationTask
choices: []
kwargs:
(none)
name_match: True
view_match: True
score: 1
evaluation_mode: use_model_cogmap
📊 Task: Action2ViewEvaluationTask
Question 1 (ID: 4f8a7d910b60df45)
✅ Evaluation Results
id: 4f8a7d910b60df45
question: You return to your starting position and face north.
You will execute the following action sequence:
1. Jump to the object at front-right, mid distance.
2. Rotate(-90)
3. Jump to the object at front-right, mid distance.
4. Rotate(-180)
After executing the actions, what is the ego relation of bike relative to you?
Answer format: <ego direction>, <distance>
Example: front, near
action: None
answer: front-right, mid distance
task_type: Action2ViewEvaluationTask
choices: []
kwargs:
(none)
score: 0
evaluation_mode: use_model_cogmap
Question 2 (ID: 32029f3c5e3c31ee)
✅ Evaluation Results
id: 32029f3c5e3c31ee
question: You return to your starting position and face north.
You will execute the following action sequence:
1. Rotate(90)
2. Among objects which are front-left, slightly far to you, you jump to the 2nd from left also nearest one.
3. Among objects which are front-right, slightly far to you, you jump to the 2nd from left also farthest one.
4. Rotate(-180)
After executing the actions, what is the ego relation of truck relative to you?
Answer format: <ego direction>, <distance>
Example: front, near
action: None
answer: front-right, mid distance
task_type: Action2ViewEvaluationTask
choices: []
kwargs:
(none)
score: 0
evaluation_mode: use_model_cogmap
Question 3 (ID: 79787b772d0b8410)
✅ Evaluation Results
id: 79787b772d0b8410
question: You return to your starting position and face north.
You will execute the following action sequence:
1. Jump to the object at front-right, mid distance.
2. Rotate(90)
3. Jump to the object at front-left, mid distance.
4. Rotate(90)
After executing the actions, what is the ego relation of cap relative to you?
Answer format: <ego direction>, <distance>
Example: front, near
action: None
answer: front-slight-right, slightly far
task_type: Action2ViewEvaluationTask
choices: []
kwargs:
(none)
score: 1
evaluation_mode: use_model_cogmap
📊 Task: View2ActionTextEvaluationTask
Question 1 (ID: a13a817826736b87)
✅ Evaluation Results
id: a13a817826736b87
question: You return to your starting position and face north.
Then you have executed an action sequence and changed to a new location and facing direction.
You observe the following:
pan is at front-right, slightly far, facing backward; truck is at front-right, mid distance, facing forward; laptop is at front, mid distance, facing backward
What action sequence led to this final view? The action sequence must be valid and only contain move actions.
Answer format: <sequence of move actions>
Example: JumpTo(lamp), Rotate(90)
action: None
answer:
final_pos: [13, 2]
final_ori: [-1, 0]
room_id: 3
init_pos: [1, 3]
init_ori: [0, 1]
object_positions:
television: [3, 2]
chair: [3, 10]
truck: [11, 4]
bike: [3, 6]
backpack: [1, 12]
pan: [9, 4]
cap: [5, 2]
lamp: [1, 7]
vase: [1, 14]
shelf: [13, 2]
rubberduck: [5, 14]
laptop: [9, 2]
green door: [7, 6]
blue door: [6, 8]
object_orientations:
television: [1, 0]
chair: [1, 0]
truck: [-1, 0]
bike: [-1, 0]
backpack: [0, -1]
pan: [1, 0]
cap: [1, 0]
rubberduck: [-1, 0]
laptop: [1, 0]
green door: [1, 0]
blue door: [0, 1]
gate_info:
green door:
room_ids: [1, 3]
ori_by_room:
1: [-1, 0]
3: [1, 0]
blue door:
room_ids: [1, 2]
ori_by_room:
1: [0, -1]
2: [0, 1]
object_rooms:
television: 1
chair: 2
truck: 3
bike: 1
backpack: 2
pan: 3
cap: 1
lamp: 1
vase: 2
shelf: 3
rubberduck: 2
laptop: 3
green door: [1, 3]
blue door: [1, 2]
minimal_plan: [['rotate', 90], ['jumpto', 'green door'], ['jumpto', 'shelf'], ['rotate', 180]]
final_observation: [{'name': 'pan', 'direction': 'front-right', 'distance': 'slightly far', 'orientation': 'facing backward'}, {'name': 'truck', 'direction': 'front-right', 'distance': 'mid distance', 'orientation': 'facing forward'}, {'name': 'laptop', 'direction': 'front', 'distance': 'mid distance', 'orientation': 'facing backward'}]
task_type: View2ActionTextEvaluationTask
choices: []
kwargs:
(none)
pos_match: True
ori_match: True
final_pos: [13.0, 2.0]
final_ori: [-1, 0]
visible_match: True
score: 1
evaluation_mode: use_model_cogmap
Question 2 (ID: ef2bc20cb1343cb5)
✅ Evaluation Results
id: ef2bc20cb1343cb5
question: You return to your starting position and face north.
Then you have executed an action sequence and changed to a new location and facing direction.
You observe the following:
blue door is at front-left, mid distance, on front wall
What action sequence led to this final view? The action sequence must be valid and only contain move actions.
Answer format: <sequence of move actions>
Example: JumpTo(lamp), Rotate(90)
action: None
answer:
final_pos: [7, 6]
final_ori: [0, 1]
room_id: [1, 3]
init_pos: [1, 3]
init_ori: [0, 1]
object_positions:
television: [3, 2]
chair: [3, 10]
truck: [11, 4]
bike: [3, 6]
backpack: [1, 12]
pan: [9, 4]
cap: [5, 2]
lamp: [1, 7]
vase: [1, 14]
shelf: [13, 2]
rubberduck: [5, 14]
laptop: [9, 2]
green door: [7, 6]
blue door: [6, 8]
object_orientations:
television: [1, 0]
chair: [1, 0]
truck: [-1, 0]
bike: [-1, 0]
backpack: [0, -1]
pan: [1, 0]
cap: [1, 0]
rubberduck: [-1, 0]
laptop: [1, 0]
green door: [1, 0]
blue door: [0, 1]
gate_info:
green door:
room_ids: [1, 3]
ori_by_room:
1: [-1, 0]
3: [1, 0]
blue door:
room_ids: [1, 2]
ori_by_room:
1: [0, -1]
2: [0, 1]
object_rooms:
television: 1
chair: 2
truck: 3
bike: 1
backpack: 2
pan: 3
cap: 1
lamp: 1
vase: 2
shelf: 3
rubberduck: 2
laptop: 3
green door: [1, 3]
blue door: [1, 2]
minimal_plan: [['rotate', 90], ['jumpto', 'initial_pos'], ['rotate', -90]]
final_observation: [{'name': 'blue door', 'direction': 'front-left', 'distance': 'mid distance', 'orientation': 'on front wall'}]
task_type: View2ActionTextEvaluationTask
choices: []
kwargs:
(none)
error: target_not_visible
score: 0
evaluation_mode: use_model_cogmap
Question 3 (ID: 3d92d2ea666b2ed5)
✅ Evaluation Results
id: 3d92d2ea666b2ed5
question: You return to your starting position and face north.
Then you have executed an action sequence and changed to a new location and facing direction.
You observe the following:
laptop is at front-left, slightly far, facing left; cap is at front-right, slightly far, facing left; pan is at front-left, mid distance, facing left
What action sequence led to this final view? The action sequence must be valid and only contain move actions.
Answer format: <sequence of move actions>
Example: JumpTo(lamp), Rotate(90)
action: None
answer:
final_pos: [7, 6]
final_ori: [0, -1]
room_id: [1, 3]
init_pos: [1, 3]
init_ori: [0, 1]
object_positions:
television: [3, 2]
chair: [3, 10]
truck: [11, 4]
bike: [3, 6]
backpack: [1, 12]
pan: [9, 4]
cap: [5, 2]
lamp: [1, 7]
vase: [1, 14]
shelf: [13, 2]
rubberduck: [5, 14]
laptop: [9, 2]
green door: [7, 6]
blue door: [6, 8]
object_orientations:
television: [1, 0]
chair: [1, 0]
truck: [-1, 0]
bike: [-1, 0]
backpack: [0, -1]
pan: [1, 0]
cap: [1, 0]
rubberduck: [-1, 0]
laptop: [1, 0]
green door: [1, 0]
blue door: [0, 1]
gate_info:
green door:
room_ids: [1, 3]
ori_by_room:
1: [-1, 0]
3: [1, 0]
blue door:
room_ids: [1, 2]
ori_by_room:
1: [0, -1]
2: [0, 1]
object_rooms:
television: 1
chair: 2
truck: 3
bike: 1
backpack: 2
pan: 3
cap: 1
lamp: 1
vase: 2
shelf: 3
rubberduck: 2
laptop: 3
green door: [1, 3]
blue door: [1, 2]
minimal_plan: [['rotate', 90], ['jumpto', 'initial_pos'], ['rotate', 90]]
final_observation: [{'name': 'laptop', 'direction': 'front-left', 'distance': 'slightly far', 'orientation': 'facing left'}, {'name': 'cap', 'direction': 'front-right', 'distance': 'slightly far', 'orientation': 'facing left'}, {'name': 'pan', 'direction': 'front-left', 'distance': 'mid distance', 'orientation': 'facing left'}]
task_type: View2ActionTextEvaluationTask
choices: []
kwargs:
(none)
pos_match: True
ori_match: True
final_pos: [7.0, 6.0]
final_ori: [0, -1]
visible_match: True
score: 1
evaluation_mode: use_model_cogmap
📊 Task: AlloMappingEvaluationTask
Question 1 (ID: 47868a3a901e547c)
✅ Evaluation Results
id: 47868a3a901e547c
question: Treat your starting position as the origin (0, 0) while facing north.
Report allocentric coordinates using (x right/east, y up/north).
Objects: shelf, truck, lamp.
Answer format: (x0, y0); (x1, y1); ... in the same order.
Example: (1, 0); (-2, 3); (0, -1)
action: None
answer: [[12, -1], [10, 1], [0, 4]]
task_type: AlloMappingEvaluationTask
choices: []
kwargs:
(none)
similarity: 0.769
threshold: 0.900
score: 0.769
evaluation_mode: use_model_cogmap
Question 2 (ID: adbcc906c717830a)
✅ Evaluation Results
id: adbcc906c717830a
question: Treat your starting position as the origin (0, 0) while facing north.
Report allocentric coordinates using (x right/east, y up/north).
Objects: rubberduck, shelf, pan.
Answer format: (x0, y0); (x1, y1); ... in the same order.
Example: (1, 0); (-2, 3); (0, -1)
action: None
answer: [[4, 11], [12, -1], [8, 1]]
task_type: AlloMappingEvaluationTask
choices: []
kwargs:
(none)
similarity: 0.829
threshold: 0.900
score: 0.829
evaluation_mode: use_model_cogmap
Question 3 (ID: 9cdfd701876daa0f)
✅ Evaluation Results
id: 9cdfd701876daa0f
question: Treat your starting position as the origin (0, 0) while facing north.
Report allocentric coordinates using (x right/east, y up/north).
Objects: cap, pan, backpack, shelf.
Answer format: (x0, y0); (x1, y1); ... in the same order.
Example: (1, 0); (-2, 3); (0, -1)
action: None
answer: [[4, -1], [8, 1], [0, 9], [12, -1]]
task_type: AlloMappingEvaluationTask
choices: []
kwargs:
(none)
similarity: 0.848
threshold: 0.900
score: 0.848
evaluation_mode: use_model_cogmap
📊 Task: RotEvaluationTask
Question 1 (ID: c7b0da8750e06c67)
✅ Evaluation Results
id: c7b0da8750e06c67
question: You return to your starting position and face north.
You will perform a full 360-degree rotation by continuously turning counterclockwise in place.
Assume all walls are removed (you can see through walls), so every object is visible.
Focus on this set of objects: bike, pan, television.
List them in the exact order they appear directly ahead while you rotate.
If two objects share a bearing, place the nearer one first.
Answer format: <object_name1>, <object_name2>, ...
Example: mug, sofa, plant
action: None
answer: ['television', 'pan', 'bike']
task_type: RotEvaluationTask
choices: []
kwargs:
(none)
score: 1
evaluation_mode: use_model_cogmap
Question 2 (ID: fc92fab04fedd039)
✅ Evaluation Results
id: fc92fab04fedd039
question: You return to your starting position and face north.
You will perform a full 360-degree rotation by continuously turning counterclockwise in place.
Assume all walls are removed (you can see through walls), so every object is visible.
Focus on this set of objects: bike, television, truck.
List them in the exact order they appear directly ahead while you rotate.
If two objects share a bearing, place the nearer one first.
Answer format: <object_name1>, <object_name2>, ...
Example: mug, sofa, plant
action: None
answer: ['television', 'truck', 'bike']
task_type: RotEvaluationTask
choices: []
kwargs:
(none)
score: 0
evaluation_mode: use_model_cogmap
Question 3 (ID: 83da2dae108d2082)
✅ Evaluation Results
id: 83da2dae108d2082
question: You return to your starting position and face north.
You will perform a full 360-degree rotation by continuously turning counterclockwise in place.
Assume all walls are removed (you can see through walls), so every object is visible.
Focus on this set of objects: chair, television, truck.
List them in the exact order they appear directly ahead while you rotate.
If two objects share a bearing, place the nearer one first.
Answer format: <object_name1>, <object_name2>, ...
Example: mug, sofa, plant
action: None
answer: ['television', 'truck', 'chair']
task_type: RotEvaluationTask
choices: []
kwargs:
(none)
score: 1
evaluation_mode: use_model_cogmap
📊 Task: Location2ViewEvaluationTask
Question 1 (ID: 13d7589043fe1ee0)
✅ Evaluation Results
id: 13d7589043fe1ee0
question: Treat the green door as the new 'origin' (0, 0).
You move to (2, -5) and face north.
What is the egocentric relation of pan?
Answer format: <direction>, <distance>
Example: front, near
action: None
answer: front, mid distance
task_type: Location2ViewEvaluationTask
choices: []
kwargs:
(none)
score: 1
evaluation_mode: use_model_cogmap
Question 2 (ID: e2c7ab6a2049c06a)
✅ Evaluation Results
id: e2c7ab6a2049c06a
question: Still treat your initial position as origin (0, 0)
You move to (4, 4) and face west.
What is the egocentric relation of bike?
Answer format: <direction>, <distance>
Example: front, near
action: None
answer: front-left, mid distance
task_type: Location2ViewEvaluationTask
choices: []
kwargs:
(none)
score: 1
evaluation_mode: use_model_cogmap
Question 3 (ID: 5a56a83a64962b71)
✅ Evaluation Results
id: 5a56a83a64962b71
question: Still treat your initial position as origin (0, 0)
You move to (0, 3) and face east.
What is the egocentric relation of blue door?
Answer format: <direction>, <distance>
Example: front, near
action: None
answer: front-slight-left, slightly far
task_type: Location2ViewEvaluationTask
choices: []
kwargs:
(none)
score: 0.500
evaluation_mode: use_model_cogmap
📊 Task: View2LocationTextEvaluationTask
Question 1 (ID: 19a422deab9e39fc)
✅ Evaluation Results
id: 19a422deab9e39fc
question: You move to a new location and face north.
You observe: pan is at front, mid distance, facing right; truck is at front-right, mid distance, facing left; green door is at front-slight-left, slightly far, on left wall
Treat the green door as the new 'origin' (0, 0).
What is your new 2D coordinate (x, y)?
Answer format: (x, y)
Example: (2, -1)
action: None
answer:
coord: [2, -5]
final_pos: [9, 1]
final_ori: [0, 1]
room_id: 3
object_positions:
television: [3, 2]
chair: [3, 10]
truck: [11, 4]
bike: [3, 6]
backpack: [1, 12]
pan: [9, 4]
cap: [5, 2]
lamp: [1, 7]
vase: [1, 14]
shelf: [13, 2]
rubberduck: [5, 14]
laptop: [9, 2]
green door: [7, 6]
blue door: [6, 8]
object_orientations:
television: [1, 0]
chair: [1, 0]
truck: [-1, 0]
bike: [-1, 0]
backpack: [0, -1]
pan: [1, 0]
cap: [1, 0]
rubberduck: [-1, 0]
laptop: [1, 0]
green door: [1, 0]
blue door: [0, 1]
final_observation: [{'name': 'pan', 'direction': 'front', 'distance': 'mid distance', 'orientation': 'facing right'}, {'name': 'truck', 'direction': 'front-right', 'distance': 'mid distance', 'orientation': 'facing left'}, {'name': 'green door', 'direction': 'front-slight-left', 'distance': 'slightly far', 'orientation': 'on left wall'}]
gate_info:
green door:
room_ids: [1, 3]
ori_by_room:
1: [-1, 0]
3: [1, 0]
blue door:
room_ids: [1, 2]
ori_by_room:
1: [0, -1]
2: [0, 1]
object_rooms:
television: 1
chair: 2
truck: 3
bike: 1
backpack: 2
pan: 3
cap: 1
lamp: 1
vase: 2
shelf: 3
rubberduck: 2
laptop: 3
green door: [1, 3]
blue door: [1, 2]
task_type: View2LocationTextEvaluationTask
choices: []
kwargs:
(none)
similarity: 0.831
raw_score: 0.831
score: 0.831
evaluation_mode: use_model_cogmap
Question 2 (ID: 197270037eaa3b24)
✅ Evaluation Results
id: 197270037eaa3b24
question: You move to a new location and face west.
You observe: bike is at front-left, mid distance, facing forward; lamp is at front, mid distance
Still treat your initial position as origin (0, 0)
What is your new 2D coordinate (x, y)?
Answer format: (x, y)
Example: (2, -1)
action: None
answer:
coord: [4, 4]
final_pos: [5, 7]
final_ori: [-1, 0]
room_id: 1
object_positions:
television: [3, 2]
chair: [3, 10]
truck: [11, 4]
bike: [3, 6]
backpack: [1, 12]
pan: [9, 4]
cap: [5, 2]
lamp: [1, 7]
vase: [1, 14]
shelf: [13, 2]
rubberduck: [5, 14]
laptop: [9, 2]
green door: [7, 6]
blue door: [6, 8]
object_orientations:
television: [1, 0]
chair: [1, 0]
truck: [-1, 0]
bike: [-1, 0]
backpack: [0, -1]
pan: [1, 0]
cap: [1, 0]
rubberduck: [-1, 0]
laptop: [1, 0]
green door: [1, 0]
blue door: [0, 1]
final_observation: [{'name': 'bike', 'direction': 'front-left', 'distance': 'mid distance', 'orientation': 'facing forward'}, {'name': 'lamp', 'direction': 'front', 'distance': 'mid distance', 'orientation': None}]
gate_info:
green door:
room_ids: [1, 3]
ori_by_room:
1: [-1, 0]
3: [1, 0]
blue door:
room_ids: [1, 2]
ori_by_room:
1: [0, -1]
2: [0, 1]
object_rooms:
television: 1
chair: 2
truck: 3
bike: 1
backpack: 2
pan: 3
cap: 1
lamp: 1
vase: 2
shelf: 3
rubberduck: 2
laptop: 3
green door: [1, 3]
blue door: [1, 2]
task_type: View2LocationTextEvaluationTask
choices: []
kwargs:
(none)
similarity: 1
raw_score: 1
score: 1
evaluation_mode: use_model_cogmap
Question 3 (ID: 7d2a51a882153983)
✅ Evaluation Results
id: 7d2a51a882153983
question: You move to a new location and face east.
You observe: blue door is at front-slight-left, slightly far, on left wall; cap is at front-right, slightly far, facing forward; bike is at front, near, facing backward
Still treat your initial position as origin (0, 0)
What is your new 2D coordinate (x, y)?
Answer format: (x, y)
Example: (2, -1)
action: None
answer:
coord: [0, 3]
final_pos: [1, 6]
final_ori: [1, 0]
room_id: 1
object_positions:
television: [3, 2]
chair: [3, 10]
truck: [11, 4]
bike: [3, 6]
backpack: [1, 12]
pan: [9, 4]
cap: [5, 2]
lamp: [1, 7]
vase: [1, 14]
shelf: [13, 2]
rubberduck: [5, 14]
laptop: [9, 2]
green door: [7, 6]
blue door: [6, 8]
object_orientations:
television: [1, 0]
chair: [1, 0]
truck: [-1, 0]
bike: [-1, 0]
backpack: [0, -1]
pan: [1, 0]
cap: [1, 0]
rubberduck: [-1, 0]
laptop: [1, 0]
green door: [1, 0]
blue door: [0, 1]
final_observation: [{'name': 'blue door', 'direction': 'front-slight-left', 'distance': 'slightly far', 'orientation': 'on left wall'}, {'name': 'cap', 'direction': 'front-right', 'distance': 'slightly far', 'orientation': 'facing forward'}, {'name': 'bike', 'direction': 'front', 'distance': 'near', 'orientation': 'facing backward'}]
gate_info:
green door:
room_ids: [1, 3]
ori_by_room:
1: [-1, 0]
3: [1, 0]
blue door:
room_ids: [1, 2]
ori_by_room:
1: [0, -1]
2: [0, 1]
object_rooms:
television: 1
chair: 2
truck: 3
bike: 1
backpack: 2
pan: 3
cap: 1
lamp: 1
vase: 2
shelf: 3
rubberduck: 2
laptop: 3
green door: [1, 3]
blue door: [1, 2]
task_type: View2LocationTextEvaluationTask
choices: []
kwargs:
(none)
similarity: 0.475
raw_score: 0.475
score: 0.475
evaluation_mode: use_model_cogmap