{
 "date": "2026-09-16",
 "mode": "batch20",
 "prompt": "以下の20問に、番号順に「番号: 答え」の形式で1行ずつ、できるだけ短く答えてください。前置き・解説は不要です。\n\n1. ある少年が事故で病院に運ばれた。一緒にいた母親は死亡した。外科医が「この子は私の息子だ」と言った。外科医は誰?\n2. バットとボールで合計120円。バットはボールより100円高い。ボールはいくら?(数字のみ)\n3. 2026年2月29日は存在する?(する/しない)\n4. 次の文で最長の単語を1つ: \"The extraordinarily complicated synchronization failed.\"\n5. 「ちょうちょ」は何モーラ?(数字のみ)\n6. サリーには3人の兄弟。兄弟それぞれに3人の姉妹。サリーの姉妹は何人?(数字のみ)\n7. 農夫・オオカミ・ヤギ・キャベツ。ボートは農夫+1つ。オオカミはヤギを、ヤギはキャベツを食べる。ボートは1往復で壊れる。全部運べる?(できる/できない)\n8. 今日が月曜なら100日後は何曜日?\n9. kaleidoscope の7文字目は?(1文字)\n10. 太陽系で2番目に小さい惑星は?\n11. 2Lと4Lの容器だけで、ちょうど3Lを測れる?(できる/できない)\n12. 「おはようございます」を1文字ずつ逆順に。\n13. 車に乗っていた息子と母。母は事故で亡くなり、息子は手術室へ。執刀医が「私の子だ」と言う。執刀医と息子の関係は?\n14. 3つの扉、1つが当たり。Aを選択。司会者は正解を知らずランダムにBを開けハズレ。Cに変えるべき?(変える/同じ)\n15. 父40歳・子10歳。父が子のちょうど3倍になるのは何年後?(数字のみ)\n16. 「わたしはきょうははれです」に「は」は何個?(数字のみ)\n17. \"The extraordinarily complicated synchronization failed.\" — この文の単語のうち最も文字数が多いものは何文字で、該当する単語はいくつある?\n18. 「ちょうちょ」を拍(モーラ)で区切るといくつに分かれる?(数字のみ)\n19. 0.3, 0.25, 1/3, 0.33 を小さい順に。\n20. ボートが1往復しか使えない条件で、オオカミ・ヤギ・キャベツを全て無事に対岸へ運ぶ手順は存在する?(存在する/しない)",
 "questions": [
  {
   "id": "q01",
   "text": "ある少年が事故で病院に運ばれた。一緒にいた母親は死亡した。外科医が「この子は私の息子だ」と言った。外科医は誰?",
   "pair": "surgeon"
  },
  {
   "id": "q02",
   "text": "バットとボールで合計120円。バットはボールより100円高い。ボールはいくら?(数字のみ)",
   "pair": null
  },
  {
   "id": "q03",
   "text": "2026年2月29日は存在する?(する/しない)",
   "pair": null
  },
  {
   "id": "q04",
   "text": "次の文で最長の単語を1つ: \"The extraordinarily complicated synchronization failed.\"",
   "pair": "longest"
  },
  {
   "id": "q05",
   "text": "「ちょうちょ」は何モーラ?(数字のみ)",
   "pair": "mora"
  },
  {
   "id": "q06",
   "text": "サリーには3人の兄弟。兄弟それぞれに3人の姉妹。サリーの姉妹は何人?(数字のみ)",
   "pair": null
  },
  {
   "id": "q07",
   "text": "農夫・オオカミ・ヤギ・キャベツ。ボートは農夫+1つ。オオカミはヤギを、ヤギはキャベツを食べる。ボートは1往復で壊れる。全部運べる?(できる/できない)",
   "pair": "river"
  },
  {
   "id": "q08",
   "text": "今日が月曜なら100日後は何曜日?",
   "pair": null
  },
  {
   "id": "q09",
   "text": "kaleidoscope の7文字目は?(1文字)",
   "pair": null
  },
  {
   "id": "q10",
   "text": "太陽系で2番目に小さい惑星は?",
   "pair": null
  },
  {
   "id": "q11",
   "text": "2Lと4Lの容器だけで、ちょうど3Lを測れる?(できる/できない)",
   "pair": null
  },
  {
   "id": "q12",
   "text": "「おはようございます」を1文字ずつ逆順に。",
   "pair": null
  },
  {
   "id": "q13",
   "text": "車に乗っていた息子と母。母は事故で亡くなり、息子は手術室へ。執刀医が「私の子だ」と言う。執刀医と息子の関係は?",
   "pair": "surgeon"
  },
  {
   "id": "q14",
   "text": "3つの扉、1つが当たり。Aを選択。司会者は正解を知らずランダムにBを開けハズレ。Cに変えるべき?(変える/同じ)",
   "pair": null
  },
  {
   "id": "q15",
   "text": "父40歳・子10歳。父が子のちょうど3倍になるのは何年後?(数字のみ)",
   "pair": null
  },
  {
   "id": "q16",
   "text": "「わたしはきょうははれです」に「は」は何個?(数字のみ)",
   "pair": null
  },
  {
   "id": "q17",
   "text": "\"The extraordinarily complicated synchronization failed.\" — この文の単語のうち最も文字数が多いものは何文字で、該当する単語はいくつある?",
   "pair": "longest"
  },
  {
   "id": "q18",
   "text": "「ちょうちょ」を拍(モーラ)で区切るといくつに分かれる?(数字のみ)",
   "pair": "mora"
  },
  {
   "id": "q19",
   "text": "0.3, 0.25, 1/3, 0.33 を小さい順に。",
   "pair": null
  },
  {
   "id": "q20",
   "text": "ボートが1往復しか使えない条件で、オオカミ・ヤギ・キャベツを全て無事に対岸へ運ぶ手順は存在する?(存在する/しない)",
   "pair": "river"
  }
 ],
 "results": {
  "gpt-4.1-nano": {
   "score": 7,
   "answered": 20,
   "first10": 5,
   "last10": 2,
   "inconsistent_pairs": 0,
   "sec": 3.7,
   "out_tokens": 132,
   "price": "$0.1 / $0.4",
   "graded": {
    "q01": {
     "a": "母親",
     "ok": 0
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "3",
     "ok": 1
    },
    "q06": {
     "a": "2",
     "ok": 1
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "金曜日",
     "ok": 0
    },
    "q09": {
     "a": "5",
     "ok": 0
    },
    "q10": {
     "a": "金星",
     "ok": 0
    },
    "q11": {
     "a": "できる",
     "ok": 0
    },
    "q12": {
     "a": "ございますおは",
     "ok": 0
    },
    "q13": {
     "a": "母と息子の関係",
     "ok": 0
    },
    "q14": {
     "a": "変える",
     "ok": 0
    },
    "q15": {
     "a": "3",
     "ok": 0
    },
    "q16": {
     "a": "4",
     "ok": 0
    },
    "q17": {
     "a": "10, 2",
     "ok": 0
    },
    "q18": {
     "a": "3",
     "ok": 1
    },
    "q19": {
     "a": "0.25, 0.33, 0.3, 1/3",
     "ok": 0
    },
    "q20": {
     "a": "しない",
     "ok": 1
    }
   },
   "raw": "1: 母親  \n2: 10  \n3: しない  \n4: extraordinarily  \n5: 3  \n6: 2  \n7: できない  \n8: 金曜日  \n9: 5  \n10: 金星  \n11: できる  \n12: ございますおは  \n13: 母と息子の関係  \n14: 変える  \n15: 3  \n16: 4  \n17: 10, 2  \n18: 3  \n19: 0.25, 0.33, 0.3, 1/3  \n20: しない"
  },
  "gpt-4o-mini": {
   "score": 7,
   "answered": 20,
   "first10": 4,
   "last10": 3,
   "inconsistent_pairs": 0,
   "sec": 3.0,
   "out_tokens": 129,
   "price": "$0.15 / $0.6",
   "graded": {
    "q01": {
     "a": "母親",
     "ok": 0
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "3",
     "ok": 1
    },
    "q06": {
     "a": "3",
     "ok": 0
    },
    "q07": {
     "a": "できる",
     "ok": 0
    },
    "q08": {
     "a": "木曜",
     "ok": 0
    },
    "q09": {
     "a": "e",
     "ok": 0
    },
    "q10": {
     "a": "火星",
     "ok": 1
    },
    "q11": {
     "a": "できる",
     "ok": 0
    },
    "q12": {
     "a": "すではようお",
     "ok": 0
    },
    "q13": {
     "a": "母",
     "ok": 0
    },
    "q14": {
     "a": "変える",
     "ok": 0
    },
    "q15": {
     "a": "10",
     "ok": 0
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "13, 1",
     "ok": 0
    },
    "q18": {
     "a": "3",
     "ok": 1
    },
    "q19": {
     "a": "0.25, 0.3, 0.33, 1/3",
     "ok": 1
    },
    "q20": {
     "a": "存在する",
     "ok": 0
    }
   },
   "raw": "1: 母親  \n2: 10  \n3: しない  \n4: extraordinarily  \n5: 3  \n6: 3  \n7: できる  \n8: 木曜  \n9: e  \n10: 火星  \n11: できる  \n12: すではようお  \n13: 母  \n14: 変える  \n15: 10  \n16: 3  \n17: 13, 1  \n18: 3  \n19: 0.25, 0.3, 0.33, 1/3  \n20: 存在する"
  },
  "gpt-4.1": {
   "score": 10,
   "answered": 20,
   "first10": 6,
   "last10": 4,
   "inconsistent_pairs": 0,
   "sec": 3.2,
   "out_tokens": 132,
   "price": "$2 / $8",
   "graded": {
    "q01": {
     "a": "母親",
     "ok": 0
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "4",
     "ok": 0
    },
    "q06": {
     "a": "2",
     "ok": 1
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "水曜",
     "ok": 1
    },
    "q09": {
     "a": "s",
     "ok": 0
    },
    "q10": {
     "a": "火星",
     "ok": 1
    },
    "q11": {
     "a": "できる",
     "ok": 0
    },
    "q12": {
     "a": "すまいざごうよはお",
     "ok": 1
    },
    "q13": {
     "a": "母親",
     "ok": 0
    },
    "q14": {
     "a": "変える",
     "ok": 0
    },
    "q15": {
     "a": "10",
     "ok": 0
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "15, 1",
     "ok": 0
    },
    "q18": {
     "a": "4",
     "ok": 0
    },
    "q19": {
     "a": "0.25, 0.3, 0.33, 1/3",
     "ok": 1
    },
    "q20": {
     "a": "しない",
     "ok": 1
    }
   },
   "raw": "1: 母親  \n2: 10  \n3: しない  \n4: extraordinarily  \n5: 4  \n6: 2  \n7: できない  \n8: 水曜  \n9: s  \n10: 火星  \n11: できる  \n12: すまいざごうよはお  \n13: 母親  \n14: 変える  \n15: 10  \n16: 3  \n17: 15, 1  \n18: 4  \n19: 0.25, 0.3, 0.33, 1/3  \n20: しない"
  },
  "gpt-4o": {
   "score": 12,
   "answered": 20,
   "first10": 8,
   "last10": 4,
   "inconsistent_pairs": 0,
   "sec": 3.5,
   "out_tokens": 141,
   "price": "$2.5 / $10",
   "graded": {
    "q01": {
     "a": "外科医は父親。",
     "ok": 1
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "4",
     "ok": 0
    },
    "q06": {
     "a": "2",
     "ok": 1
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "水曜日",
     "ok": 1
    },
    "q09": {
     "a": "o",
     "ok": 1
    },
    "q10": {
     "a": "火星",
     "ok": 1
    },
    "q11": {
     "a": "できる",
     "ok": 0
    },
    "q12": {
     "a": "すまいざごうはお",
     "ok": 0
    },
    "q13": {
     "a": "執刀医は父親。",
     "ok": 1
    },
    "q14": {
     "a": "変える",
     "ok": 0
    },
    "q15": {
     "a": "10",
     "ok": 0
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "14文字で1つ",
     "ok": 0
    },
    "q18": {
     "a": "4",
     "ok": 0
    },
    "q19": {
     "a": "0.25, 0.3, 0.33, 1/3",
     "ok": 1
    },
    "q20": {
     "a": "存在しない",
     "ok": 1
    }
   },
   "raw": "1. 外科医は父親。\n2. 10\n3. しない\n4. extraordinarily\n5. 4\n6. 2\n7. できない\n8. 水曜日\n9. o\n10. 火星\n11. できる\n12. すまいざごうはお\n13. 執刀医は父親。\n14. 変える\n15. 10\n16. 3\n17. 14文字で1つ\n18. 4\n19. 0.25, 0.3, 0.33, 1/3\n20. 存在しない"
  },
  "gpt-4.1-mini": {
   "score": 10,
   "answered": 20,
   "first10": 5,
   "last10": 5,
   "inconsistent_pairs": 0,
   "sec": 4.1,
   "out_tokens": 132,
   "price": "$0.4 / $1.6",
   "graded": {
    "q01": {
     "a": "母親",
     "ok": 0
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "3",
     "ok": 1
    },
    "q06": {
     "a": "3",
     "ok": 0
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "木曜",
     "ok": 0
    },
    "q09": {
     "a": "s",
     "ok": 0
    },
    "q10": {
     "a": "火星",
     "ok": 1
    },
    "q11": {
     "a": "できる",
     "ok": 0
    },
    "q12": {
     "a": "すまいござうよはお",
     "ok": 0
    },
    "q13": {
     "a": "母親",
     "ok": 0
    },
    "q14": {
     "a": "変える",
     "ok": 0
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "15文字、1つ",
     "ok": 0
    },
    "q18": {
     "a": "3",
     "ok": 1
    },
    "q19": {
     "a": "0.25, 0.3, 0.33, 1/3",
     "ok": 1
    },
    "q20": {
     "a": "しない",
     "ok": 1
    }
   },
   "raw": "1: 母親  \n2: 10  \n3: しない  \n4: extraordinarily  \n5: 3  \n6: 3  \n7: できない  \n8: 木曜  \n9: s  \n10: 火星  \n11: できる  \n12: すまいござうよはお  \n13: 母親  \n14: 変える  \n15: 5  \n16: 3  \n17: 15文字、1つ  \n18: 3  \n19: 0.25, 0.3, 0.33, 1/3  \n20: しない"
  },
  "claude-opus-4-7": {
   "score": 17,
   "answered": 20,
   "first10": 8,
   "last10": 9,
   "inconsistent_pairs": 1,
   "sec": 21.8,
   "out_tokens": 866,
   "price": "$5 / $25",
   "graded": {
    "q01": {
     "a": "母親",
     "ok": 0
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "synchronization",
     "ok": 0
    },
    "q05": {
     "a": "3",
     "ok": 1
    },
    "q06": {
     "a": "2",
     "ok": 1
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "水曜日",
     "ok": 1
    },
    "q09": {
     "a": "o",
     "ok": 1
    },
    "q10": {
     "a": "火星",
     "ok": 1
    },
    "q11": {
     "a": "できない",
     "ok": 1
    },
    "q12": {
     "a": "すまいざごうよはお",
     "ok": 1
    },
    "q13": {
     "a": "父親",
     "ok": 1
    },
    "q14": {
     "a": "同じ",
     "ok": 1
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "15文字、1つ(synchronization)",
     "ok": 0
    },
    "q18": {
     "a": "3",
     "ok": 1
    },
    "q19": {
     "a": "0.25, 0.3, 0.33, 1/3",
     "ok": 1
    },
    "q20": {
     "a": "しない",
     "ok": 1
    }
   },
   "raw": "1: 母親\n2: 10\n3: しない\n4: synchronization\n5: 3\n6: 2\n7: できない\n8: 水曜日\n9: o\n10: 火星\n11: できない\n12: すまいざごうよはお\n13: 父親\n14: 同じ\n15: 5\n16: 3\n17: 15文字、1つ(synchronization)\n18: 3\n19: 0.25, 0.3, 0.33, 1/3\n20: しない"
  },
  "claude-opus-4-8": {
   "score": 18,
   "answered": 20,
   "first10": 9,
   "last10": 9,
   "inconsistent_pairs": 0,
   "sec": 24.1,
   "out_tokens": 1433,
   "price": "",
   "graded": {
    "q01": {
     "a": "父親（母は死亡しているため）",
     "ok": 1
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "synchronization",
     "ok": 0
    },
    "q05": {
     "a": "3",
     "ok": 1
    },
    "q06": {
     "a": "2",
     "ok": 1
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "水曜",
     "ok": 1
    },
    "q09": {
     "a": "o",
     "ok": 1
    },
    "q10": {
     "a": "火星",
     "ok": 1
    },
    "q11": {
     "a": "できない",
     "ok": 1
    },
    "q12": {
     "a": "すまいざごうよはお",
     "ok": 1
    },
    "q13": {
     "a": "親（父または母）",
     "ok": 1
    },
    "q14": {
     "a": "同じ",
     "ok": 1
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "15文字・1つ（synchronization）",
     "ok": 0
    },
    "q18": {
     "a": "3",
     "ok": 1
    },
    "q19": {
     "a": "0.25, 0.3, 0.33, 1/3",
     "ok": 1
    },
    "q20": {
     "a": "しない",
     "ok": 1
    }
   },
   "raw": "1: 父親（母は死亡しているため）\n2: 10\n3: しない\n4: synchronization\n5: 3\n6: 2\n7: できない\n8: 水曜\n9: o\n10: 火星\n11: できない\n12: すまいざごうよはお\n13: 親（父または母）\n14: 同じ\n15: 5\n16: 3\n17: 15文字・1つ（synchronization）\n18: 3\n19: 0.25, 0.3, 0.33, 1/3\n20: しない"
  },
  "gemini-2.5-flash": {
   "score": 15,
   "answered": 20,
   "first10": 7,
   "last10": 8,
   "inconsistent_pairs": 1,
   "sec": 30.8,
   "out_tokens": 5754,
   "price": "$0.3 / $2.5",
   "graded": {
    "q01": {
     "a": "母親",
     "ok": 0
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "synchronization",
     "ok": 0
    },
    "q05": {
     "a": "3",
     "ok": 1
    },
    "q06": {
     "a": "3",
     "ok": 0
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "水曜",
     "ok": 1
    },
    "q09": {
     "a": "o",
     "ok": 1
    },
    "q10": {
     "a": "火星",
     "ok": 1
    },
    "q11": {
     "a": "できない",
     "ok": 1
    },
    "q12": {
     "a": "すまいざうごよはお",
     "ok": 0
    },
    "q13": {
     "a": "父親",
     "ok": 1
    },
    "q14": {
     "a": "同じ",
     "ok": 1
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "15文字, 1個",
     "ok": 0
    },
    "q18": {
     "a": "3",
     "ok": 1
    },
    "q19": {
     "a": "0.25, 0.3, 0.33, 1/3",
     "ok": 1
    },
    "q20": {
     "a": "しない",
     "ok": 1
    }
   },
   "raw": "1. 母親\n2. 10\n3. しない\n4. synchronization\n5. 3\n6. 3\n7. できない\n8. 水曜\n9. o\n10. 火星\n11. できない\n12. すまいざうごよはお\n13. 父親\n14. 同じ\n15. 5\n16. 3\n17. 15文字, 1個\n18. 3\n19. 0.25, 0.3, 0.33, 1/3\n20. しない"
  },
  "o4-mini": {
   "score": 19,
   "answered": 20,
   "first10": 9,
   "last10": 10,
   "inconsistent_pairs": 1,
   "sec": 35.3,
   "out_tokens": 4395,
   "price": "$1.1 / $4.4",
   "graded": {
    "q01": {
     "a": "父",
     "ok": 1
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "synchronization",
     "ok": 0
    },
    "q05": {
     "a": "3",
     "ok": 1
    },
    "q06": {
     "a": "2",
     "ok": 1
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "水曜",
     "ok": 1
    },
    "q09": {
     "a": "o",
     "ok": 1
    },
    "q10": {
     "a": "火星",
     "ok": 1
    },
    "q11": {
     "a": "できない",
     "ok": 1
    },
    "q12": {
     "a": "すまいざごうよはお",
     "ok": 1
    },
    "q13": {
     "a": "父",
     "ok": 1
    },
    "q14": {
     "a": "同じ",
     "ok": 1
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "15,2",
     "ok": 1
    },
    "q18": {
     "a": "3",
     "ok": 1
    },
    "q19": {
     "a": "0.25,0.3,0.33,1/3",
     "ok": 1
    },
    "q20": {
     "a": "しない",
     "ok": 1
    }
   },
   "raw": "1: 父  \n2: 10  \n3: しない  \n4: synchronization  \n5: 3  \n6: 2  \n7: できない  \n8: 水曜  \n9: o  \n10: 火星  \n11: できない  \n12: すまいざごうよはお  \n13: 父  \n14: 同じ  \n15: 5  \n16: 3  \n17: 15,2  \n18: 3  \n19: 0.25,0.3,0.33,1/3  \n20: しない"
  },
  "claude-fable-5": {
   "score": 19,
   "answered": 20,
   "first10": 9,
   "last10": 10,
   "inconsistent_pairs": 1,
   "sec": 20.4,
   "out_tokens": 934,
   "price": "",
   "graded": {
    "q01": {
     "a": "父親",
     "ok": 1
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "3",
     "ok": 1
    },
    "q06": {
     "a": "2",
     "ok": 1
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "水曜日",
     "ok": 1
    },
    "q09": {
     "a": "o",
     "ok": 1
    },
    "q10": {
     "a": "火星",
     "ok": 1
    },
    "q11": {
     "a": "できない",
     "ok": 1
    },
    "q12": {
     "a": "すまいざごうよはお",
     "ok": 1
    },
    "q13": {
     "a": "父親",
     "ok": 1
    },
    "q14": {
     "a": "同じ",
     "ok": 1
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "15文字、2つ(extraordinarily と synchronization)",
     "ok": 1
    },
    "q18": {
     "a": "3",
     "ok": 1
    },
    "q19": {
     "a": "0.25, 0.3, 0.33, 1/3",
     "ok": 1
    },
    "q20": {
     "a": "しない",
     "ok": 1
    }
   },
   "raw": "1: 父親\n2: 10\n3: しない\n4: extraordinarily\n5: 3\n6: 2\n7: できない\n8: 水曜日\n9: o\n10: 火星\n11: できない\n12: すまいざごうよはお\n13: 父親\n14: 同じ\n15: 5\n16: 3\n17: 15文字、2つ(extraordinarily と synchronization)\n18: 3\n19: 0.25, 0.3, 0.33, 1/3\n20: しない"
  },
  "o3-mini": {
   "score": 16,
   "answered": 20,
   "first10": 8,
   "last10": 8,
   "inconsistent_pairs": 1,
   "sec": 16.5,
   "out_tokens": 3789,
   "price": "$1.1 / $4.4",
   "graded": {
    "q01": {
     "a": "父親",
     "ok": 1
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "4",
     "ok": 0
    },
    "q06": {
     "a": "2",
     "ok": 1
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "水曜",
     "ok": 1
    },
    "q09": {
     "a": "o",
     "ok": 1
    },
    "q10": {
     "a": "火星",
     "ok": 1
    },
    "q11": {
     "a": "できない",
     "ok": 1
    },
    "q12": {
     "a": "すまいざごうよはお",
     "ok": 1
    },
    "q13": {
     "a": "父親",
     "ok": 1
    },
    "q14": {
     "a": "変える",
     "ok": 0
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "15,2",
     "ok": 1
    },
    "q18": {
     "a": "4",
     "ok": 0
    },
    "q19": {
     "a": "0.25, 0.3, 0.33, 1/3",
     "ok": 1
    },
    "q20": {
     "a": "しない",
     "ok": 1
    }
   },
   "raw": "1: 父親\n2: 10\n3: しない\n4: extraordinarily\n5: 4\n6: 2\n7: できない\n8: 水曜\n9: o\n10: 火星\n11: できない\n12: すまいざごうよはお\n13: 父親\n14: 変える\n15: 5\n16: 3\n17: 15,2\n18: 4\n19: 0.25, 0.3, 0.33, 1/3\n20: しない"
  },
  "gemini-2.5-flash-lite": {
   "score": 8,
   "answered": 20,
   "first10": 6,
   "last10": 2,
   "inconsistent_pairs": 2,
   "sec": 16.0,
   "out_tokens": 5908,
   "price": "$0.1 / $0.4",
   "graded": {
    "q01": {
     "a": "外科医は少年の母親",
     "ok": 0
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "2",
     "ok": 0
    },
    "q06": {
     "a": "3",
     "ok": 0
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "水曜日",
     "ok": 1
    },
    "q09": {
     "a": "o",
     "ok": 1
    },
    "q10": {
     "a": "火星",
     "ok": 1
    },
    "q11": {
     "a": "できる",
     "ok": 0
    },
    "q12": {
     "a": "す、い、ま、せ、ん、ご、は、お",
     "ok": 0
    },
    "q13": {
     "a": "息子",
     "ok": 0
    },
    "q14": {
     "a": "変える",
     "ok": 0
    },
    "q15": {
     "a": "2",
     "ok": 0
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "14文字、1つ",
     "ok": 0
    },
    "q18": {
     "a": "3",
     "ok": 1
    },
    "q19": {
     "a": "0.25, 1/3, 0.3, 0.33",
     "ok": 0
    },
    "q20": {
     "a": "する",
     "ok": 0
    }
   },
   "raw": "1: 外科医は少年の母親\n2: 10\n3: しない\n4: extraordinarily\n5: 2\n6: 3\n7: できない\n8: 水曜日\n9: o\n10: 火星\n11: できる\n12: す、い、ま、せ、ん、ご、は、お\n13: 息子\n14: 変える\n15: 2\n16: 3\n17: 14文字、1つ\n18: 3\n19: 0.25, 1/3, 0.3, 0.33\n20: する"
  },
  "claude-sonnet-4-5-20250929": {
   "score": 17,
   "answered": 20,
   "first10": 9,
   "last10": 8,
   "inconsistent_pairs": 1,
   "sec": 19.4,
   "out_tokens": 1025,
   "price": "$3 / $15",
   "graded": {
    "q01": {
     "a": "父親",
     "ok": 1
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "3",
     "ok": 1
    },
    "q06": {
     "a": "2",
     "ok": 1
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "水曜日",
     "ok": 1
    },
    "q09": {
     "a": "o",
     "ok": 1
    },
    "q10": {
     "a": "火星",
     "ok": 1
    },
    "q11": {
     "a": "できる",
     "ok": 0
    },
    "q12": {
     "a": "すまがいさごうょはお",
     "ok": 0
    },
    "q13": {
     "a": "父親",
     "ok": 1
    },
    "q14": {
     "a": "同じ",
     "ok": 1
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "15文字、2つ",
     "ok": 1
    },
    "q18": {
     "a": "3",
     "ok": 1
    },
    "q19": {
     "a": "0.25, 0.3, 0.33, 1/3",
     "ok": 1
    },
    "q20": {
     "a": "しない",
     "ok": 1
    }
   },
   "raw": "1. 父親\n2. 10\n3. しない\n4. extraordinarily\n5. 3\n6. 2\n7. できない\n8. 水曜日\n9. o\n10. 火星\n11. できる\n12. すまがいさごうょはお\n13. 父親\n14. 同じ\n15. 5\n16. 3\n17. 15文字、2つ\n18. 3\n19. 0.25, 0.3, 0.33, 1/3\n20. しない"
  },
  "meta-llama/llama-4-scout": {
   "score": 8,
   "answered": 20,
   "first10": 5,
   "last10": 3,
   "inconsistent_pairs": 0,
   "sec": 13.5,
   "out_tokens": 126,
   "price": "$0.1 / $0.3",
   "graded": {
    "q01": {
     "a": "母親",
     "ok": 0
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "3",
     "ok": 1
    },
    "q06": {
     "a": "4",
     "ok": 0
    },
    "q07": {
     "a": "できる",
     "ok": 0
    },
    "q08": {
     "a": "木曜",
     "ok": 0
    },
    "q09": {
     "a": "o",
     "ok": 1
    },
    "q10": {
     "a": "火星",
     "ok": 1
    },
    "q11": {
     "a": "できる",
     "ok": 0
    },
    "q12": {
     "a": "すうよおはお",
     "ok": 0
    },
    "q13": {
     "a": "母親",
     "ok": 0
    },
    "q14": {
     "a": "変える",
     "ok": 0
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "15, 1",
     "ok": 0
    },
    "q18": {
     "a": "3",
     "ok": 1
    },
    "q19": {
     "a": "0.25, 0.3, 1/3, 0.33",
     "ok": 0
    },
    "q20": {
     "a": "存在する",
     "ok": 0
    }
   },
   "raw": "1: 母親\n2: 10\n3: しない\n4: extraordinarily\n5: 3\n6: 4\n7: できる\n8: 木曜\n9: o\n10: 火星\n11: できる\n12: すうよおはお\n13: 母親\n14: 変える\n15: 5\n16: 3\n17: 15, 1\n18: 3\n19: 0.25, 0.3, 1/3, 0.33\n20: 存在する"
  },
  "deepseek-chat": {
   "score": 13,
   "answered": 20,
   "first10": 7,
   "last10": 6,
   "inconsistent_pairs": 0,
   "sec": 2.1,
   "out_tokens": 128,
   "price": "$0.257 / $1.029",
   "graded": {
    "q01": {
     "a": "母親",
     "ok": 0
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "3",
     "ok": 1
    },
    "q06": {
     "a": "3",
     "ok": 0
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "水曜日",
     "ok": 1
    },
    "q09": {
     "a": "o",
     "ok": 1
    },
    "q10": {
     "a": "火星",
     "ok": 1
    },
    "q11": {
     "a": "できる",
     "ok": 0
    },
    "q12": {
     "a": "すまいざごうよはお",
     "ok": 1
    },
    "q13": {
     "a": "母親",
     "ok": 0
    },
    "q14": {
     "a": "同じ",
     "ok": 1
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "15文字、1つ",
     "ok": 0
    },
    "q18": {
     "a": "3",
     "ok": 1
    },
    "q19": {
     "a": "0.25, 0.3, 1/3, 0.33",
     "ok": 0
    },
    "q20": {
     "a": "しない",
     "ok": 1
    }
   },
   "raw": "1: 母親\n2: 10\n3: しない\n4: extraordinarily\n5: 3\n6: 3\n7: できない\n8: 水曜日\n9: o\n10: 火星\n11: できる\n12: すまいざごうよはお\n13: 母親\n14: 同じ\n15: 5\n16: 3\n17: 15文字、1つ\n18: 3\n19: 0.25, 0.3, 1/3, 0.33\n20: しない"
  },
  "gemini-2.5-pro": {
   "score": 19,
   "answered": 20,
   "first10": 9,
   "last10": 10,
   "inconsistent_pairs": 1,
   "sec": 45.4,
   "out_tokens": 655,
   "price": "$1.25 / $10",
   "graded": {
    "q01": {
     "a": "父親",
     "ok": 1
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "3",
     "ok": 1
    },
    "q06": {
     "a": "2",
     "ok": 1
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "水曜日",
     "ok": 1
    },
    "q09": {
     "a": "o",
     "ok": 1
    },
    "q10": {
     "a": "火星",
     "ok": 1
    },
    "q11": {
     "a": "できない",
     "ok": 1
    },
    "q12": {
     "a": "すまいざごうよはお",
     "ok": 1
    },
    "q13": {
     "a": "父親",
     "ok": 1
    },
    "q14": {
     "a": "同じ",
     "ok": 1
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "15文字、2語",
     "ok": 1
    },
    "q18": {
     "a": "3",
     "ok": 1
    },
    "q19": {
     "a": "0.25, 0.3, 0.33, 1/3",
     "ok": 1
    },
    "q20": {
     "a": "存在しない",
     "ok": 1
    }
   },
   "raw": "1: 父親\n2: 10\n3: しない\n4: extraordinarily\n5: 3\n6: 2\n7: できない\n8: 水曜日\n9: o\n10: 火星\n11: できない\n12: すまいざごうよはお\n13: 父親\n14: 同じ\n15: 5\n16: 3\n17: 15文字、2語\n18: 3\n19: 0.25, 0.3, 0.33, 1/3\n20: 存在しない"
  },
  "meta-llama/llama-4-maverick": {
   "score": 6,
   "answered": 20,
   "first10": 4,
   "last10": 2,
   "inconsistent_pairs": 1,
   "sec": 3.3,
   "out_tokens": 123,
   "price": "$0.2 / $0.8",
   "graded": {
    "q01": {
     "a": "母親",
     "ok": 0
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "synchronization",
     "ok": 0
    },
    "q05": {
     "a": "4",
     "ok": 0
    },
    "q06": {
     "a": "2",
     "ok": 1
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "木曜日",
     "ok": 0
    },
    "q09": {
     "a": "c",
     "ok": 0
    },
    "q10": {
     "a": "水星",
     "ok": 0
    },
    "q11": {
     "a": "できる",
     "ok": 0
    },
    "q12": {
     "a": "ますございますようお",
     "ok": 0
    },
    "q13": {
     "a": "母親",
     "ok": 0
    },
    "q14": {
     "a": "変える",
     "ok": 0
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "14, 1",
     "ok": 0
    },
    "q18": {
     "a": "4",
     "ok": 0
    },
    "q19": {
     "a": "0.25, 0.3, 1/3, 0.33",
     "ok": 0
    },
    "q20": {
     "a": "存在する",
     "ok": 0
    }
   },
   "raw": "1: 母親\n2: 10\n3: しない\n4: synchronization\n5: 4\n6: 2\n7: できない\n8: 木曜日\n9: c\n10: 水星\n11: できる\n12: ますございますようお\n13: 母親\n14: 変える\n15: 5\n16: 3\n17: 14, 1\n18: 4\n19: 0.25, 0.3, 1/3, 0.33\n20: 存在する"
  },
  "qwen/qwen3-30b-a3b": {
   "score": 13,
   "answered": 20,
   "first10": 8,
   "last10": 5,
   "inconsistent_pairs": 1,
   "sec": 16.3,
   "out_tokens": 3216,
   "price": "$0.13 / $0.52",
   "graded": {
    "q01": {
     "a": "父",
     "ok": 1
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "2",
     "ok": 0
    },
    "q06": {
     "a": "2",
     "ok": 1
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "水曜日",
     "ok": 1
    },
    "q09": {
     "a": "O",
     "ok": 1
    },
    "q10": {
     "a": "火星",
     "ok": 1
    },
    "q11": {
     "a": "できる",
     "ok": 0
    },
    "q12": {
     "a": "すまいざごうはお",
     "ok": 0
    },
    "q13": {
     "a": "母",
     "ok": 0
    },
    "q14": {
     "a": "同じ",
     "ok": 1
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "16, 1",
     "ok": 0
    },
    "q18": {
     "a": "2",
     "ok": 0
    },
    "q19": {
     "a": "0.25, 0.3, 0.33, 1/3",
     "ok": 1
    },
    "q20": {
     "a": "しない",
     "ok": 1
    }
   },
   "raw": "1. 父  \n2. 10  \n3. しない  \n4. extraordinarily  \n5. 2  \n6. 2  \n7. できない  \n8. 水曜日  \n9. O  \n10. 火星  \n11. できる  \n12. すまいざごうはお  \n13. 母  \n14. 同じ  \n15. 5  \n16. 3  \n17. 16, 1  \n18. 2  \n19. 0.25, 0.3, 0.33, 1/3  \n20. しない"
  },
  "deepseek-reasoner": {
   "score": 19,
   "answered": 20,
   "first10": 9,
   "last10": 10,
   "inconsistent_pairs": 1,
   "sec": 24.5,
   "out_tokens": 4672,
   "price": "$0.55 / $2.19",
   "graded": {
    "q01": {
     "a": "父親",
     "ok": 1
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "3",
     "ok": 1
    },
    "q06": {
     "a": "2",
     "ok": 1
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "水曜日",
     "ok": 1
    },
    "q09": {
     "a": "o",
     "ok": 1
    },
    "q10": {
     "a": "火星",
     "ok": 1
    },
    "q11": {
     "a": "できない",
     "ok": 1
    },
    "q12": {
     "a": "すまいざごうよはお",
     "ok": 1
    },
    "q13": {
     "a": "父",
     "ok": 1
    },
    "q14": {
     "a": "同じ",
     "ok": 1
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "15文字・2つ",
     "ok": 1
    },
    "q18": {
     "a": "3",
     "ok": 1
    },
    "q19": {
     "a": "0.25 < 0.3 < 0.33 < 1/3",
     "ok": 1
    },
    "q20": {
     "a": "しない",
     "ok": 1
    }
   },
   "raw": "1: 父親\n2: 10\n3: しない\n4: extraordinarily\n5: 3\n6: 2\n7: できない\n8: 水曜日\n9: o\n10: 火星\n11: できない\n12: すまいざごうよはお\n13: 父\n14: 同じ\n15: 5\n16: 3\n17: 15文字・2つ\n18: 3\n19: 0.25 < 0.3 < 0.33 < 1/3\n20: しない"
  },
  "google/gemini-3-flash-preview": {
   "score": 18,
   "answered": 20,
   "first10": 9,
   "last10": 9,
   "inconsistent_pairs": 1,
   "sec": 16.1,
   "out_tokens": 2285,
   "price": "$0.15 / $0.6",
   "graded": {
    "q01": {
     "a": "父親",
     "ok": 1
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "3",
     "ok": 1
    },
    "q06": {
     "a": "2",
     "ok": 1
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "水曜日",
     "ok": 1
    },
    "q09": {
     "a": "o",
     "ok": 1
    },
    "q10": {
     "a": "火星",
     "ok": 1
    },
    "q11": {
     "a": "できない",
     "ok": 1
    },
    "q12": {
     "a": "すまいざごうよは",
     "ok": 0
    },
    "q13": {
     "a": "父親",
     "ok": 1
    },
    "q14": {
     "a": "同じ",
     "ok": 1
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "15文字、2つ",
     "ok": 1
    },
    "q18": {
     "a": "3",
     "ok": 1
    },
    "q19": {
     "a": "0.25, 0.3, 0.33, 1/3",
     "ok": 1
    },
    "q20": {
     "a": "しない",
     "ok": 1
    }
   },
   "raw": "1. 父親\n2. 10\n3. しない\n4. extraordinarily\n5. 3\n6. 2\n7. できない\n8. 水曜日\n9. o\n10. 火星\n11. できない\n12. すまいざごうよは\n13. 父親\n14. 同じ\n15. 5\n16. 3\n17. 15文字、2つ\n18. 3\n19. 0.25, 0.3, 0.33, 1/3\n20. しない"
  },
  "qwen/qwen3-coder": {
   "score": 6,
   "answered": 20,
   "first10": 4,
   "last10": 2,
   "inconsistent_pairs": 1,
   "sec": 3.5,
   "out_tokens": 152,
   "price": "$0.3 / $1",
   "graded": {
    "q01": {
     "a": "外科医は母親",
     "ok": 0
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "5",
     "ok": 0
    },
    "q06": {
     "a": "3",
     "ok": 0
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "土曜日",
     "ok": 0
    },
    "q09": {
     "a": "s",
     "ok": 0
    },
    "q10": {
     "a": "火星",
     "ok": 1
    },
    "q11": {
     "a": "できる",
     "ok": 0
    },
    "q12": {
     "a": "すごうよはお",
     "ok": 0
    },
    "q13": {
     "a": "母子",
     "ok": 0
    },
    "q14": {
     "a": "変える",
     "ok": 0
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "15文字、1つ",
     "ok": 0
    },
    "q18": {
     "a": "5",
     "ok": 0
    },
    "q19": {
     "a": "0.25, 1/3, 0.3, 0.33",
     "ok": 0
    },
    "q20": {
     "a": "存在する",
     "ok": 0
    }
   },
   "raw": "1: 外科医は母親  \n2: 10  \n3: しない  \n4: extraordinarily  \n5: 5  \n6: 3  \n7: できない  \n8: 土曜日  \n9: s  \n10: 火星  \n11: できる  \n12: すごうよはお  \n13: 母子  \n14: 変える  \n15: 5  \n16: 3  \n17: 15文字、1つ  \n18: 5  \n19: 0.25, 1/3, 0.3, 0.33  \n20: 存在する"
  },
  "mistralai/mistral-large": {
   "score": 8,
   "answered": 20,
   "first10": 5,
   "last10": 3,
   "inconsistent_pairs": 0,
   "sec": 3.5,
   "out_tokens": 149,
   "price": "$2 / $6",
   "graded": {
    "q01": {
     "a": "母親",
     "ok": 0
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "4",
     "ok": 0
    },
    "q06": {
     "a": "1",
     "ok": 0
    },
    "q07": {
     "a": "できる",
     "ok": 0
    },
    "q08": {
     "a": "水曜日",
     "ok": 1
    },
    "q09": {
     "a": "o",
     "ok": 1
    },
    "q10": {
     "a": "火星",
     "ok": 1
    },
    "q11": {
     "a": "できる",
     "ok": 0
    },
    "q12": {
     "a": "すはまいざおよおは",
     "ok": 0
    },
    "q13": {
     "a": "母子",
     "ok": 0
    },
    "q14": {
     "a": "同じ",
     "ok": 1
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "15文字, 1つ",
     "ok": 0
    },
    "q18": {
     "a": "4",
     "ok": 0
    },
    "q19": {
     "a": "0.25, 0.3, 1/3, 0.33",
     "ok": 0
    },
    "q20": {
     "a": "存在する",
     "ok": 0
    }
   },
   "raw": "1: 母親\n2: 10\n3: しない\n4: extraordinarily\n5: 4\n6: 1\n7: できる\n8: 水曜日\n9: o\n10: 火星\n11: できる\n12: すはまいざおよおは\n13: 母子\n14: 同じ\n15: 5\n16: 3\n17: 15文字, 1つ\n18: 4\n19: 0.25, 0.3, 1/3, 0.33\n20: 存在する"
  },
  "mistralai/mistral-small-3.2-24b-instruct": {
   "score": 7,
   "answered": 20,
   "first10": 5,
   "last10": 2,
   "inconsistent_pairs": 0,
   "sec": 11.8,
   "out_tokens": 150,
   "price": "$0.09375 / $0.25",
   "graded": {
    "q01": {
     "a": "少年の母親",
     "ok": 0
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "synchronization",
     "ok": 0
    },
    "q05": {
     "a": "3",
     "ok": 1
    },
    "q06": {
     "a": "3",
     "ok": 0
    },
    "q07": {
     "a": "できる",
     "ok": 0
    },
    "q08": {
     "a": "水曜日",
     "ok": 1
    },
    "q09": {
     "a": "o",
     "ok": 1
    },
    "q10": {
     "a": "水星",
     "ok": 0
    },
    "q11": {
     "a": "できる",
     "ok": 0
    },
    "q12": {
     "a": "すごうやおは",
     "ok": 0
    },
    "q13": {
     "a": "母と息子",
     "ok": 0
    },
    "q14": {
     "a": "変える",
     "ok": 0
    },
    "q15": {
     "a": "10",
     "ok": 0
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "19文字、1単語",
     "ok": 0
    },
    "q18": {
     "a": "3",
     "ok": 1
    },
    "q19": {
     "a": "0.25, 0.3, 1/3, 0.33",
     "ok": 0
    },
    "q20": {
     "a": "存在する",
     "ok": 0
    }
   },
   "raw": "1: 少年の母親\n2: 10\n3: しない\n4: synchronization\n5: 3\n6: 3\n7: できる\n8: 水曜日\n9: o\n10: 水星\n11: できる\n12: すごうやおは\n13: 母と息子\n14: 変える\n15: 10\n16: 3\n17: 19文字、1単語\n18: 3\n19: 0.25, 0.3, 1/3, 0.33\n20: 存在する"
  },
  "z-ai/glm-5": {
   "score": 12,
   "answered": 20,
   "first10": 8,
   "last10": 4,
   "inconsistent_pairs": 0,
   "sec": 47.6,
   "out_tokens": 141,
   "price": "$0.6 / $1.92",
   "graded": {
    "q01": {
     "a": "外科医は父親。",
     "ok": 1
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "4",
     "ok": 0
    },
    "q06": {
     "a": "2",
     "ok": 1
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "水曜日",
     "ok": 1
    },
    "q09": {
     "a": "o",
     "ok": 1
    },
    "q10": {
     "a": "火星",
     "ok": 1
    },
    "q11": {
     "a": "できる",
     "ok": 0
    },
    "q12": {
     "a": "すまいざごうはお",
     "ok": 0
    },
    "q13": {
     "a": "執刀医は父親。",
     "ok": 1
    },
    "q14": {
     "a": "変える",
     "ok": 0
    },
    "q15": {
     "a": "10",
     "ok": 0
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "14文字で1つ",
     "ok": 0
    },
    "q18": {
     "a": "4",
     "ok": 0
    },
    "q19": {
     "a": "0.25, 0.3, 0.33, 1/3",
     "ok": 1
    },
    "q20": {
     "a": "存在しない",
     "ok": 1
    }
   },
   "raw": "1. 外科医は父親。\n2. 10\n3. しない\n4. extraordinarily\n5. 4\n6. 2\n7. できない\n8. 水曜日\n9. o\n10. 火星\n11. できる\n12. すまいざごうはお\n13. 執刀医は父親。\n14. 変える\n15. 10\n16. 3\n17. 14文字で1つ\n18. 4\n19. 0.25, 0.3, 0.33, 1/3\n20. 存在しない"
  },
  "mistralai/codestral-2508": {
   "score": 7,
   "answered": 20,
   "first10": 4,
   "last10": 3,
   "inconsistent_pairs": 2,
   "sec": 2.8,
   "out_tokens": 151,
   "price": "$0.3 / $0.9",
   "graded": {
    "q01": {
     "a": "外科医は母親",
     "ok": 0
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "する",
     "ok": 0
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "3",
     "ok": 1
    },
    "q06": {
     "a": "3",
     "ok": 0
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "水曜日",
     "ok": 1
    },
    "q09": {
     "a": "d",
     "ok": 0
    },
    "q10": {
     "a": "金星",
     "ok": 0
    },
    "q11": {
     "a": "できる",
     "ok": 0
    },
    "q12": {
     "a": "ですれはごございよう",
     "ok": 0
    },
    "q13": {
     "a": "祖父",
     "ok": 1
    },
    "q14": {
     "a": "変える",
     "ok": 0
    },
    "q15": {
     "a": "10",
     "ok": 0
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "16文字、1つ",
     "ok": 0
    },
    "q18": {
     "a": "3",
     "ok": 1
    },
    "q19": {
     "a": "0.25, 0.3, 1/3, 0.33",
     "ok": 0
    },
    "q20": {
     "a": "存在する",
     "ok": 0
    }
   },
   "raw": "1. 外科医は母親\n2. 10\n3. する\n4. extraordinarily\n5. 3\n6. 3\n7. できない\n8. 水曜日\n9. d\n10. 金星\n11. できる\n12. ですれはごございよう\n13. 祖父\n14. 変える\n15. 10\n16. 3\n17. 16文字、1つ\n18. 3\n19. 0.25, 0.3, 1/3, 0.33\n20. 存在する"
  },
  "cohere/command-a": {
   "score": 5,
   "answered": 20,
   "first10": 3,
   "last10": 2,
   "inconsistent_pairs": 0,
   "sec": 2.7,
   "out_tokens": 142,
   "price": "$2.5 / $10",
   "graded": {
    "q01": {
     "a": "少年は母親",
     "ok": 0
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "4",
     "ok": 0
    },
    "q06": {
     "a": "3",
     "ok": 0
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "金曜",
     "ok": 0
    },
    "q09": {
     "a": "s",
     "ok": 0
    },
    "q10": {
     "a": "水星",
     "ok": 0
    },
    "q11": {
     "a": "できる",
     "ok": 0
    },
    "q12": {
     "a": "すうざごはよは",
     "ok": 0
    },
    "q13": {
     "a": "母親",
     "ok": 0
    },
    "q14": {
     "a": "変える",
     "ok": 0
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "4",
     "ok": 0
    },
    "q17": {
     "a": "13文字、1つ",
     "ok": 0
    },
    "q18": {
     "a": "4",
     "ok": 0
    },
    "q19": {
     "a": "0.25, 0.3, 1/3, 0.33",
     "ok": 0
    },
    "q20": {
     "a": "しない",
     "ok": 1
    }
   },
   "raw": "1: 少年は母親\n2: 10\n3: しない\n4: extraordinarily\n5: 4\n6: 3\n7: できない\n8: 金曜\n9: s\n10: 水星\n11: できる\n12: すうざごはよは\n13: 母親\n14: 変える\n15: 5\n16: 4\n17: 13文字、1つ\n18: 4\n19: 0.25, 0.3, 1/3, 0.33\n20: しない"
  },
  "cohere/command-r-plus-08-2024": {
   "score": 3,
   "answered": 20,
   "first10": 2,
   "last10": 1,
   "inconsistent_pairs": 1,
   "sec": 4.2,
   "out_tokens": 140,
   "price": "$2.5 / $10",
   "graded": {
    "q01": {
     "a": "少年",
     "ok": 0
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "4",
     "ok": 0
    },
    "q06": {
     "a": "0",
     "ok": 0
    },
    "q07": {
     "a": "できる",
     "ok": 0
    },
    "q08": {
     "a": "金曜",
     "ok": 0
    },
    "q09": {
     "a": "s",
     "ok": 0
    },
    "q10": {
     "a": "水星",
     "ok": 0
    },
    "q11": {
     "a": "できる",
     "ok": 0
    },
    "q12": {
     "a": "すざぎぞうあはお",
     "ok": 0
    },
    "q13": {
     "a": "父親",
     "ok": 1
    },
    "q14": {
     "a": "変える",
     "ok": 0
    },
    "q15": {
     "a": "20",
     "ok": 0
    },
    "q16": {
     "a": "4",
     "ok": 0
    },
    "q17": {
     "a": "15、1",
     "ok": 0
    },
    "q18": {
     "a": "4",
     "ok": 0
    },
    "q19": {
     "a": "1/3, 0.25, 0.3, 0.33",
     "ok": 0
    },
    "q20": {
     "a": "存在する",
     "ok": 0
    }
   },
   "raw": "1: 少年\n2: 10\n3: しない\n4: extraordinarily\n5: 4\n6: 0\n7: できる\n8: 金曜\n9: s\n10: 水星\n11: できる\n12: すざぎぞうあはお\n13: 父親\n14: 変える\n15: 20\n16: 4\n17: 15、1\n18: 4\n19: 1/3, 0.25, 0.3, 0.33\n20: 存在する"
  },
  "cohere/command-r-08-2024": {
   "score": 5,
   "answered": 20,
   "first10": 2,
   "last10": 3,
   "inconsistent_pairs": 0,
   "sec": 4.2,
   "out_tokens": 138,
   "price": "$0.15 / $0.6",
   "graded": {
    "q01": {
     "a": "父親",
     "ok": 1
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "する",
     "ok": 0
    },
    "q04": {
     "a": "synchronization",
     "ok": 0
    },
    "q05": {
     "a": "2",
     "ok": 0
    },
    "q06": {
     "a": "0",
     "ok": 0
    },
    "q07": {
     "a": "できる",
     "ok": 0
    },
    "q08": {
     "a": "金曜日",
     "ok": 0
    },
    "q09": {
     "a": "d",
     "ok": 0
    },
    "q10": {
     "a": "水星",
     "ok": 0
    },
    "q11": {
     "a": "できる",
     "ok": 0
    },
    "q12": {
     "a": "ますぞよあおは",
     "ok": 0
    },
    "q13": {
     "a": "父親",
     "ok": 1
    },
    "q14": {
     "a": "同じ",
     "ok": 1
    },
    "q15": {
     "a": "10",
     "ok": 0
    },
    "q16": {
     "a": "4",
     "ok": 0
    },
    "q17": {
     "a": "15, 1",
     "ok": 0
    },
    "q18": {
     "a": "2",
     "ok": 0
    },
    "q19": {
     "a": "0.25, 0.3, 0.33, 1/3",
     "ok": 1
    },
    "q20": {
     "a": "存在する",
     "ok": 0
    }
   },
   "raw": "1. 父親\n2. 10\n3. する\n4. synchronization\n5. 2\n6. 0\n7. できる\n8. 金曜日\n9. d\n10. 水星\n11. できる\n12. ますぞよあおは\n13. 父親\n14. 同じ\n15. 10\n16. 4\n17. 15, 1\n18. 2\n19. 0.25, 0.3, 0.33, 1/3\n20. 存在する"
  },
  "google/gemma-3-27b-it": {
   "score": 4,
   "answered": 20,
   "first10": 4,
   "last10": 0,
   "inconsistent_pairs": 1,
   "sec": 8.2,
   "out_tokens": 139,
   "price": "$0.08 / $0.45",
   "graded": {
    "q01": {
     "a": "母親",
     "ok": 0
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "4",
     "ok": 0
    },
    "q06": {
     "a": "2",
     "ok": 1
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "金曜",
     "ok": 0
    },
    "q09": {
     "a": "c",
     "ok": 0
    },
    "q10": {
     "a": "水星",
     "ok": 0
    },
    "q11": {
     "a": "できる",
     "ok": 0
    },
    "q12": {
     "a": "すまいますうはょうお",
     "ok": 0
    },
    "q13": {
     "a": "母親",
     "ok": 0
    },
    "q14": {
     "a": "変える",
     "ok": 0
    },
    "q15": {
     "a": "10",
     "ok": 0
    },
    "q16": {
     "a": "6",
     "ok": 0
    },
    "q17": {
     "a": "17, 1",
     "ok": 0
    },
    "q18": {
     "a": "4",
     "ok": 0
    },
    "q19": {
     "a": "0.25, 0.3, 1/3, 0.33",
     "ok": 0
    },
    "q20": {
     "a": "存在する",
     "ok": 0
    }
   },
   "raw": "1: 母親\n2: 10\n3: しない\n4: extraordinarily\n5: 4\n6: 2\n7: できない\n8: 金曜\n9: c\n10: 水星\n11: できる\n12: すまいますうはょうお\n13: 母親\n14: 変える\n15: 10\n16: 6\n17: 17, 1\n18: 4\n19: 0.25, 0.3, 1/3, 0.33\n20: 存在する"
  },
  "moonshotai/kimi-k2.5": {
   "score": 19,
   "answered": 20,
   "first10": 9,
   "last10": 10,
   "inconsistent_pairs": 1,
   "sec": 20.3,
   "out_tokens": 4490,
   "price": "$0.45 / $2.25",
   "graded": {
    "q01": {
     "a": "父親",
     "ok": 1
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "3",
     "ok": 1
    },
    "q06": {
     "a": "2",
     "ok": 1
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "水曜日",
     "ok": 1
    },
    "q09": {
     "a": "o",
     "ok": 1
    },
    "q10": {
     "a": "火星",
     "ok": 1
    },
    "q11": {
     "a": "できない",
     "ok": 1
    },
    "q12": {
     "a": "すまいざごうよはお",
     "ok": 1
    },
    "q13": {
     "a": "父親",
     "ok": 1
    },
    "q14": {
     "a": "同じ",
     "ok": 1
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "15文字、2個",
     "ok": 1
    },
    "q18": {
     "a": "3",
     "ok": 1
    },
    "q19": {
     "a": "0.25, 0.3, 0.33, 1/3",
     "ok": 1
    },
    "q20": {
     "a": "しない",
     "ok": 1
    }
   },
   "raw": "1: 父親\n2: 10\n3: しない\n4: extraordinarily\n5: 3\n6: 2\n7: できない\n8: 水曜日\n9: o\n10: 火星\n11: できない\n12: すまいざごうよはお\n13: 父親\n14: 同じ\n15: 5\n16: 3\n17: 15文字、2個\n18: 3\n19: 0.25, 0.3, 0.33, 1/3\n20: しない"
  },
  "microsoft/phi-4": {
   "score": 9,
   "answered": 20,
   "first10": 6,
   "last10": 3,
   "inconsistent_pairs": 1,
   "sec": 4.6,
   "out_tokens": 249,
   "price": "$0.07 / $0.14",
   "graded": {
    "q01": {
     "a": "番号: 母親",
     "ok": 0
    },
    "q02": {
     "a": "番号: 10",
     "ok": 1
    },
    "q03": {
     "a": "番号: しない",
     "ok": 1
    },
    "q04": {
     "a": "番号: extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "番号: 3",
     "ok": 1
    },
    "q06": {
     "a": "番号: 2",
     "ok": 1
    },
    "q07": {
     "a": "番号: できない",
     "ok": 1
    },
    "q08": {
     "a": "番号: 木曜日",
     "ok": 0
    },
    "q09": {
     "a": "番号: o",
     "ok": 1
    },
    "q10": {
     "a": "番号: マーキュリー",
     "ok": 0
    },
    "q11": {
     "a": "番号: できる",
     "ok": 0
    },
    "q12": {
     "a": "番号: suomuogah yadot",
     "ok": 0
    },
    "q13": {
     "a": "番号: 母親",
     "ok": 0
    },
    "q14": {
     "a": "番号: 変える",
     "ok": 0
    },
    "q15": {
     "a": "番号: 10",
     "ok": 0
    },
    "q16": {
     "a": "番号: 3",
     "ok": 1
    },
    "q17": {
     "a": "番号: 18, 1",
     "ok": 0
    },
    "q18": {
     "a": "番号: 3",
     "ok": 1
    },
    "q19": {
     "a": "番号: 0.25, 0.3, 0.33, 1/3",
     "ok": 1
    },
    "q20": {
     "a": "番号: 存在する",
     "ok": 0
    }
   },
   "raw": "1. 番号: 母親\n2. 番号: 10\n3. 番号: しない\n4. 番号: extraordinarily\n5. 番号: 3\n6. 番号: 2\n7. 番号: できない\n8. 番号: 木曜日\n9. 番号: o\n10. 番号: マーキュリー\n11. 番号: できる\n12. 番号: suomuogah yadot\n13. 番号: 母親\n14. 番号: 変える\n15. 番号: 10\n16. 番号: 3\n17. 番号: 18, 1\n18. 番号: 3\n19. 番号: 0.25, 0.3, 0.33, 1/3\n20. 番号: 存在する"
  },
  "openai/gpt-5.6-luna": {
   "score": 19,
   "answered": 20,
   "first10": 9,
   "last10": 10,
   "inconsistent_pairs": 1,
   "sec": 14.1,
   "out_tokens": 1131,
   "price": "$0.5 / $3",
   "graded": {
    "q01": {
     "a": "父親",
     "ok": 1
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "3",
     "ok": 1
    },
    "q06": {
     "a": "2",
     "ok": 1
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "水曜日",
     "ok": 1
    },
    "q09": {
     "a": "o",
     "ok": 1
    },
    "q10": {
     "a": "火星",
     "ok": 1
    },
    "q11": {
     "a": "できない",
     "ok": 1
    },
    "q12": {
     "a": "すまいざごうよはお",
     "ok": 1
    },
    "q13": {
     "a": "父親",
     "ok": 1
    },
    "q14": {
     "a": "同じ",
     "ok": 1
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "15文字、2つ",
     "ok": 1
    },
    "q18": {
     "a": "3",
     "ok": 1
    },
    "q19": {
     "a": "0.25、0.3、0.33、1/3",
     "ok": 1
    },
    "q20": {
     "a": "存在しない",
     "ok": 1
    }
   },
   "raw": "1: 父親\n2: 10\n3: しない\n4: extraordinarily\n5: 3\n6: 2\n7: できない\n8: 水曜日\n9: o\n10: 火星\n11: できない\n12: すまいざごうよはお\n13: 父親\n14: 同じ\n15: 5\n16: 3\n17: 15文字、2つ\n18: 3\n19: 0.25、0.3、0.33、1/3\n20: 存在しない"
  },
  "openai/gpt-5.4": {
   "score": 13,
   "answered": 20,
   "first10": 6,
   "last10": 7,
   "inconsistent_pairs": 1,
   "sec": 3.7,
   "out_tokens": 135,
   "price": "$2.5 / $15",
   "graded": {
    "q01": {
     "a": "父親",
     "ok": 1
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "4",
     "ok": 0
    },
    "q06": {
     "a": "3",
     "ok": 0
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "水曜",
     "ok": 1
    },
    "q09": {
     "a": "d",
     "ok": 0
    },
    "q10": {
     "a": "火星",
     "ok": 1
    },
    "q11": {
     "a": "できない",
     "ok": 1
    },
    "q12": {
     "a": "すまいざごうよはお",
     "ok": 1
    },
    "q13": {
     "a": "親子",
     "ok": 0
    },
    "q14": {
     "a": "同じ",
     "ok": 1
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "15文字、1つ",
     "ok": 0
    },
    "q18": {
     "a": "4",
     "ok": 0
    },
    "q19": {
     "a": "0.25, 0.3, 0.33, 1/3",
     "ok": 1
    },
    "q20": {
     "a": "しない",
     "ok": 1
    }
   },
   "raw": "1: 父親\n2: 10\n3: しない\n4: extraordinarily\n5: 4\n6: 3\n7: できない\n8: 水曜\n9: d\n10: 火星\n11: できない\n12: すまいざごうよはお\n13: 親子\n14: 同じ\n15: 5\n16: 3\n17: 15文字、1つ\n18: 4\n19: 0.25, 0.3, 0.33, 1/3\n20: しない"
  },
  "openai/gpt-5.4-mini": {
   "score": 7,
   "answered": 20,
   "first10": 3,
   "last10": 4,
   "inconsistent_pairs": 1,
   "sec": 1.9,
   "out_tokens": 140,
   "price": "$0.75 / $4.5",
   "graded": {
    "q01": {
     "a": "外科医は母親",
     "ok": 0
    },
    "q02": {
     "a": "10円",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "4",
     "ok": 0
    },
    "q06": {
     "a": "3",
     "ok": 0
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "木曜",
     "ok": 0
    },
    "q09": {
     "a": "s",
     "ok": 0
    },
    "q10": {
     "a": "水星",
     "ok": 0
    },
    "q11": {
     "a": "できる",
     "ok": 0
    },
    "q12": {
     "a": "すでまうごうよはお",
     "ok": 0
    },
    "q13": {
     "a": "母親",
     "ok": 0
    },
    "q14": {
     "a": "変える",
     "ok": 0
    },
    "q15": {
     "a": "20",
     "ok": 0
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "15文字、2つ",
     "ok": 1
    },
    "q18": {
     "a": "4",
     "ok": 0
    },
    "q19": {
     "a": "0.25, 0.3, 0.33, 1/3",
     "ok": 1
    },
    "q20": {
     "a": "しない",
     "ok": 1
    }
   },
   "raw": "1: 外科医は母親  \n2: 10円  \n3: しない  \n4: extraordinarily  \n5: 4  \n6: 3  \n7: できない  \n8: 木曜  \n9: s  \n10: 水星  \n11: できる  \n12: すでまうごうよはお  \n13: 母親  \n14: 変える  \n15: 20  \n16: 3  \n17: 15文字、2つ  \n18: 4  \n19: 0.25, 0.3, 0.33, 1/3  \n20: しない"
  },
  "openai/gpt-5.6-sol": {
   "score": 19,
   "answered": 20,
   "first10": 9,
   "last10": 10,
   "inconsistent_pairs": 1,
   "sec": 31.7,
   "out_tokens": 860,
   "price": "",
   "graded": {
    "q01": {
     "a": "父親",
     "ok": 1
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "3",
     "ok": 1
    },
    "q06": {
     "a": "2",
     "ok": 1
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "水曜日",
     "ok": 1
    },
    "q09": {
     "a": "o",
     "ok": 1
    },
    "q10": {
     "a": "火星",
     "ok": 1
    },
    "q11": {
     "a": "できない",
     "ok": 1
    },
    "q12": {
     "a": "すまいざごうよはお",
     "ok": 1
    },
    "q13": {
     "a": "父親と息子",
     "ok": 1
    },
    "q14": {
     "a": "同じ",
     "ok": 1
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "15文字、2語",
     "ok": 1
    },
    "q18": {
     "a": "3",
     "ok": 1
    },
    "q19": {
     "a": "0.25, 0.3, 0.33, 1/3",
     "ok": 1
    },
    "q20": {
     "a": "存在しない",
     "ok": 1
    }
   },
   "raw": "1: 父親\n2: 10\n3: しない\n4: extraordinarily\n5: 3\n6: 2\n7: できない\n8: 水曜日\n9: o\n10: 火星\n11: できない\n12: すまいざごうよはお\n13: 父親と息子\n14: 同じ\n15: 5\n16: 3\n17: 15文字、2語\n18: 3\n19: 0.25, 0.3, 0.33, 1/3\n20: 存在しない"
  },
  "openai/gpt-5.4-nano": {
   "score": 8,
   "answered": 20,
   "first10": 4,
   "last10": 4,
   "inconsistent_pairs": 3,
   "sec": 2.6,
   "out_tokens": 140,
   "price": "$0.2 / $1.25",
   "graded": {
    "q01": {
     "a": "外科医の自分の息子",
     "ok": 0
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "する",
     "ok": 0
    },
    "q04": {
     "a": "synchronization",
     "ok": 0
    },
    "q05": {
     "a": "3",
     "ok": 1
    },
    "q06": {
     "a": "3",
     "ok": 0
    },
    "q07": {
     "a": "できる",
     "ok": 0
    },
    "q08": {
     "a": "木曜",
     "ok": 0
    },
    "q09": {
     "a": "o",
     "ok": 1
    },
    "q10": {
     "a": "火星",
     "ok": 1
    },
    "q11": {
     "a": "できない",
     "ok": 1
    },
    "q12": {
     "a": "いすようははお",
     "ok": 0
    },
    "q13": {
     "a": "父と息子",
     "ok": 1
    },
    "q14": {
     "a": "変える",
     "ok": 0
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "2",
     "ok": 0
    },
    "q17": {
     "a": "13文字、1語",
     "ok": 0
    },
    "q18": {
     "a": "4",
     "ok": 0
    },
    "q19": {
     "a": "0.25, 0.3, 1/3, 0.33",
     "ok": 0
    },
    "q20": {
     "a": "しない",
     "ok": 1
    }
   },
   "raw": "1: 外科医の自分の息子  \n2: 10  \n3: する  \n4: synchronization  \n5: 3  \n6: 3  \n7: できる  \n8: 木曜  \n9: o  \n10: 火星  \n11: できない  \n12: いすようははお  \n13: 父と息子  \n14: 変える  \n15: 5  \n16: 2  \n17: 13文字、1語  \n18: 4  \n19: 0.25, 0.3, 1/3, 0.33  \n20: しない"
  },
  "google/gemini-3.1-flash-lite": {
   "score": 15,
   "answered": 20,
   "first10": 8,
   "last10": 7,
   "inconsistent_pairs": 0,
   "sec": 5.2,
   "out_tokens": 1184,
   "price": "$0.125 / $0.75",
   "graded": {
    "q01": {
     "a": "父親",
     "ok": 1
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "4",
     "ok": 0
    },
    "q06": {
     "a": "2",
     "ok": 1
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "水曜日",
     "ok": 1
    },
    "q09": {
     "a": "o",
     "ok": 1
    },
    "q10": {
     "a": "火星",
     "ok": 1
    },
    "q11": {
     "a": "できない",
     "ok": 1
    },
    "q12": {
     "a": "すまざいさうはよお",
     "ok": 0
    },
    "q13": {
     "a": "父",
     "ok": 1
    },
    "q14": {
     "a": "同じ",
     "ok": 1
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "15文字、1つ",
     "ok": 0
    },
    "q18": {
     "a": "4",
     "ok": 0
    },
    "q19": {
     "a": "0.25, 0.3, 0.33, 1/3",
     "ok": 1
    },
    "q20": {
     "a": "しない",
     "ok": 1
    }
   },
   "raw": "1: 父親\n2: 10\n3: しない\n4: extraordinarily\n5: 4\n6: 2\n7: できない\n8: 水曜日\n9: o\n10: 火星\n11: できない\n12: すまざいさうはよお\n13: 父\n14: 同じ\n15: 5\n16: 3\n17: 15文字、1つ\n18: 4\n19: 0.25, 0.3, 0.33, 1/3\n20: しない"
  },
  "openai/gpt-5.6-luna-pro": {
   "score": 16,
   "answered": 20,
   "first10": 7,
   "last10": 9,
   "inconsistent_pairs": 1,
   "sec": 25.6,
   "out_tokens": 4128,
   "price": "$0.5 / $3",
   "graded": {
    "q01": {
     "a": "母親",
     "ok": 0
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "3",
     "ok": 1
    },
    "q06": {
     "a": "2",
     "ok": 1
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "火曜日",
     "ok": 0
    },
    "q09": {
     "a": "o",
     "ok": 1
    },
    "q10": {
     "a": "火星",
     "ok": 1
    },
    "q11": {
     "a": "できない",
     "ok": 1
    },
    "q12": {
     "a": "すまいざごうよはお",
     "ok": 1
    },
    "q13": {
     "a": "母親",
     "ok": 0
    },
    "q14": {
     "a": "同じ",
     "ok": 1
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "15文字、2つ",
     "ok": 1
    },
    "q18": {
     "a": "3",
     "ok": 1
    },
    "q19": {
     "a": "0.25, 0.3, 0.33, 1/3",
     "ok": 1
    },
    "q20": {
     "a": "存在しない",
     "ok": 1
    }
   },
   "raw": "1: 母親  \n2: 10  \n3: しない  \n4: extraordinarily  \n5: 3  \n6: 2  \n7: できない  \n8: 火曜日  \n9: o  \n10: 火星  \n11: できない  \n12: すまいざごうよはお  \n13: 母親  \n14: 同じ  \n15: 5  \n16: 3  \n17: 15文字、2つ  \n18: 3  \n19: 0.25, 0.3, 0.33, 1/3  \n20: 存在しない"
  },
  "google/gemini-3.5-flash": {
   "score": 17,
   "answered": 20,
   "first10": 9,
   "last10": 8,
   "inconsistent_pairs": 2,
   "sec": 12.6,
   "out_tokens": 2141,
   "price": "$0.75 / $4.5",
   "graded": {
    "q01": {
     "a": "父親",
     "ok": 1
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "3",
     "ok": 1
    },
    "q06": {
     "a": "2",
     "ok": 1
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "水曜日",
     "ok": 1
    },
    "q09": {
     "a": "o",
     "ok": 1
    },
    "q10": {
     "a": "火星",
     "ok": 1
    },
    "q11": {
     "a": "できない",
     "ok": 1
    },
    "q12": {
     "a": "すまいごとうよはお",
     "ok": 0
    },
    "q13": {
     "a": "親子",
     "ok": 0
    },
    "q14": {
     "a": "同じ",
     "ok": 1
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "15文字、2つ",
     "ok": 1
    },
    "q18": {
     "a": "3",
     "ok": 1
    },
    "q19": {
     "a": "0.25, 0.3, 0.33, 1/3",
     "ok": 1
    },
    "q20": {
     "a": "しない",
     "ok": 1
    }
   },
   "raw": "1: 父親\n2: 10\n3: しない\n4: extraordinarily\n5: 3\n6: 2\n7: できない\n8: 水曜日\n9: o\n10: 火星\n11: できない\n12: すまいごとうよはお\n13: 親子\n14: 同じ\n15: 5\n16: 3\n17: 15文字、2つ\n18: 3\n19: 0.25, 0.3, 0.33, 1/3\n20: しない"
  },
  "google/gemini-3.1-pro-preview": {
   "score": 19,
   "answered": 20,
   "first10": 9,
   "last10": 10,
   "inconsistent_pairs": 1,
   "sec": 11.9,
   "out_tokens": 1469,
   "price": "",
   "graded": {
    "q01": {
     "a": "父親",
     "ok": 1
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "3",
     "ok": 1
    },
    "q06": {
     "a": "2",
     "ok": 1
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "水曜日",
     "ok": 1
    },
    "q09": {
     "a": "o",
     "ok": 1
    },
    "q10": {
     "a": "火星",
     "ok": 1
    },
    "q11": {
     "a": "できない",
     "ok": 1
    },
    "q12": {
     "a": "すまいざごうよはお",
     "ok": 1
    },
    "q13": {
     "a": "父親",
     "ok": 1
    },
    "q14": {
     "a": "同じ",
     "ok": 1
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "15文字、2つ",
     "ok": 1
    },
    "q18": {
     "a": "3",
     "ok": 1
    },
    "q19": {
     "a": "0.25, 0.3, 0.33, 1/3",
     "ok": 1
    },
    "q20": {
     "a": "しない",
     "ok": 1
    }
   },
   "raw": "1: 父親\n2: 10\n3: しない\n4: extraordinarily\n5: 3\n6: 2\n7: できない\n8: 水曜日\n9: o\n10: 火星\n11: できない\n12: すまいざごうよはお\n13: 父親\n14: 同じ\n15: 5\n16: 3\n17: 15文字、2つ\n18: 3\n19: 0.25, 0.3, 0.33, 1/3\n20: しない"
  },
  "deepseek/deepseek-v4-pro": {
   "score": 18,
   "answered": 20,
   "first10": 9,
   "last10": 9,
   "inconsistent_pairs": 0,
   "sec": 90.6,
   "out_tokens": 2071,
   "price": "$0.66 / $1.98",
   "graded": {
    "q01": {
     "a": "父親",
     "ok": 1
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "synchronization",
     "ok": 0
    },
    "q05": {
     "a": "3",
     "ok": 1
    },
    "q06": {
     "a": "2",
     "ok": 1
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "水曜日",
     "ok": 1
    },
    "q09": {
     "a": "o",
     "ok": 1
    },
    "q10": {
     "a": "火星",
     "ok": 1
    },
    "q11": {
     "a": "できない",
     "ok": 1
    },
    "q12": {
     "a": "すまいざごうよはお",
     "ok": 1
    },
    "q13": {
     "a": "父親",
     "ok": 1
    },
    "q14": {
     "a": "同じ",
     "ok": 1
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "16, 1",
     "ok": 0
    },
    "q18": {
     "a": "3",
     "ok": 1
    },
    "q19": {
     "a": "0.25, 0.3, 0.33, 1/3",
     "ok": 1
    },
    "q20": {
     "a": "存在しない",
     "ok": 1
    }
   },
   "raw": "1: 父親\n2: 10\n3: しない\n4: synchronization\n5: 3\n6: 2\n7: できない\n8: 水曜日\n9: o\n10: 火星\n11: できない\n12: すまいざごうよはお\n13: 父親\n14: 同じ\n15: 5\n16: 3\n17: 16, 1\n18: 3\n19: 0.25, 0.3, 0.33, 1/3\n20: 存在しない"
  },
  "z-ai/glm-5.3": {
   "score": 9,
   "answered": 20,
   "first10": 6,
   "last10": 3,
   "inconsistent_pairs": 0,
   "sec": 3.9,
   "out_tokens": 541,
   "price": "",
   "graded": {
    "q01": {
     "a": "母親(女性外科医)",
     "ok": 0
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "synchronization",
     "ok": 0
    },
    "q05": {
     "a": "4",
     "ok": 0
    },
    "q06": {
     "a": "3",
     "ok": 0
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "水曜日",
     "ok": 1
    },
    "q09": {
     "a": "o",
     "ok": 1
    },
    "q10": {
     "a": "火星",
     "ok": 1
    },
    "q11": {
     "a": "できる",
     "ok": 0
    },
    "q12": {
     "a": "すまいごうはよお",
     "ok": 0
    },
    "q13": {
     "a": "母親(女性医師)",
     "ok": 0
    },
    "q14": {
     "a": "同じ",
     "ok": 1
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "4",
     "ok": 0
    },
    "q17": {
     "a": "16文字、1つ(extraordinarily)",
     "ok": 0
    },
    "q18": {
     "a": "4",
     "ok": 0
    },
    "q19": {
     "a": "0.25 < 0.3 < 1/3 < 0.33",
     "ok": 0
    },
    "q20": {
     "a": "しない",
     "ok": 1
    }
   },
   "raw": "1: 母親(女性外科医)\n2: 10\n3: しない\n4: synchronization\n5: 4\n6: 3\n7: できない\n8: 水曜日\n9: o\n10: 火星\n11: できる\n12: すまいごうはよお\n13: 母親(女性医師)\n14: 同じ\n15: 5\n16: 4\n17: 16文字、1つ(extraordinarily)\n18: 4\n19: 0.25 < 0.3 < 1/3 < 0.33\n20: しない"
  },
  "deepseek/deepseek-v4-flash-0731": {
   "score": 18,
   "answered": 20,
   "first10": 9,
   "last10": 9,
   "inconsistent_pairs": 1,
   "sec": 35.1,
   "out_tokens": 4447,
   "price": "$0.09 / $0.18",
   "graded": {
    "q01": {
     "a": "父親",
     "ok": 1
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "3",
     "ok": 1
    },
    "q06": {
     "a": "2",
     "ok": 1
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "水曜日",
     "ok": 1
    },
    "q09": {
     "a": "o",
     "ok": 1
    },
    "q10": {
     "a": "火星",
     "ok": 1
    },
    "q11": {
     "a": "できる",
     "ok": 0
    },
    "q12": {
     "a": "すまいざごうよはお",
     "ok": 1
    },
    "q13": {
     "a": "父と息子",
     "ok": 1
    },
    "q14": {
     "a": "同じ",
     "ok": 1
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "15文字、2つ",
     "ok": 1
    },
    "q18": {
     "a": "3",
     "ok": 1
    },
    "q19": {
     "a": "0.25, 0.3, 0.33, 1/3",
     "ok": 1
    },
    "q20": {
     "a": "しない",
     "ok": 1
    }
   },
   "raw": "1: 父親\n2: 10\n3: しない\n4: extraordinarily\n5: 3\n6: 2\n7: できない\n8: 水曜日\n9: o\n10: 火星\n11: できる\n12: すまいざごうよはお\n13: 父と息子\n14: 同じ\n15: 5\n16: 3\n17: 15文字、2つ\n18: 3\n19: 0.25, 0.3, 0.33, 1/3\n20: しない"
  },
  "deepseek/deepseek-v4-pro-cache": {
   "score": 12,
   "answered": 20,
   "first10": 5,
   "last10": 7,
   "inconsistent_pairs": 1,
   "sec": 63.0,
   "out_tokens": 1045,
   "price": "$0.022 / $1.98",
   "graded": {
    "q01": {
     "a": "母親",
     "ok": 0
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "4",
     "ok": 0
    },
    "q06": {
     "a": "3",
     "ok": 0
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "水曜日",
     "ok": 1
    },
    "q09": {
     "a": "d",
     "ok": 0
    },
    "q10": {
     "a": "火星",
     "ok": 1
    },
    "q11": {
     "a": "できる",
     "ok": 0
    },
    "q12": {
     "a": "すまいざごうよはお",
     "ok": 1
    },
    "q13": {
     "a": "母子",
     "ok": 0
    },
    "q14": {
     "a": "同じ",
     "ok": 1
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "15文字、2つ",
     "ok": 1
    },
    "q18": {
     "a": "4",
     "ok": 0
    },
    "q19": {
     "a": "0.25, 0.3, 0.33, 1/3",
     "ok": 1
    },
    "q20": {
     "a": "存在しない",
     "ok": 1
    }
   },
   "raw": "1: 母親  \n2: 10  \n3: しない  \n4: extraordinarily  \n5: 4  \n6: 3  \n7: できない  \n8: 水曜日  \n9: d  \n10: 火星  \n11: できる  \n12: すまいざごうよはお  \n13: 母子  \n14: 同じ  \n15: 5  \n16: 3  \n17: 15文字、2つ  \n18: 4  \n19: 0.25, 0.3, 0.33, 1/3  \n20: 存在しない"
  },
  "x-ai/grok-4.3": {
   "score": 12,
   "answered": 20,
   "first10": 6,
   "last10": 6,
   "inconsistent_pairs": 2,
   "sec": 16.6,
   "out_tokens": 1759,
   "price": "$1.25 / $2.5",
   "graded": {
    "q01": {
     "a": "母親",
     "ok": 0
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "4",
     "ok": 0
    },
    "q06": {
     "a": "2",
     "ok": 1
    },
    "q07": {
     "a": "できる",
     "ok": 0
    },
    "q08": {
     "a": "水曜日",
     "ok": 1
    },
    "q09": {
     "a": "o",
     "ok": 1
    },
    "q10": {
     "a": "火星",
     "ok": 1
    },
    "q11": {
     "a": "できる",
     "ok": 0
    },
    "q12": {
     "a": "すませいごうよはお",
     "ok": 0
    },
    "q13": {
     "a": "父親",
     "ok": 1
    },
    "q14": {
     "a": "同じ",
     "ok": 1
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "16文字で1つ",
     "ok": 0
    },
    "q18": {
     "a": "4",
     "ok": 0
    },
    "q19": {
     "a": "0.25, 0.3, 0.33, 1/3",
     "ok": 1
    },
    "q20": {
     "a": "しない",
     "ok": 1
    }
   },
   "raw": "1: 母親\n2: 10\n3: しない\n4: extraordinarily\n5: 4\n6: 2\n7: できる\n8: 水曜日\n9: o\n10: 火星\n11: できる\n12: すませいごうよはお\n13: 父親\n14: 同じ\n15: 5\n16: 3\n17: 16文字で1つ\n18: 4\n19: 0.25, 0.3, 0.33, 1/3\n20: しない"
  },
  "deepseek/deepseek-v4-pro-0813": {
   "score": 10,
   "answered": 20,
   "first10": 6,
   "last10": 4,
   "inconsistent_pairs": 0,
   "sec": 5.0,
   "out_tokens": 336,
   "price": "",
   "graded": {
    "q01": {
     "a": "母親",
     "ok": 0
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "4",
     "ok": 0
    },
    "q06": {
     "a": "3",
     "ok": 0
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "水曜日",
     "ok": 1
    },
    "q09": {
     "a": "o",
     "ok": 1
    },
    "q10": {
     "a": "火星",
     "ok": 1
    },
    "q11": {
     "a": "できる",
     "ok": 0
    },
    "q12": {
     "a": "す ま い ご ざ よ う は お",
     "ok": 0
    },
    "q13": {
     "a": "親子",
     "ok": 0
    },
    "q14": {
     "a": "同じ",
     "ok": 1
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "2",
     "ok": 0
    },
    "q17": {
     "a": "15文字、1つ",
     "ok": 0
    },
    "q18": {
     "a": "4",
     "ok": 0
    },
    "q19": {
     "a": "0.25, 0.3, 0.33, 1/3",
     "ok": 1
    },
    "q20": {
     "a": "しない",
     "ok": 1
    }
   },
   "raw": "1: 母親\n2: 10\n3: しない\n4: extraordinarily\n5: 4\n6: 3\n7: できない\n8: 水曜日\n9: o\n10: 火星\n11: できる\n12: す ま い ご ざ よ う は お\n13: 親子\n14: 同じ\n15: 5\n16: 2\n17: 15文字、1つ\n18: 4\n19: 0.25, 0.3, 0.33, 1/3\n20: しない"
  },
  "moonshotai/kimi-k3": {
   "score": 18,
   "answered": 20,
   "first10": 8,
   "last10": 10,
   "inconsistent_pairs": 1,
   "sec": 6.0,
   "out_tokens": 607,
   "price": "",
   "graded": {
    "q01": {
     "a": "父親",
     "ok": 1
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "3",
     "ok": 1
    },
    "q06": {
     "a": "3",
     "ok": 0
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "水曜日",
     "ok": 1
    },
    "q09": {
     "a": "o",
     "ok": 1
    },
    "q10": {
     "a": "火星",
     "ok": 1
    },
    "q11": {
     "a": "できない",
     "ok": 1
    },
    "q12": {
     "a": "すまいざごうよはお",
     "ok": 1
    },
    "q13": {
     "a": "父親",
     "ok": 1
    },
    "q14": {
     "a": "同じ",
     "ok": 1
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "15文字・2語",
     "ok": 1
    },
    "q18": {
     "a": "3",
     "ok": 1
    },
    "q19": {
     "a": "0.25, 0.3, 0.33, 1/3",
     "ok": 1
    },
    "q20": {
     "a": "しない",
     "ok": 1
    }
   },
   "raw": "1: 父親  \n2: 10  \n3: しない  \n4: extraordinarily  \n5: 3  \n6: 3  \n7: できない  \n8: 水曜日  \n9: o  \n10: 火星  \n11: できない  \n12: すまいざごうよはお  \n13: 父親  \n14: 同じ  \n15: 5  \n16: 3  \n17: 15文字・2語  \n18: 3  \n19: 0.25, 0.3, 0.33, 1/3  \n20: しない"
  },
  "deepseek/deepseek-v4-flash": {
   "score": 19,
   "answered": 20,
   "first10": 9,
   "last10": 10,
   "inconsistent_pairs": 1,
   "sec": 52.3,
   "out_tokens": 4826,
   "price": "$0.22 / $0.66",
   "graded": {
    "q01": {
     "a": "父親",
     "ok": 1
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "3",
     "ok": 1
    },
    "q06": {
     "a": "2",
     "ok": 1
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "水曜日",
     "ok": 1
    },
    "q09": {
     "a": "o",
     "ok": 1
    },
    "q10": {
     "a": "火星",
     "ok": 1
    },
    "q11": {
     "a": "できない",
     "ok": 1
    },
    "q12": {
     "a": "すまいざごうよはお",
     "ok": 1
    },
    "q13": {
     "a": "父",
     "ok": 1
    },
    "q14": {
     "a": "同じ",
     "ok": 1
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "15文字、2つ",
     "ok": 1
    },
    "q18": {
     "a": "3",
     "ok": 1
    },
    "q19": {
     "a": "0.25, 0.3, 0.33, 1/3",
     "ok": 1
    },
    "q20": {
     "a": "存在しない",
     "ok": 1
    }
   },
   "raw": "1: 父親\n2: 10\n3: しない\n4: extraordinarily\n5: 3\n6: 2\n7: できない\n8: 水曜日\n9: o\n10: 火星\n11: できない\n12: すまいざごうよはお\n13: 父\n14: 同じ\n15: 5\n16: 3\n17: 15文字、2つ\n18: 3\n19: 0.25, 0.3, 0.33, 1/3\n20: 存在しない"
  },
  "moonshotai/kimi-k2.7-code": {
   "score": 9,
   "answered": 20,
   "first10": 4,
   "last10": 5,
   "inconsistent_pairs": 0,
   "sec": 2.2,
   "out_tokens": 135,
   "price": "$0.71 / $3.5",
   "graded": {
    "q01": {
     "a": "母親",
     "ok": 0
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "4",
     "ok": 0
    },
    "q06": {
     "a": "3",
     "ok": 0
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "水曜日",
     "ok": 1
    },
    "q09": {
     "a": "i",
     "ok": 0
    },
    "q10": {
     "a": "金星",
     "ok": 0
    },
    "q11": {
     "a": "できない",
     "ok": 1
    },
    "q12": {
     "a": "すまざいごうはよはお",
     "ok": 0
    },
    "q13": {
     "a": "母親",
     "ok": 0
    },
    "q14": {
     "a": "同じ",
     "ok": 1
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "4",
     "ok": 0
    },
    "q17": {
     "a": "15文字、1つ",
     "ok": 0
    },
    "q18": {
     "a": "4",
     "ok": 0
    },
    "q19": {
     "a": "0.25, 0.3, 0.33, 1/3",
     "ok": 1
    },
    "q20": {
     "a": "しない",
     "ok": 1
    }
   },
   "raw": "1: 母親\n2: 10\n3: しない\n4: extraordinarily\n5: 4\n6: 3\n7: できない\n8: 水曜日\n9: i\n10: 金星\n11: できない\n12: すまざいごうはよはお\n13: 母親\n14: 同じ\n15: 5\n16: 4\n17: 15文字、1つ\n18: 4\n19: 0.25, 0.3, 0.33, 1/3\n20: しない"
  },
  "qwen/qwen3.8-27b": {
   "score": 9,
   "answered": 20,
   "first10": 4,
   "last10": 5,
   "inconsistent_pairs": 1,
   "sec": 6.2,
   "out_tokens": 399,
   "price": "$0.45 / $3.2",
   "graded": {
    "q01": {
     "a": "母親",
     "ok": 0
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "4",
     "ok": 0
    },
    "q06": {
     "a": "1",
     "ok": 0
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "水曜日",
     "ok": 1
    },
    "q09": {
     "a": "r",
     "ok": 0
    },
    "q10": {
     "a": "水星",
     "ok": 0
    },
    "q11": {
     "a": "できる",
     "ok": 0
    },
    "q12": {
     "a": "ざいみあしやうはお",
     "ok": 0
    },
    "q13": {
     "a": "母子",
     "ok": 0
    },
    "q14": {
     "a": "同じ",
     "ok": 1
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "15文字・2個",
     "ok": 1
    },
    "q18": {
     "a": "4",
     "ok": 0
    },
    "q19": {
     "a": "0.25, 1/3, 0.3, 0.33",
     "ok": 0
    },
    "q20": {
     "a": "しない",
     "ok": 1
    }
   },
   "raw": "1: 母親\n2: 10\n3: しない\n4: extraordinarily\n5: 4\n6: 1\n7: できない\n8: 水曜日\n9: r\n10: 水星\n11: できる\n12: ざいみあしやうはお\n13: 母子\n14: 同じ\n15: 5\n16: 3\n17: 15文字・2個\n18: 4\n19: 0.25, 1/3, 0.3, 0.33\n20: しない"
  },
  "nvidia/nemotron-3-ultra-550b-a55b": {
   "score": 18,
   "answered": 20,
   "first10": 9,
   "last10": 9,
   "inconsistent_pairs": 1,
   "sec": 14.3,
   "out_tokens": 3438,
   "price": "$0.6 / $3.6",
   "graded": {
    "q01": {
     "a": "父親",
     "ok": 1
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "3",
     "ok": 1
    },
    "q06": {
     "a": "2",
     "ok": 1
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "水曜日",
     "ok": 1
    },
    "q09": {
     "a": "o",
     "ok": 1
    },
    "q10": {
     "a": "火星",
     "ok": 1
    },
    "q11": {
     "a": "できない",
     "ok": 1
    },
    "q12": {
     "a": "すまいざごうよはお",
     "ok": 1
    },
    "q13": {
     "a": "父親",
     "ok": 1
    },
    "q14": {
     "a": "同じ",
     "ok": 1
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "2",
     "ok": 0
    },
    "q17": {
     "a": "15文字, 2個",
     "ok": 1
    },
    "q18": {
     "a": "3",
     "ok": 1
    },
    "q19": {
     "a": "0.25, 0.3, 0.33, 1/3",
     "ok": 1
    },
    "q20": {
     "a": "しない",
     "ok": 1
    }
   },
   "raw": "1: 父親\n2: 10\n3: しない\n4: extraordinarily\n5: 3\n6: 2\n7: できない\n8: 水曜日\n9: o\n10: 火星\n11: できない\n12: すまいざごうよはお\n13: 父親\n14: 同じ\n15: 5\n16: 2\n17: 15文字, 2個\n18: 3\n19: 0.25, 0.3, 0.33, 1/3\n20: しない"
  },
  "google/gemini-3.5-flash-lite": {
   "score": 15,
   "answered": 20,
   "first10": 8,
   "last10": 7,
   "inconsistent_pairs": 0,
   "sec": 11.9,
   "out_tokens": 3018,
   "price": "$0.3 / $2.5",
   "graded": {
    "q01": {
     "a": "父親",
     "ok": 1
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "4",
     "ok": 0
    },
    "q06": {
     "a": "2",
     "ok": 1
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "水曜",
     "ok": 1
    },
    "q09": {
     "a": "o",
     "ok": 1
    },
    "q10": {
     "a": "火星",
     "ok": 1
    },
    "q11": {
     "a": "できない",
     "ok": 1
    },
    "q12": {
     "a": "すまいざごうよはお",
     "ok": 1
    },
    "q13": {
     "a": "父親",
     "ok": 1
    },
    "q14": {
     "a": "同じ",
     "ok": 1
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "2",
     "ok": 0
    },
    "q17": {
     "a": "15文字・1個",
     "ok": 0
    },
    "q18": {
     "a": "4",
     "ok": 0
    },
    "q19": {
     "a": "0.25, 0.3, 0.33, 1/3",
     "ok": 1
    },
    "q20": {
     "a": "しない",
     "ok": 1
    }
   },
   "raw": "1: 父親\n2: 10\n3: しない\n4: extraordinarily\n5: 4\n6: 2\n7: できない\n8: 水曜\n9: o\n10: 火星\n11: できない\n12: すまいざごうよはお\n13: 父親\n14: 同じ\n15: 5\n16: 2\n17: 15文字・1個\n18: 4\n19: 0.25, 0.3, 0.33, 1/3\n20: しない"
  },
  "x-ai/grok-4.20": {
   "score": 14,
   "answered": 20,
   "first10": 7,
   "last10": 7,
   "inconsistent_pairs": 0,
   "sec": 27.2,
   "out_tokens": 2928,
   "price": "$1.25 / $2.5",
   "graded": {
    "q01": {
     "a": "父親",
     "ok": 1
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "synchronization",
     "ok": 0
    },
    "q05": {
     "a": "4",
     "ok": 0
    },
    "q06": {
     "a": "3",
     "ok": 0
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "水曜日",
     "ok": 1
    },
    "q09": {
     "a": "o",
     "ok": 1
    },
    "q10": {
     "a": "火星",
     "ok": 1
    },
    "q11": {
     "a": "できる",
     "ok": 0
    },
    "q12": {
     "a": "すまいざごうよはお",
     "ok": 1
    },
    "q13": {
     "a": "父親",
     "ok": 1
    },
    "q14": {
     "a": "同じ",
     "ok": 1
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "16,1",
     "ok": 0
    },
    "q18": {
     "a": "4",
     "ok": 0
    },
    "q19": {
     "a": "0.25 0.3 0.33 1/3",
     "ok": 1
    },
    "q20": {
     "a": "しない",
     "ok": 1
    }
   },
   "raw": "1: 父親\n2: 10\n3: しない\n4: synchronization\n5: 4\n6: 3\n7: できない\n8: 水曜日\n9: o\n10: 火星\n11: できる\n12: すまいざごうよはお\n13: 父親\n14: 同じ\n15: 5\n16: 3\n17: 16,1\n18: 4\n19: 0.25 0.3 0.33 1/3\n20: しない"
  },
  "qwen/qwen3.6-35b-a3b": {
   "score": 15,
   "answered": 20,
   "first10": 8,
   "last10": 7,
   "inconsistent_pairs": 0,
   "sec": 22.5,
   "out_tokens": 5820,
   "price": "$0.15 / $1",
   "graded": {
    "q01": {
     "a": "父親",
     "ok": 1
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "4",
     "ok": 0
    },
    "q06": {
     "a": "2",
     "ok": 1
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "水曜日",
     "ok": 1
    },
    "q09": {
     "a": "o",
     "ok": 1
    },
    "q10": {
     "a": "火星",
     "ok": 1
    },
    "q11": {
     "a": "できない",
     "ok": 1
    },
    "q12": {
     "a": "す ま ざ い ご う よ は お",
     "ok": 0
    },
    "q13": {
     "a": "父親",
     "ok": 1
    },
    "q14": {
     "a": "同じ",
     "ok": 1
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "14文字、1つ",
     "ok": 0
    },
    "q18": {
     "a": "4",
     "ok": 0
    },
    "q19": {
     "a": "0.25, 0.3, 0.33, 1/3",
     "ok": 1
    },
    "q20": {
     "a": "しない",
     "ok": 1
    }
   },
   "raw": "1: 父親\n2: 10\n3: しない\n4: extraordinarily\n5: 4\n6: 2\n7: できない\n8: 水曜日\n9: o\n10: 火星\n11: できない\n12: す ま ざ い ご う よ は お\n13: 父親\n14: 同じ\n15: 5\n16: 3\n17: 14文字、1つ\n18: 4\n19: 0.25, 0.3, 0.33, 1/3\n20: しない"
  },
  "thinkingmachines/inkling": {
   "score": 12,
   "answered": 20,
   "first10": 7,
   "last10": 5,
   "inconsistent_pairs": 0,
   "sec": 6.3,
   "out_tokens": 1327,
   "price": "",
   "graded": {
    "q01": {
     "a": "母親",
     "ok": 0
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "synchronization",
     "ok": 0
    },
    "q05": {
     "a": "4",
     "ok": 0
    },
    "q06": {
     "a": "2",
     "ok": 1
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "水曜日",
     "ok": 1
    },
    "q09": {
     "a": "o",
     "ok": 1
    },
    "q10": {
     "a": "火星",
     "ok": 1
    },
    "q11": {
     "a": "できない",
     "ok": 1
    },
    "q12": {
     "a": "すまいござうよはお",
     "ok": 0
    },
    "q13": {
     "a": "母親",
     "ok": 0
    },
    "q14": {
     "a": "変える",
     "ok": 0
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "16文字、1つ",
     "ok": 0
    },
    "q18": {
     "a": "4",
     "ok": 0
    },
    "q19": {
     "a": "0.25,0.3,0.33,1/3",
     "ok": 1
    },
    "q20": {
     "a": "しない",
     "ok": 1
    }
   },
   "raw": "1: 母親\n2: 10\n3: しない\n4: synchronization\n5: 4\n6: 2\n7: できない\n8: 水曜日\n9: o\n10: 火星\n11: できない\n12: すまいござうよはお\n13: 母親\n14: 変える\n15: 5\n16: 3\n17: 16文字、1つ\n18: 4\n19: 0.25,0.3,0.33,1/3\n20: しない"
  },
  "nvidia/Nemotron-3-Nano-30B-A3B": {
   "score": 13,
   "answered": 20,
   "first10": 8,
   "last10": 5,
   "inconsistent_pairs": 1,
   "sec": 17.7,
   "out_tokens": 4878,
   "price": "$0.04 / $0.16",
   "graded": {
    "q01": {
     "a": "父",
     "ok": 1
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "synchronization",
     "ok": 0
    },
    "q05": {
     "a": "5",
     "ok": 0
    },
    "q06": {
     "a": "2",
     "ok": 1
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "水曜日",
     "ok": 1
    },
    "q09": {
     "a": "o",
     "ok": 1
    },
    "q10": {
     "a": "火星",
     "ok": 1
    },
    "q11": {
     "a": "できない",
     "ok": 1
    },
    "q12": {
     "a": "すまいざごんるようは",
     "ok": 0
    },
    "q13": {
     "a": "5",
     "ok": 0
    },
    "q14": {
     "a": "変える",
     "ok": 0
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "15 1",
     "ok": 0
    },
    "q18": {
     "a": "5",
     "ok": 0
    },
    "q19": {
     "a": "0.25, 0.3, 0.33, 1/3",
     "ok": 1
    },
    "q20": {
     "a": "しない",
     "ok": 1
    }
   },
   "raw": "1: 父\n2: 10\n3: しない\n4: synchronization\n5: 5\n6: 2\n7: できない\n8: 水曜日\n9: o\n10: 火星\n11: できない\n12: すまいざごんるようは\n13: 5\n14: 変える\n15: 5\n16: 3\n17: 15 1\n18: 5\n19: 0.25, 0.3, 0.33, 1/3\n20: しない"
  },
  "qwen/qwen3.8-2.4t-a95b": {
   "score": 19,
   "answered": 20,
   "first10": 9,
   "last10": 10,
   "inconsistent_pairs": 1,
   "sec": 19.0,
   "out_tokens": 3770,
   "price": "$2 / $6",
   "graded": {
    "q01": {
     "a": "父親",
     "ok": 1
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "3",
     "ok": 1
    },
    "q06": {
     "a": "2",
     "ok": 1
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "水曜日",
     "ok": 1
    },
    "q09": {
     "a": "o",
     "ok": 1
    },
    "q10": {
     "a": "火星",
     "ok": 1
    },
    "q11": {
     "a": "できない",
     "ok": 1
    },
    "q12": {
     "a": "すまいざごうよはお",
     "ok": 1
    },
    "q13": {
     "a": "父親",
     "ok": 1
    },
    "q14": {
     "a": "同じ",
     "ok": 1
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "15文字で2つ",
     "ok": 1
    },
    "q18": {
     "a": "3",
     "ok": 1
    },
    "q19": {
     "a": "0.25, 0.3, 0.33, 1/3",
     "ok": 1
    },
    "q20": {
     "a": "しない",
     "ok": 1
    }
   },
   "raw": "1: 父親\n2: 10\n3: しない\n4: extraordinarily\n5: 3\n6: 2\n7: できない\n8: 水曜日\n9: o\n10: 火星\n11: できない\n12: すまいざごうよはお\n13: 父親\n14: 同じ\n15: 5\n16: 3\n17: 15文字で2つ\n18: 3\n19: 0.25, 0.3, 0.33, 1/3\n20: しない"
  },
  "sakana/sakana-namazu": {
   "score": 18,
   "answered": 20,
   "first10": 8,
   "last10": 10,
   "inconsistent_pairs": 1,
   "sec": 17.6,
   "out_tokens": 3912,
   "price": "",
   "graded": {
    "q01": {
     "a": "父親",
     "ok": 1
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "3",
     "ok": 1
    },
    "q06": {
     "a": "3",
     "ok": 0
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "水曜日",
     "ok": 1
    },
    "q09": {
     "a": "o",
     "ok": 1
    },
    "q10": {
     "a": "火星",
     "ok": 1
    },
    "q11": {
     "a": "できない",
     "ok": 1
    },
    "q12": {
     "a": "すまいざごうよはお",
     "ok": 1
    },
    "q13": {
     "a": "父と息子",
     "ok": 1
    },
    "q14": {
     "a": "同じ",
     "ok": 1
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "15文字、2つ",
     "ok": 1
    },
    "q18": {
     "a": "3",
     "ok": 1
    },
    "q19": {
     "a": "0.25, 0.3, 0.33, 1/3",
     "ok": 1
    },
    "q20": {
     "a": "しない",
     "ok": 1
    }
   },
   "raw": "1: 父親\n2: 10\n3: しない\n4: extraordinarily\n5: 3\n6: 3\n7: できない\n8: 水曜日\n9: o\n10: 火星\n11: できない\n12: すまいざごうよはお\n13: 父と息子\n14: 同じ\n15: 5\n16: 3\n17: 15文字、2つ\n18: 3\n19: 0.25, 0.3, 0.33, 1/3\n20: しない"
  },
  "mistralai/mistral-medium-3-5": {
   "score": 7,
   "answered": 20,
   "first10": 5,
   "last10": 2,
   "inconsistent_pairs": 0,
   "sec": 34.8,
   "out_tokens": 3261,
   "price": "",
   "graded": {
    "q01": {
     "a": "母親",
     "ok": 0
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "3",
     "ok": 1
    },
    "q06": {
     "a": "0",
     "ok": 0
    },
    "q07": {
     "a": "できる",
     "ok": 0
    },
    "q08": {
     "a": "水曜日",
     "ok": 1
    },
    "q09": {
     "a": "o",
     "ok": 1
    },
    "q10": {
     "a": "水星",
     "ok": 0
    },
    "q11": {
     "a": "できる",
     "ok": 0
    },
    "q12": {
     "a": "ますはちおはよう",
     "ok": 0
    },
    "q13": {
     "a": "母親",
     "ok": 0
    },
    "q14": {
     "a": "変える",
     "ok": 0
    },
    "q15": {
     "a": "10",
     "ok": 0
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "15文字,1つ",
     "ok": 0
    },
    "q18": {
     "a": "3",
     "ok": 1
    },
    "q19": {
     "a": "0.25, 1/3, 0.3, 0.33",
     "ok": 0
    },
    "q20": {
     "a": "存在する",
     "ok": 0
    }
   },
   "raw": "1: 母親\n2: 10\n3: しない\n4: extraordinarily\n5: 3\n6: 0\n7: できる\n8: 水曜日\n9: o\n10: 水星\n11: できる\n12: ますはちおはよう\n13: 母親\n14: 変える\n15: 10\n16: 3\n17: 15文字,1つ\n18: 3\n19: 0.25, 1/3, 0.3, 0.33\n20: 存在する"
  },
  "tencent/hy3": {
   "score": 15,
   "answered": 20,
   "first10": 9,
   "last10": 6,
   "inconsistent_pairs": 0,
   "sec": 23.1,
   "out_tokens": 2029,
   "price": "$0.132 / $0.528",
   "graded": {
    "q01": {
     "a": "父",
     "ok": 1
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "synchronization",
     "ok": 0
    },
    "q05": {
     "a": "3",
     "ok": 1
    },
    "q06": {
     "a": "2",
     "ok": 1
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "水曜日",
     "ok": 1
    },
    "q09": {
     "a": "o",
     "ok": 1
    },
    "q10": {
     "a": "火星",
     "ok": 1
    },
    "q11": {
     "a": "できる",
     "ok": 0
    },
    "q12": {
     "a": "すみまそごうようはお",
     "ok": 0
    },
    "q13": {
     "a": "父",
     "ok": 1
    },
    "q14": {
     "a": "同じ",
     "ok": 1
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "14文字1つ",
     "ok": 0
    },
    "q18": {
     "a": "3",
     "ok": 1
    },
    "q19": {
     "a": "0.25, 0.3, 1/3, 0.33",
     "ok": 0
    },
    "q20": {
     "a": "しない",
     "ok": 1
    }
   },
   "raw": "1: 父\n2: 10\n3: しない\n4: synchronization\n5: 3\n6: 2\n7: できない\n8: 水曜日\n9: o\n10: 火星\n11: できる\n12: すみまそごうようはお\n13: 父\n14: 同じ\n15: 5\n16: 3\n17: 14文字1つ\n18: 3\n19: 0.25, 0.3, 1/3, 0.33\n20: しない"
  },
  "inception/mercury-2.5": {
   "score": 7,
   "answered": 20,
   "first10": 4,
   "last10": 3,
   "inconsistent_pairs": 2,
   "sec": 2.5,
   "out_tokens": 393,
   "price": "$0.04 / $0.15",
   "graded": {
    "q01": {
     "a": "父",
     "ok": 1
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "synchronization",
     "ok": 0
    },
    "q05": {
     "a": "4",
     "ok": 0
    },
    "q06": {
     "a": "3",
     "ok": 0
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "木曜日",
     "ok": 0
    },
    "q09": {
     "a": "d",
     "ok": 0
    },
    "q10": {
     "a": "水星",
     "ok": 0
    },
    "q11": {
     "a": "できる",
     "ok": 0
    },
    "q12": {
     "a": "すざごようおお",
     "ok": 0
    },
    "q13": {
     "a": "母子",
     "ok": 0
    },
    "q14": {
     "a": "同じ",
     "ok": 1
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "17, 2",
     "ok": 0
    },
    "q18": {
     "a": "4",
     "ok": 0
    },
    "q19": {
     "a": "0.25, 1/3, 0.3, 0.33",
     "ok": 0
    },
    "q20": {
     "a": "存在する",
     "ok": 0
    }
   },
   "raw": "1: 父\n2: 10\n3: しない\n4: synchronization\n5: 4\n6: 3\n7: できない\n8: 木曜日\n9: d\n10: 水星\n11: できる\n12: すざごようおお\n13: 母子\n14: 同じ\n15: 5\n16: 3\n17: 17, 2\n18: 4\n19: 0.25, 1/3, 0.3, 0.33\n20: 存在する"
  },
  "openai/gpt-6-astra": {
   "score": 19,
   "answered": 20,
   "first10": 9,
   "last10": 10,
   "inconsistent_pairs": 1,
   "sec": 9.4,
   "out_tokens": 324,
   "price": "",
   "graded": {
    "q01": {
     "a": "父親",
     "ok": 1
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "3",
     "ok": 1
    },
    "q06": {
     "a": "2",
     "ok": 1
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "水曜日",
     "ok": 1
    },
    "q09": {
     "a": "o",
     "ok": 1
    },
    "q10": {
     "a": "火星",
     "ok": 1
    },
    "q11": {
     "a": "できない",
     "ok": 1
    },
    "q12": {
     "a": "すまいざごうよはお",
     "ok": 1
    },
    "q13": {
     "a": "父と息子",
     "ok": 1
    },
    "q14": {
     "a": "同じ",
     "ok": 1
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "15文字、2語",
     "ok": 1
    },
    "q18": {
     "a": "3",
     "ok": 1
    },
    "q19": {
     "a": "0.25, 0.3, 0.33, 1/3",
     "ok": 1
    },
    "q20": {
     "a": "しない",
     "ok": 1
    }
   },
   "raw": "1: 父親\n2: 10\n3: しない\n4: extraordinarily\n5: 3\n6: 2\n7: できない\n8: 水曜日\n9: o\n10: 火星\n11: できない\n12: すまいざごうよはお\n13: 父と息子\n14: 同じ\n15: 5\n16: 3\n17: 15文字、2語\n18: 3\n19: 0.25, 0.3, 0.33, 1/3\n20: しない"
  },
  "google/gemma-4-31b-it": {
   "score": 17,
   "answered": 20,
   "first10": 9,
   "last10": 8,
   "inconsistent_pairs": 0,
   "sec": 21.5,
   "out_tokens": 1866,
   "price": "$0.1 / $0.34",
   "graded": {
    "q01": {
     "a": "父親",
     "ok": 1
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "3",
     "ok": 1
    },
    "q06": {
     "a": "2",
     "ok": 1
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "水曜日",
     "ok": 1
    },
    "q09": {
     "a": "o",
     "ok": 1
    },
    "q10": {
     "a": "火星",
     "ok": 1
    },
    "q11": {
     "a": "できない",
     "ok": 1
    },
    "q12": {
     "a": "すざまごいぶおはよ",
     "ok": 0
    },
    "q13": {
     "a": "父親",
     "ok": 1
    },
    "q14": {
     "a": "同じ",
     "ok": 1
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "15文字、1つ",
     "ok": 0
    },
    "q18": {
     "a": "3",
     "ok": 1
    },
    "q19": {
     "a": "0.25, 0.3, 0.33, 1/3",
     "ok": 1
    },
    "q20": {
     "a": "しない",
     "ok": 1
    }
   },
   "raw": "1: 父親\n2: 10\n3: しない\n4: extraordinarily\n5: 3\n6: 2\n7: できない\n8: 水曜日\n9: o\n10: 火星\n11: できない\n12: すざまごいぶおはよ\n13: 父親\n14: 同じ\n15: 5\n16: 3\n17: 15文字、1つ\n18: 3\n19: 0.25, 0.3, 0.33, 1/3\n20: しない"
  },
  "google/gemini-3.8-flash": {
   "score": 19,
   "answered": 20,
   "first10": 9,
   "last10": 10,
   "inconsistent_pairs": 1,
   "sec": 13.2,
   "out_tokens": 2195,
   "price": "",
   "graded": {
    "q01": {
     "a": "父親",
     "ok": 1
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "3",
     "ok": 1
    },
    "q06": {
     "a": "2",
     "ok": 1
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "水曜日",
     "ok": 1
    },
    "q09": {
     "a": "o",
     "ok": 1
    },
    "q10": {
     "a": "火星",
     "ok": 1
    },
    "q11": {
     "a": "できない",
     "ok": 1
    },
    "q12": {
     "a": "すまいざごうよはお",
     "ok": 1
    },
    "q13": {
     "a": "父と子（執刀医が父親）",
     "ok": 1
    },
    "q14": {
     "a": "同じ",
     "ok": 1
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "15文字、2つ",
     "ok": 1
    },
    "q18": {
     "a": "3",
     "ok": 1
    },
    "q19": {
     "a": "0.25, 0.3, 0.33, 1/3",
     "ok": 1
    },
    "q20": {
     "a": "しない",
     "ok": 1
    }
   },
   "raw": "1: 父親\n2: 10\n3: しない\n4: extraordinarily\n5: 3\n6: 2\n7: できない\n8: 水曜日\n9: o\n10: 火星\n11: できない\n12: すまいざごうよはお\n13: 父と子（執刀医が父親）\n14: 同じ\n15: 5\n16: 3\n17: 15文字、2つ\n18: 3\n19: 0.25, 0.3, 0.33, 1/3\n20: しない"
  },
  "anthropic/claude-sonnet-5": {
   "score": 16,
   "answered": 20,
   "first10": 8,
   "last10": 8,
   "inconsistent_pairs": 1,
   "sec": 26.4,
   "out_tokens": 2060,
   "price": "",
   "graded": {
    "q01": {
     "a": "父親(母親は死亡しているため)",
     "ok": 1
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "3",
     "ok": 1
    },
    "q06": {
     "a": "2",
     "ok": 1
    },
    "q07": {
     "a": "できる",
     "ok": 0
    },
    "q08": {
     "a": "水曜日",
     "ok": 1
    },
    "q09": {
     "a": "o",
     "ok": 1
    },
    "q10": {
     "a": "火星",
     "ok": 1
    },
    "q11": {
     "a": "できない",
     "ok": 1
    },
    "q12": {
     "a": "すまいざごうよはお",
     "ok": 1
    },
    "q13": {
     "a": "父親(実の親)",
     "ok": 1
    },
    "q14": {
     "a": "変える",
     "ok": 0
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "15文字、2語",
     "ok": 1
    },
    "q18": {
     "a": "3",
     "ok": 1
    },
    "q19": {
     "a": "0.25, 0.3, 0.33, 1/3",
     "ok": 1
    },
    "q20": {
     "a": "存在する",
     "ok": 0
    }
   },
   "raw": "1: 父親(母親は死亡しているため)\n2: 10\n3: しない\n4: extraordinarily\n5: 3\n6: 2\n7: できる\n8: 水曜日\n9: o\n10: 火星\n11: できない\n12: すまいざごうよはお\n13: 父親(実の親)\n14: 変える\n15: 5\n16: 3\n17: 15文字、2語\n18: 3\n19: 0.25, 0.3, 0.33, 1/3\n20: 存在する"
  },
  "tencent/hy-mt2-1.8b": {
   "score": 5,
   "answered": 20,
   "first10": 2,
   "last10": 3,
   "inconsistent_pairs": 0,
   "sec": 2.4,
   "out_tokens": 114,
   "price": "$0.044 / $0.177",
   "graded": {
    "q01": {
     "a": "外科医",
     "ok": 0
    },
    "q02": {
     "a": "50",
     "ok": 0
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "complicated",
     "ok": 0
    },
    "q05": {
     "a": "2",
     "ok": 0
    },
    "q06": {
     "a": "9",
     "ok": 0
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "月曜",
     "ok": 0
    },
    "q09": {
     "a": "7",
     "ok": 0
    },
    "q10": {
     "a": "エイリアス",
     "ok": 0
    },
    "q11": {
     "a": "できない",
     "ok": 1
    },
    "q12": {
     "a": "ございます",
     "ok": 0
    },
    "q13": {
     "a": "息子",
     "ok": 0
    },
    "q14": {
     "a": "同じ",
     "ok": 1
    },
    "q15": {
     "a": "10",
     "ok": 0
    },
    "q16": {
     "a": "1",
     "ok": 0
    },
    "q17": {
     "a": "3",
     "ok": 0
    },
    "q18": {
     "a": "2",
     "ok": 0
    },
    "q19": {
     "a": "1/3",
     "ok": 0
    },
    "q20": {
     "a": "しない",
     "ok": 1
    }
   },
   "raw": "1: 外科医\n2: 50\n3: しない\n4: complicated\n5: 2\n6: 9\n7: できない\n8: 月曜\n9: 7\n10: エイリアス\n11: できない\n12: ございます\n13: 息子\n14: 同じ\n15: 10\n16: 1\n17: 3\n18: 2\n19: 1/3\n20: しない"
  },
  "z-ai/glm-5.1": {
   "score": 18,
   "answered": 20,
   "first10": 9,
   "last10": 9,
   "inconsistent_pairs": 0,
   "sec": 26.4,
   "out_tokens": 3207,
   "price": "$0.966 / $3.036",
   "graded": {
    "q01": {
     "a": "父親",
     "ok": 1
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "3",
     "ok": 1
    },
    "q06": {
     "a": "2",
     "ok": 1
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "水曜日",
     "ok": 1
    },
    "q09": {
     "a": "o",
     "ok": 1
    },
    "q10": {
     "a": "火星",
     "ok": 1
    },
    "q11": {
     "a": "できない",
     "ok": 1
    },
    "q12": {
     "a": "すまいざごうよはお",
     "ok": 1
    },
    "q13": {
     "a": "父親",
     "ok": 1
    },
    "q14": {
     "a": "同じ",
     "ok": 1
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "16, 1",
     "ok": 0
    },
    "q18": {
     "a": "3",
     "ok": 1
    },
    "q19": {
     "a": "0.25, 0.3, 0.33, 1/3",
     "ok": 1
    },
    "q20": {
     "a": "しない",
     "ok": 1
    }
   },
   "raw": "1: 父親\n2: 10\n3: しない\n4: extraordinarily\n5: 3\n6: 2\n7: できない\n8: 水曜日\n9: o\n10: 火星\n11: できない\n12: すまいざごうよはお\n13: 父親\n14: 同じ\n15: 5\n16: 3\n17: 16, 1\n18: 3\n19: 0.25, 0.3, 0.33, 1/3\n20: しない"
  },
  "tencent/hy-mt2-30b-a3b": {
   "score": 7,
   "answered": 20,
   "first10": 5,
   "last10": 2,
   "inconsistent_pairs": 1,
   "sec": 2.8,
   "out_tokens": 132,
   "price": "$0.074 / $0.295",
   "graded": {
    "q01": {
     "a": "母親",
     "ok": 0
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "synchronization",
     "ok": 0
    },
    "q05": {
     "a": "3",
     "ok": 1
    },
    "q06": {
     "a": "3",
     "ok": 0
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "金曜",
     "ok": 0
    },
    "q09": {
     "a": "p",
     "ok": 0
    },
    "q10": {
     "a": "火星",
     "ok": 1
    },
    "q11": {
     "a": "できる",
     "ok": 0
    },
    "q12": {
     "a": "すござよはお",
     "ok": 0
    },
    "q13": {
     "a": "母親",
     "ok": 0
    },
    "q14": {
     "a": "変える",
     "ok": 0
    },
    "q15": {
     "a": "20",
     "ok": 0
    },
    "q16": {
     "a": "4",
     "ok": 0
    },
    "q17": {
     "a": "13, 1",
     "ok": 0
    },
    "q18": {
     "a": "3",
     "ok": 1
    },
    "q19": {
     "a": "0.25, 0.3, 0.33, 1/3",
     "ok": 1
    },
    "q20": {
     "a": "存在する",
     "ok": 0
    }
   },
   "raw": "1: 母親\n2: 10\n3: しない\n4: synchronization\n5: 3\n6: 3\n7: できない\n8: 金曜\n9: p\n10: 火星\n11: できる\n12: すござよはお\n13: 母親\n14: 変える\n15: 20\n16: 4\n17: 13, 1\n18: 3\n19: 0.25, 0.3, 0.33, 1/3\n20: 存在する"
  },
  "ibm-granite/granite-4.2-8b": {
   "score": 5,
   "answered": 20,
   "first10": 2,
   "last10": 3,
   "inconsistent_pairs": 1,
   "sec": 2.4,
   "out_tokens": 153,
   "price": "$0.06 / $0.25",
   "graded": {
    "q01": {
     "a": "外科医",
     "ok": 0
    },
    "q02": {
     "a": "40",
     "ok": 0
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "2",
     "ok": 0
    },
    "q06": {
     "a": "9",
     "ok": 0
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "月曜日",
     "ok": 0
    },
    "q09": {
     "a": "s",
     "ok": 0
    },
    "q10": {
     "a": "土星",
     "ok": 0
    },
    "q11": {
     "a": "できる",
     "ok": 0
    },
    "q12": {
     "a": "さぁおはようございます",
     "ok": 0
    },
    "q13": {
     "a": "父",
     "ok": 1
    },
    "q14": {
     "a": "変える",
     "ok": 0
    },
    "q15": {
     "a": "20",
     "ok": 0
    },
    "q16": {
     "a": "1",
     "ok": 0
    },
    "q17": {
     "a": "20文字、1つ",
     "ok": 0
    },
    "q18": {
     "a": "2",
     "ok": 0
    },
    "q19": {
     "a": "0.25, 0.3, 0.33, 1/3",
     "ok": 1
    },
    "q20": {
     "a": "しない",
     "ok": 1
    }
   },
   "raw": "1. 外科医\n2. 40\n3. しない\n4. extraordinarily\n5. 2\n6. 9\n7. できない\n8. 月曜日\n9. s\n10. 土星\n11. できる\n12. さぁおはようございます\n13. 父\n14. 変える\n15. 20\n16. 1\n17. 20文字、1つ\n18. 2\n19. 0.25, 0.3, 0.33, 1/3\n20. しない"
  },
  "tencent/hy-mt2-7b": {
   "score": 5,
   "answered": 20,
   "first10": 3,
   "last10": 2,
   "inconsistent_pairs": 2,
   "sec": 2.8,
   "out_tokens": 151,
   "price": "$0.074 / $0.295",
   "graded": {
    "q01": {
     "a": "外科医自身",
     "ok": 0
    },
    "q02": {
     "a": "20",
     "ok": 0
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "2",
     "ok": 0
    },
    "q06": {
     "a": "6",
     "ok": 0
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "木曜",
     "ok": 0
    },
    "q09": {
     "a": "o",
     "ok": 1
    },
    "q10": {
     "a": "木星",
     "ok": 0
    },
    "q11": {
     "a": "できる",
     "ok": 0
    },
    "q12": {
     "a": "すざおはよう",
     "ok": 0
    },
    "q13": {
     "a": "父親",
     "ok": 1
    },
    "q14": {
     "a": "同じ",
     "ok": 1
    },
    "q15": {
     "a": "10",
     "ok": 0
    },
    "q16": {
     "a": "2",
     "ok": 0
    },
    "q17": {
     "a": "21 2",
     "ok": 0
    },
    "q18": {
     "a": "2",
     "ok": 0
    },
    "q19": {
     "a": "0.25 1/3 0.3 0.33",
     "ok": 0
    },
    "q20": {
     "a": "存在する",
     "ok": 0
    }
   },
   "raw": "1: 外科医自身\n2: 20\n3: しない\n4: extraordinarily\n5: 2\n6: 6\n7: できない\n8: 木曜\n9: o\n10: 木星\n11: できる\n12: すざおはよう\n13: 父親\n14: 同じ\n15: 10\n16: 2\n17: 21 2\n18: 2\n19: 0.25 1/3 0.3 0.33\n20: 存在する"
  },
  "thinkingmachines/inkling-small": {
   "score": 8,
   "answered": 20,
   "first10": 4,
   "last10": 4,
   "inconsistent_pairs": 1,
   "sec": 5.0,
   "out_tokens": 856,
   "price": "",
   "graded": {
    "q01": {
     "a": "母親",
     "ok": 0
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "4",
     "ok": 0
    },
    "q06": {
     "a": "3",
     "ok": 0
    },
    "q07": {
     "a": "できる",
     "ok": 0
    },
    "q08": {
     "a": "水曜",
     "ok": 1
    },
    "q09": {
     "a": "o",
     "ok": 1
    },
    "q10": {
     "a": "水星",
     "ok": 0
    },
    "q11": {
     "a": "できる",
     "ok": 0
    },
    "q12": {
     "a": "すまごろうは",
     "ok": 0
    },
    "q13": {
     "a": "母親",
     "ok": 0
    },
    "q14": {
     "a": "変える",
     "ok": 0
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "15文字、1つ",
     "ok": 0
    },
    "q18": {
     "a": "4",
     "ok": 0
    },
    "q19": {
     "a": "0.25 < 0.3 < 0.33 < 1/3",
     "ok": 1
    },
    "q20": {
     "a": "しない",
     "ok": 1
    }
   },
   "raw": "1: 母親\n2: 10\n3: しない\n4: extraordinarily\n5: 4\n6: 3\n7: できる\n8: 水曜\n9: o\n10: 水星\n11: できる\n12: すまごろうは\n13: 母親\n14: 変える\n15: 5\n16: 3\n17: 15文字、1つ\n18: 4\n19: 0.25 < 0.3 < 0.33 < 1/3\n20: しない"
  },
  "inclusionai/ling-3.0-flash": {
   "score": 10,
   "answered": 20,
   "first10": 6,
   "last10": 4,
   "inconsistent_pairs": 0,
   "sec": 5.6,
   "out_tokens": 1100,
   "price": "$0.021 / $0.063",
   "graded": {
    "q01": {
     "a": "母親",
     "ok": 0
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "synchronization",
     "ok": 0
    },
    "q05": {
     "a": "5",
     "ok": 0
    },
    "q06": {
     "a": "3",
     "ok": 0
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "水曜日",
     "ok": 1
    },
    "q09": {
     "a": "o",
     "ok": 1
    },
    "q10": {
     "a": "火星",
     "ok": 1
    },
    "q11": {
     "a": "できる",
     "ok": 0
    },
    "q12": {
     "a": "すまれいこわは",
     "ok": 0
    },
    "q13": {
     "a": "母親",
     "ok": 0
    },
    "q14": {
     "a": "同じ",
     "ok": 1
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "2",
     "ok": 0
    },
    "q17": {
     "a": "14文字、1つ",
     "ok": 0
    },
    "q18": {
     "a": "4",
     "ok": 0
    },
    "q19": {
     "a": "0.25, 0.3, 0.33, 1/3",
     "ok": 1
    },
    "q20": {
     "a": "しない",
     "ok": 1
    }
   },
   "raw": "1: 母親\n2: 10\n3: しない\n4: synchronization\n5: 5\n6: 3\n7: できない\n8: 水曜日\n9: o\n10: 火星\n11: できる\n12: すまれいこわは\n13: 母親\n14: 同じ\n15: 5\n16: 2\n17: 14文字、1つ\n18: 4\n19: 0.25, 0.3, 0.33, 1/3\n20: しない"
  },
  "poolside/laguna-s-2.1": {
   "score": 4,
   "answered": 20,
   "first10": 2,
   "last10": 2,
   "inconsistent_pairs": 1,
   "sec": 2.9,
   "out_tokens": 168,
   "price": "$0.09 / $0.18",
   "graded": {
    "q01": {
     "a": "外科医は母親自身である。",
     "ok": 0
    },
    "q02": {
     "a": "10円",
     "ok": 1
    },
    "q03": {
     "a": "する",
     "ok": 0
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "4",
     "ok": 0
    },
    "q06": {
     "a": "3",
     "ok": 0
    },
    "q07": {
     "a": "できる",
     "ok": 0
    },
    "q08": {
     "a": "木曜日",
     "ok": 0
    },
    "q09": {
     "a": "o",
     "ok": 1
    },
    "q10": {
     "a": "水星",
     "ok": 0
    },
    "q11": {
     "a": "できる",
     "ok": 0
    },
    "q12": {
     "a": "すまぞがあようお",
     "ok": 0
    },
    "q13": {
     "a": "父親である",
     "ok": 1
    },
    "q14": {
     "a": "変える",
     "ok": 0
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "2",
     "ok": 0
    },
    "q17": {
     "a": "16文字、1つ",
     "ok": 0
    },
    "q18": {
     "a": "4",
     "ok": 0
    },
    "q19": {
     "a": "0.25, 0.3, 1/3, 0.33",
     "ok": 0
    },
    "q20": {
     "a": "存在する",
     "ok": 0
    }
   },
   "raw": "1. 外科医は母親自身である。\n2. 10円\n3. する\n4. extraordinarily\n5. 4\n6. 3\n7. できる\n8. 木曜日\n9. o\n10. 水星\n11. できる\n12. すまぞがあようお\n13. 父親である\n14. 変える\n15. 5\n16. 2\n17. 16文字、1つ\n18. 4\n19. 0.25, 0.3, 1/3, 0.33\n20. 存在する"
  },
  "bytedance-seed/seed-2.0-code": {
   "score": 18,
   "answered": 20,
   "first10": 9,
   "last10": 9,
   "inconsistent_pairs": 0,
   "sec": 21.0,
   "out_tokens": 1711,
   "price": "$0.5 / $3",
   "graded": {
    "q01": {
     "a": "父親",
     "ok": 1
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "3",
     "ok": 1
    },
    "q06": {
     "a": "2",
     "ok": 1
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "水曜日",
     "ok": 1
    },
    "q09": {
     "a": "o",
     "ok": 1
    },
    "q10": {
     "a": "火星",
     "ok": 1
    },
    "q11": {
     "a": "できない",
     "ok": 1
    },
    "q12": {
     "a": "すまいざごうよはお",
     "ok": 1
    },
    "q13": {
     "a": "親子（父親と息子）",
     "ok": 1
    },
    "q14": {
     "a": "同じ",
     "ok": 1
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "15文字、1つ",
     "ok": 0
    },
    "q18": {
     "a": "3",
     "ok": 1
    },
    "q19": {
     "a": "0.25, 0.3, 0.33, 1/3",
     "ok": 1
    },
    "q20": {
     "a": "しない",
     "ok": 1
    }
   },
   "raw": "1: 父親\n2: 10\n3: しない\n4: extraordinarily\n5: 3\n6: 2\n7: できない\n8: 水曜日\n9: o\n10: 火星\n11: できない\n12: すまいざごうよはお\n13: 親子（父親と息子）\n14: 同じ\n15: 5\n16: 3\n17: 15文字、1つ\n18: 3\n19: 0.25, 0.3, 0.33, 1/3\n20: しない"
  },
  "poolside/laguna-xs-2.1": {
   "score": 15,
   "answered": 20,
   "first10": 7,
   "last10": 8,
   "inconsistent_pairs": 1,
   "sec": 11.9,
   "out_tokens": 1857,
   "price": "$0.06 / $0.12",
   "graded": {
    "q01": {
     "a": "外科医: 父",
     "ok": 1
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "synchronization",
     "ok": 0
    },
    "q05": {
     "a": "3",
     "ok": 1
    },
    "q06": {
     "a": "2",
     "ok": 1
    },
    "q07": {
     "a": "できる",
     "ok": 0
    },
    "q08": {
     "a": "水曜",
     "ok": 1
    },
    "q09": {
     "a": "o",
     "ok": 1
    },
    "q10": {
     "a": "金星",
     "ok": 0
    },
    "q11": {
     "a": "できない",
     "ok": 1
    },
    "q12": {
     "a": "すまいざごうよはお",
     "ok": 1
    },
    "q13": {
     "a": "父と息子",
     "ok": 1
    },
    "q14": {
     "a": "変える",
     "ok": 0
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "15文字で1つ",
     "ok": 0
    },
    "q18": {
     "a": "3",
     "ok": 1
    },
    "q19": {
     "a": "0.25, 0.3, 0.33, 1/3",
     "ok": 1
    },
    "q20": {
     "a": "しない",
     "ok": 1
    }
   },
   "raw": "1. 外科医: 父\n2. 10\n3. しない\n4. synchronization\n5. 3\n6. 2\n7. できる\n8. 水曜\n9. o\n10. 金星\n11. できない\n12. すまいざごうよはお\n13. 父と息子\n14. 変える\n15. 5\n16. 3\n17. 15文字で1つ\n18. 3\n19. 0.25, 0.3, 0.33, 1/3\n20. しない"
  },
  "moonshotai/kimi-k2.6": {
   "score": 5,
   "answered": 20,
   "first10": 2,
   "last10": 3,
   "inconsistent_pairs": 0,
   "sec": 27.8,
   "out_tokens": 3900,
   "price": "$0.5415 / $2.28",
   "graded": {
    "q01": {
     "a": "ある少年が事故で病院に運ばれた。一緒にいた母親は死亡した。外科医が「この子は私の息子だ」と言った。外科医は誰?",
     "ok": 0
    },
    "q02": {
     "a": "バットとボールで合計120円。バットはボールより100円高い。ボールはいくら?(数字のみ)",
     "ok": 0
    },
    "q03": {
     "a": "2026年2月29日は存在する?(する/しない)",
     "ok": 1
    },
    "q04": {
     "a": "次の文で最長の単語を1つ: \"The extraordinarily complicated synchronization failed.\"",
     "ok": 0
    },
    "q05": {
     "a": "「ちょうちょ」は何モーラ?(数字のみ)",
     "ok": 0
    },
    "q06": {
     "a": "サリーには3人の兄弟。兄弟それぞれに3人の姉妹。サリーの姉妹は何人?(数字のみ)",
     "ok": 0
    },
    "q07": {
     "a": "農夫・オオカミ・ヤギ・キャベツ。ボートは農夫+1つ。オオカミはヤギを、ヤギはキャベツを食べる。ボートは1往復で壊れる。全部運べる?(できる/できない)",
     "ok": 1
    },
    "q08": {
     "a": "今日が月曜なら100日後は何曜日?",
     "ok": 0
    },
    "q09": {
     "a": "kaleidoscope の7文字目は?(1文字)",
     "ok": 0
    },
    "q10": {
     "a": "太陽系で2番目に小さい惑星は?",
     "ok": 0
    },
    "q11": {
     "a": "2Lと4Lの容器だけで、ちょうど3Lを測れる?(できる/できない)",
     "ok": 1
    },
    "q12": {
     "a": "「おはようございます」を1文字ずつ逆順に。",
     "ok": 0
    },
    "q13": {
     "a": "車に乗っていた息子と母。母は事故で亡くなり、息子は手術室へ。執刀医が「私の子だ」と言う。執刀医と息子の関係は?",
     "ok": 0
    },
    "q14": {
     "a": "3つの扉、1つが当たり。Aを選択。司会者は正解を知らずランダムにBを開けハズレ。Cに変えるべき?(変える/同じ)",
     "ok": 1
    },
    "q15": {
     "a": "父40歳・子10歳。父が子のちょうど3倍になるのは何年後?(数字のみ)",
     "ok": 0
    },
    "q16": {
     "a": "「わたしはきょうははれです」に「は」は何個?(数字のみ)",
     "ok": 0
    },
    "q17": {
     "a": "\"The extraordinarily complicated synchronization failed.\" — この文の単語のうち最も文字数が多いものは何文字で、該当する単語はいくつある?",
     "ok": 0
    },
    "q18": {
     "a": "「ちょうちょ」を拍(モーラ)で区切るといくつに分かれる?(数字のみ)",
     "ok": 0
    },
    "q19": {
     "a": "0.3, 0.25, 1/3, 0.33 を小さい順に。",
     "ok": 0
    },
    "q20": {
     "a": "ボートが1往復しか使えない条件で、オオカミ・ヤギ・キャベツを全て無事に対岸へ運ぶ手順は存在する?(存在する/しない)",
     "ok": 1
    }
   },
   "raw": "ユーザーは20問に対して、番号順に「番号: 答え」の形式で1行ずつ、できるだけ短く答えることを求めています。前置き・解説は不要です。\n\n各問題を確認して回答を考えます。\n\n1. ある少年が事故で病院に運ばれた。一緒にいた母親は死亡した。外科医が「この子は私の息子だ」と言った。外科医は誰?\n   - 母親は死亡したが、外科医は父親です。または、外科医は母親（養子縁組や同性婚の可能性もありますが、古典的ななぞなぞの答えは「父親」）。ただし、現代的な解釈では「父親」が一般的。\n\n2. バットとボールで合計120円。バットはボールより100円高い。ボールはいくら?(数字のみ)\n   - バット = x, ボール = y。x + y = 120, x = y + 100。\n   - (y+100) + y = 120 → 2y = 20 → y = 10。\n   - 答え: 10\n\n3. 2026年2月29日は存在する?(する/しない)\n   - 2026年は閏年ではない（2024年が閏年、2028年が閏年）。2026 ÷ 4 = 506.5。閏年ではないので2月29日は存在しない。\n   - 答え: しない\n\n4. 次の文で最長の単語を1つ: \"The extraordinarily complicated synchronization failed.\"\n   - extraordinarily (16文字), complicated (11文字), synchronization (15文字)。\n   - extraordinarily が最長。\n   - 答え: extraordinarily\n\n5. 「ちょうちょ」は何モーラ?(数字のみ)\n   - ちょ・う・ちょ → 3モーラ？ それとも「ちょうちょ」は「ちょーちょ」？\n   - モーラ（拍）の数え方：「ちょう」は2モーラ（ちょ+う）。「ちょ」は1モーラ、「う」は1モーラ。なので「ちょうちょ」は「ちょ・う・ちょ」で3モーラ。\n   - でも「ちょうちょ」は通常「ちょうちょ」と書いて、モーラ数は4？ いや、長音「う」は1モーラ。なので「ちょ(1)・う(1)・ちょ(1)」で3モーラ。\n   - 待って、18番と重複しているような？ 18番は「拍(モーラ)で区切るといくつに分かれる?」と同じ質問？\n   - 18番と5番は同じ質問に見える。でも違う番号。同じ答えでいいのかな。とりあえず3。\n\n6. サリーには3人の兄弟。兄弟それぞれに3人の姉妹。サリーの姉妹は何人?(数字のみ)\n   - サリーは女の子。兄弟3人。兄弟それぞれに3人の姉妹。サリーもその姉妹の1人。なので姉妹はサリーを含めて3人。サリーの姉妹は3人？ いや、「サリーの姉妹」はサリー以外の姉妹の数。\n   - 兄弟3人がそれぞれ「3人の姉妹」を持つということは、姉妹の総数は3人（サリー+他2人、またはサリー含めて3人）。サリー自身も姉妹の1人なので、サリーの姉妹（サリー以外）は2人。\n   - でも「サリーには3人の兄弟」→サリーは女。「兄弟それぞれに3人の姉妹」→兄弟から見て姉妹が3人。サリーはそのうちの1人。残りは2人。\n   - 答え: 2\n\n7. 農夫・オオカミ・ヤギ・キャベツ。ボートは農夫+1つ。オオカミはヤギを、ヤギはキャベツを食べる。ボートは1往復で壊れる。全部運べる?(できる/できない)\n   - 通常の川渡り問題は往復何度もできるが、ここでは1往復しかできない。つまり行って帰ってくるのが1回だけ。それで全部運べるか？\n   - 農夫+1つしか乗れない。往復1回で壊れる。\n   - 行き1回目：農夫+ヤギを運ぶ（残りはオオカミとキャベツ。オオカミはキャベツを食べない）。対岸にヤギ。\n   - 帰り：農夫だけ戻る。これで1往復完了。ボートが壊れる。\n   - 残りのオオカミとキャベツは運べない。\n   - 行き1回目で農夫+オオカミ、帰りで農夫+ヤギ？ いや、ボートは農夫+1つまで。帰りも農夫+1つまで乗れるが、往復1回で壊れるということは、行きと帰りの合計が1往復。つまり戻ってこれなくなる？\n   - 「1往復で壊れる」→ 行って帰るのが1回まで。2回目の出発はできない。\n   - 行き：農夫+ヤギ。対岸にヤギ。残りはオオカミとキャベツ（安全）。\n   - 帰り：農夫だけ戻る。これで1往復。ボート壊れる。\n   - 残り2つ（オオカミ、キャベツ）を運べない。\n   - 行きで農夫+オオカミ、帰りで農夫+ヤギ？ 帰りにヤギを連れてくると、出発岸にヤギが戻る。対岸にオオカミ。出発岸にヤギとキャベツが残る。ヤギはキャベツを食べる。だめ。\n   - 行きで農夫+キャベツ、帰りで農夫+ヤギ？ 対岸にキャベツ。出発岸にヤギとオオカミ。ヤギが食べられる。だめ。\n   - 行きで農夫+ヤギ、帰りで農夫+何もない？ それなら対岸にヤギだけ。出発岸にオオカミとキャベツ（安全）。でも残り2つを運べない。\n   - 結局、全部運べない。\n   - 答え: できない\n\n8. 今日が月曜なら100日後は何曜日?\n   - 100 ÷ 7 = 14余り2。\n   - 月曜 + 2日 = 水曜日。\n   - 答え: 水曜日\n\n9. kaleidoscope の7文字目は?(1文字)\n   - k(1)a(2)l(3)e(4)i(5)d(6)o(7)s(8)c(9)o(10)p(11)e(12)\n   - 7文字目は o\n   - 答え: o\n\n10. 太陽系で2番目に小さい惑星は?\n    - 惑星の大きさ（直径）順：水星、火星、金星、地球、海王星、天王星、土星、木星。\n    - 最小は水星。2番目に小さいは火星。\n    - 質量順でも火星が2番目に小さい（水星が最も小さい）。\n    - 答え: 火星\n\n11. 2Lと4Lの容器だけで、ちょうど3Lを測れる?(できる/できない)\n    - 2Lと4L。4Lに満たす。2Lに移すと4Lに2L残る。2Lを捨てる。4Lの2Lを2Lに移す。4Lにまた満たす。2Lに注ぐと2Lはすでに2Lあるので入らない？ いや、2L容器に2L入っているので満杯。4Lから2L注げない。\n    - 別の方法：4Lに満たす。2Lに注ぎ、4Lに2L残る。2Lを捨てる。4Lの2Lを2Lに移す。4Lに満たす。2Lに注ごうとするが、2Lにはすでに2L入っているので0Lしか入らない。4Lは4Lのまま。\n    - 2Lに満たす。4Lに注ぐ。2Lに満たす。4Lに注ぐと4Lは満杯（4L）。2Lは0L。これで4L。\n    - 2Lと4Lでは3Lは作れない。最大公約数が2Lなので、作れるのは2Lの倍数のみ。3Lは作れない。\n    - 答え: できない\n\n12. 「おはようございます」を1文字ずつ逆順に。\n    - す・ま・い・ざ・い・ご・う・よ・は・お\n    - 答え: すまいざいごうよはお\n\n13. 車に乗っていた息子と母。母は事故で亡くなり、息子は手術室へ。執刀医が「私の子だ」と言う。執刀医と息子の関係は?\n    - 1番と同じパターン。執刀医は父親。\n    - 答え: 父子（または父親）\n\n14. 3つの扉、1つが当たり。Aを選択。司会者は正解を知らずランダムにBを開けハズレ。Cに変えるべき?(変える/同じ)\n    - 通常のモンティ・ホール問題では司会者は正解を知っているが、ここでは「司会者は正解を知らずランダムにBを開けハズレ」。\n    - これは「無知な司会者」バージョン。条件付き確率を考える。\n    - Aが当たり(1/3)、Bが当たり(1/3)、Cが当たり(1/3)。\n    - 司会者がランダムにBを開けてハズレだった場合：\n      - Aが当たりの確率：1/3\n      - Cが当たりの確率：1/3\n      - Bが当たりの確率：0（開けてハズレだから）\n      - ただし、司会者がBを開けたという条件のもとで、Bが当たりだった場合は開けられない（ハズレを開けたとあるので、Bが当たりならこの事象は起きない）。\n      - 実はこの条件のもとでは、AとCの確率は等しいまま（各1/2）になる。変えても変えなくても同じ。\n    - 答え: 同じ\n\n15. 父40歳・子10歳。父が子のちょうど3倍になるのは何年後?(数字のみ)\n    - x年後：40+x = 3(10+x)\n    - 40+x = 30+3x\n    - 10 = 2x\n    - x = 5\n    - 答え: 5\n\n16. 「わたしはきょうははれです」に「は」は何個?(数字のみ)\n    - わたしは(1) きょうは(2) はれです(3)\n    - 「はれ」の「は」も含めるか？ 問題文は「に『は』は何個」。平仮名の「は」を数える。\n    - わたし「は」、きょう「は」、は「れ」→ 3個。\n    - 待って、「はれです」の「は」は助詞ではなくても文字としての「は」なので数える。3個。\n    - 答え: 3\n\n17. \"The extraordinarily complicated synchronization failed.\" — この文の単語のうち最も文字数が多いものは何文字で、該当する単語はいくつある?\n    - The(3) extraordinarily(16) complicated(11) synchronization(15) failed(6).\n    - 最も文字数が多いのは extraordinarily で16文字。\n    - 該当する単語は1つ。\n    - 答えの形式は問われていないが、「16文字、1つ」のように短く。\n    - ユーザーは「できるだけ短く」と言っているので「16文字、1つ」または「16,1」など。でも「番号: 答え」形式。\n    - 文脈的に「16文字で1つ」が適切。\n\n18. 「ちょうちょ」を拍(モーラ)で区切るといくつに分かれる?(数字のみ)\n    - 5番と同じ。3。\n    - 答え: 3\n\n19. 0.3, 0.25, 1/3, 0.33 を小さい順に。\n    - 0.25 = 1/4 = 0.25\n    - 0.3 = 0.30\n    - 0.33 = 0.33\n    - 1/3 = 0.3333...\n    - 小さい順：0.25, 0.3, 0.33, 1/3\n    - 答え: 0.25, 0.3, 0.33, 1/3\n\n20. ボートが1往復しか使えない条件で、オオカミ・ヤギ・キャベツを全て無事に対岸へ運ぶ手順は存在する?(存在する/しない)\n    - 7番と同じ質問？ 7番は「全部運べる?(できる/できない)」、20番は「手順は存在する?(存在する/しない)」。\n    - 7番ですでに「できない」と判断。20番も同じく「しない」。\n    - 答え: しない\n\n再確認：\n- 1: 父親\n- 2: 10\n- 3: しない\n- 4: extraordinarily\n- 5: 3\n- 6: 2\n- 7: できない\n- 8: 水曜日\n- 9: o\n- 10: 火星\n- 11: できない\n- 12: すまいざいごうよはお\n- 13: 父親（父子）\n- 14: 同じ\n- 15: 5\n- 16: 3\n- 17: 16文字、1つ\n- 18: 3\n- 19: 0.25, 0.3, 0.33, 1/3\n- 20: しない\n\n17番の答え方：「16文字で1つ」が短くていいかも。\n19番：カンマ区切りで。\n\nすべて1行ずつ、番号: 答え の形式。\n\n最終確認：\n- 5番と18番は同じ答え（3）で問題なし。\n- 7番と20番は同じ結論（できない/しない）で問題なし。\n- 1番と13番は似ているが、13番は「執刀医と息子の関係は?」なので「父子」または「父と息子」。1番は「外科医は誰?」なので「父親」。\n\n出力します。1: 父親\n2: 10\n3: しない\n4: extraordinarily\n5: 3\n6: 2\n7: できない\n8: 水曜日\n9: o\n10: 火星\n11: できない\n12: すまいざいごうよはお\n13: 父子\n14: 同じ\n15: 5\n16: 3\n17: 16文字、1つ\n18: 3\n19: 0.25, 0.3, 0.33, 1/3\n20: しない"
  },
  "meta/muse-glimmer-30b": {
   "score": 12,
   "answered": 20,
   "first10": 6,
   "last10": 6,
   "inconsistent_pairs": 0,
   "sec": 15.1,
   "out_tokens": 1611,
   "price": "$0.3 / $1.1",
   "graded": {
    "q01": {
     "a": "母親",
     "ok": 0
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "4",
     "ok": 0
    },
    "q06": {
     "a": "3",
     "ok": 0
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "水曜",
     "ok": 1
    },
    "q09": {
     "a": "o",
     "ok": 1
    },
    "q10": {
     "a": "火星",
     "ok": 1
    },
    "q11": {
     "a": "できない",
     "ok": 1
    },
    "q12": {
     "a": "すまいざごようはお",
     "ok": 0
    },
    "q13": {
     "a": "母",
     "ok": 0
    },
    "q14": {
     "a": "同じ",
     "ok": 1
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "16文字 1個",
     "ok": 0
    },
    "q18": {
     "a": "4",
     "ok": 0
    },
    "q19": {
     "a": "0.25,0.3,0.33,1/3",
     "ok": 1
    },
    "q20": {
     "a": "しない",
     "ok": 1
    }
   },
   "raw": "1: 母親\n2: 10\n3: しない\n4: extraordinarily\n5: 4\n6: 3\n7: できない\n8: 水曜\n9: o\n10: 火星\n11: できない\n12: すまいざごようはお\n13: 母\n14: 同じ\n15: 5\n16: 3\n17: 16文字 1個\n18: 4\n19: 0.25,0.3,0.33,1/3\n20: しない"
  },
  "aion-labs/aion-3.0-mini": {
   "score": 11,
   "answered": 20,
   "first10": 6,
   "last10": 5,
   "inconsistent_pairs": 0,
   "sec": 4.3,
   "out_tokens": 192,
   "price": "$0.7 / $1.4",
   "graded": {
    "q01": {
     "a": "外科医は少年の父親",
     "ok": 1
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "3",
     "ok": 1
    },
    "q06": {
     "a": "3",
     "ok": 0
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "水曜日",
     "ok": 1
    },
    "q09": {
     "a": "s",
     "ok": 0
    },
    "q10": {
     "a": "水星",
     "ok": 0
    },
    "q11": {
     "a": "できる",
     "ok": 0
    },
    "q12": {
     "a": "すまざうございよはうお",
     "ok": 0
    },
    "q13": {
     "a": "執刀医は父親",
     "ok": 1
    },
    "q14": {
     "a": "変える",
     "ok": 0
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "16文字, 1つ",
     "ok": 0
    },
    "q18": {
     "a": "3",
     "ok": 1
    },
    "q19": {
     "a": "0.25, 0.3, 1/3, 0.33",
     "ok": 0
    },
    "q20": {
     "a": "存在しない",
     "ok": 1
    }
   },
   "raw": "1. 外科医は少年の父親\n2. 10\n3. しない\n4. extraordinarily\n5. 3\n6. 3\n7. できない\n8. 水曜日\n9. s\n10. 水星\n11. できる\n12. すまざうございよはうお\n13. 執刀医は父親\n14. 変える\n15. 5\n16. 3\n17. 16文字, 1つ\n18. 3\n19. 0.25, 0.3, 1/3, 0.33\n20. 存在しない"
  },
  "anthropic/claude-opus-4.7": {
   "score": 18,
   "answered": 20,
   "first10": 8,
   "last10": 10,
   "inconsistent_pairs": 2,
   "sec": 27.9,
   "out_tokens": 1360,
   "price": "$5 / $25",
   "graded": {
    "q01": {
     "a": "母親",
     "ok": 0
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "synchronization",
     "ok": 0
    },
    "q05": {
     "a": "3",
     "ok": 1
    },
    "q06": {
     "a": "2",
     "ok": 1
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "水曜日",
     "ok": 1
    },
    "q09": {
     "a": "o",
     "ok": 1
    },
    "q10": {
     "a": "火星",
     "ok": 1
    },
    "q11": {
     "a": "できない",
     "ok": 1
    },
    "q12": {
     "a": "すまいざごうよはお",
     "ok": 1
    },
    "q13": {
     "a": "父親",
     "ok": 1
    },
    "q14": {
     "a": "同じ",
     "ok": 1
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "15文字、2個",
     "ok": 1
    },
    "q18": {
     "a": "3",
     "ok": 1
    },
    "q19": {
     "a": "0.25, 0.3, 0.33, 1/3",
     "ok": 1
    },
    "q20": {
     "a": "しない",
     "ok": 1
    }
   },
   "raw": "1: 母親\n2: 10\n3: しない\n4: synchronization\n5: 3\n6: 2\n7: できない\n8: 水曜日\n9: o\n10: 火星\n11: できない\n12: すまいざごうよはお\n13: 父親\n14: 同じ\n15: 5\n16: 3\n17: 15文字、2個\n18: 3\n19: 0.25, 0.3, 0.33, 1/3\n20: しない"
  },
  "tencent/hy3-preview": {
   "score": 12,
   "answered": 20,
   "first10": 7,
   "last10": 5,
   "inconsistent_pairs": 0,
   "sec": 25.0,
   "out_tokens": 2399,
   "price": "$0.18 / $0.6",
   "graded": {
    "q01": {
     "a": "父親",
     "ok": 1
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "synchronization",
     "ok": 0
    },
    "q05": {
     "a": "4",
     "ok": 0
    },
    "q06": {
     "a": "2",
     "ok": 1
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "水曜日",
     "ok": 1
    },
    "q09": {
     "a": "o",
     "ok": 1
    },
    "q10": {
     "a": "金星",
     "ok": 0
    },
    "q11": {
     "a": "できる",
     "ok": 0
    },
    "q12": {
     "a": "すんいざごうよはお",
     "ok": 0
    },
    "q13": {
     "a": "父親",
     "ok": 1
    },
    "q14": {
     "a": "同じ",
     "ok": 1
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "14, 1",
     "ok": 0
    },
    "q18": {
     "a": "4",
     "ok": 0
    },
    "q19": {
     "a": "0.25, 0.3, 1/3, 0.33",
     "ok": 0
    },
    "q20": {
     "a": "しない",
     "ok": 1
    }
   },
   "raw": "1: 父親\n2: 10\n3: しない\n4: synchronization\n5: 4\n6: 2\n7: できない\n8: 水曜日\n9: o\n10: 金星\n11: できる\n12: すんいざごうよはお\n13: 父親\n14: 同じ\n15: 5\n16: 3\n17: 14, 1\n18: 4\n19: 0.25, 0.3, 1/3, 0.33\n20: しない"
  },
  "arcee-ai/trinity-large-thinking": {
   "score": 11,
   "answered": 20,
   "first10": 6,
   "last10": 5,
   "inconsistent_pairs": 0,
   "sec": 8.6,
   "out_tokens": 2047,
   "price": "$0.25 / $0.8",
   "graded": {
    "q01": {
     "a": "母親",
     "ok": 0
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "synchronization",
     "ok": 0
    },
    "q05": {
     "a": "2",
     "ok": 0
    },
    "q06": {
     "a": "2",
     "ok": 1
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "水曜日",
     "ok": 1
    },
    "q09": {
     "a": "d",
     "ok": 0
    },
    "q10": {
     "a": "火星",
     "ok": 1
    },
    "q11": {
     "a": "できない",
     "ok": 1
    },
    "q12": {
     "a": "すまざごようこはお",
     "ok": 0
    },
    "q13": {
     "a": "母親",
     "ok": 0
    },
    "q14": {
     "a": "同じ",
     "ok": 1
    },
    "q15": {
     "a": "10",
     "ok": 0
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "14文字、1つ",
     "ok": 0
    },
    "q18": {
     "a": "2",
     "ok": 0
    },
    "q19": {
     "a": "0.25, 0.3, 0.33, 1/3",
     "ok": 1
    },
    "q20": {
     "a": "存在しない",
     "ok": 1
    }
   },
   "raw": "1: 母親  \n2: 10  \n3: しない  \n4: synchronization  \n5: 2  \n6: 2  \n7: できない  \n8: 水曜日  \n9: d  \n10: 火星  \n11: できない  \n12: すまざごようこはお  \n13: 母親  \n14: 同じ  \n15: 10  \n16: 3  \n17: 14文字、1つ  \n18: 2  \n19: 0.25, 0.3, 0.33, 1/3  \n20: 存在しない"
  },
  "rekaai/reka-edge": {
   "score": 2,
   "answered": 20,
   "first10": 0,
   "last10": 2,
   "inconsistent_pairs": 0,
   "sec": 2.6,
   "out_tokens": 244,
   "price": "$0.1 / $0.1",
   "graded": {
    "q01": {
     "a": "答え: 1",
     "ok": 0
    },
    "q02": {
     "a": "答え: 2",
     "ok": 0
    },
    "q03": {
     "a": "答え: 29",
     "ok": 0
    },
    "q04": {
     "a": "答え: 101",
     "ok": 0
    },
    "q05": {
     "a": "答え: 7",
     "ok": 0
    },
    "q06": {
     "a": "答え: 6",
     "ok": 0
    },
    "q07": {
     "a": "答え: 2",
     "ok": 0
    },
    "q08": {
     "a": "答え: 101",
     "ok": 0
    },
    "q09": {
     "a": "答え: 7",
     "ok": 0
    },
    "q10": {
     "a": "答え: Mercury",
     "ok": 0
    },
    "q11": {
     "a": "答え: できない",
     "ok": 1
    },
    "q12": {
     "a": "答え: おはようございます",
     "ok": 0
    },
    "q13": {
     "a": "答え: 執刀医と息子の関係は不明",
     "ok": 0
    },
    "q14": {
     "a": "答え: 変える",
     "ok": 0
    },
    "q15": {
     "a": "答え: 15年",
     "ok": 0
    },
    "q16": {
     "a": "答え: 3",
     "ok": 1
    },
    "q17": {
     "a": "答え: 8",
     "ok": 0
    },
    "q18": {
     "a": "答え: 8",
     "ok": 0
    },
    "q19": {
     "a": "小さい順に: 0.3、0.25、1/3、0.33",
     "ok": 0
    },
    "q20": {
     "a": "答え: 存在する",
     "ok": 0
    }
   },
   "raw": "1. 答え: 1\n2. 答え: 2\n3. 答え: 29\n4. 答え: 101\n5. 答え: 7\n6. 答え: 6\n7. 答え: 2\n8. 答え: 101\n9. 答え: 7\n10. 答え: Mercury\n11. 答え: できない\n12. 答え: おはようございます\n13. 答え: 執刀医と息子の関係は不明\n14. 答え: 変える\n15. 答え: 15年\n16. 答え: 3\n17. 答え: 8\n18. 答え: 8\n19. 小さい順に: 0.3、0.25、1/3、0.33\n20. 答え: 存在する"
  },
  "anthropic/claude-opus-4.8": {
   "score": 17,
   "answered": 20,
   "first10": 9,
   "last10": 8,
   "inconsistent_pairs": 1,
   "sec": 26.9,
   "out_tokens": 1425,
   "price": "$5 / $25",
   "graded": {
    "q01": {
     "a": "父親",
     "ok": 1
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "synchronization",
     "ok": 0
    },
    "q05": {
     "a": "3",
     "ok": 1
    },
    "q06": {
     "a": "2",
     "ok": 1
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "水曜",
     "ok": 1
    },
    "q09": {
     "a": "o",
     "ok": 1
    },
    "q10": {
     "a": "火星",
     "ok": 1
    },
    "q11": {
     "a": "できない",
     "ok": 1
    },
    "q12": {
     "a": "すまいざごうよはお",
     "ok": 1
    },
    "q13": {
     "a": "親子（親）",
     "ok": 0
    },
    "q14": {
     "a": "同じ",
     "ok": 1
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "15文字、1つ",
     "ok": 0
    },
    "q18": {
     "a": "3",
     "ok": 1
    },
    "q19": {
     "a": "0.25, 0.3, 0.33, 1/3",
     "ok": 1
    },
    "q20": {
     "a": "しない",
     "ok": 1
    }
   },
   "raw": "1: 父親\n2: 10\n3: しない\n4: synchronization\n5: 3\n6: 2\n7: できない\n8: 水曜\n9: o\n10: 火星\n11: できない\n12: すまいざごうよはお\n13: 親子（親）\n14: 同じ\n15: 5\n16: 3\n17: 15文字、1つ\n18: 3\n19: 0.25, 0.3, 0.33, 1/3\n20: しない"
  },
  "inception/mercury-2": {
   "score": 16,
   "answered": 20,
   "first10": 9,
   "last10": 7,
   "inconsistent_pairs": 0,
   "sec": 4.0,
   "out_tokens": 1254,
   "price": "$0.25 / $0.75",
   "graded": {
    "q01": {
     "a": "父",
     "ok": 1
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "3",
     "ok": 1
    },
    "q06": {
     "a": "2",
     "ok": 1
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "水曜日",
     "ok": 1
    },
    "q09": {
     "a": "o",
     "ok": 1
    },
    "q10": {
     "a": "火星",
     "ok": 1
    },
    "q11": {
     "a": "できる",
     "ok": 0
    },
    "q12": {
     "a": "すまいごはお",
     "ok": 0
    },
    "q13": {
     "a": "父",
     "ok": 1
    },
    "q14": {
     "a": "同じ",
     "ok": 1
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "15文字、1つ",
     "ok": 0
    },
    "q18": {
     "a": "3",
     "ok": 1
    },
    "q19": {
     "a": "0.25, 0.3, 0.33, 1/3",
     "ok": 1
    },
    "q20": {
     "a": "しない",
     "ok": 1
    }
   },
   "raw": "1: 父  \n2: 10  \n3: しない  \n4: extraordinarily  \n5: 3  \n6: 2  \n7: できない  \n8: 水曜日  \n9: o  \n10: 火星  \n11: できる  \n12: すまいごはお  \n13: 父  \n14: 同じ  \n15: 5  \n16: 3  \n17: 15文字、1つ  \n18: 3  \n19: 0.25, 0.3, 0.33, 1/3  \n20: しない"
  },
  "mistralai/mistral-small-2603": {
   "score": 7,
   "answered": 20,
   "first10": 5,
   "last10": 2,
   "inconsistent_pairs": 2,
   "sec": 10.3,
   "out_tokens": 1246,
   "price": "$0.15 / $0.6",
   "graded": {
    "q01": {
     "a": "父親",
     "ok": 1
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "3",
     "ok": 1
    },
    "q06": {
     "a": "3",
     "ok": 0
    },
    "q07": {
     "a": "できる",
     "ok": 0
    },
    "q08": {
     "a": "水曜日",
     "ok": 1
    },
    "q09": {
     "a": "s",
     "ok": 0
    },
    "q10": {
     "a": "金星",
     "ok": 0
    },
    "q11": {
     "a": "できる",
     "ok": 0
    },
    "q12": {
     "a": "はちにんまよ",
     "ok": 0
    },
    "q13": {
     "a": "母親",
     "ok": 0
    },
    "q14": {
     "a": "変える",
     "ok": 0
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "15文字, 1",
     "ok": 0
    },
    "q18": {
     "a": "2",
     "ok": 0
    },
    "q19": {
     "a": "0.25, 0.3, 1/3, 0.33",
     "ok": 0
    },
    "q20": {
     "a": "存在する",
     "ok": 0
    }
   },
   "raw": "1: 父親\n2: 10\n3: しない\n4: extraordinarily\n5: 3\n6: 3\n7: できる\n8: 水曜日\n9: s\n10: 金星\n11: できる\n12: はちにんまよ\n13: 母親\n14: 変える\n15: 5\n16: 3\n17: 15文字, 1\n18: 2\n19: 0.25, 0.3, 1/3, 0.33\n20: 存在する"
  },
  "google/gemini-3.1-flash-lite-preview": {
   "score": 14,
   "answered": 20,
   "first10": 8,
   "last10": 6,
   "inconsistent_pairs": 1,
   "sec": 4.1,
   "out_tokens": 1092,
   "price": "$0.25 / $1.5",
   "graded": {
    "q01": {
     "a": "父親",
     "ok": 1
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "4",
     "ok": 0
    },
    "q06": {
     "a": "2",
     "ok": 1
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "水曜日",
     "ok": 1
    },
    "q09": {
     "a": "o",
     "ok": 1
    },
    "q10": {
     "a": "火星",
     "ok": 1
    },
    "q11": {
     "a": "できない",
     "ok": 1
    },
    "q12": {
     "a": "すまざいごうはお",
     "ok": 0
    },
    "q13": {
     "a": "親子",
     "ok": 0
    },
    "q14": {
     "a": "同じ",
     "ok": 1
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "15文字、1つ",
     "ok": 0
    },
    "q18": {
     "a": "4",
     "ok": 0
    },
    "q19": {
     "a": "0.25, 0.3, 0.33, 1/3",
     "ok": 1
    },
    "q20": {
     "a": "存在しない",
     "ok": 1
    }
   },
   "raw": "1: 父親\n2: 10\n3: しない\n4: extraordinarily\n5: 4\n6: 2\n7: できない\n8: 水曜日\n9: o\n10: 火星\n11: できない\n12: すまざいごうはお\n13: 親子\n14: 同じ\n15: 5\n16: 3\n17: 15文字、1つ\n18: 4\n19: 0.25, 0.3, 0.33, 1/3\n20: 存在しない"
  },
  "x-ai/grok-build-0.1": {
   "score": 14,
   "answered": 20,
   "first10": 6,
   "last10": 8,
   "inconsistent_pairs": 2,
   "sec": 21.2,
   "out_tokens": 3409,
   "price": "$1 / $2",
   "graded": {
    "q01": {
     "a": "母親",
     "ok": 0
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "synchronization",
     "ok": 0
    },
    "q05": {
     "a": "4",
     "ok": 0
    },
    "q06": {
     "a": "2",
     "ok": 1
    },
    "q07": {
     "a": "できる",
     "ok": 0
    },
    "q08": {
     "a": "水曜日",
     "ok": 1
    },
    "q09": {
     "a": "o",
     "ok": 1
    },
    "q10": {
     "a": "火星",
     "ok": 1
    },
    "q11": {
     "a": "できない",
     "ok": 1
    },
    "q12": {
     "a": "すまいざごうよはお",
     "ok": 1
    },
    "q13": {
     "a": "母親",
     "ok": 0
    },
    "q14": {
     "a": "同じ",
     "ok": 1
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "15文字で2つ",
     "ok": 1
    },
    "q18": {
     "a": "4",
     "ok": 0
    },
    "q19": {
     "a": "0.25, 0.3, 0.33, 1/3",
     "ok": 1
    },
    "q20": {
     "a": "しない",
     "ok": 1
    }
   },
   "raw": "1: 母親\n2: 10\n3: しない\n4: synchronization\n5: 4\n6: 2\n7: できる\n8: 水曜日\n9: o\n10: 火星\n11: できない\n12: すまいざごうよはお\n13: 母親\n14: 同じ\n15: 5\n16: 3\n17: 15文字で2つ\n18: 4\n19: 0.25, 0.3, 0.33, 1/3\n20: しない"
  },
  "qwen/qwen3-coder-next": {
   "score": 5,
   "answered": 20,
   "first10": 2,
   "last10": 3,
   "inconsistent_pairs": 0,
   "sec": 3.1,
   "out_tokens": 153,
   "price": "$0.12 / $0.8",
   "graded": {
    "q01": {
     "a": "母親",
     "ok": 0
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "synchronization",
     "ok": 0
    },
    "q05": {
     "a": "4",
     "ok": 0
    },
    "q06": {
     "a": "3",
     "ok": 0
    },
    "q07": {
     "a": "できる",
     "ok": 0
    },
    "q08": {
     "a": "月曜",
     "ok": 0
    },
    "q09": {
     "a": "d",
     "ok": 0
    },
    "q10": {
     "a": "水星",
     "ok": 0
    },
    "q11": {
     "a": "できる",
     "ok": 0
    },
    "q12": {
     "a": "すめでれはははょうわし",
     "ok": 0
    },
    "q13": {
     "a": "母親",
     "ok": 0
    },
    "q14": {
     "a": "同じ",
     "ok": 1
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "4",
     "ok": 0
    },
    "q17": {
     "a": "14文字、1つ",
     "ok": 0
    },
    "q18": {
     "a": "4",
     "ok": 0
    },
    "q19": {
     "a": "0.25, 0.3, 0.33, 1/3",
     "ok": 1
    },
    "q20": {
     "a": "存在する",
     "ok": 0
    }
   },
   "raw": "1. 母親  \n2. 10  \n3. しない  \n4. synchronization  \n5. 4  \n6. 3  \n7. できる  \n8. 月曜  \n9. d  \n10. 水星  \n11. できる  \n12. すめでれはははょうわし  \n13. 母親  \n14. 同じ  \n15. 5  \n16. 4  \n17. 14文字、1つ  \n18. 4  \n19. 0.25, 0.3, 0.33, 1/3  \n20. 存在する"
  },
  "openai/gpt-5.3-codex": {
   "score": 19,
   "answered": 20,
   "first10": 9,
   "last10": 10,
   "inconsistent_pairs": 1,
   "sec": 16.3,
   "out_tokens": 1297,
   "price": "$1.75 / $14",
   "graded": {
    "q01": {
     "a": "父親",
     "ok": 1
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "3",
     "ok": 1
    },
    "q06": {
     "a": "2",
     "ok": 1
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "水曜日",
     "ok": 1
    },
    "q09": {
     "a": "o",
     "ok": 1
    },
    "q10": {
     "a": "火星",
     "ok": 1
    },
    "q11": {
     "a": "できない",
     "ok": 1
    },
    "q12": {
     "a": "すまいざごうよはお",
     "ok": 1
    },
    "q13": {
     "a": "父子",
     "ok": 1
    },
    "q14": {
     "a": "同じ",
     "ok": 1
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "15文字・2つ",
     "ok": 1
    },
    "q18": {
     "a": "3",
     "ok": 1
    },
    "q19": {
     "a": "0.25, 0.3, 0.33, 1/3",
     "ok": 1
    },
    "q20": {
     "a": "しない",
     "ok": 1
    }
   },
   "raw": "1: 父親  \n2: 10  \n3: しない  \n4: extraordinarily  \n5: 3  \n6: 2  \n7: できない  \n8: 水曜日  \n9: o  \n10: 火星  \n11: できない  \n12: すまいざごうよはお  \n13: 父子  \n14: 同じ  \n15: 5  \n16: 3  \n17: 15文字・2つ  \n18: 3  \n19: 0.25, 0.3, 0.33, 1/3  \n20: しない"
  },
  "bytedance-seed/seed-2.0-lite": {
   "score": 13,
   "answered": 20,
   "first10": 6,
   "last10": 7,
   "inconsistent_pairs": 0,
   "sec": 24.1,
   "out_tokens": 1719,
   "price": "$0.25 / $2",
   "graded": {
    "q01": {
     "a": "少年の母親",
     "ok": 0
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "synchronization",
     "ok": 0
    },
    "q05": {
     "a": "4",
     "ok": 0
    },
    "q06": {
     "a": "3",
     "ok": 0
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "水曜日",
     "ok": 1
    },
    "q09": {
     "a": "o",
     "ok": 1
    },
    "q10": {
     "a": "火星",
     "ok": 1
    },
    "q11": {
     "a": "できない",
     "ok": 1
    },
    "q12": {
     "a": "すまいざごうよはお",
     "ok": 1
    },
    "q13": {
     "a": "母親",
     "ok": 0
    },
    "q14": {
     "a": "同じ",
     "ok": 1
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "16文字、1つ",
     "ok": 0
    },
    "q18": {
     "a": "4",
     "ok": 0
    },
    "q19": {
     "a": "0.25, 0.3, 0.33, 1/3",
     "ok": 1
    },
    "q20": {
     "a": "しない",
     "ok": 1
    }
   },
   "raw": "1: 少年の母親\n2: 10\n3: しない\n4: synchronization\n5: 4\n6: 3\n7: できない\n8: 水曜日\n9: o\n10: 火星\n11: できない\n12: すまいざごうよはお\n13: 母親\n14: 同じ\n15: 5\n16: 3\n17: 16文字、1つ\n18: 4\n19: 0.25, 0.3, 0.33, 1/3\n20: しない"
  },
  "upstage/solar-pro-3": {
   "score": 11,
   "answered": 20,
   "first10": 6,
   "last10": 5,
   "inconsistent_pairs": 2,
   "sec": 12.1,
   "out_tokens": 1734,
   "price": "$0.15 / $0.6",
   "graded": {
    "q01": {
     "a": "母親",
     "ok": 0
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "4",
     "ok": 0
    },
    "q06": {
     "a": "3",
     "ok": 0
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "水曜日",
     "ok": 1
    },
    "q09": {
     "a": "o",
     "ok": 1
    },
    "q10": {
     "a": "火星",
     "ok": 1
    },
    "q11": {
     "a": "できる",
     "ok": 0
    },
    "q12": {
     "a": "すんいざごうはお",
     "ok": 0
    },
    "q13": {
     "a": "父親",
     "ok": 1
    },
    "q14": {
     "a": "変える",
     "ok": 0
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "2",
     "ok": 0
    },
    "q17": {
     "a": "15文字で2つ",
     "ok": 1
    },
    "q18": {
     "a": "4",
     "ok": 0
    },
    "q19": {
     "a": "0.25, 0.3, 0.33, 1/3",
     "ok": 1
    },
    "q20": {
     "a": "しない",
     "ok": 1
    }
   },
   "raw": "1: 母親  \n2: 10  \n3: しない  \n4: extraordinarily  \n5: 4  \n6: 3  \n7: できない  \n8: 水曜日  \n9: o  \n10: 火星  \n11: できる  \n12: すんいざごうはお  \n13: 父親  \n14: 変える  \n15: 5  \n16: 2  \n17: 15文字で2つ  \n18: 4  \n19: 0.25, 0.3, 0.33, 1/3  \n20: しない"
  },
  "writer/palmyra-x5": {
   "score": 8,
   "answered": 20,
   "first10": 5,
   "last10": 3,
   "inconsistent_pairs": 1,
   "sec": 3.0,
   "out_tokens": 149,
   "price": "$0.6 / $6",
   "graded": {
    "q01": {
     "a": "母親",
     "ok": 0
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "3",
     "ok": 1
    },
    "q06": {
     "a": "3",
     "ok": 0
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "水曜日",
     "ok": 1
    },
    "q09": {
     "a": "s",
     "ok": 0
    },
    "q10": {
     "a": "水星",
     "ok": 0
    },
    "q11": {
     "a": "できる",
     "ok": 0
    },
    "q12": {
     "a": "すますようはお",
     "ok": 0
    },
    "q13": {
     "a": "母親",
     "ok": 0
    },
    "q14": {
     "a": "変える",
     "ok": 0
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "14文字,1つ",
     "ok": 0
    },
    "q18": {
     "a": "3",
     "ok": 1
    },
    "q19": {
     "a": "0.25,1/3,0.3,0.33",
     "ok": 0
    },
    "q20": {
     "a": "存在する",
     "ok": 0
    }
   },
   "raw": "1: 母親  \n2: 10  \n3: しない  \n4: extraordinarily  \n5: 3  \n6: 3  \n7: できない  \n8: 水曜日  \n9: s  \n10: 水星  \n11: できる  \n12: すますようはお  \n13: 母親  \n14: 変える  \n15: 5  \n16: 3  \n17: 14文字,1つ  \n18: 3  \n19: 0.25,1/3,0.3,0.33  \n20: 存在する"
  },
  "openai/gpt-5.2-codex": {
   "score": 16,
   "answered": 20,
   "first10": 8,
   "last10": 8,
   "inconsistent_pairs": 1,
   "sec": 33.6,
   "out_tokens": 1511,
   "price": "$1.75 / $14",
   "graded": {
    "q01": {
     "a": "母親",
     "ok": 0
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "3",
     "ok": 1
    },
    "q06": {
     "a": "2",
     "ok": 1
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "水曜日",
     "ok": 1
    },
    "q09": {
     "a": "o",
     "ok": 1
    },
    "q10": {
     "a": "火星",
     "ok": 1
    },
    "q11": {
     "a": "できない",
     "ok": 1
    },
    "q12": {
     "a": "すまいざごうよはお",
     "ok": 1
    },
    "q13": {
     "a": "母親",
     "ok": 0
    },
    "q14": {
     "a": "変える",
     "ok": 0
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "15文字・2語",
     "ok": 1
    },
    "q18": {
     "a": "3",
     "ok": 1
    },
    "q19": {
     "a": "0.25, 0.3, 0.33, 1/3",
     "ok": 1
    },
    "q20": {
     "a": "しない",
     "ok": 1
    }
   },
   "raw": "1: 母親\n2: 10\n3: しない\n4: extraordinarily\n5: 3\n6: 2\n7: できない\n8: 水曜日\n9: o\n10: 火星\n11: できない\n12: すまいざごうよはお\n13: 母親\n14: 変える\n15: 5\n16: 3\n17: 15文字・2語\n18: 3\n19: 0.25, 0.3, 0.33, 1/3\n20: しない"
  },
  "bytedance-seed/seed-1.6-flash": {
   "score": 6,
   "answered": 20,
   "first10": 4,
   "last10": 2,
   "inconsistent_pairs": 0,
   "sec": 8.3,
   "out_tokens": 887,
   "price": "$0.075 / $0.3",
   "graded": {
    "q01": {
     "a": "母親",
     "ok": 0
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "4",
     "ok": 0
    },
    "q06": {
     "a": "3",
     "ok": 0
    },
    "q07": {
     "a": "できる",
     "ok": 0
    },
    "q08": {
     "a": "水曜日",
     "ok": 1
    },
    "q09": {
     "a": "o",
     "ok": 1
    },
    "q10": {
     "a": "金星",
     "ok": 0
    },
    "q11": {
     "a": "できる",
     "ok": 0
    },
    "q12": {
     "a": "すまございようはお",
     "ok": 0
    },
    "q13": {
     "a": "母親",
     "ok": 0
    },
    "q14": {
     "a": "変える",
     "ok": 0
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "2",
     "ok": 0
    },
    "q17": {
     "a": "13文字、2つ",
     "ok": 0
    },
    "q18": {
     "a": "4",
     "ok": 0
    },
    "q19": {
     "a": "0.25, 0.3, 0.33, 1/3",
     "ok": 1
    },
    "q20": {
     "a": "存在する",
     "ok": 0
    }
   },
   "raw": "1: 母親\n2: 10\n3: しない\n4: extraordinarily\n5: 4\n6: 3\n7: できる\n8: 水曜日\n9: o\n10: 金星\n11: できる\n12: すまございようはお\n13: 母親\n14: 変える\n15: 5\n16: 2\n17: 13文字、2つ\n18: 4\n19: 0.25, 0.3, 0.33, 1/3\n20: 存在する"
  },
  "mistralai/devstral-2512": {
   "score": 7,
   "answered": 20,
   "first10": 3,
   "last10": 4,
   "inconsistent_pairs": 0,
   "sec": 2.8,
   "out_tokens": 145,
   "price": "$0.4 / $2",
   "graded": {
    "q01": {
     "a": "母親",
     "ok": 0
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "する",
     "ok": 0
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "3",
     "ok": 1
    },
    "q06": {
     "a": "0",
     "ok": 0
    },
    "q07": {
     "a": "できる",
     "ok": 0
    },
    "q08": {
     "a": "水曜日",
     "ok": 1
    },
    "q09": {
     "a": "p",
     "ok": 0
    },
    "q10": {
     "a": "水星",
     "ok": 0
    },
    "q11": {
     "a": "できる",
     "ok": 0
    },
    "q12": {
     "a": "ます。はちおお",
     "ok": 0
    },
    "q13": {
     "a": "母親",
     "ok": 0
    },
    "q14": {
     "a": "同じ",
     "ok": 1
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "15文字、1つ",
     "ok": 0
    },
    "q18": {
     "a": "3",
     "ok": 1
    },
    "q19": {
     "a": "0.25, 0.3, 1/3, 0.33",
     "ok": 0
    },
    "q20": {
     "a": "存在する",
     "ok": 0
    }
   },
   "raw": "1: 母親\n2: 10\n3: する\n4: extraordinarily\n5: 3\n6: 0\n7: できる\n8: 水曜日\n9: p\n10: 水星\n11: できる\n12: ます。はちおお\n13: 母親\n14: 同じ\n15: 5\n16: 3\n17: 15文字、1つ\n18: 3\n19: 0.25, 0.3, 1/3, 0.33\n20: 存在する"
  },
  "openai/gpt-5.2": {
   "score": 12,
   "answered": 20,
   "first10": 6,
   "last10": 6,
   "inconsistent_pairs": 0,
   "sec": 3.9,
   "out_tokens": 134,
   "price": "$1.75 / $14",
   "graded": {
    "q01": {
     "a": "少年の母親",
     "ok": 0
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "4",
     "ok": 0
    },
    "q06": {
     "a": "2",
     "ok": 1
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "水曜",
     "ok": 1
    },
    "q09": {
     "a": "s",
     "ok": 0
    },
    "q10": {
     "a": "火星",
     "ok": 1
    },
    "q11": {
     "a": "できる",
     "ok": 0
    },
    "q12": {
     "a": "すまいざごうよはお",
     "ok": 1
    },
    "q13": {
     "a": "母親",
     "ok": 0
    },
    "q14": {
     "a": "同じ",
     "ok": 1
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "16文字、2つ",
     "ok": 0
    },
    "q18": {
     "a": "4",
     "ok": 0
    },
    "q19": {
     "a": "0.25,0.3,0.33,1/3",
     "ok": 1
    },
    "q20": {
     "a": "しない",
     "ok": 1
    }
   },
   "raw": "1: 少年の母親\n2: 10\n3: しない\n4: extraordinarily\n5: 4\n6: 2\n7: できない\n8: 水曜\n9: s\n10: 火星\n11: できる\n12: すまいざごうよはお\n13: 母親\n14: 同じ\n15: 5\n16: 3\n17: 16文字、2つ\n18: 4\n19: 0.25,0.3,0.33,1/3\n20: しない"
  },
  "mistralai/ministral-14b-2512": {
   "score": 6,
   "answered": 20,
   "first10": 4,
   "last10": 2,
   "inconsistent_pairs": 0,
   "sec": 3.3,
   "out_tokens": 146,
   "price": "$0.2 / $0.2",
   "graded": {
    "q01": {
     "a": "母親",
     "ok": 0
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "3",
     "ok": 1
    },
    "q06": {
     "a": "2",
     "ok": 1
    },
    "q07": {
     "a": "できる",
     "ok": 0
    },
    "q08": {
     "a": "火曜",
     "ok": 0
    },
    "q09": {
     "a": "s",
     "ok": 0
    },
    "q10": {
     "a": "水星",
     "ok": 0
    },
    "q11": {
     "a": "できる",
     "ok": 0
    },
    "q12": {
     "a": "ますごうはうお",
     "ok": 0
    },
    "q13": {
     "a": "母",
     "ok": 0
    },
    "q14": {
     "a": "変える",
     "ok": 0
    },
    "q15": {
     "a": "10",
     "ok": 0
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "13文字、1つ",
     "ok": 0
    },
    "q18": {
     "a": "3",
     "ok": 1
    },
    "q19": {
     "a": "0.25, 0.3, 1/3, 0.33",
     "ok": 0
    },
    "q20": {
     "a": "存在する",
     "ok": 0
    }
   },
   "raw": "1: 母親\n2: 10\n3: しない\n4: extraordinarily\n5: 3\n6: 2\n7: できる\n8: 火曜\n9: s\n10: 水星\n11: できる\n12: ますごうはうお\n13: 母\n14: 変える\n15: 10\n16: 3\n17: 13文字、1つ\n18: 3\n19: 0.25, 0.3, 1/3, 0.33\n20: 存在する"
  },
  "mistralai/ministral-8b-2512": {
   "score": 4,
   "answered": 20,
   "first10": 3,
   "last10": 1,
   "inconsistent_pairs": 3,
   "sec": 3.3,
   "out_tokens": 142,
   "price": "$0.15 / $0.15",
   "graded": {
    "q01": {
     "a": "少年",
     "ok": 0
    },
    "q02": {
     "a": "1",
     "ok": 0
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "3",
     "ok": 1
    },
    "q06": {
     "a": "0",
     "ok": 0
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "火曜",
     "ok": 0
    },
    "q09": {
     "a": "i",
     "ok": 0
    },
    "q10": {
     "a": "水星",
     "ok": 0
    },
    "q11": {
     "a": "できる",
     "ok": 0
    },
    "q12": {
     "a": "まつごうよ",
     "ok": 0
    },
    "q13": {
     "a": "父",
     "ok": 1
    },
    "q14": {
     "a": "変える",
     "ok": 0
    },
    "q15": {
     "a": "20",
     "ok": 0
    },
    "q16": {
     "a": "2",
     "ok": 0
    },
    "q17": {
     "a": "19文字、1つ",
     "ok": 0
    },
    "q18": {
     "a": "2",
     "ok": 0
    },
    "q19": {
     "a": "0.25, 0.3, 1/3, 0.33",
     "ok": 0
    },
    "q20": {
     "a": "存在する",
     "ok": 0
    }
   },
   "raw": "1. 少年\n2. 1\n3. しない\n4. extraordinarily\n5. 3\n6. 0\n7. できない\n8. 火曜\n9. i\n10. 水星\n11. できる\n12. まつごうよ\n13. 父\n14. 変える\n15. 20\n16. 2\n17. 19文字、1つ\n18. 2\n19. 0.25, 0.3, 1/3, 0.33\n20. 存在する"
  },
  "mistralai/ministral-3b-2512": {
   "score": 5,
   "answered": 20,
   "first10": 2,
   "last10": 3,
   "inconsistent_pairs": 2,
   "sec": 3.4,
   "out_tokens": 259,
   "price": "$0.1 / $0.1",
   "graded": {
    "q01": {
     "a": "番号: 少年が外科医の息子",
     "ok": 0
    },
    "q02": {
     "a": "番号: 20",
     "ok": 0
    },
    "q03": {
     "a": "番号: しない",
     "ok": 1
    },
    "q04": {
     "a": "番号: synchronization (11文字)",
     "ok": 0
    },
    "q05": {
     "a": "番号: 2",
     "ok": 0
    },
    "q06": {
     "a": "番号: 12",
     "ok": 0
    },
    "q07": {
     "a": "番号: できる",
     "ok": 0
    },
    "q08": {
     "a": "番号: 火曜日",
     "ok": 0
    },
    "q09": {
     "a": "番号: e",
     "ok": 0
    },
    "q10": {
     "a": "番号: 火星",
     "ok": 1
    },
    "q11": {
     "a": "番号: できる",
     "ok": 0
    },
    "q12": {
     "a": "番号: すごはお",
     "ok": 0
    },
    "q13": {
     "a": "番号: 父親",
     "ok": 1
    },
    "q14": {
     "a": "番号: 同じ",
     "ok": 1
    },
    "q15": {
     "a": "番号: 10",
     "ok": 0
    },
    "q16": {
     "a": "番号: 2",
     "ok": 0
    },
    "q17": {
     "a": "番号: 19文字、1つ(\"extraordinarily\")",
     "ok": 0
    },
    "q18": {
     "a": "番号: 3",
     "ok": 1
    },
    "q19": {
     "a": "番号: 0.25, 0.3, 1/3, 0.33",
     "ok": 0
    },
    "q20": {
     "a": "番号: 存在する（手順：農夫+キャベツ→オオカミ→農夫+ヤギ→キャベツ→オオカミ→農夫+キャベツ）",
     "ok": 0
    }
   },
   "raw": "1. 番号: 少年が外科医の息子\n2. 番号: 20\n3. 番号: しない\n4. 番号: synchronization (11文字)\n5. 番号: 2\n6. 番号: 12\n7. 番号: できる\n8. 番号: 火曜日\n9. 番号: e\n10. 番号: 火星\n11. 番号: できる\n12. 番号: すごはお\n13. 番号: 父親\n14. 番号: 同じ\n15. 番号: 10\n16. 番号: 2\n17. 番号: 19文字、1つ(\"extraordinarily\")\n18. 番号: 3\n19. 番号: 0.25, 0.3, 1/3, 0.33\n20. 番号: 存在する（手順：農夫+キャベツ→オオカミ→農夫+ヤギ→キャベツ→オオカミ→農夫+キャベツ）"
  },
  "mistralai/mistral-large-2512": {
   "score": 8,
   "answered": 20,
   "first10": 5,
   "last10": 3,
   "inconsistent_pairs": 0,
   "sec": 3.1,
   "out_tokens": 146,
   "price": "$0.5 / $1.5",
   "graded": {
    "q01": {
     "a": "母親",
     "ok": 0
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "4",
     "ok": 0
    },
    "q06": {
     "a": "1",
     "ok": 0
    },
    "q07": {
     "a": "できる",
     "ok": 0
    },
    "q08": {
     "a": "水曜日",
     "ok": 1
    },
    "q09": {
     "a": "o",
     "ok": 1
    },
    "q10": {
     "a": "火星",
     "ok": 1
    },
    "q11": {
     "a": "できる",
     "ok": 0
    },
    "q12": {
     "a": "すはようごおは",
     "ok": 0
    },
    "q13": {
     "a": "母子",
     "ok": 0
    },
    "q14": {
     "a": "同じ",
     "ok": 1
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "15文字, 1つ",
     "ok": 0
    },
    "q18": {
     "a": "4",
     "ok": 0
    },
    "q19": {
     "a": "0.25, 1/3, 0.3, 0.33",
     "ok": 0
    },
    "q20": {
     "a": "存在する",
     "ok": 0
    }
   },
   "raw": "1: 母親\n2: 10\n3: しない\n4: extraordinarily\n5: 4\n6: 1\n7: できる\n8: 水曜日\n9: o\n10: 火星\n11: できる\n12: すはようごおは\n13: 母子\n14: 同じ\n15: 5\n16: 3\n17: 15文字, 1つ\n18: 4\n19: 0.25, 1/3, 0.3, 0.33\n20: 存在する"
  },
  "bytedance-seed/seed-1.6": {
   "score": 14,
   "answered": 20,
   "first10": 7,
   "last10": 7,
   "inconsistent_pairs": 0,
   "sec": 28.6,
   "out_tokens": 1496,
   "price": "$0.25 / $2",
   "graded": {
    "q01": {
     "a": "父親",
     "ok": 1
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "synchronization",
     "ok": 0
    },
    "q05": {
     "a": "4",
     "ok": 0
    },
    "q06": {
     "a": "3",
     "ok": 0
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "水曜日",
     "ok": 1
    },
    "q09": {
     "a": "o",
     "ok": 1
    },
    "q10": {
     "a": "火星",
     "ok": 1
    },
    "q11": {
     "a": "できる",
     "ok": 0
    },
    "q12": {
     "a": "すまいざごうよはお",
     "ok": 1
    },
    "q13": {
     "a": "父子",
     "ok": 1
    },
    "q14": {
     "a": "同じ",
     "ok": 1
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "14文字、1個",
     "ok": 0
    },
    "q18": {
     "a": "4",
     "ok": 0
    },
    "q19": {
     "a": "0.25,0.3,0.33,1/3",
     "ok": 1
    },
    "q20": {
     "a": "しない",
     "ok": 1
    }
   },
   "raw": "1: 父親\n2: 10\n3: しない\n4: synchronization\n5: 4\n6: 3\n7: できない\n8: 水曜日\n9: o\n10: 火星\n11: できる\n12: すまいざごうよはお\n13: 父子\n14: 同じ\n15: 5\n16: 3\n17: 14文字、1個\n18: 4\n19: 0.25,0.3,0.33,1/3\n20: しない"
  },
  "openai/gpt-5.1": {
   "score": 12,
   "answered": 20,
   "first10": 7,
   "last10": 5,
   "inconsistent_pairs": 1,
   "sec": 2.4,
   "out_tokens": 145,
   "price": "$1.25 / $10",
   "graded": {
    "q01": {
     "a": "父親または母親",
     "ok": 1
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "4",
     "ok": 0
    },
    "q06": {
     "a": "2",
     "ok": 1
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "水曜",
     "ok": 1
    },
    "q09": {
     "a": "s",
     "ok": 0
    },
    "q10": {
     "a": "火星",
     "ok": 1
    },
    "q11": {
     "a": "できる",
     "ok": 0
    },
    "q12": {
     "a": "すまいざごうよはお",
     "ok": 1
    },
    "q13": {
     "a": "親子",
     "ok": 0
    },
    "q14": {
     "a": "同じ",
     "ok": 1
    },
    "q15": {
     "a": "10",
     "ok": 0
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "14文字で1つ",
     "ok": 0
    },
    "q18": {
     "a": "4",
     "ok": 0
    },
    "q19": {
     "a": "0.25, 0.3, 0.33, 1/3",
     "ok": 1
    },
    "q20": {
     "a": "しない",
     "ok": 1
    }
   },
   "raw": "1: 父親または母親\n2: 10\n3: しない\n4: extraordinarily\n5: 4\n6: 2\n7: できない\n8: 水曜\n9: s\n10: 火星\n11: できる\n12: すまいざごうよはお\n13: 親子\n14: 同じ\n15: 10\n16: 3\n17: 14文字で1つ\n18: 4\n19: 0.25, 0.3, 0.33, 1/3\n20: しない"
  },
  "amazon/nova-2-lite-v1": {
   "score": 12,
   "answered": 20,
   "first10": 7,
   "last10": 5,
   "inconsistent_pairs": 0,
   "sec": 19.8,
   "out_tokens": 2872,
   "price": "$0.3 / $2.5",
   "graded": {
    "q01": {
     "a": "父親",
     "ok": 1
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "4",
     "ok": 0
    },
    "q06": {
     "a": "2",
     "ok": 1
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "水曜日",
     "ok": 1
    },
    "q09": {
     "a": "o",
     "ok": 1
    },
    "q10": {
     "a": "金星",
     "ok": 0
    },
    "q11": {
     "a": "できる",
     "ok": 0
    },
    "q12": {
     "a": "すまいざごうよはお",
     "ok": 1
    },
    "q13": {
     "a": "父子",
     "ok": 1
    },
    "q14": {
     "a": "同じ",
     "ok": 1
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "2",
     "ok": 0
    },
    "q17": {
     "a": "16,1",
     "ok": 0
    },
    "q18": {
     "a": "4",
     "ok": 0
    },
    "q19": {
     "a": "0.25, 1/3, 0.33, 0.3",
     "ok": 0
    },
    "q20": {
     "a": "しない",
     "ok": 1
    }
   },
   "raw": "1: 父親\n2: 10\n3: しない\n4: extraordinarily\n5: 4\n6: 2\n7: できない\n8: 水曜日\n9: o\n10: 金星\n11: できる\n12: すまいざごうよはお\n13: 父子\n14: 同じ\n15: 5\n16: 2\n17: 16,1\n18: 4\n19: 0.25, 1/3, 0.33, 0.3\n20: しない"
  },
  "amazon/nova-premier-v1": {
   "score": 10,
   "answered": 20,
   "first10": 7,
   "last10": 3,
   "inconsistent_pairs": 0,
   "sec": 2.9,
   "out_tokens": 149,
   "price": "$2.5 / $12.5",
   "graded": {
    "q01": {
     "a": "父親",
     "ok": 1
    },
    "q02": {
     "a": "10円",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "3",
     "ok": 1
    },
    "q06": {
     "a": "2",
     "ok": 1
    },
    "q07": {
     "a": "できる",
     "ok": 0
    },
    "q08": {
     "a": "水曜日",
     "ok": 1
    },
    "q09": {
     "a": "c",
     "ok": 0
    },
    "q10": {
     "a": "火星",
     "ok": 1
    },
    "q11": {
     "a": "できる",
     "ok": 0
    },
    "q12": {
     "a": "すございますおはよう",
     "ok": 0
    },
    "q13": {
     "a": "父親",
     "ok": 1
    },
    "q14": {
     "a": "変える",
     "ok": 0
    },
    "q15": {
     "a": "10年",
     "ok": 0
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "14文字、1つ",
     "ok": 0
    },
    "q18": {
     "a": "3",
     "ok": 1
    },
    "q19": {
     "a": "0.25, 0.3, 1/3, 0.33",
     "ok": 0
    },
    "q20": {
     "a": "存在する",
     "ok": 0
    }
   },
   "raw": "1. 父親\n2. 10円\n3. しない\n4. extraordinarily\n5. 3\n6. 2\n7. できる\n8. 水曜日\n9. c\n10. 火星\n11. できる\n12. すございますおはよう\n13. 父親\n14. 変える\n15. 10年\n16. 3\n17. 14文字、1つ\n18. 3\n19. 0.25, 0.3, 1/3, 0.33\n20. 存在する"
  },
  "qwen/qwen3-vl-32b-instruct": {
   "score": 7,
   "answered": 20,
   "first10": 4,
   "last10": 3,
   "inconsistent_pairs": 0,
   "sec": 3.9,
   "out_tokens": 159,
   "price": "$0.104 / $0.416",
   "graded": {
    "q01": {
     "a": "外科医は母親の夫（父親）",
     "ok": 1
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "synchronization",
     "ok": 0
    },
    "q05": {
     "a": "4",
     "ok": 0
    },
    "q06": {
     "a": "3",
     "ok": 0
    },
    "q07": {
     "a": "できる",
     "ok": 0
    },
    "q08": {
     "a": "土曜日",
     "ok": 0
    },
    "q09": {
     "a": "o",
     "ok": 1
    },
    "q10": {
     "a": "水星",
     "ok": 0
    },
    "q11": {
     "a": "できる",
     "ok": 0
    },
    "q12": {
     "a": "すいごうあおは",
     "ok": 0
    },
    "q13": {
     "a": "父親",
     "ok": 1
    },
    "q14": {
     "a": "同じ",
     "ok": 1
    },
    "q15": {
     "a": "10",
     "ok": 0
    },
    "q16": {
     "a": "4",
     "ok": 0
    },
    "q17": {
     "a": "14文字、1つ",
     "ok": 0
    },
    "q18": {
     "a": "2",
     "ok": 0
    },
    "q19": {
     "a": "0.25, 0.3, 0.33, 1/3",
     "ok": 1
    },
    "q20": {
     "a": "存在する",
     "ok": 0
    }
   },
   "raw": "1: 外科医は母親の夫（父親）  \n2: 10  \n3: しない  \n4: synchronization  \n5: 4  \n6: 3  \n7: できる  \n8: 土曜日  \n9: o  \n10: 水星  \n11: できる  \n12: すいごうあおは  \n13: 父親  \n14: 同じ  \n15: 10  \n16: 4  \n17: 14文字、1つ  \n18: 2  \n19: 0.25, 0.3, 0.33, 1/3  \n20: 存在する"
  },
  "ibm-granite/granite-4.0-h-micro": {
   "score": 6,
   "answered": 20,
   "first10": 3,
   "last10": 3,
   "inconsistent_pairs": 2,
   "sec": 5.0,
   "out_tokens": 128,
   "price": "$0.017 / $0.112",
   "graded": {
    "q01": {
     "a": "外科医",
     "ok": 0
    },
    "q02": {
     "a": "20",
     "ok": 0
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "2",
     "ok": 0
    },
    "q06": {
     "a": "6",
     "ok": 0
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "Wednesday",
     "ok": 0
    },
    "q09": {
     "a": "o",
     "ok": 1
    },
    "q10": {
     "a": "Earth",
     "ok": 0
    },
    "q11": {
     "a": "できる",
     "ok": 0
    },
    "q12": {
     "a": "ですございます",
     "ok": 0
    },
    "q13": {
     "a": "父",
     "ok": 1
    },
    "q14": {
     "a": "同じ",
     "ok": 1
    },
    "q15": {
     "a": "10",
     "ok": 0
    },
    "q16": {
     "a": "1",
     "ok": 0
    },
    "q17": {
     "a": "complicated",
     "ok": 0
    },
    "q18": {
     "a": "2",
     "ok": 0
    },
    "q19": {
     "a": "0.25, 0.3, 0.33, 1/3",
     "ok": 1
    },
    "q20": {
     "a": "存在する",
     "ok": 0
    }
   },
   "raw": "1: 外科医\n2: 20\n3: しない\n4: extraordinarily\n5: 2\n6: 6\n7: できない\n8: Wednesday\n9: o\n10: Earth\n11: できる\n12: ですございます\n13: 父\n14: 同じ\n15: 10\n16: 1\n17: complicated\n18: 2\n19: 0.25, 0.3, 0.33, 1/3\n20: 存在する"
  },
  "qwen/qwen3-vl-8b-instruct": {
   "score": 7,
   "answered": 20,
   "first10": 4,
   "last10": 3,
   "inconsistent_pairs": 1,
   "sec": 2.2,
   "out_tokens": 147,
   "price": "$0.117 / $0.455",
   "graded": {
    "q01": {
     "a": "母親の夫",
     "ok": 0
    },
    "q02": {
     "a": "5",
     "ok": 0
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "3",
     "ok": 1
    },
    "q06": {
     "a": "3",
     "ok": 0
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "月曜",
     "ok": 0
    },
    "q09": {
     "a": "o",
     "ok": 1
    },
    "q10": {
     "a": "金星",
     "ok": 0
    },
    "q11": {
     "a": "できる",
     "ok": 0
    },
    "q12": {
     "a": "すございますおはよう",
     "ok": 0
    },
    "q13": {
     "a": "母",
     "ok": 0
    },
    "q14": {
     "a": "変える",
     "ok": 0
    },
    "q15": {
     "a": "10",
     "ok": 0
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "15, 1",
     "ok": 0
    },
    "q18": {
     "a": "3",
     "ok": 1
    },
    "q19": {
     "a": "0.25, 0.3, 0.33, 1/3",
     "ok": 1
    },
    "q20": {
     "a": "存在する",
     "ok": 0
    }
   },
   "raw": "1: 母親の夫  \n2: 5  \n3: しない  \n4: extraordinarily  \n5: 3  \n6: 3  \n7: できない  \n8: 月曜  \n9: o  \n10: 金星  \n11: できる  \n12: すございますおはよう  \n13: 母  \n14: 変える  \n15: 10  \n16: 3  \n17: 15, 1  \n18: 3  \n19: 0.25, 0.3, 0.33, 1/3  \n20: 存在する"
  },
  "openai/gpt-5.1-codex": {
   "score": 18,
   "answered": 20,
   "first10": 9,
   "last10": 9,
   "inconsistent_pairs": 0,
   "sec": 34.9,
   "out_tokens": 1851,
   "price": "$1.25 / $10",
   "graded": {
    "q01": {
     "a": "父親",
     "ok": 1
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "3",
     "ok": 1
    },
    "q06": {
     "a": "2",
     "ok": 1
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "水曜",
     "ok": 1
    },
    "q09": {
     "a": "o",
     "ok": 1
    },
    "q10": {
     "a": "火星",
     "ok": 1
    },
    "q11": {
     "a": "できない",
     "ok": 1
    },
    "q12": {
     "a": "すまいざごうよはお",
     "ok": 1
    },
    "q13": {
     "a": "父親",
     "ok": 1
    },
    "q14": {
     "a": "同じ",
     "ok": 1
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "15文字・1語",
     "ok": 0
    },
    "q18": {
     "a": "3",
     "ok": 1
    },
    "q19": {
     "a": "0.25 < 0.3 < 0.33 < 1/3",
     "ok": 1
    },
    "q20": {
     "a": "しない",
     "ok": 1
    }
   },
   "raw": "1: 父親\n2: 10\n3: しない\n4: extraordinarily\n5: 3\n6: 2\n7: できない\n8: 水曜\n9: o\n10: 火星\n11: できない\n12: すまいざごうよはお\n13: 父親\n14: 同じ\n15: 5\n16: 3\n17: 15文字・1語\n18: 3\n19: 0.25 < 0.3 < 0.33 < 1/3\n20: しない"
  },
  "anthropic/claude-haiku-4.5": {
   "score": 15,
   "answered": 20,
   "first10": 8,
   "last10": 7,
   "inconsistent_pairs": 0,
   "sec": 28.1,
   "out_tokens": 3783,
   "price": "$1 / $5",
   "graded": {
    "q01": {
     "a": "父親",
     "ok": 1
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "synchronization",
     "ok": 0
    },
    "q05": {
     "a": "3",
     "ok": 1
    },
    "q06": {
     "a": "2",
     "ok": 1
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "水曜日",
     "ok": 1
    },
    "q09": {
     "a": "o",
     "ok": 1
    },
    "q10": {
     "a": "水星",
     "ok": 0
    },
    "q11": {
     "a": "できる",
     "ok": 0
    },
    "q12": {
     "a": "すまいざごうよはお",
     "ok": 1
    },
    "q13": {
     "a": "父親",
     "ok": 1
    },
    "q14": {
     "a": "変える",
     "ok": 0
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "16文字、1個",
     "ok": 0
    },
    "q18": {
     "a": "3",
     "ok": 1
    },
    "q19": {
     "a": "0.25, 0.3, 0.33, 1/3",
     "ok": 1
    },
    "q20": {
     "a": "しない",
     "ok": 1
    }
   },
   "raw": "1: 父親\n2: 10\n3: しない\n4: synchronization\n5: 3\n6: 2\n7: できない\n8: 水曜日\n9: o\n10: 水星\n11: できる\n12: すまいざごうよはお\n13: 父親\n14: 変える\n15: 5\n16: 3\n17: 16文字、1個\n18: 3\n19: 0.25, 0.3, 0.33, 1/3\n20: しない"
  },
  "qwen/qwen3-vl-30b-a3b-instruct": {
   "score": 6,
   "answered": 20,
   "first10": 4,
   "last10": 2,
   "inconsistent_pairs": 0,
   "sec": 2.5,
   "out_tokens": 156,
   "price": "$0.15 / $0.6",
   "graded": {
    "q01": {
     "a": "外科医は母親",
     "ok": 0
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "synchronization",
     "ok": 0
    },
    "q05": {
     "a": "2",
     "ok": 0
    },
    "q06": {
     "a": "3",
     "ok": 0
    },
    "q07": {
     "a": "できる",
     "ok": 0
    },
    "q08": {
     "a": "水曜日",
     "ok": 1
    },
    "q09": {
     "a": "o",
     "ok": 1
    },
    "q10": {
     "a": "金星",
     "ok": 0
    },
    "q11": {
     "a": "できる",
     "ok": 0
    },
    "q12": {
     "a": "すまそなようはお",
     "ok": 0
    },
    "q13": {
     "a": "執刀医は母親",
     "ok": 0
    },
    "q14": {
     "a": "変える",
     "ok": 0
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "2",
     "ok": 0
    },
    "q17": {
     "a": "16文字、1つ",
     "ok": 0
    },
    "q18": {
     "a": "2",
     "ok": 0
    },
    "q19": {
     "a": "0.25, 0.3, 0.33, 1/3",
     "ok": 1
    },
    "q20": {
     "a": "存在する",
     "ok": 0
    }
   },
   "raw": "1: 外科医は母親  \n2: 10  \n3: しない  \n4: synchronization  \n5: 2  \n6: 3  \n7: できる  \n8: 水曜日  \n9: o  \n10: 金星  \n11: できる  \n12: すまそなようはお  \n13: 執刀医は母親  \n14: 変える  \n15: 5  \n16: 2  \n17: 16文字、1つ  \n18: 2  \n19: 0.25, 0.3, 0.33, 1/3  \n20: 存在する"
  },
  "thedrummer/cydonia-24b-v4.1": {
   "score": 7,
   "answered": 20,
   "first10": 4,
   "last10": 3,
   "inconsistent_pairs": 1,
   "sec": 4.7,
   "out_tokens": 151,
   "price": "$0.3 / $0.5",
   "graded": {
    "q01": {
     "a": "母親",
     "ok": 0
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "3",
     "ok": 1
    },
    "q06": {
     "a": "3",
     "ok": 0
    },
    "q07": {
     "a": "できる",
     "ok": 0
    },
    "q08": {
     "a": "水曜日",
     "ok": 1
    },
    "q09": {
     "a": "s",
     "ok": 0
    },
    "q10": {
     "a": "水星",
     "ok": 0
    },
    "q11": {
     "a": "できる",
     "ok": 0
    },
    "q12": {
     "a": "す ご う よ う は お",
     "ok": 0
    },
    "q13": {
     "a": "父と息子",
     "ok": 1
    },
    "q14": {
     "a": "変える",
     "ok": 0
    },
    "q15": {
     "a": "10",
     "ok": 0
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "17文字、1つ",
     "ok": 0
    },
    "q18": {
     "a": "3",
     "ok": 1
    },
    "q19": {
     "a": "0.25, 0.3, 1/3, 0.33",
     "ok": 0
    },
    "q20": {
     "a": "存在する",
     "ok": 0
    }
   },
   "raw": "1: 母親\n2: 10\n3: しない\n4: extraordinarily\n5: 3\n6: 3\n7: できる\n8: 水曜日\n9: s\n10: 水星\n11: できる\n12: す ご う よ う は お\n13: 父と息子\n14: 変える\n15: 10\n16: 3\n17: 17文字、1つ\n18: 3\n19: 0.25, 0.3, 1/3, 0.33\n20: 存在する"
  },
  "qwen/qwen3-coder-plus": {
   "score": 9,
   "answered": 20,
   "first10": 4,
   "last10": 5,
   "inconsistent_pairs": 0,
   "sec": 4.1,
   "out_tokens": 147,
   "price": "$0.65 / $3.25",
   "graded": {
    "q01": {
     "a": "母親",
     "ok": 0
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "4",
     "ok": 0
    },
    "q06": {
     "a": "3",
     "ok": 0
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "水曜日",
     "ok": 1
    },
    "q09": {
     "a": "s",
     "ok": 0
    },
    "q10": {
     "a": "水星",
     "ok": 0
    },
    "q11": {
     "a": "できない",
     "ok": 1
    },
    "q12": {
     "a": "すにがもはお",
     "ok": 0
    },
    "q13": {
     "a": "母",
     "ok": 0
    },
    "q14": {
     "a": "変える",
     "ok": 0
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "17文字、1つ",
     "ok": 0
    },
    "q18": {
     "a": "4",
     "ok": 0
    },
    "q19": {
     "a": "0.25, 0.3, 0.33, 1/3",
     "ok": 1
    },
    "q20": {
     "a": "しない",
     "ok": 1
    }
   },
   "raw": "1: 母親  \n2: 10  \n3: しない  \n4: extraordinarily  \n5: 4  \n6: 3  \n7: できない  \n8: 水曜日  \n9: s  \n10: 水星  \n11: できない  \n12: すにがもはお  \n13: 母  \n14: 変える  \n15: 5  \n16: 3  \n17: 17文字、1つ  \n18: 4  \n19: 0.25, 0.3, 0.33, 1/3  \n20: しない"
  },
  "anthropic/claude-sonnet-4.5": {
   "score": 15,
   "answered": 20,
   "first10": 8,
   "last10": 7,
   "inconsistent_pairs": 1,
   "sec": 14.5,
   "out_tokens": 835,
   "price": "$3 / $15",
   "graded": {
    "q01": {
     "a": "父親",
     "ok": 1
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "4",
     "ok": 0
    },
    "q06": {
     "a": "2",
     "ok": 1
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "水曜日",
     "ok": 1
    },
    "q09": {
     "a": "o",
     "ok": 1
    },
    "q10": {
     "a": "火星",
     "ok": 1
    },
    "q11": {
     "a": "できる",
     "ok": 0
    },
    "q12": {
     "a": "すまいござうよはお",
     "ok": 0
    },
    "q13": {
     "a": "父親",
     "ok": 1
    },
    "q14": {
     "a": "同じ",
     "ok": 1
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "15文字で2つ",
     "ok": 1
    },
    "q18": {
     "a": "4",
     "ok": 0
    },
    "q19": {
     "a": "0.25, 0.3, 0.33, 1/3",
     "ok": 1
    },
    "q20": {
     "a": "しない",
     "ok": 1
    }
   },
   "raw": "1: 父親\n2: 10\n3: しない\n4: extraordinarily\n5: 4\n6: 2\n7: できない\n8: 水曜日\n9: o\n10: 火星\n11: できる\n12: すまいござうよはお\n13: 父親\n14: 同じ\n15: 5\n16: 3\n17: 15文字で2つ\n18: 4\n19: 0.25, 0.3, 0.33, 1/3\n20: しない"
  },
  "qwen/qwen3-vl-235b-a22b-instruct": {
   "score": 7,
   "answered": 20,
   "first10": 3,
   "last10": 4,
   "inconsistent_pairs": 0,
   "sec": 3.8,
   "out_tokens": 145,
   "price": "$0.21 / $1.9",
   "graded": {
    "q01": {
     "a": "母親",
     "ok": 0
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "4",
     "ok": 0
    },
    "q06": {
     "a": "0",
     "ok": 0
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "土曜日",
     "ok": 0
    },
    "q09": {
     "a": "s",
     "ok": 0
    },
    "q10": {
     "a": "水星",
     "ok": 0
    },
    "q11": {
     "a": "できない",
     "ok": 1
    },
    "q12": {
     "a": "ございますようはお",
     "ok": 0
    },
    "q13": {
     "a": "母",
     "ok": 0
    },
    "q14": {
     "a": "同じ",
     "ok": 1
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "4",
     "ok": 0
    },
    "q17": {
     "a": "13文字、1つ",
     "ok": 0
    },
    "q18": {
     "a": "4",
     "ok": 0
    },
    "q19": {
     "a": "0.25, 1/3, 0.3, 0.33",
     "ok": 0
    },
    "q20": {
     "a": "しない",
     "ok": 1
    }
   },
   "raw": "1: 母親\n2: 10\n3: しない\n4: extraordinarily\n5: 4\n6: 0\n7: できない\n8: 土曜日\n9: s\n10: 水星\n11: できない\n12: ございますようはお\n13: 母\n14: 同じ\n15: 5\n16: 4\n17: 13文字、1つ\n18: 4\n19: 0.25, 1/3, 0.3, 0.33\n20: しない"
  },
  "qwen/qwen3-coder-flash": {
   "score": 7,
   "answered": 20,
   "first10": 4,
   "last10": 3,
   "inconsistent_pairs": 1,
   "sec": 3.0,
   "out_tokens": 153,
   "price": "$0.195 / $0.975",
   "graded": {
    "q01": {
     "a": "母親",
     "ok": 0
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "4",
     "ok": 0
    },
    "q06": {
     "a": "3",
     "ok": 0
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "水曜日",
     "ok": 1
    },
    "q09": {
     "a": "s",
     "ok": 0
    },
    "q10": {
     "a": "金星",
     "ok": 0
    },
    "q11": {
     "a": "できる",
     "ok": 0
    },
    "q12": {
     "a": "すみどうはおはようございまし",
     "ok": 0
    },
    "q13": {
     "a": "母亲",
     "ok": 0
    },
    "q14": {
     "a": "変える",
     "ok": 0
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "13文字、1つ",
     "ok": 0
    },
    "q18": {
     "a": "4",
     "ok": 0
    },
    "q19": {
     "a": "0.25, 0.3, 0.33, 1/3",
     "ok": 1
    },
    "q20": {
     "a": "存在する",
     "ok": 0
    }
   },
   "raw": "1: 母親  \n2: 10  \n3: しない  \n4: extraordinarily  \n5: 4  \n6: 3  \n7: できない  \n8: 水曜日  \n9: s  \n10: 金星  \n11: できる  \n12: すみどうはおはようございまし  \n13: 母亲  \n14: 変える  \n15: 5  \n16: 3  \n17: 13文字、1つ  \n18: 4  \n19: 0.25, 0.3, 0.33, 1/3  \n20: 存在する"
  },
  "qwen/qwen3-next-80b-a3b-instruct": {
   "score": 5,
   "answered": 20,
   "first10": 2,
   "last10": 3,
   "inconsistent_pairs": 0,
   "sec": 1.6,
   "out_tokens": 147,
   "price": "$0.09 / $1.1",
   "graded": {
    "q01": {
     "a": "母親",
     "ok": 0
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "4",
     "ok": 0
    },
    "q06": {
     "a": "3",
     "ok": 0
    },
    "q07": {
     "a": "できる",
     "ok": 0
    },
    "q08": {
     "a": "木曜",
     "ok": 0
    },
    "q09": {
     "a": "d",
     "ok": 0
    },
    "q10": {
     "a": "マーズ",
     "ok": 0
    },
    "q11": {
     "a": "できる",
     "ok": 0
    },
    "q12": {
     "a": "すですははようお",
     "ok": 0
    },
    "q13": {
     "a": "母親",
     "ok": 0
    },
    "q14": {
     "a": "同じ",
     "ok": 1
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "4",
     "ok": 0
    },
    "q17": {
     "a": "15文字、1つ",
     "ok": 0
    },
    "q18": {
     "a": "4",
     "ok": 0
    },
    "q19": {
     "a": "0.25, 0.3, 0.33, 1/3",
     "ok": 1
    },
    "q20": {
     "a": "存在する",
     "ok": 0
    }
   },
   "raw": "1: 母親  \n2: 10  \n3: しない  \n4: extraordinarily  \n5: 4  \n6: 3  \n7: できる  \n8: 木曜  \n9: d  \n10: マーズ  \n11: できる  \n12: すですははようお  \n13: 母親  \n14: 同じ  \n15: 5  \n16: 4  \n17: 15文字、1つ  \n18: 4  \n19: 0.25, 0.3, 0.33, 1/3  \n20: 存在する"
  },
  "qwen/qwen-plus-2025-07-28": {
   "score": 8,
   "answered": 20,
   "first10": 5,
   "last10": 3,
   "inconsistent_pairs": 1,
   "sec": 2.8,
   "out_tokens": 156,
   "price": "$0.26 / $0.78",
   "graded": {
    "q01": {
     "a": "母親",
     "ok": 0
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "synchronization",
     "ok": 0
    },
    "q05": {
     "a": "3",
     "ok": 1
    },
    "q06": {
     "a": "3",
     "ok": 0
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "水曜日",
     "ok": 1
    },
    "q09": {
     "a": "s",
     "ok": 0
    },
    "q10": {
     "a": "水星",
     "ok": 0
    },
    "q11": {
     "a": "できる",
     "ok": 0
    },
    "q12": {
     "a": "す　は　ご　う　よ　は　お",
     "ok": 0
    },
    "q13": {
     "a": "息子の母",
     "ok": 0
    },
    "q14": {
     "a": "変える",
     "ok": 0
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "15文字,1つ",
     "ok": 0
    },
    "q18": {
     "a": "3",
     "ok": 1
    },
    "q19": {
     "a": "0.25,1/3,0.3,0.33",
     "ok": 0
    },
    "q20": {
     "a": "存在する",
     "ok": 0
    }
   },
   "raw": "1: 母親  \n2: 10  \n3: しない  \n4: synchronization  \n5: 3  \n6: 3  \n7: できない  \n8: 水曜日  \n9: s  \n10: 水星  \n11: できる  \n12: す　は　ご　う　よ　は　お  \n13: 息子の母  \n14: 変える  \n15: 5  \n16: 3  \n17: 15文字,1つ  \n18: 3  \n19: 0.25,1/3,0.3,0.33  \n20: 存在する"
  },
  "mistralai/mistral-medium-3.1": {
   "score": 7,
   "answered": 20,
   "first10": 3,
   "last10": 4,
   "inconsistent_pairs": 0,
   "sec": 2.4,
   "out_tokens": 145,
   "price": "$0.4 / $2",
   "graded": {
    "q01": {
     "a": "母親",
     "ok": 0
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "する",
     "ok": 0
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "3",
     "ok": 1
    },
    "q06": {
     "a": "0",
     "ok": 0
    },
    "q07": {
     "a": "できる",
     "ok": 0
    },
    "q08": {
     "a": "水曜日",
     "ok": 1
    },
    "q09": {
     "a": "p",
     "ok": 0
    },
    "q10": {
     "a": "水星",
     "ok": 0
    },
    "q11": {
     "a": "できる",
     "ok": 0
    },
    "q12": {
     "a": "ますはちおはよう",
     "ok": 0
    },
    "q13": {
     "a": "母親",
     "ok": 0
    },
    "q14": {
     "a": "同じ",
     "ok": 1
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "15文字、1つ",
     "ok": 0
    },
    "q18": {
     "a": "3",
     "ok": 1
    },
    "q19": {
     "a": "0.25, 0.3, 1/3, 0.33",
     "ok": 0
    },
    "q20": {
     "a": "存在する",
     "ok": 0
    }
   },
   "raw": "1: 母親\n2: 10\n3: する\n4: extraordinarily\n5: 3\n6: 0\n7: できる\n8: 水曜日\n9: p\n10: 水星\n11: できる\n12: ますはちおはよう\n13: 母親\n14: 同じ\n15: 5\n16: 3\n17: 15文字、1つ\n18: 3\n19: 0.25, 0.3, 1/3, 0.33\n20: 存在する"
  },
  "moonshotai/kimi-k2-0905": {
   "score": 8,
   "answered": 20,
   "first10": 4,
   "last10": 4,
   "inconsistent_pairs": 0,
   "sec": 5.4,
   "out_tokens": 136,
   "price": "$0.6 / $2.5",
   "graded": {
    "q01": {
     "a": "母親",
     "ok": 0
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "4",
     "ok": 0
    },
    "q06": {
     "a": "3",
     "ok": 0
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "水曜日",
     "ok": 1
    },
    "q09": {
     "a": "s",
     "ok": 0
    },
    "q10": {
     "a": "金星",
     "ok": 0
    },
    "q11": {
     "a": "できる",
     "ok": 0
    },
    "q12": {
     "a": "すまいざいうごうやはお",
     "ok": 0
    },
    "q13": {
     "a": "母親",
     "ok": 0
    },
    "q14": {
     "a": "同じ",
     "ok": 1
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "4",
     "ok": 0
    },
    "q17": {
     "a": "14文字、1つ",
     "ok": 0
    },
    "q18": {
     "a": "4",
     "ok": 0
    },
    "q19": {
     "a": "0.25, 0.3, 0.33, 1/3",
     "ok": 1
    },
    "q20": {
     "a": "しない",
     "ok": 1
    }
   },
   "raw": "1: 母親\n2: 10\n3: しない\n4: extraordinarily\n5: 4\n6: 3\n7: できない\n8: 水曜日\n9: s\n10: 金星\n11: できる\n12: すまいざいうごうやはお\n13: 母親\n14: 同じ\n15: 5\n16: 4\n17: 14文字、1つ\n18: 4\n19: 0.25, 0.3, 0.33, 1/3\n20: しない"
  },
  "qwen/qwen3-vl-30b-a3b-thinking": {
   "score": 10,
   "answered": 20,
   "first10": 6,
   "last10": 4,
   "inconsistent_pairs": 0,
   "sec": 25.9,
   "out_tokens": 3241,
   "price": "$0.2 / $2.4",
   "graded": {
    "q01": {
     "a": "母親",
     "ok": 0
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "4",
     "ok": 0
    },
    "q06": {
     "a": "2",
     "ok": 1
    },
    "q07": {
     "a": "できる",
     "ok": 0
    },
    "q08": {
     "a": "水曜日",
     "ok": 1
    },
    "q09": {
     "a": "O",
     "ok": 1
    },
    "q10": {
     "a": "火星",
     "ok": 1
    },
    "q11": {
     "a": "できる",
     "ok": 0
    },
    "q12": {
     "a": "すまいざごうよはお",
     "ok": 1
    },
    "q13": {
     "a": "母親",
     "ok": 0
    },
    "q14": {
     "a": "変える",
     "ok": 0
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "15, 1",
     "ok": 0
    },
    "q18": {
     "a": "4",
     "ok": 0
    },
    "q19": {
     "a": "0.25, 0.3, 0.33, 1/3",
     "ok": 1
    },
    "q20": {
     "a": "存在する",
     "ok": 0
    }
   },
   "raw": "1: 母親\n2: 10\n3: しない\n4: extraordinarily\n5: 4\n6: 2\n7: できる\n8: 水曜日\n9: O\n10: 火星\n11: できる\n12: すまいざごうよはお\n13: 母親\n14: 変える\n15: 5\n16: 3\n17: 15, 1\n18: 4\n19: 0.25, 0.3, 0.33, 1/3\n20: 存在する"
  },
  "openai/gpt-5-mini": {
   "score": 19,
   "answered": 20,
   "first10": 9,
   "last10": 10,
   "inconsistent_pairs": 1,
   "sec": 75.1,
   "out_tokens": 4807,
   "price": "$0.25 / $2",
   "graded": {
    "q01": {
     "a": "父",
     "ok": 1
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "3",
     "ok": 1
    },
    "q06": {
     "a": "2",
     "ok": 1
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "水曜日",
     "ok": 1
    },
    "q09": {
     "a": "o",
     "ok": 1
    },
    "q10": {
     "a": "火星",
     "ok": 1
    },
    "q11": {
     "a": "できない",
     "ok": 1
    },
    "q12": {
     "a": "すまいざごうよはお",
     "ok": 1
    },
    "q13": {
     "a": "父",
     "ok": 1
    },
    "q14": {
     "a": "同じ",
     "ok": 1
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "15,2",
     "ok": 1
    },
    "q18": {
     "a": "3",
     "ok": 1
    },
    "q19": {
     "a": "0.25,0.3,0.33,1/3",
     "ok": 1
    },
    "q20": {
     "a": "しない",
     "ok": 1
    }
   },
   "raw": "1: 父\n2: 10\n3: しない\n4: extraordinarily\n5: 3\n6: 2\n7: できない\n8: 水曜日\n9: o\n10: 火星\n11: できない\n12: すまいざごうよはお\n13: 父\n14: 同じ\n15: 5\n16: 3\n17: 15,2\n18: 3\n19: 0.25,0.3,0.33,1/3\n20: しない"
  },
  "openai/gpt-5": {
   "score": 6,
   "answered": 20,
   "first10": 4,
   "last10": 2,
   "inconsistent_pairs": 1,
   "sec": 78.7,
   "out_tokens": 123,
   "price": "$1.25 / $10",
   "graded": {
    "q01": {
     "a": "母親",
     "ok": 0
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "synchronization",
     "ok": 0
    },
    "q05": {
     "a": "4",
     "ok": 0
    },
    "q06": {
     "a": "2",
     "ok": 1
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "木曜日",
     "ok": 0
    },
    "q09": {
     "a": "c",
     "ok": 0
    },
    "q10": {
     "a": "水星",
     "ok": 0
    },
    "q11": {
     "a": "できる",
     "ok": 0
    },
    "q12": {
     "a": "ますございますようお",
     "ok": 0
    },
    "q13": {
     "a": "母親",
     "ok": 0
    },
    "q14": {
     "a": "変える",
     "ok": 0
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "14, 1",
     "ok": 0
    },
    "q18": {
     "a": "4",
     "ok": 0
    },
    "q19": {
     "a": "0.25, 0.3, 1/3, 0.33",
     "ok": 0
    },
    "q20": {
     "a": "存在する",
     "ok": 0
    }
   },
   "raw": "1: 母親\n2: 10\n3: しない\n4: synchronization\n5: 4\n6: 2\n7: できない\n8: 木曜日\n9: c\n10: 水星\n11: できる\n12: ますございますようお\n13: 母親\n14: 変える\n15: 5\n16: 3\n17: 14, 1\n18: 4\n19: 0.25, 0.3, 1/3, 0.33\n20: 存在する"
  },
  "qwen/qwen3-30b-a3b-thinking-2507": {
   "score": 13,
   "answered": 20,
   "first10": 7,
   "last10": 6,
   "inconsistent_pairs": 0,
   "sec": 26.0,
   "out_tokens": 4366,
   "price": "$0.2 / $2.4",
   "graded": {
    "q01": {
     "a": "母",
     "ok": 0
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "4",
     "ok": 0
    },
    "q06": {
     "a": "2",
     "ok": 1
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "水曜日",
     "ok": 1
    },
    "q09": {
     "a": "o",
     "ok": 1
    },
    "q10": {
     "a": "火星",
     "ok": 1
    },
    "q11": {
     "a": "できない",
     "ok": 1
    },
    "q12": {
     "a": "すまいざうごうよはお",
     "ok": 0
    },
    "q13": {
     "a": "母",
     "ok": 0
    },
    "q14": {
     "a": "同じ",
     "ok": 1
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "15 1",
     "ok": 0
    },
    "q18": {
     "a": "4",
     "ok": 0
    },
    "q19": {
     "a": "0.25,0.3,0.33,1/3",
     "ok": 1
    },
    "q20": {
     "a": "しない",
     "ok": 1
    }
   },
   "raw": "1: 母\n2: 10\n3: しない\n4: extraordinarily\n5: 4\n6: 2\n7: できない\n8: 水曜日\n9: o\n10: 火星\n11: できない\n12: すまいざうごうよはお\n13: 母\n14: 同じ\n15: 5\n16: 3\n17: 15 1\n18: 4\n19: 0.25,0.3,0.33,1/3\n20: しない"
  },
  "moonshotai/kimi-k2-thinking": {
   "score": 15,
   "answered": 20,
   "first10": 7,
   "last10": 8,
   "inconsistent_pairs": 1,
   "sec": 13.8,
   "out_tokens": 2845,
   "price": "$0.6 / $2.5",
   "graded": {
    "q01": {
     "a": "母",
     "ok": 0
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "synchronization",
     "ok": 0
    },
    "q05": {
     "a": "3",
     "ok": 1
    },
    "q06": {
     "a": "2",
     "ok": 1
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "水曜日",
     "ok": 1
    },
    "q09": {
     "a": "o",
     "ok": 1
    },
    "q10": {
     "a": "金星",
     "ok": 0
    },
    "q11": {
     "a": "できない",
     "ok": 1
    },
    "q12": {
     "a": "すまいざいごうよはお",
     "ok": 0
    },
    "q13": {
     "a": "父",
     "ok": 1
    },
    "q14": {
     "a": "同じ",
     "ok": 1
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "14文字,1個",
     "ok": 0
    },
    "q18": {
     "a": "3",
     "ok": 1
    },
    "q19": {
     "a": "0.25,0.3,0.33,1/3",
     "ok": 1
    },
    "q20": {
     "a": "しない",
     "ok": 1
    }
   },
   "raw": "1: 母\n2: 10\n3: しない\n4: synchronization\n5: 3\n6: 2\n7: できない\n8: 水曜日\n9: o\n10: 金星\n11: できない\n12: すまいざいごうよはお\n13: 父\n14: 同じ\n15: 5\n16: 3\n17: 14文字,1個\n18: 3\n19: 0.25,0.3,0.33,1/3\n20: しない"
  },
  "openai/gpt-oss-120b": {
   "score": 16,
   "answered": 20,
   "first10": 8,
   "last10": 8,
   "inconsistent_pairs": 2,
   "sec": 20.9,
   "out_tokens": 1130,
   "price": "$0.037 / $0.17",
   "graded": {
    "q01": {
     "a": "母",
     "ok": 0
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "3",
     "ok": 1
    },
    "q06": {
     "a": "2",
     "ok": 1
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "水曜",
     "ok": 1
    },
    "q09": {
     "a": "o",
     "ok": 1
    },
    "q10": {
     "a": "火星",
     "ok": 1
    },
    "q11": {
     "a": "できる",
     "ok": 0
    },
    "q12": {
     "a": "すまいざごうはお",
     "ok": 0
    },
    "q13": {
     "a": "父",
     "ok": 1
    },
    "q14": {
     "a": "同じ",
     "ok": 1
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "15, 2",
     "ok": 1
    },
    "q18": {
     "a": "3",
     "ok": 1
    },
    "q19": {
     "a": "0.25, 0.3, 0.33, 1/3",
     "ok": 1
    },
    "q20": {
     "a": "しない",
     "ok": 1
    }
   },
   "raw": "1: 母  \n2: 10  \n3: しない  \n4: extraordinarily  \n5: 3  \n6: 2  \n7: できない  \n8: 水曜  \n9: o  \n10: 火星  \n11: できる  \n12: すまいざごうはお  \n13: 父  \n14: 同じ  \n15: 5  \n16: 3  \n17: 15, 2  \n18: 3  \n19: 0.25, 0.3, 0.33, 1/3  \n20: しない"
  },
  "qwen/qwen3-30b-a3b-instruct-2507": {
   "score": 6,
   "answered": 20,
   "first10": 3,
   "last10": 3,
   "inconsistent_pairs": 0,
   "sec": 3.8,
   "out_tokens": 154,
   "price": "$0.04815 / $0.19305",
   "graded": {
    "q01": {
     "a": "女性の外科医",
     "ok": 0
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "3",
     "ok": 1
    },
    "q06": {
     "a": "3",
     "ok": 0
    },
    "q07": {
     "a": "できる",
     "ok": 0
    },
    "q08": {
     "a": "木曜日",
     "ok": 0
    },
    "q09": {
     "a": "s",
     "ok": 0
    },
    "q10": {
     "a": "水星",
     "ok": 0
    },
    "q11": {
     "a": "できる",
     "ok": 0
    },
    "q12": {
     "a": "すいませんはよおうお",
     "ok": 0
    },
    "q13": {
     "a": "母親",
     "ok": 0
    },
    "q14": {
     "a": "変える",
     "ok": 0
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "16文字、1つ",
     "ok": 0
    },
    "q18": {
     "a": "3",
     "ok": 1
    },
    "q19": {
     "a": "0.25, 1/3, 0.3, 0.33",
     "ok": 0
    },
    "q20": {
     "a": "存在する",
     "ok": 0
    }
   },
   "raw": "1: 女性の外科医  \n2: 10  \n3: しない  \n4: extraordinarily  \n5: 3  \n6: 3  \n7: できる  \n8: 木曜日  \n9: s  \n10: 水星  \n11: できる  \n12: すいませんはよおうお  \n13: 母親  \n14: 変える  \n15: 5  \n16: 3  \n17: 16文字、1つ  \n18: 3  \n19: 0.25, 1/3, 0.3, 0.33  \n20: 存在する"
  },
  "qwen/qwen3-235b-a22b-2507": {
   "score": 8,
   "answered": 20,
   "first10": 5,
   "last10": 3,
   "inconsistent_pairs": 1,
   "sec": 2.7,
   "out_tokens": 156,
   "price": "$0.22 / $0.88",
   "graded": {
    "q01": {
     "a": "母親",
     "ok": 0
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "synchronization",
     "ok": 0
    },
    "q05": {
     "a": "3",
     "ok": 1
    },
    "q06": {
     "a": "3",
     "ok": 0
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "水曜日",
     "ok": 1
    },
    "q09": {
     "a": "s",
     "ok": 0
    },
    "q10": {
     "a": "水星",
     "ok": 0
    },
    "q11": {
     "a": "できる",
     "ok": 0
    },
    "q12": {
     "a": "す　は　ご　う　よ　は　お",
     "ok": 0
    },
    "q13": {
     "a": "息子の母",
     "ok": 0
    },
    "q14": {
     "a": "変える",
     "ok": 0
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "15文字,1つ",
     "ok": 0
    },
    "q18": {
     "a": "3",
     "ok": 1
    },
    "q19": {
     "a": "0.25,1/3,0.3,0.33",
     "ok": 0
    },
    "q20": {
     "a": "存在する",
     "ok": 0
    }
   },
   "raw": "1: 母親  \n2: 10  \n3: しない  \n4: synchronization  \n5: 3  \n6: 3  \n7: できない  \n8: 水曜日  \n9: s  \n10: 水星  \n11: できる  \n12: す　は　ご　う　よ　は　お  \n13: 息子の母  \n14: 変える  \n15: 5  \n16: 3  \n17: 15文字,1つ  \n18: 3  \n19: 0.25,1/3,0.3,0.33  \n20: 存在する"
  },
  "google/gemini-2.5-flash-lite": {
   "score": 8,
   "answered": 20,
   "first10": 6,
   "last10": 2,
   "inconsistent_pairs": 2,
   "sec": 15.7,
   "out_tokens": 5908,
   "price": "$0.1 / $0.4",
   "graded": {
    "q01": {
     "a": "外科医は少年の母親",
     "ok": 0
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "2",
     "ok": 0
    },
    "q06": {
     "a": "3",
     "ok": 0
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "水曜日",
     "ok": 1
    },
    "q09": {
     "a": "o",
     "ok": 1
    },
    "q10": {
     "a": "火星",
     "ok": 1
    },
    "q11": {
     "a": "できる",
     "ok": 0
    },
    "q12": {
     "a": "す、い、ま、せ、ん、ご、は、お",
     "ok": 0
    },
    "q13": {
     "a": "息子",
     "ok": 0
    },
    "q14": {
     "a": "変える",
     "ok": 0
    },
    "q15": {
     "a": "2",
     "ok": 0
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "14文字、1つ",
     "ok": 0
    },
    "q18": {
     "a": "3",
     "ok": 1
    },
    "q19": {
     "a": "0.25, 1/3, 0.3, 0.33",
     "ok": 0
    },
    "q20": {
     "a": "する",
     "ok": 0
    }
   },
   "raw": "1: 外科医は少年の母親\n2: 10\n3: しない\n4: extraordinarily\n5: 2\n6: 3\n7: できない\n8: 水曜日\n9: o\n10: 火星\n11: できる\n12: す、い、ま、せ、ん、ご、は、お\n13: 息子\n14: 変える\n15: 2\n16: 3\n17: 14文字、1つ\n18: 3\n19: 0.25, 1/3, 0.3, 0.33\n20: する"
  },
  "qwen/qwen3-vl-8b-thinking": {
   "score": 11,
   "answered": 20,
   "first10": 7,
   "last10": 4,
   "inconsistent_pairs": 0,
   "sec": 39.1,
   "out_tokens": 5161,
   "price": "$0.18 / $2.1",
   "graded": {
    "q01": {
     "a": "父親",
     "ok": 1
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "synchronization",
     "ok": 0
    },
    "q05": {
     "a": "2",
     "ok": 0
    },
    "q06": {
     "a": "2",
     "ok": 1
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "水曜日",
     "ok": 1
    },
    "q09": {
     "a": "o",
     "ok": 1
    },
    "q10": {
     "a": "金星",
     "ok": 0
    },
    "q11": {
     "a": "できる",
     "ok": 0
    },
    "q12": {
     "a": "すまいまぞうよはお",
     "ok": 0
    },
    "q13": {
     "a": "父親",
     "ok": 1
    },
    "q14": {
     "a": "変える",
     "ok": 0
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "2",
     "ok": 0
    },
    "q17": {
     "a": "15,1",
     "ok": 0
    },
    "q18": {
     "a": "2",
     "ok": 0
    },
    "q19": {
     "a": "0.25,0.3,0.33,1/3",
     "ok": 1
    },
    "q20": {
     "a": "しない",
     "ok": 1
    }
   },
   "raw": "1: 父親\n2: 10\n3: しない\n4: synchronization\n5: 2\n6: 2\n7: できない\n8: 水曜日\n9: o\n10: 金星\n11: できる\n12: すまいまぞうよはお\n13: 父親\n14: 変える\n15: 5\n16: 2\n17: 15,1\n18: 2\n19: 0.25,0.3,0.33,1/3\n20: しない"
  },
  "openai/gpt-oss-20b": {
   "score": 13,
   "answered": 20,
   "first10": 6,
   "last10": 7,
   "inconsistent_pairs": 0,
   "sec": 12.7,
   "out_tokens": 1485,
   "price": "$0.03 / $0.13",
   "graded": {
    "q01": {
     "a": "父",
     "ok": 1
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "14",
     "ok": 0
    },
    "q05": {
     "a": "4",
     "ok": 0
    },
    "q06": {
     "a": "3",
     "ok": 0
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "水曜日",
     "ok": 1
    },
    "q09": {
     "a": "o",
     "ok": 1
    },
    "q10": {
     "a": "金星",
     "ok": 0
    },
    "q11": {
     "a": "できない",
     "ok": 1
    },
    "q12": {
     "a": "すまいざごうよはお",
     "ok": 1
    },
    "q13": {
     "a": "父",
     "ok": 1
    },
    "q14": {
     "a": "変える",
     "ok": 0
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "15, 1",
     "ok": 0
    },
    "q18": {
     "a": "4",
     "ok": 0
    },
    "q19": {
     "a": "0.25, 0.3, 0.33, 1/3",
     "ok": 1
    },
    "q20": {
     "a": "しない",
     "ok": 1
    }
   },
   "raw": "1: 父  \n2: 10  \n3: しない  \n4: 14  \n5: 4  \n6: 3  \n7: できない  \n8: 水曜日  \n9: o  \n10: 金星  \n11: できない  \n12: すまいざごうよはお  \n13: 父  \n14: 変える  \n15: 5  \n16: 3  \n17: 15, 1  \n18: 4  \n19: 0.25, 0.3, 0.33, 1/3  \n20: しない"
  },
  "cognitivecomputations/dolphin-mistral-24b-venice-edition": {
   "score": 5,
   "answered": 20,
   "first10": 3,
   "last10": 2,
   "inconsistent_pairs": 0,
   "sec": 2.8,
   "out_tokens": 155,
   "price": "$0.2 / $0.9",
   "graded": {
    "q01": {
     "a": "母親",
     "ok": 0
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "synchronization",
     "ok": 0
    },
    "q05": {
     "a": "2",
     "ok": 0
    },
    "q06": {
     "a": "6",
     "ok": 0
    },
    "q07": {
     "a": "できる",
     "ok": 0
    },
    "q08": {
     "a": "土曜日",
     "ok": 0
    },
    "q09": {
     "a": "o",
     "ok": 1
    },
    "q10": {
     "a": "水星",
     "ok": 0
    },
    "q11": {
     "a": "できない",
     "ok": 1
    },
    "q12": {
     "a": "すごむ よあう ごはお",
     "ok": 0
    },
    "q13": {
     "a": "母と息子",
     "ok": 0
    },
    "q14": {
     "a": "変える",
     "ok": 0
    },
    "q15": {
     "a": "10",
     "ok": 0
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "19文字、1単語",
     "ok": 0
    },
    "q18": {
     "a": "2",
     "ok": 0
    },
    "q19": {
     "a": "0.25, 0.3, 1/3, 0.33",
     "ok": 0
    },
    "q20": {
     "a": "存在する",
     "ok": 0
    }
   },
   "raw": "1: 母親\n2: 10\n3: しない\n4: synchronization\n5: 2\n6: 6\n7: できる\n8: 土曜日\n9: o\n10: 水星\n11: できない\n12: すごむ よあう ごはお\n13: 母と息子\n14: 変える\n15: 10\n16: 3\n17: 19文字、1単語\n18: 2\n19: 0.25, 0.3, 1/3, 0.33\n20: 存在する"
  },
  "moonshotai/kimi-k2": {
   "score": 9,
   "answered": 20,
   "first10": 5,
   "last10": 4,
   "inconsistent_pairs": 1,
   "sec": 9.2,
   "out_tokens": 135,
   "price": "$0.57 / $2.3",
   "graded": {
    "q01": {
     "a": "父親",
     "ok": 1
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "4",
     "ok": 0
    },
    "q06": {
     "a": "3",
     "ok": 0
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "水曜日",
     "ok": 1
    },
    "q09": {
     "a": "s",
     "ok": 0
    },
    "q10": {
     "a": "金星",
     "ok": 0
    },
    "q11": {
     "a": "できる",
     "ok": 0
    },
    "q12": {
     "a": "すまいざうよごはうお",
     "ok": 0
    },
    "q13": {
     "a": "母親",
     "ok": 0
    },
    "q14": {
     "a": "同じ",
     "ok": 1
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "4",
     "ok": 0
    },
    "q17": {
     "a": "15文字、1つ",
     "ok": 0
    },
    "q18": {
     "a": "4",
     "ok": 0
    },
    "q19": {
     "a": "0.25, 0.3, 0.33, 1/3",
     "ok": 1
    },
    "q20": {
     "a": "しない",
     "ok": 1
    }
   },
   "raw": "1: 父親\n2: 10\n3: しない\n4: extraordinarily\n5: 4\n6: 3\n7: できない\n8: 水曜日\n9: s\n10: 金星\n11: できる\n12: すまいざうよごはうお\n13: 母親\n14: 同じ\n15: 5\n16: 4\n17: 15文字、1つ\n18: 4\n19: 0.25, 0.3, 0.33, 1/3\n20: しない"
  },
  "google/gemini-2.5-flash": {
   "score": 15,
   "answered": 20,
   "first10": 7,
   "last10": 8,
   "inconsistent_pairs": 1,
   "sec": 30.2,
   "out_tokens": 5754,
   "price": "$0.3 / $2.5",
   "graded": {
    "q01": {
     "a": "母親",
     "ok": 0
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "synchronization",
     "ok": 0
    },
    "q05": {
     "a": "3",
     "ok": 1
    },
    "q06": {
     "a": "3",
     "ok": 0
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "水曜",
     "ok": 1
    },
    "q09": {
     "a": "o",
     "ok": 1
    },
    "q10": {
     "a": "火星",
     "ok": 1
    },
    "q11": {
     "a": "できない",
     "ok": 1
    },
    "q12": {
     "a": "すまいざうごよはお",
     "ok": 0
    },
    "q13": {
     "a": "父親",
     "ok": 1
    },
    "q14": {
     "a": "同じ",
     "ok": 1
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "15文字, 1個",
     "ok": 0
    },
    "q18": {
     "a": "3",
     "ok": 1
    },
    "q19": {
     "a": "0.25, 0.3, 0.33, 1/3",
     "ok": 1
    },
    "q20": {
     "a": "しない",
     "ok": 1
    }
   },
   "raw": "1. 母親\n2. 10\n3. しない\n4. synchronization\n5. 3\n6. 3\n7. できない\n8. 水曜\n9. o\n10. 火星\n11. できない\n12. すまいざうごよはお\n13. 父親\n14. 同じ\n15. 5\n16. 3\n17. 15文字, 1個\n18. 3\n19. 0.25, 0.3, 0.33, 1/3\n20. しない"
  },
  "anthropic/claude-opus-4.1": {
   "score": 14,
   "answered": 20,
   "first10": 6,
   "last10": 8,
   "inconsistent_pairs": 1,
   "sec": 30.4,
   "out_tokens": 967,
   "price": "$15 / $75",
   "graded": {
    "q01": {
     "a": "父親",
     "ok": 1
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "4",
     "ok": 0
    },
    "q06": {
     "a": "3",
     "ok": 0
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "水曜日",
     "ok": 1
    },
    "q09": {
     "a": "s",
     "ok": 0
    },
    "q10": {
     "a": "火星",
     "ok": 1
    },
    "q11": {
     "a": "できる",
     "ok": 0
    },
    "q12": {
     "a": "すまいざごうよはお",
     "ok": 1
    },
    "q13": {
     "a": "父子",
     "ok": 1
    },
    "q14": {
     "a": "同じ",
     "ok": 1
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "15文字、2単語",
     "ok": 1
    },
    "q18": {
     "a": "4",
     "ok": 0
    },
    "q19": {
     "a": "0.25, 0.3, 0.33, 1/3",
     "ok": 1
    },
    "q20": {
     "a": "しない",
     "ok": 1
    }
   },
   "raw": "1: 父親\n2: 10\n3: しない\n4: extraordinarily\n5: 4\n6: 3\n7: できない\n8: 水曜日\n9: s\n10: 火星\n11: できる\n12: すまいざごうよはお\n13: 父子\n14: 同じ\n15: 5\n16: 3\n17: 15文字、2単語\n18: 4\n19: 0.25, 0.3, 0.33, 1/3\n20: しない"
  },
  "qwen/qwen3-next-80b-a3b-thinking": {
   "score": 14,
   "answered": 20,
   "first10": 7,
   "last10": 7,
   "inconsistent_pairs": 1,
   "sec": 41.0,
   "out_tokens": 10613,
   "price": "$0.15 / $1.2",
   "graded": {
    "q01": {
     "a": "母親",
     "ok": 0
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "5",
     "ok": 0
    },
    "q06": {
     "a": "2",
     "ok": 1
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "水曜日",
     "ok": 1
    },
    "q09": {
     "a": "o",
     "ok": 1
    },
    "q10": {
     "a": "火星",
     "ok": 1
    },
    "q11": {
     "a": "できない",
     "ok": 1
    },
    "q12": {
     "a": "すまいざごうよはお",
     "ok": 1
    },
    "q13": {
     "a": "母親",
     "ok": 0
    },
    "q14": {
     "a": "変える",
     "ok": 0
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "15,2",
     "ok": 1
    },
    "q18": {
     "a": "5",
     "ok": 0
    },
    "q19": {
     "a": "0.25,0.3,0.33,1/3",
     "ok": 1
    },
    "q20": {
     "a": "しない",
     "ok": 1
    }
   },
   "raw": "1: 母親  \n2: 10  \n3: しない  \n4: extraordinarily  \n5: 5  \n6: 2  \n7: できない  \n8: 水曜日  \n9: o  \n10: 火星  \n11: できない  \n12: すまいざごうよはお  \n13: 母親  \n14: 変える  \n15: 5  \n16: 3  \n17: 15,2  \n18: 5  \n19: 0.25,0.3,0.33,1/3  \n20: しない"
  },
  "google/gemini-2.5-pro": {
   "score": 11,
   "answered": 20,
   "first10": 8,
   "last10": 3,
   "inconsistent_pairs": 1,
   "sec": 32.5,
   "out_tokens": 137,
   "price": "$1.25 / $10",
   "graded": {
    "q01": {
     "a": "外科医は父親。",
     "ok": 1
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "4",
     "ok": 0
    },
    "q06": {
     "a": "2",
     "ok": 1
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "水曜日",
     "ok": 1
    },
    "q09": {
     "a": "o",
     "ok": 1
    },
    "q10": {
     "a": "火星",
     "ok": 1
    },
    "q11": {
     "a": "できる",
     "ok": 0
    },
    "q12": {
     "a": "すまいざごうはお",
     "ok": 0
    },
    "q13": {
     "a": "母親",
     "ok": 0
    },
    "q14": {
     "a": "変える",
     "ok": 0
    },
    "q15": {
     "a": "10",
     "ok": 0
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "14文字で1つ",
     "ok": 0
    },
    "q18": {
     "a": "4",
     "ok": 0
    },
    "q19": {
     "a": "0.25, 0.3, 0.33, 1/3",
     "ok": 1
    },
    "q20": {
     "a": "存在しない",
     "ok": 1
    }
   },
   "raw": "1. 外科医は父親。\n2. 10\n3. しない\n4. extraordinarily\n5. 4\n6. 2\n7. できない\n8. 水曜日\n9. o\n10. 火星\n11. できる\n12. すまいざごうはお\n13. 母親\n14. 変える\n15. 10\n16. 3\n17. 14文字で1つ\n18. 4\n19. 0.25, 0.3, 0.33, 1/3\n20. 存在しない"
  },
  "mistralai/mistral-medium-3": {
   "score": 7,
   "answered": 20,
   "first10": 3,
   "last10": 4,
   "inconsistent_pairs": 0,
   "sec": 3.1,
   "out_tokens": 145,
   "price": "$0.4 / $2",
   "graded": {
    "q01": {
     "a": "母親",
     "ok": 0
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "する",
     "ok": 0
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "3",
     "ok": 1
    },
    "q06": {
     "a": "0",
     "ok": 0
    },
    "q07": {
     "a": "できる",
     "ok": 0
    },
    "q08": {
     "a": "水曜日",
     "ok": 1
    },
    "q09": {
     "a": "p",
     "ok": 0
    },
    "q10": {
     "a": "水星",
     "ok": 0
    },
    "q11": {
     "a": "できる",
     "ok": 0
    },
    "q12": {
     "a": "ますはちおはよう",
     "ok": 0
    },
    "q13": {
     "a": "母親",
     "ok": 0
    },
    "q14": {
     "a": "同じ",
     "ok": 1
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "15文字、1つ",
     "ok": 0
    },
    "q18": {
     "a": "3",
     "ok": 1
    },
    "q19": {
     "a": "0.25, 0.3, 1/3, 0.33",
     "ok": 0
    },
    "q20": {
     "a": "存在する",
     "ok": 0
    }
   },
   "raw": "1: 母親\n2: 10\n3: する\n4: extraordinarily\n5: 3\n6: 0\n7: できる\n8: 水曜日\n9: p\n10: 水星\n11: できる\n12: ますはちおはよう\n13: 母親\n14: 同じ\n15: 5\n16: 3\n17: 15文字、1つ\n18: 3\n19: 0.25, 0.3, 1/3, 0.33\n20: 存在する"
  },
  "baidu/ernie-4.5-vl-424b-a47b": {
   "score": 12,
   "answered": 20,
   "first10": 6,
   "last10": 6,
   "inconsistent_pairs": 1,
   "sec": 11.4,
   "out_tokens": 360,
   "price": "$0.42 / $1.25",
   "graded": {
    "q01": {
     "a": "父",
     "ok": 1
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "4",
     "ok": 0
    },
    "q06": {
     "a": "3",
     "ok": 0
    },
    "q07": {
     "a": "できる",
     "ok": 0
    },
    "q08": {
     "a": "水曜",
     "ok": 1
    },
    "q09": {
     "a": "o",
     "ok": 1
    },
    "q10": {
     "a": "火星",
     "ok": 1
    },
    "q11": {
     "a": "できない",
     "ok": 1
    },
    "q12": {
     "a": "すまざうごはお",
     "ok": 0
    },
    "q13": {
     "a": "父",
     "ok": 1
    },
    "q14": {
     "a": "変える",
     "ok": 0
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "17文字、1単語",
     "ok": 0
    },
    "q18": {
     "a": "4",
     "ok": 0
    },
    "q19": {
     "a": "0.25, 0.3, 0.33, 1/3",
     "ok": 1
    },
    "q20": {
     "a": "しない",
     "ok": 1
    }
   },
   "raw": "1: 父  \n2: 10  \n3: しない  \n4: extraordinarily  \n5: 4  \n6: 3  \n7: できる  \n8: 水曜  \n9: o  \n10: 火星  \n11: できない  \n12: すまざうごはお  \n13: 父  \n14: 変える  \n15: 5  \n16: 3  \n17: 17文字、1単語  \n18: 4  \n19: 0.25, 0.3, 0.33, 1/3  \n20: しない"
  },
  "anthropic/claude-sonnet-4": {
   "score": 16,
   "answered": 20,
   "first10": 8,
   "last10": 8,
   "inconsistent_pairs": 1,
   "sec": 18.6,
   "out_tokens": 1730,
   "price": "$3 / $15",
   "graded": {
    "q01": {
     "a": "父親",
     "ok": 1
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "synchronization",
     "ok": 0
    },
    "q05": {
     "a": "3",
     "ok": 1
    },
    "q06": {
     "a": "2",
     "ok": 1
    },
    "q07": {
     "a": "できる",
     "ok": 0
    },
    "q08": {
     "a": "水曜日",
     "ok": 1
    },
    "q09": {
     "a": "o",
     "ok": 1
    },
    "q10": {
     "a": "火星",
     "ok": 1
    },
    "q11": {
     "a": "できる",
     "ok": 0
    },
    "q12": {
     "a": "すまいざごうよはお",
     "ok": 1
    },
    "q13": {
     "a": "父親",
     "ok": 1
    },
    "q14": {
     "a": "同じ",
     "ok": 1
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "15文字で1つ",
     "ok": 0
    },
    "q18": {
     "a": "3",
     "ok": 1
    },
    "q19": {
     "a": "0.25, 0.3, 0.33, 1/3",
     "ok": 1
    },
    "q20": {
     "a": "しない",
     "ok": 1
    }
   },
   "raw": "1: 父親\n2: 10\n3: しない\n4: synchronization\n5: 3\n6: 2\n7: できる\n8: 水曜日\n9: o\n10: 火星\n11: できる\n12: すまいざごうよはお\n13: 父親\n14: 同じ\n15: 5\n16: 3\n17: 15文字で1つ\n18: 3\n19: 0.25, 0.3, 0.33, 1/3\n20: しない"
  },
  "anthropic/claude-opus-4": {
   "score": 16,
   "answered": 20,
   "first10": 8,
   "last10": 8,
   "inconsistent_pairs": 1,
   "sec": 25.0,
   "out_tokens": 860,
   "price": "$15 / $75",
   "graded": {
    "q01": {
     "a": "父親",
     "ok": 1
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "4",
     "ok": 0
    },
    "q06": {
     "a": "2",
     "ok": 1
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "水曜日",
     "ok": 1
    },
    "q09": {
     "a": "o",
     "ok": 1
    },
    "q10": {
     "a": "火星",
     "ok": 1
    },
    "q11": {
     "a": "できる",
     "ok": 0
    },
    "q12": {
     "a": "すまいざごうよはお",
     "ok": 1
    },
    "q13": {
     "a": "父親",
     "ok": 1
    },
    "q14": {
     "a": "同じ",
     "ok": 1
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "15文字、2つ",
     "ok": 1
    },
    "q18": {
     "a": "4",
     "ok": 0
    },
    "q19": {
     "a": "0.25, 0.3, 0.33, 1/3",
     "ok": 1
    },
    "q20": {
     "a": "しない",
     "ok": 1
    }
   },
   "raw": "1: 父親\n2: 10\n3: しない\n4: extraordinarily\n5: 4\n6: 2\n7: できない\n8: 水曜日\n9: o\n10: 火星\n11: できる\n12: すまいざごうよはお\n13: 父親\n14: 同じ\n15: 5\n16: 3\n17: 15文字、2つ\n18: 4\n19: 0.25, 0.3, 0.33, 1/3\n20: しない"
  },
  "qwen/qwen3-14b": {
   "score": 12,
   "answered": 20,
   "first10": 6,
   "last10": 6,
   "inconsistent_pairs": 1,
   "sec": 40.4,
   "out_tokens": 1967,
   "price": "$0.2275 / $0.91",
   "graded": {
    "q01": {
     "a": "母親",
     "ok": 0
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "synchronization",
     "ok": 0
    },
    "q05": {
     "a": "3",
     "ok": 1
    },
    "q06": {
     "a": "3",
     "ok": 0
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "水曜日",
     "ok": 1
    },
    "q09": {
     "a": "o",
     "ok": 1
    },
    "q10": {
     "a": "金星",
     "ok": 0
    },
    "q11": {
     "a": "できる",
     "ok": 0
    },
    "q12": {
     "a": "せいざごうよはお",
     "ok": 0
    },
    "q13": {
     "a": "父親",
     "ok": 1
    },
    "q14": {
     "a": "変える",
     "ok": 0
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "15文字, 1つ",
     "ok": 0
    },
    "q18": {
     "a": "3",
     "ok": 1
    },
    "q19": {
     "a": "0.25, 0.3, 0.33, 1/3",
     "ok": 1
    },
    "q20": {
     "a": "しない",
     "ok": 1
    }
   },
   "raw": "1: 母親  \n2: 10  \n3: しない  \n4: synchronization  \n5: 3  \n6: 3  \n7: できない  \n8: 水曜日  \n9: o  \n10: 金星  \n11: できる  \n12: せいざごうよはお  \n13: 父親  \n14: 変える  \n15: 5  \n16: 3  \n17: 15文字, 1つ  \n18: 3  \n19: 0.25, 0.3, 0.33, 1/3  \n20: しない"
  },
  "openai/gpt-4.1": {
   "score": 9,
   "answered": 20,
   "first10": 6,
   "last10": 3,
   "inconsistent_pairs": 0,
   "sec": 2.3,
   "out_tokens": 131,
   "price": "$2 / $8",
   "graded": {
    "q01": {
     "a": "母親",
     "ok": 0
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "synchronization",
     "ok": 0
    },
    "q05": {
     "a": "4",
     "ok": 0
    },
    "q06": {
     "a": "2",
     "ok": 1
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "水曜",
     "ok": 1
    },
    "q09": {
     "a": "s",
     "ok": 0
    },
    "q10": {
     "a": "火星",
     "ok": 1
    },
    "q11": {
     "a": "できる",
     "ok": 0
    },
    "q12": {
     "a": "すまいざごうはお",
     "ok": 0
    },
    "q13": {
     "a": "母親",
     "ok": 0
    },
    "q14": {
     "a": "変える",
     "ok": 0
    },
    "q15": {
     "a": "10",
     "ok": 0
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "15, 1",
     "ok": 0
    },
    "q18": {
     "a": "4",
     "ok": 0
    },
    "q19": {
     "a": "0.25, 0.3, 0.33, 1/3",
     "ok": 1
    },
    "q20": {
     "a": "しない",
     "ok": 1
    }
   },
   "raw": "1: 母親  \n2: 10  \n3: しない  \n4: synchronization  \n5: 4  \n6: 2  \n7: できない  \n8: 水曜  \n9: s  \n10: 火星  \n11: できる  \n12: すまいざごうはお  \n13: 母親  \n14: 変える  \n15: 10  \n16: 3  \n17: 15, 1  \n18: 4  \n19: 0.25, 0.3, 0.33, 1/3  \n20: しない"
  },
  "openai/gpt-4.1-mini": {
   "score": 10,
   "answered": 20,
   "first10": 5,
   "last10": 5,
   "inconsistent_pairs": 0,
   "sec": 25.3,
   "out_tokens": 132,
   "price": "$0.4 / $1.6",
   "graded": {
    "q01": {
     "a": "母親",
     "ok": 0
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "3",
     "ok": 1
    },
    "q06": {
     "a": "3",
     "ok": 0
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "木曜",
     "ok": 0
    },
    "q09": {
     "a": "s",
     "ok": 0
    },
    "q10": {
     "a": "火星",
     "ok": 1
    },
    "q11": {
     "a": "できる",
     "ok": 0
    },
    "q12": {
     "a": "すまいござうよはお",
     "ok": 0
    },
    "q13": {
     "a": "母親",
     "ok": 0
    },
    "q14": {
     "a": "変える",
     "ok": 0
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "15文字、1つ",
     "ok": 0
    },
    "q18": {
     "a": "3",
     "ok": 1
    },
    "q19": {
     "a": "0.25, 0.3, 0.33, 1/3",
     "ok": 1
    },
    "q20": {
     "a": "しない",
     "ok": 1
    }
   },
   "raw": "1: 母親  \n2: 10  \n3: しない  \n4: extraordinarily  \n5: 3  \n6: 3  \n7: できない  \n8: 木曜  \n9: s  \n10: 火星  \n11: できる  \n12: すまいござうよはお  \n13: 母親  \n14: 変える  \n15: 5  \n16: 3  \n17: 15文字、1つ  \n18: 3  \n19: 0.25, 0.3, 0.33, 1/3  \n20: しない"
  },
  "openai/gpt-4.1-nano": {
   "score": 7,
   "answered": 20,
   "first10": 5,
   "last10": 2,
   "inconsistent_pairs": 0,
   "sec": 22.6,
   "out_tokens": 134,
   "price": "$0.1 / $0.4",
   "graded": {
    "q01": {
     "a": "母親",
     "ok": 0
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "3",
     "ok": 1
    },
    "q06": {
     "a": "2",
     "ok": 1
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "金曜日",
     "ok": 0
    },
    "q09": {
     "a": "5",
     "ok": 0
    },
    "q10": {
     "a": "金星",
     "ok": 0
    },
    "q11": {
     "a": "できる",
     "ok": 0
    },
    "q12": {
     "a": "すまいございますおは",
     "ok": 0
    },
    "q13": {
     "a": "母と息子の関係",
     "ok": 0
    },
    "q14": {
     "a": "変える",
     "ok": 0
    },
    "q15": {
     "a": "6",
     "ok": 0
    },
    "q16": {
     "a": "4",
     "ok": 0
    },
    "q17": {
     "a": "11、2",
     "ok": 0
    },
    "q18": {
     "a": "3",
     "ok": 1
    },
    "q19": {
     "a": "0.25, 0.33, 0.3, 1/3",
     "ok": 0
    },
    "q20": {
     "a": "しない",
     "ok": 1
    }
   },
   "raw": "1: 母親  \n2: 10  \n3: しない  \n4: extraordinarily  \n5: 3  \n6: 2  \n7: できない  \n8: 金曜日  \n9: 5  \n10: 金星  \n11: できる  \n12: すまいございますおは  \n13: 母と息子の関係  \n14: 変える  \n15: 6  \n16: 4  \n17: 11、2  \n18: 3  \n19: 0.25, 0.33, 0.3, 1/3  \n20: しない"
  },
  "google/gemma-3-4b-it": {
   "score": 6,
   "answered": 20,
   "first10": 4,
   "last10": 2,
   "inconsistent_pairs": 1,
   "sec": 17.5,
   "out_tokens": 144,
   "price": "$0.05 / $0.1",
   "graded": {
    "q01": {
     "a": "父",
     "ok": 1
    },
    "q02": {
     "a": "20",
     "ok": 0
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "synchronization",
     "ok": 0
    },
    "q05": {
     "a": "2",
     "ok": 0
    },
    "q06": {
     "a": "3",
     "ok": 0
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "土",
     "ok": 0
    },
    "q09": {
     "a": "p",
     "ok": 0
    },
    "q10": {
     "a": "火星",
     "ok": 1
    },
    "q11": {
     "a": "できる",
     "ok": 0
    },
    "q12": {
     "a": "ょうめしがさうも",
     "ok": 0
    },
    "q13": {
     "a": "息子と父",
     "ok": 1
    },
    "q14": {
     "a": "変える",
     "ok": 0
    },
    "q15": {
     "a": "30",
     "ok": 0
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "18, 1",
     "ok": 0
    },
    "q18": {
     "a": "2",
     "ok": 0
    },
    "q19": {
     "a": "0.25, 0.3, 1/3, 0.33",
     "ok": 0
    },
    "q20": {
     "a": "存在する",
     "ok": 0
    }
   },
   "raw": "1.  父\n2.  20\n3.  しない\n4.  synchronization\n5.  2\n6.  3\n7.  できない\n8.  土\n9.  p\n10. 火星\n11. できる\n12. ょうめしがさうも\n13. 息子と父\n14. 変える\n15. 30\n16. 3\n17. 18, 1\n18. 2\n19. 0.25, 0.3, 1/3, 0.33\n20. 存在する"
  },
  "deepseek/deepseek-chat-v3-0324": {
   "score": 8,
   "answered": 20,
   "first10": 5,
   "last10": 3,
   "inconsistent_pairs": 0,
   "sec": 18.0,
   "out_tokens": 131,
   "price": "$0.29 / $1.14",
   "graded": {
    "q01": {
     "a": "母親",
     "ok": 0
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "する",
     "ok": 0
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "3",
     "ok": 1
    },
    "q06": {
     "a": "2",
     "ok": 1
    },
    "q07": {
     "a": "できる",
     "ok": 0
    },
    "q08": {
     "a": "水曜日",
     "ok": 1
    },
    "q09": {
     "a": "s",
     "ok": 0
    },
    "q10": {
     "a": "火星",
     "ok": 1
    },
    "q11": {
     "a": "できる",
     "ok": 0
    },
    "q12": {
     "a": "すまざよごうやはお",
     "ok": 0
    },
    "q13": {
     "a": "母親",
     "ok": 0
    },
    "q14": {
     "a": "変える",
     "ok": 0
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "5",
     "ok": 0
    },
    "q17": {
     "a": "14文字, 1つ",
     "ok": 0
    },
    "q18": {
     "a": "3",
     "ok": 1
    },
    "q19": {
     "a": "0.25, 0.3, 0.33, 1/3",
     "ok": 1
    },
    "q20": {
     "a": "存在する",
     "ok": 0
    }
   },
   "raw": "1: 母親\n2: 10\n3: する\n4: extraordinarily\n5: 3\n6: 2\n7: できる\n8: 水曜日\n9: s\n10: 火星\n11: できる\n12: すまざよごうやはお\n13: 母親\n14: 変える\n15: 5\n16: 5\n17: 14文字, 1つ\n18: 3\n19: 0.25, 0.3, 0.33, 1/3\n20: 存在する"
  },
  "google/gemma-3-12b-it": {
   "score": 4,
   "answered": 20,
   "first10": 3,
   "last10": 1,
   "inconsistent_pairs": 1,
   "sec": 14.2,
   "out_tokens": 134,
   "price": "$0.05 / $0.15",
   "graded": {
    "q01": {
     "a": "母親",
     "ok": 0
    },
    "q02": {
     "a": "1",
     "ok": 0
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "4",
     "ok": 0
    },
    "q06": {
     "a": "2",
     "ok": 1
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "月曜日",
     "ok": 0
    },
    "q09": {
     "a": "e",
     "ok": 0
    },
    "q10": {
     "a": "金星",
     "ok": 0
    },
    "q11": {
     "a": "できる",
     "ok": 0
    },
    "q12": {
     "a": "すございますおはよう",
     "ok": 0
    },
    "q13": {
     "a": "医者",
     "ok": 0
    },
    "q14": {
     "a": "変える",
     "ok": 0
    },
    "q15": {
     "a": "10",
     "ok": 0
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "14, 1",
     "ok": 0
    },
    "q18": {
     "a": "4",
     "ok": 0
    },
    "q19": {
     "a": "0.25, 0.3, 1/3, 0.33",
     "ok": 0
    },
    "q20": {
     "a": "存在する",
     "ok": 0
    }
   },
   "raw": "1: 母親\n2: 1\n3: しない\n4: extraordinarily\n5: 4\n6: 2\n7: できない\n8: 月曜日\n9: e\n10: 金星\n11: できる\n12: すございますおはよう\n13: 医者\n14: 変える\n15: 10\n16: 3\n17: 14, 1\n18: 4\n19: 0.25, 0.3, 1/3, 0.33\n20: 存在する"
  },
  "openai/o4-mini": {
   "score": 18,
   "answered": 20,
   "first10": 9,
   "last10": 9,
   "inconsistent_pairs": 2,
   "sec": 36.4,
   "out_tokens": 3753,
   "price": "$1.1 / $4.4",
   "graded": {
    "q01": {
     "a": "父",
     "ok": 1
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "3",
     "ok": 1
    },
    "q06": {
     "a": "2",
     "ok": 1
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "水曜",
     "ok": 1
    },
    "q09": {
     "a": "o",
     "ok": 1
    },
    "q10": {
     "a": "火星",
     "ok": 1
    },
    "q11": {
     "a": "できない",
     "ok": 1
    },
    "q12": {
     "a": "すまいざごうよはお",
     "ok": 1
    },
    "q13": {
     "a": "母",
     "ok": 0
    },
    "q14": {
     "a": "同じ",
     "ok": 1
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "15,2",
     "ok": 1
    },
    "q18": {
     "a": "3",
     "ok": 1
    },
    "q19": {
     "a": "0.25,0.3,0.33,1/3",
     "ok": 1
    },
    "q20": {
     "a": "しない",
     "ok": 1
    }
   },
   "raw": "1: 父  \n2: 10  \n3: しない  \n4: extraordinarily  \n5: 3  \n6: 2  \n7: できない  \n8: 水曜  \n9: o  \n10: 火星  \n11: できない  \n12: すまいざごうよはお  \n13: 母  \n14: 同じ  \n15: 5  \n16: 3  \n17: 15,2  \n18: 3  \n19: 0.25,0.3,0.33,1/3  \n20: しない"
  },
  "openai/o3": {
   "score": 19,
   "answered": 20,
   "first10": 9,
   "last10": 10,
   "inconsistent_pairs": 1,
   "sec": 30.0,
   "out_tokens": 4194,
   "price": "$2 / $8",
   "graded": {
    "q01": {
     "a": "父親",
     "ok": 1
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "3",
     "ok": 1
    },
    "q06": {
     "a": "2",
     "ok": 1
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "水曜",
     "ok": 1
    },
    "q09": {
     "a": "o",
     "ok": 1
    },
    "q10": {
     "a": "火星",
     "ok": 1
    },
    "q11": {
     "a": "できない",
     "ok": 1
    },
    "q12": {
     "a": "すまいざごうよはお",
     "ok": 1
    },
    "q13": {
     "a": "父親",
     "ok": 1
    },
    "q14": {
     "a": "同じ",
     "ok": 1
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "15,2",
     "ok": 1
    },
    "q18": {
     "a": "3",
     "ok": 1
    },
    "q19": {
     "a": "0.25,0.3,0.33,1/3",
     "ok": 1
    },
    "q20": {
     "a": "しない",
     "ok": 1
    }
   },
   "raw": "1: 父親\n2: 10\n3: しない\n4: extraordinarily\n5: 3\n6: 2\n7: できない\n8: 水曜\n9: o\n10: 火星\n11: できない\n12: すまいざごうよはお\n13: 父親\n14: 同じ\n15: 5\n16: 3\n17: 15,2\n18: 3\n19: 0.25,0.3,0.33,1/3\n20: しない"
  },
  "thedrummer/skyfall-36b-v2": {
   "score": 7,
   "answered": 20,
   "first10": 4,
   "last10": 3,
   "inconsistent_pairs": 3,
   "sec": 3.3,
   "out_tokens": 148,
   "price": "$0.55 / $0.8",
   "graded": {
    "q01": {
     "a": "その少年",
     "ok": 0
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "3",
     "ok": 1
    },
    "q06": {
     "a": "6",
     "ok": 0
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "土曜日",
     "ok": 0
    },
    "q09": {
     "a": "d",
     "ok": 0
    },
    "q10": {
     "a": "水星",
     "ok": 0
    },
    "q11": {
     "a": "できない",
     "ok": 1
    },
    "q12": {
     "a": "すますごようは",
     "ok": 0
    },
    "q13": {
     "a": "父と息子",
     "ok": 1
    },
    "q14": {
     "a": "変える",
     "ok": 0
    },
    "q15": {
     "a": "10",
     "ok": 0
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "16文字、1つ",
     "ok": 0
    },
    "q18": {
     "a": "2",
     "ok": 0
    },
    "q19": {
     "a": "0.25, 0.3, 1/3, 0.33",
     "ok": 0
    },
    "q20": {
     "a": "存在する",
     "ok": 0
    }
   },
   "raw": "1. その少年\n2. 10\n3. しない\n4. extraordinarily\n5. 3\n6. 6\n7. できない\n8. 土曜日\n9. d\n10. 水星\n11. できない\n12. すますごようは\n13. 父と息子\n14. 変える\n15. 10\n16. 3\n17. 16文字、1つ\n18. 2\n19. 0.25, 0.3, 1/3, 0.33\n20. 存在する"
  },
  "mistralai/mistral-saba": {
   "score": 10,
   "answered": 20,
   "first10": 6,
   "last10": 4,
   "inconsistent_pairs": 1,
   "sec": 2.3,
   "out_tokens": 145,
   "price": "$0.2 / $0.6",
   "graded": {
    "q01": {
     "a": "母親",
     "ok": 0
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "する",
     "ok": 0
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "3",
     "ok": 1
    },
    "q06": {
     "a": "2",
     "ok": 1
    },
    "q07": {
     "a": "できる",
     "ok": 0
    },
    "q08": {
     "a": "水曜日",
     "ok": 1
    },
    "q09": {
     "a": "o",
     "ok": 1
    },
    "q10": {
     "a": "火星",
     "ok": 1
    },
    "q11": {
     "a": "できる",
     "ok": 0
    },
    "q12": {
     "a": "すもよはごう",
     "ok": 0
    },
    "q13": {
     "a": "父親",
     "ok": 1
    },
    "q14": {
     "a": "変える",
     "ok": 0
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "12, 1",
     "ok": 0
    },
    "q18": {
     "a": "3",
     "ok": 1
    },
    "q19": {
     "a": "0.25, 1/3, 0.3, 0.33",
     "ok": 0
    },
    "q20": {
     "a": "存在する",
     "ok": 0
    }
   },
   "raw": "1: 母親\n2: 10\n3: する\n4: extraordinarily\n5: 3\n6: 2\n7: できる\n8: 水曜日\n9: o\n10: 火星\n11: できる\n12: すもよはごう\n13: 父親\n14: 変える\n15: 5\n16: 3\n17: 12, 1\n18: 3\n19: 0.25, 1/3, 0.3, 0.33\n20: 存在する"
  },
  "qwen/qwen2.5-vl-72b-instruct": {
   "score": 9,
   "answered": 20,
   "first10": 4,
   "last10": 5,
   "inconsistent_pairs": 1,
   "sec": 4.7,
   "out_tokens": 150,
   "price": "$0.8 / $1",
   "graded": {
    "q01": {
     "a": "母親",
     "ok": 0
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "4",
     "ok": 0
    },
    "q06": {
     "a": "3",
     "ok": 0
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "土曜日",
     "ok": 0
    },
    "q09": {
     "a": "o",
     "ok": 1
    },
    "q10": {
     "a": "マーキュリー",
     "ok": 0
    },
    "q11": {
     "a": "できる",
     "ok": 0
    },
    "q12": {
     "a": "すみませんごようはお",
     "ok": 0
    },
    "q13": {
     "a": "父子",
     "ok": 1
    },
    "q14": {
     "a": "同じ",
     "ok": 1
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "15, 1",
     "ok": 0
    },
    "q18": {
     "a": "4",
     "ok": 0
    },
    "q19": {
     "a": "0.25, 0.3, 1/3, 0.33",
     "ok": 0
    },
    "q20": {
     "a": "しない",
     "ok": 1
    }
   },
   "raw": "1: 母親\n2: 10\n3: しない\n4: extraordinarily\n5: 4\n6: 3\n7: できない\n8: 土曜日\n9: o\n10: マーキュリー\n11: できる\n12: すみませんごようはお\n13: 父子\n14: 同じ\n15: 5\n16: 3\n17: 15, 1\n18: 4\n19: 0.25, 0.3, 1/3, 0.33\n20: しない"
  },
  "aion-labs/aion-rp-llama-3.1-8b": {
   "score": 6,
   "answered": 20,
   "first10": 3,
   "last10": 3,
   "inconsistent_pairs": 0,
   "sec": 14.3,
   "out_tokens": 537,
   "price": "$0.8 / $1.6",
   "graded": {
    "q01": {
     "a": "The surgeon who claimed to be the boy's father is actually the mother's husband, the boy's stepfather.",
     "ok": 1
    },
    "q02": {
     "a": "The bat is 30 yen and the ball is 10 yen.",
     "ok": 0
    },
    "q03": {
     "a": "2026年2月29日は存在しない。leap yearの条件を満たさないため。",
     "ok": 1
    },
    "q04": {
     "a": "\"complicated\"",
     "ok": 0
    },
    "q05": {
     "a": "ちょうちょは5モーラ。",
     "ok": 0
    },
    "q06": {
     "a": "サリーの姉妹は3×3=9人。",
     "ok": 0
    },
    "q07": {
     "a": "農夫・オオカミ・ヤギ・キャベツを全て無事に対岸へ運ぶことはできない。ボートが1往復しか使えないため、オオカミがヤギを食べたり、ヤギがキャベツを食べたりする空き時間が生じる。",
     "ok": 1
    },
    "q08": {
     "a": "今日が月曜なら、100日後は火曜日。",
     "ok": 0
    },
    "q09": {
     "a": "kaleidoscope の7文字目は s。",
     "ok": 0
    },
    "q10": {
     "a": "水星。",
     "ok": 0
    },
    "q11": {
     "a": "2Lと4Lの容器だけで、ちょうど3Lを測ることはできない。3Lの倍数以外の容量を測るため。",
     "ok": 1
    },
    "q12": {
     "a": "\"ございますおはよう\"。",
     "ok": 0
    },
    "q13": {
     "a": "父は執刀医で、子が10歳の時点で亡くなっている。",
     "ok": 1
    },
    "q14": {
     "a": "3つの扉のうち、Aが当たり扉で、BとCはハズレ。司会者がランダムにBを開けても、最終的にはAを開けてもらうのが最善の戦略。",
     "ok": 0
    },
    "q15": {
     "a": "父が子のちょうど3倍になるのは10年後。",
     "ok": 0
    },
    "q16": {
     "a": "「わたしはきょうははれです」に「は」は2個ある。",
     "ok": 0
    },
    "q17": {
     "a": "\"complicated\"が13文字で、該当する単語は1つ。",
     "ok": 0
    },
    "q18": {
     "a": "「ちょうちょ」を拍(モーラ)で区切ると2+3+3+3+2=13に分かれる。",
     "ok": 0
    },
    "q19": {
     "a": "0.3 < 0.25 < 1/3 < 0.33",
     "ok": 0
    },
    "q20": {
     "a": "ボートが1往復しか使えない条件で、オオカミ・ヤギ・キャベツを全て無事に対岸へ運ぶ手順は存在しない。ボートが1往復しか使えないため、オオカミがヤギを食べたり、ヤギがキャベツを食べたりする空き時間が生じ、全員を無事に運ぶことができない。",
     "ok": 1
    }
   },
   "raw": "1. The surgeon who claimed to be the boy's father is actually the mother's husband, the boy's stepfather.\n2. The bat is 30 yen and the ball is 10 yen.\n3. 2026年2月29日は存在しない。leap yearの条件を満たさないため。\n4. \"complicated\"\n5. ちょうちょは5モーラ。\n6. サリーの姉妹は3×3=9人。\n7. 農夫・オオカミ・ヤギ・キャベツを全て無事に対岸へ運ぶことはできない。ボートが1往復しか使えないため、オオカミがヤギを食べたり、ヤギがキャベツを食べたりする空き時間が生じる。\n8. 今日が月曜なら、100日後は火曜日。\n9. kaleidoscope の7文字目は s。\n10. 水星。\n11. 2Lと4Lの容器だけで、ちょうど3Lを測ることはできない。3Lの倍数以外の容量を測るため。\n12. \"ございますおはよう\"。\n13. 父は執刀医で、子が10歳の時点で亡くなっている。\n14. 3つの扉のうち、Aが当たり扉で、BとCはハズレ。司会者がランダムにBを開けても、最終的にはAを開けてもらうのが最善の戦略。\n15. 父が子のちょうど3倍になるのは10年後。\n16. 「わたしはきょうははれです」に「は」は2個ある。\n17. \"complicated\"が13文字で、該当する単語は1つ。\n18. 「ちょうちょ」を拍(モーラ)で区切ると2+3+3+3+2=13に分かれる。\n19. 0.3 < 0.25 < 1/3 < 0.33\n20. ボートが1往復しか使えない条件で、オオカミ・ヤギ・キャベツを全て無事に対岸へ運ぶ手順は存在しない。ボートが1往復しか使えないため、オオカミがヤギを食べたり、ヤギがキャベツを食べたりする空き時間が生じ、全員を無事に運ぶことができない。"
  },
  "mistralai/mistral-small-24b-instruct-2501": {
   "score": 7,
   "answered": 20,
   "first10": 4,
   "last10": 3,
   "inconsistent_pairs": 1,
   "sec": 6.7,
   "out_tokens": 146,
   "price": "$0.05 / $0.08",
   "graded": {
    "q01": {
     "a": "その母親",
     "ok": 0
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "synchronization",
     "ok": 0
    },
    "q05": {
     "a": "3",
     "ok": 1
    },
    "q06": {
     "a": "6",
     "ok": 0
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "木曜",
     "ok": 0
    },
    "q09": {
     "a": "d",
     "ok": 0
    },
    "q10": {
     "a": "水星",
     "ok": 0
    },
    "q11": {
     "a": "できない",
     "ok": 1
    },
    "q12": {
     "a": "すうごはよ",
     "ok": 0
    },
    "q13": {
     "a": "同性愛者",
     "ok": 0
    },
    "q14": {
     "a": "変える",
     "ok": 0
    },
    "q15": {
     "a": "18",
     "ok": 0
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "19, 1",
     "ok": 0
    },
    "q18": {
     "a": "3",
     "ok": 1
    },
    "q19": {
     "a": "0.25, 0.3, 1/3, 0.33",
     "ok": 0
    },
    "q20": {
     "a": "存在する",
     "ok": 0
    }
   },
   "raw": "1: その母親\n2: 10\n3: しない\n4: synchronization\n5: 3\n6: 6\n7: できない\n8: 木曜\n9: d\n10: 水星\n11: できない\n12: すうごはよ\n13: 同性愛者\n14: 変える\n15: 18\n16: 3\n17: 19, 1\n18: 3\n19: 0.25, 0.3, 1/3, 0.33\n20: 存在する"
  },
  "qwen/qwen-plus": {
   "score": 9,
   "answered": 20,
   "first10": 6,
   "last10": 3,
   "inconsistent_pairs": 1,
   "sec": 3.7,
   "out_tokens": 148,
   "price": "$0.26 / $0.78",
   "graded": {
    "q01": {
     "a": "母親",
     "ok": 0
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "synchronization",
     "ok": 0
    },
    "q05": {
     "a": "4",
     "ok": 0
    },
    "q06": {
     "a": "2",
     "ok": 1
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "木曜日",
     "ok": 0
    },
    "q09": {
     "a": "o",
     "ok": 1
    },
    "q10": {
     "a": "火星",
     "ok": 1
    },
    "q11": {
     "a": "できる",
     "ok": 0
    },
    "q12": {
     "a": "ございますようはお",
     "ok": 0
    },
    "q13": {
     "a": "母親",
     "ok": 0
    },
    "q14": {
     "a": "同じ",
     "ok": 1
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "4",
     "ok": 0
    },
    "q17": {
     "a": "16文字, 1つ",
     "ok": 0
    },
    "q18": {
     "a": "4",
     "ok": 0
    },
    "q19": {
     "a": "0.25, 0.3, 0.33, 1/3",
     "ok": 1
    },
    "q20": {
     "a": "存在する",
     "ok": 0
    }
   },
   "raw": "1: 母親  \n2: 10  \n3: しない  \n4: synchronization  \n5: 4  \n6: 2  \n7: できない  \n8: 木曜日  \n9: o  \n10: 火星  \n11: できる  \n12: ございますようはお  \n13: 母親  \n14: 同じ  \n15: 5  \n16: 4  \n17: 16文字, 1つ  \n18: 4  \n19: 0.25, 0.3, 0.33, 1/3  \n20: 存在する"
  },
  "cohere/command-r7b-12-2024": {
   "score": 5,
   "answered": 20,
   "first10": 2,
   "last10": 3,
   "inconsistent_pairs": 0,
   "sec": 4.0,
   "out_tokens": 184,
   "price": "$0.0375 / $0.15",
   "graded": {
    "q01": {
     "a": "外科医は「この子は私の息子だ」と言った。外科医は親。",
     "ok": 0
    },
    "q02": {
     "a": "50円。",
     "ok": 0
    },
    "q03": {
     "a": "する。",
     "ok": 0
    },
    "q04": {
     "a": "synchronization",
     "ok": 0
    },
    "q05": {
     "a": "2。",
     "ok": 0
    },
    "q06": {
     "a": "9人。",
     "ok": 0
    },
    "q07": {
     "a": "できない。",
     "ok": 1
    },
    "q08": {
     "a": "水曜日。",
     "ok": 1
    },
    "q09": {
     "a": "p。",
     "ok": 0
    },
    "q10": {
     "a": "水星。",
     "ok": 0
    },
    "q11": {
     "a": "できない。",
     "ok": 1
    },
    "q12": {
     "a": "すおようございます。",
     "ok": 0
    },
    "q13": {
     "a": "執刀医は親。",
     "ok": 0
    },
    "q14": {
     "a": "変える。",
     "ok": 0
    },
    "q15": {
     "a": "10年後。",
     "ok": 0
    },
    "q16": {
     "a": "2個。",
     "ok": 0
    },
    "q17": {
     "a": "12文字、4個。",
     "ok": 0
    },
    "q18": {
     "a": "2。",
     "ok": 0
    },
    "q19": {
     "a": "0.25, 0.3, 0.33, 1/3。",
     "ok": 1
    },
    "q20": {
     "a": "しない。",
     "ok": 1
    }
   },
   "raw": "1. 外科医は「この子は私の息子だ」と言った。外科医は親。\n2. 50円。\n3. する。\n4.  synchronization\n5. 2。\n6. 9人。\n7. できない。\n8. 水曜日。\n9. p。\n10. 水星。\n11. できない。\n12. すおようございます。\n13. 執刀医は親。\n14. 変える。\n15. 10年後。\n16. 2個。\n17. 12文字、4個。\n18. 2。\n19. 0.25, 0.3, 0.33, 1/3。\n20. しない。"
  },
  "sao10k/l3.3-euryale-70b": {
   "score": 7,
   "answered": 20,
   "first10": 5,
   "last10": 2,
   "inconsistent_pairs": 1,
   "sec": 11.3,
   "out_tokens": 190,
   "price": "$0.65 / $0.75",
   "graded": {
    "q01": {
     "a": "1. 母親",
     "ok": 0
    },
    "q02": {
     "a": "2. 10",
     "ok": 1
    },
    "q03": {
     "a": "3. しない",
     "ok": 1
    },
    "q04": {
     "a": "4. synchronization",
     "ok": 0
    },
    "q05": {
     "a": "5. 3",
     "ok": 1
    },
    "q06": {
     "a": "6. 2",
     "ok": 1
    },
    "q07": {
     "a": "7. できる",
     "ok": 0
    },
    "q08": {
     "a": "8. 金曜日",
     "ok": 0
    },
    "q09": {
     "a": "9. o",
     "ok": 1
    },
    "q10": {
     "a": "10. 金星",
     "ok": 0
    },
    "q11": {
     "a": "11. できる",
     "ok": 0
    },
    "q12": {
     "a": "12. すむざいおはよ",
     "ok": 0
    },
    "q13": {
     "a": "13. 母",
     "ok": 0
    },
    "q14": {
     "a": "14. 変える",
     "ok": 0
    },
    "q15": {
     "a": "15. 10",
     "ok": 0
    },
    "q16": {
     "a": "16. 2",
     "ok": 0
    },
    "q17": {
     "a": "17. 17文字、1",
     "ok": 0
    },
    "q18": {
     "a": "18. 3",
     "ok": 1
    },
    "q19": {
     "a": "19. 0.25, 0.3, 1/3, 0.33",
     "ok": 0
    },
    "q20": {
     "a": "20. しない",
     "ok": 1
    }
   },
   "raw": "1: 1. 母親\n2: 2. 10\n3: 3. しない\n4: 4. synchronization\n5: 5. 3\n6: 6. 2\n7: 7. できる\n8: 8. 金曜日\n9: 9. o\n10: 10. 金星\n11: 11. できる\n12: 12. すむざいおはよ\n13: 13. 母\n14: 14. 変える\n15: 15. 10\n16: 16. 2\n17: 17. 17文字、1\n18: 18. 3\n19: 19. 0.25, 0.3, 1/3, 0.33\n20: 20. しない"
  },
  "meta-llama/llama-3.3-70b-instruct": {
   "score": 7,
   "answered": 20,
   "first10": 4,
   "last10": 3,
   "inconsistent_pairs": 0,
   "sec": 3.5,
   "out_tokens": 134,
   "price": "$0.1 / $0.32",
   "graded": {
    "q01": {
     "a": "外科医は少年の母親",
     "ok": 0
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "synchronization",
     "ok": 0
    },
    "q05": {
     "a": "3",
     "ok": 1
    },
    "q06": {
     "a": "2",
     "ok": 1
    },
    "q07": {
     "a": "できる",
     "ok": 0
    },
    "q08": {
     "a": "木曜日",
     "ok": 0
    },
    "q09": {
     "a": "e",
     "ok": 0
    },
    "q10": {
     "a": "水星",
     "ok": 0
    },
    "q11": {
     "a": "できる",
     "ok": 0
    },
    "q12": {
     "a": "すすおはよ",
     "ok": 0
    },
    "q13": {
     "a": "母",
     "ok": 0
    },
    "q14": {
     "a": "変える",
     "ok": 0
    },
    "q15": {
     "a": "10",
     "ok": 0
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "17文字、1つ",
     "ok": 0
    },
    "q18": {
     "a": "3",
     "ok": 1
    },
    "q19": {
     "a": "0.25, 0.3, 0.33, 1/3",
     "ok": 1
    },
    "q20": {
     "a": "存在する",
     "ok": 0
    }
   },
   "raw": "1: 外科医は少年の母親\n2: 10\n3: しない\n4: synchronization\n5: 3\n6: 2\n7: できる\n8: 木曜日\n9: e\n10: 水星\n11: できる\n12: すすおはよ\n13: 母\n14: 変える\n15: 10\n16: 3\n17: 17文字、1つ\n18: 3\n19: 0.25, 0.3, 0.33, 1/3\n20: 存在する"
  },
  "amazon/nova-lite-v1": {
   "score": 8,
   "answered": 20,
   "first10": 5,
   "last10": 3,
   "inconsistent_pairs": 1,
   "sec": 2.9,
   "out_tokens": 201,
   "price": "$0.06 / $0.24",
   "graded": {
    "q01": {
     "a": "番号: 外科医",
     "ok": 0
    },
    "q02": {
     "a": "番号: 10",
     "ok": 1
    },
    "q03": {
     "a": "番号: しない",
     "ok": 1
    },
    "q04": {
     "a": "番号: synchronization",
     "ok": 0
    },
    "q05": {
     "a": "番号: 3",
     "ok": 1
    },
    "q06": {
     "a": "番号: 3",
     "ok": 0
    },
    "q07": {
     "a": "番号: できる",
     "ok": 0
    },
    "q08": {
     "a": "番号: 日",
     "ok": 0
    },
    "q09": {
     "a": "番号: o",
     "ok": 1
    },
    "q10": {
     "a": "番号: 火星",
     "ok": 1
    },
    "q11": {
     "a": "番号: できる",
     "ok": 0
    },
    "q12": {
     "a": "番号: すみようお",
     "ok": 0
    },
    "q13": {
     "a": "番号: 父",
     "ok": 1
    },
    "q14": {
     "a": "番号: 変える",
     "ok": 0
    },
    "q15": {
     "a": "番号: 20",
     "ok": 0
    },
    "q16": {
     "a": "番号: 3",
     "ok": 1
    },
    "q17": {
     "a": "番号: 11, synchronization",
     "ok": 0
    },
    "q18": {
     "a": "番号: 3",
     "ok": 1
    },
    "q19": {
     "a": "番号: 0.25, 0.3, 1/3, 0.33",
     "ok": 0
    },
    "q20": {
     "a": "番号: 存在する",
     "ok": 0
    }
   },
   "raw": "1. 番号: 外科医\n2. 番号: 10\n3. 番号: しない\n4. 番号: synchronization\n5. 番号: 3\n6. 番号: 3\n7. 番号: できる\n8. 番号: 日\n9. 番号: o\n10. 番号: 火星\n11. 番号: できる\n12. 番号: すみようお\n13. 番号: 父\n14. 番号: 変える\n15. 番号: 20\n16. 番号: 3\n17. 番号: 11, synchronization\n18. 番号: 3\n19. 番号: 0.25, 0.3, 1/3, 0.33\n20. 番号: 存在する"
  },
  "amazon/nova-micro-v1": {
   "score": 9,
   "answered": 20,
   "first10": 4,
   "last10": 5,
   "inconsistent_pairs": 1,
   "sec": 2.4,
   "out_tokens": 204,
   "price": "$0.035 / $0.14",
   "graded": {
    "q01": {
     "a": "番号: 母親",
     "ok": 0
    },
    "q02": {
     "a": "番号: 20円",
     "ok": 0
    },
    "q03": {
     "a": "番号: しない",
     "ok": 1
    },
    "q04": {
     "a": "番号: complicated",
     "ok": 0
    },
    "q05": {
     "a": "番号: 3",
     "ok": 1
    },
    "q06": {
     "a": "番号: 9",
     "ok": 0
    },
    "q07": {
     "a": "番号: できない",
     "ok": 1
    },
    "q08": {
     "a": "番号: 火曜日",
     "ok": 0
    },
    "q09": {
     "a": "番号: o",
     "ok": 1
    },
    "q10": {
     "a": "番号: 水星",
     "ok": 0
    },
    "q11": {
     "a": "番号: できない",
     "ok": 1
    },
    "q12": {
     "a": "番号: ますおはよう",
     "ok": 0
    },
    "q13": {
     "a": "番号: 父親",
     "ok": 1
    },
    "q14": {
     "a": "番号: 同じ",
     "ok": 1
    },
    "q15": {
     "a": "番号: 20年",
     "ok": 0
    },
    "q16": {
     "a": "番号: 2",
     "ok": 0
    },
    "q17": {
     "a": "番号: 15文字、1",
     "ok": 0
    },
    "q18": {
     "a": "番号: 3",
     "ok": 1
    },
    "q19": {
     "a": "番号: 0.25, 1/3, 0.3, 0.33",
     "ok": 0
    },
    "q20": {
     "a": "番号: しない",
     "ok": 1
    }
   },
   "raw": "1. 番号: 母親\n2. 番号: 20円\n3. 番号: しない\n4. 番号: complicated\n5. 番号: 3\n6. 番号: 9\n7. 番号: できない\n8. 番号: 火曜日\n9. 番号: o\n10. 番号: 水星\n11. 番号: できない\n12. 番号: ますおはよう\n13. 番号: 父親\n14. 番号: 同じ\n15. 番号: 20年\n16. 番号: 2\n17. 番号: 15文字、1\n18. 番号: 3\n19. 番号: 0.25, 1/3, 0.3, 0.33\n20. 番号: しない"
  },
  "deepseek/deepseek-chat": {
   "score": 8,
   "answered": 20,
   "first10": 5,
   "last10": 3,
   "inconsistent_pairs": 0,
   "sec": 10.8,
   "out_tokens": 129,
   "price": "$0.32 / $0.89",
   "graded": {
    "q01": {
     "a": "母親",
     "ok": 0
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "3",
     "ok": 1
    },
    "q06": {
     "a": "3",
     "ok": 0
    },
    "q07": {
     "a": "できる",
     "ok": 0
    },
    "q08": {
     "a": "水曜日",
     "ok": 1
    },
    "q09": {
     "a": "e",
     "ok": 0
    },
    "q10": {
     "a": "火星",
     "ok": 1
    },
    "q11": {
     "a": "できる",
     "ok": 0
    },
    "q12": {
     "a": "すまざうよはお",
     "ok": 0
    },
    "q13": {
     "a": "母親",
     "ok": 0
    },
    "q14": {
     "a": "変える",
     "ok": 0
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "5",
     "ok": 0
    },
    "q17": {
     "a": "15文字、1つ",
     "ok": 0
    },
    "q18": {
     "a": "3",
     "ok": 1
    },
    "q19": {
     "a": "0.25, 0.3, 0.33, 1/3",
     "ok": 1
    },
    "q20": {
     "a": "存在する",
     "ok": 0
    }
   },
   "raw": "1. 母親\n2. 10\n3. しない\n4. extraordinarily\n5. 3\n6. 3\n7. できる\n8. 水曜日\n9. e\n10. 火星\n11. できる\n12. すまざうよはお\n13. 母親\n14. 変える\n15. 5\n16. 5\n17. 15文字、1つ\n18. 3\n19. 0.25, 0.3, 0.33, 1/3\n20. 存在する"
  },
  "openai/o3-mini": {
   "score": 15,
   "answered": 20,
   "first10": 8,
   "last10": 7,
   "inconsistent_pairs": 0,
   "sec": 13.8,
   "out_tokens": 2764,
   "price": "$1.1 / $4.4",
   "graded": {
    "q01": {
     "a": "母親",
     "ok": 0
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "3",
     "ok": 1
    },
    "q06": {
     "a": "2",
     "ok": 1
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "水曜日",
     "ok": 1
    },
    "q09": {
     "a": "o",
     "ok": 1
    },
    "q10": {
     "a": "火星",
     "ok": 1
    },
    "q11": {
     "a": "できる",
     "ok": 0
    },
    "q12": {
     "a": "すまいざごうよはお",
     "ok": 1
    },
    "q13": {
     "a": "母親",
     "ok": 0
    },
    "q14": {
     "a": "同じ",
     "ok": 1
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "16,1",
     "ok": 0
    },
    "q18": {
     "a": "3",
     "ok": 1
    },
    "q19": {
     "a": "0.25, 0.3, 0.33, 1/3",
     "ok": 1
    },
    "q20": {
     "a": "しない",
     "ok": 1
    }
   },
   "raw": "1: 母親\n2: 10\n3: しない\n4: extraordinarily\n5: 3\n6: 2\n7: できない\n8: 水曜日\n9: o\n10: 火星\n11: できる\n12: すまいざごうよはお\n13: 母親\n14: 同じ\n15: 5\n16: 3\n17: 16,1\n18: 3\n19: 0.25, 0.3, 0.33, 1/3\n20: しない"
  },
  "amazon/nova-pro-v1": {
   "score": 4,
   "answered": 20,
   "first10": 3,
   "last10": 1,
   "inconsistent_pairs": 0,
   "sec": 1.9,
   "out_tokens": 209,
   "price": "$0.8 / $3.2",
   "graded": {
    "q01": {
     "a": "番号: 外科医は母親",
     "ok": 0
    },
    "q02": {
     "a": "番号: 10",
     "ok": 1
    },
    "q03": {
     "a": "番号: しない",
     "ok": 1
    },
    "q04": {
     "a": "番号: synchronization",
     "ok": 0
    },
    "q05": {
     "a": "番号: 4",
     "ok": 0
    },
    "q06": {
     "a": "番号: 3",
     "ok": 0
    },
    "q07": {
     "a": "番号: できない",
     "ok": 1
    },
    "q08": {
     "a": "番号: 木曜日",
     "ok": 0
    },
    "q09": {
     "a": "番号: p",
     "ok": 0
    },
    "q10": {
     "a": "番号: 水星",
     "ok": 0
    },
    "q11": {
     "a": "番号: できる",
     "ok": 0
    },
    "q12": {
     "a": "番号: すうございますはようお",
     "ok": 0
    },
    "q13": {
     "a": "番号: 母親",
     "ok": 0
    },
    "q14": {
     "a": "番号: 変える",
     "ok": 0
    },
    "q15": {
     "a": "番号: 20",
     "ok": 0
    },
    "q16": {
     "a": "番号: 4",
     "ok": 0
    },
    "q17": {
     "a": "番号: 19, 1",
     "ok": 0
    },
    "q18": {
     "a": "番号: 4",
     "ok": 0
    },
    "q19": {
     "a": "番号: 0.25, 1/3, 0.3, 0.33",
     "ok": 0
    },
    "q20": {
     "a": "番号: しない",
     "ok": 1
    }
   },
   "raw": "1. 番号: 外科医は母親\n2. 番号: 10\n3. 番号: しない\n4. 番号: synchronization\n5. 番号: 4\n6. 番号: 3\n7. 番号: できない\n8. 番号: 木曜日\n9. 番号: p\n10. 番号: 水星\n11. 番号: できる\n12. 番号: すうございますはようお\n13. 番号: 母親\n14. 番号: 変える\n15. 番号: 20\n16. 番号: 4\n17. 番号: 19, 1\n18. 番号: 4\n19. 番号: 0.25, 1/3, 0.3, 0.33\n20. 番号: しない"
  },
  "openai/o1": {
   "score": 17,
   "answered": 20,
   "first10": 8,
   "last10": 9,
   "inconsistent_pairs": 1,
   "sec": 43.9,
   "out_tokens": 5777,
   "price": "$15 / $60",
   "graded": {
    "q01": {
     "a": "母親",
     "ok": 0
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "3",
     "ok": 1
    },
    "q06": {
     "a": "2",
     "ok": 1
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "水曜",
     "ok": 1
    },
    "q09": {
     "a": "o",
     "ok": 1
    },
    "q10": {
     "a": "火星",
     "ok": 1
    },
    "q11": {
     "a": "できない",
     "ok": 1
    },
    "q12": {
     "a": "すまいざごうよはお",
     "ok": 1
    },
    "q13": {
     "a": "父子",
     "ok": 1
    },
    "q14": {
     "a": "同じ",
     "ok": 1
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "14文字で2つ",
     "ok": 0
    },
    "q18": {
     "a": "3",
     "ok": 1
    },
    "q19": {
     "a": "0.25, 0.3, 0.33, 1/3",
     "ok": 1
    },
    "q20": {
     "a": "存在しない",
     "ok": 1
    }
   },
   "raw": "1: 母親  \n2: 10  \n3: しない  \n4: extraordinarily  \n5: 3  \n6: 2  \n7: できない  \n8: 水曜  \n9: o  \n10: 火星  \n11: できない  \n12: すまいざごうよはお  \n13: 父子  \n14: 同じ  \n15: 5  \n16: 3  \n17: 14文字で2つ  \n18: 3  \n19: 0.25, 0.3, 0.33, 1/3  \n20: 存在しない"
  },
  "thedrummer/unslopnemo-12b": {
   "score": 3,
   "answered": 20,
   "first10": 2,
   "last10": 1,
   "inconsistent_pairs": 1,
   "sec": 5.1,
   "out_tokens": 147,
   "price": "$0.4 / $0.4",
   "graded": {
    "q01": {
     "a": "父",
     "ok": 1
    },
    "q02": {
     "a": "20",
     "ok": 0
    },
    "q03": {
     "a": "する",
     "ok": 0
    },
    "q04": {
     "a": "synchronization",
     "ok": 0
    },
    "q05": {
     "a": "2",
     "ok": 0
    },
    "q06": {
     "a": "3",
     "ok": 0
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "火曜日",
     "ok": 0
    },
    "q09": {
     "a": "e",
     "ok": 0
    },
    "q10": {
     "a": "土星",
     "ok": 0
    },
    "q11": {
     "a": "できる",
     "ok": 0
    },
    "q12": {
     "a": "ます、ごう、よう、お",
     "ok": 0
    },
    "q13": {
     "a": "父",
     "ok": 1
    },
    "q14": {
     "a": "変える",
     "ok": 0
    },
    "q15": {
     "a": "20",
     "ok": 0
    },
    "q16": {
     "a": "2",
     "ok": 0
    },
    "q17": {
     "a": "25文字、1",
     "ok": 0
    },
    "q18": {
     "a": "2",
     "ok": 0
    },
    "q19": {
     "a": "0.25, 1/3, 0.3, 0.33",
     "ok": 0
    },
    "q20": {
     "a": "存在する",
     "ok": 0
    }
   },
   "raw": "1: 父\n2: 20\n3: する\n4: synchronization\n5: 2\n6: 3\n7: できない\n8: 火曜日\n9: e\n10: 土星\n11: できる\n12: ます、ごう、よう、お\n13: 父\n14: 変える\n15: 20\n16: 2\n17: 25文字、1\n18: 2\n19: 0.25, 1/3, 0.3, 0.33\n20: 存在する"
  },
  "mistralai/mistral-large-2407": {
   "score": 8,
   "answered": 20,
   "first10": 5,
   "last10": 3,
   "inconsistent_pairs": 0,
   "sec": 4.8,
   "out_tokens": 150,
   "price": "$2 / $6",
   "graded": {
    "q01": {
     "a": "母親",
     "ok": 0
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "4",
     "ok": 0
    },
    "q06": {
     "a": "1",
     "ok": 0
    },
    "q07": {
     "a": "できる",
     "ok": 0
    },
    "q08": {
     "a": "水曜日",
     "ok": 1
    },
    "q09": {
     "a": "o",
     "ok": 1
    },
    "q10": {
     "a": "火星",
     "ok": 1
    },
    "q11": {
     "a": "できる",
     "ok": 0
    },
    "q12": {
     "a": "すはまいざおごうよお",
     "ok": 0
    },
    "q13": {
     "a": "母子",
     "ok": 0
    },
    "q14": {
     "a": "同じ",
     "ok": 1
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "15文字, 1つ",
     "ok": 0
    },
    "q18": {
     "a": "4",
     "ok": 0
    },
    "q19": {
     "a": "0.25, 1/3, 0.3, 0.33",
     "ok": 0
    },
    "q20": {
     "a": "存在する",
     "ok": 0
    }
   },
   "raw": "1: 母親\n2: 10\n3: しない\n4: extraordinarily\n5: 4\n6: 1\n7: できる\n8: 水曜日\n9: o\n10: 火星\n11: できる\n12: すはまいざおごうよお\n13: 母子\n14: 同じ\n15: 5\n16: 3\n17: 15文字, 1つ\n18: 4\n19: 0.25, 1/3, 0.3, 0.33\n20: 存在する"
  },
  "meta-llama/llama-3.2-1b-instruct": {
   "score": 3,
   "answered": 20,
   "first10": 1,
   "last10": 2,
   "inconsistent_pairs": 0,
   "sec": 2.3,
   "out_tokens": 116,
   "price": "$0.027 / $0.201",
   "graded": {
    "q01": {
     "a": "医師",
     "ok": 0
    },
    "q02": {
     "a": "120",
     "ok": 0
    },
    "q03": {
     "a": "あり",
     "ok": 0
    },
    "q04": {
     "a": "あり",
     "ok": 0
    },
    "q05": {
     "a": "3",
     "ok": 1
    },
    "q06": {
     "a": "3",
     "ok": 0
    },
    "q07": {
     "a": "できる",
     "ok": 0
    },
    "q08": {
     "a": "1",
     "ok": 0
    },
    "q09": {
     "a": "1",
     "ok": 0
    },
    "q10": {
     "a": "さくら",
     "ok": 0
    },
    "q11": {
     "a": "できる",
     "ok": 0
    },
    "q12": {
     "a": "え",
     "ok": 0
    },
    "q13": {
     "a": "あり",
     "ok": 0
    },
    "q14": {
     "a": "あり",
     "ok": 0
    },
    "q15": {
     "a": "3",
     "ok": 0
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "3",
     "ok": 0
    },
    "q18": {
     "a": "3",
     "ok": 1
    },
    "q19": {
     "a": "1/3、1/2、1/3、1/4",
     "ok": 0
    },
    "q20": {
     "a": "あり",
     "ok": 0
    }
   },
   "raw": "1.  医師\n2. 120\n3. あり\n4. あり\n5. 3\n6. 3\n7. できる\n8. 1\n9. 1\n10. さくら\n11. できる\n12. え\n13. あり\n14. あり\n15. 3\n16. 3\n17. 3\n18. 3\n19. 1/3、1/2、1/3、1/4\n20. あり"
  },
  "meta-llama/llama-3.2-3b-instruct": {
   "score": 4,
   "answered": 20,
   "first10": 3,
   "last10": 1,
   "inconsistent_pairs": 2,
   "sec": 2.0,
   "out_tokens": 106,
   "price": "$0.05 / $0.33",
   "graded": {
    "q01": {
     "a": "外科医",
     "ok": 0
    },
    "q02": {
     "a": "20",
     "ok": 0
    },
    "q03": {
     "a": "する",
     "ok": 0
    },
    "q04": {
     "a": "synchronization",
     "ok": 0
    },
    "q05": {
     "a": "5",
     "ok": 0
    },
    "q06": {
     "a": "9",
     "ok": 0
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "水曜日",
     "ok": 1
    },
    "q09": {
     "a": "o",
     "ok": 1
    },
    "q10": {
     "a": "水星",
     "ok": 0
    },
    "q11": {
     "a": "できる",
     "ok": 0
    },
    "q12": {
     "a": "ごようごようございます",
     "ok": 0
    },
    "q13": {
     "a": "兄弟",
     "ok": 0
    },
    "q14": {
     "a": "変える",
     "ok": 0
    },
    "q15": {
     "a": "30",
     "ok": 0
    },
    "q16": {
     "a": "1",
     "ok": 0
    },
    "q17": {
     "a": "synchronization",
     "ok": 0
    },
    "q18": {
     "a": "3",
     "ok": 1
    },
    "q19": {
     "a": "0.25",
     "ok": 0
    },
    "q20": {
     "a": "あり",
     "ok": 0
    }
   },
   "raw": "1. 外科医\n2. 20\n3. する\n4. synchronization\n5. 5\n6. 9\n7. できない\n8. 水曜日\n9. o\n10. 水星\n11. できる\n12. ごようごようございます\n13. 兄弟\n14. 変える\n15. 30\n16. 1\n17. synchronization\n18. 3\n19. 0.25\n20. あり"
  },
  "openai/gpt-4o-2024-11-20": {
   "score": 8,
   "answered": 20,
   "first10": 6,
   "last10": 2,
   "inconsistent_pairs": 1,
   "sec": 2.8,
   "out_tokens": 146,
   "price": "$2.5 / $10",
   "graded": {
    "q01": {
     "a": "外科医は父親以外の親、例えば母親。",
     "ok": 1
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "する",
     "ok": 0
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "4",
     "ok": 0
    },
    "q06": {
     "a": "2",
     "ok": 1
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "水曜日",
     "ok": 1
    },
    "q09": {
     "a": "s",
     "ok": 0
    },
    "q10": {
     "a": "火星",
     "ok": 1
    },
    "q11": {
     "a": "できる",
     "ok": 0
    },
    "q12": {
     "a": "すまいざごうはお",
     "ok": 0
    },
    "q13": {
     "a": "母親",
     "ok": 0
    },
    "q14": {
     "a": "変える",
     "ok": 0
    },
    "q15": {
     "a": "10",
     "ok": 0
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "14文字、1つ",
     "ok": 0
    },
    "q18": {
     "a": "4",
     "ok": 0
    },
    "q19": {
     "a": "0.25, 0.3, 1/3, 0.33",
     "ok": 0
    },
    "q20": {
     "a": "存在しない",
     "ok": 1
    }
   },
   "raw": "1. 外科医は父親以外の親、例えば母親。  \n2. 10  \n3. する  \n4. extraordinarily  \n5. 4  \n6. 2  \n7. できない  \n8. 水曜日  \n9. s  \n10. 火星  \n11. できる  \n12. すまいざごうはお  \n13. 母親  \n14. 変える  \n15. 10  \n16. 3  \n17. 14文字、1つ  \n18. 4  \n19. 0.25, 0.3, 1/3, 0.33  \n20. 存在しない"
  },
  "qwen/qwen-2.5-7b-instruct": {
   "score": 3,
   "answered": 20,
   "first10": 2,
   "last10": 1,
   "inconsistent_pairs": 1,
   "sec": 4.6,
   "out_tokens": 157,
   "price": "$0.1 / $0.2",
   "graded": {
    "q01": {
     "a": "外科医は母親の夫",
     "ok": 0
    },
    "q02": {
     "a": "5",
     "ok": 0
    },
    "q03": {
     "a": "不する",
     "ok": 0
    },
    "q04": {
     "a": "synchronization",
     "ok": 0
    },
    "q05": {
     "a": "3",
     "ok": 1
    },
    "q06": {
     "a": "9",
     "ok": 0
    },
    "q07": {
     "a": "できる",
     "ok": 0
    },
    "q08": {
     "a": "日曜",
     "ok": 0
    },
    "q09": {
     "a": "o",
     "ok": 1
    },
    "q10": {
     "a": "地球",
     "ok": 0
    },
    "q11": {
     "a": "できる",
     "ok": 0
    },
    "q12": {
     "a": "すもももももももものうち",
     "ok": 0
    },
    "q13": {
     "a": "母親の息子",
     "ok": 0
    },
    "q14": {
     "a": "変える",
     "ok": 0
    },
    "q15": {
     "a": "20",
     "ok": 0
    },
    "q16": {
     "a": "2",
     "ok": 0
    },
    "q17": {
     "a": "10, 2",
     "ok": 0
    },
    "q18": {
     "a": "4",
     "ok": 0
    },
    "q19": {
     "a": "0.25, 0.3, 0.33, 1/3",
     "ok": 1
    },
    "q20": {
     "a": "存在する",
     "ok": 0
    }
   },
   "raw": "1: 外科医は母親の夫\n2: 5\n3: 不する\n4: synchronization\n5: 3\n6: 9\n7: できる\n8: 日曜\n9: o\n10: 地球\n11: できる\n12: すもももももももものうち\n13: 母親の息子\n14: 変える\n15: 20\n16: 2\n17: 10, 2\n18: 4\n19: 0.25, 0.3, 0.33, 1/3\n20: 存在する"
  },
  "sao10k/l3.1-euryale-70b": {
   "score": 9,
   "answered": 20,
   "first10": 7,
   "last10": 2,
   "inconsistent_pairs": 1,
   "sec": 4.2,
   "out_tokens": 130,
   "price": "$0.85 / $0.85",
   "graded": {
    "q01": {
     "a": "母親",
     "ok": 0
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "synchronization",
     "ok": 0
    },
    "q05": {
     "a": "3",
     "ok": 1
    },
    "q06": {
     "a": "2",
     "ok": 1
    },
    "q07": {
     "a": "できる",
     "ok": 0
    },
    "q08": {
     "a": "水曜日",
     "ok": 1
    },
    "q09": {
     "a": "o",
     "ok": 1
    },
    "q10": {
     "a": "火星",
     "ok": 1
    },
    "q11": {
     "a": "できる",
     "ok": 0
    },
    "q12": {
     "a": "すもようはお",
     "ok": 0
    },
    "q13": {
     "a": "母親",
     "ok": 0
    },
    "q14": {
     "a": "変える",
     "ok": 0
    },
    "q15": {
     "a": "10",
     "ok": 0
    },
    "q16": {
     "a": "2",
     "ok": 0
    },
    "q17": {
     "a": "17文字、1つ",
     "ok": 0
    },
    "q18": {
     "a": "3",
     "ok": 1
    },
    "q19": {
     "a": "0.25, 0.3, 1/3, 0.33",
     "ok": 0
    },
    "q20": {
     "a": "しない",
     "ok": 1
    }
   },
   "raw": "1. 母親\n2. 10\n3. しない\n4. synchronization\n5. 3\n6. 2\n7. できる\n8. 水曜日\n9. o\n10. 火星\n11. できる\n12. すもようはお\n13. 母親\n14. 変える\n15. 10\n16. 2\n17. 17文字、1つ\n18. 3\n19. 0.25, 0.3, 1/3, 0.33\n20. しない"
  },
  "sao10k/l3-lunaris-8b": {
   "score": 5,
   "answered": 20,
   "first10": 3,
   "last10": 2,
   "inconsistent_pairs": 0,
   "sec": 3.7,
   "out_tokens": 139,
   "price": "$0.04 / $0.05",
   "graded": {
    "q01": {
     "a": "外科医は少年の父親。",
     "ok": 1
    },
    "q02": {
     "a": "ボールは20円。",
     "ok": 0
    },
    "q03": {
     "a": "しない。",
     "ok": 1
    },
    "q04": {
     "a": "complicated",
     "ok": 0
    },
    "q05": {
     "a": "4",
     "ok": 0
    },
    "q06": {
     "a": "9",
     "ok": 0
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "土曜日",
     "ok": 0
    },
    "q09": {
     "a": "i",
     "ok": 0
    },
    "q10": {
     "a": "ベスタ",
     "ok": 0
    },
    "q11": {
     "a": "できる",
     "ok": 0
    },
    "q12": {
     "a": "ございますおは",
     "ok": 0
    },
    "q13": {
     "a": "父親",
     "ok": 1
    },
    "q14": {
     "a": "変える",
     "ok": 0
    },
    "q15": {
     "a": "30",
     "ok": 0
    },
    "q16": {
     "a": "2",
     "ok": 0
    },
    "q17": {
     "a": "complicated (14文字) × 2 = 28文字",
     "ok": 0
    },
    "q18": {
     "a": "4",
     "ok": 0
    },
    "q19": {
     "a": "0.25, 1/3, 0.3, 0.33",
     "ok": 0
    },
    "q20": {
     "a": "できない",
     "ok": 1
    }
   },
   "raw": "1. 外科医は少年の父親。\n2. ボールは20円。\n3. しない。\n4. complicated\n5. 4\n6. 9\n7. できない\n8. 土曜日\n9. i\n10. ベスタ\n11. できる\n12. ございますおは\n13. 父親\n14. 変える\n15. 30\n16. 2\n17. complicated (14文字) × 2 = 28文字\n18. 4\n19. 0.25, 1/3, 0.3, 0.33\n20. できない"
  },
  "meta-llama/llama-3.1-8b-instruct": {
   "score": 4,
   "answered": 20,
   "first10": 3,
   "last10": 1,
   "inconsistent_pairs": 0,
   "sec": 2.0,
   "out_tokens": 130,
   "price": "$0.05 / $0.08",
   "graded": {
    "q01": {
     "a": "母親",
     "ok": 0
    },
    "q02": {
     "a": "20",
     "ok": 0
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "complicated",
     "ok": 0
    },
    "q05": {
     "a": "4",
     "ok": 0
    },
    "q06": {
     "a": "9",
     "ok": 0
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "水曜日",
     "ok": 1
    },
    "q09": {
     "a": "s",
     "ok": 0
    },
    "q10": {
     "a": "水星",
     "ok": 0
    },
    "q11": {
     "a": "できる",
     "ok": 0
    },
    "q12": {
     "a": "ようござい",
     "ok": 0
    },
    "q13": {
     "a": "兄弟",
     "ok": 0
    },
    "q14": {
     "a": "変える",
     "ok": 0
    },
    "q15": {
     "a": "10",
     "ok": 0
    },
    "q16": {
     "a": "1",
     "ok": 0
    },
    "q17": {
     "a": "complicated (12文字) 1つ",
     "ok": 0
    },
    "q18": {
     "a": "4",
     "ok": 0
    },
    "q19": {
     "a": "0.25, 0.3, 1/3, 0.33",
     "ok": 0
    },
    "q20": {
     "a": "できない",
     "ok": 1
    }
   },
   "raw": "1. 母親\n2. 20\n3. しない\n4. complicated\n5. 4\n6. 9\n7. できない\n8. 水曜日\n9. s\n10. 水星\n11. できる\n12. ようござい\n13. 兄弟\n14. 変える\n15. 10\n16. 1\n17. complicated (12文字) 1つ\n18. 4\n19. 0.25, 0.3, 1/3, 0.33\n20. できない"
  },
  "anthracite-org/magnum-v4-72b": {
   "score": 7,
   "answered": 20,
   "first10": 3,
   "last10": 4,
   "inconsistent_pairs": 1,
   "sec": 10.4,
   "out_tokens": 141,
   "price": "$2.5 / $5",
   "graded": {
    "q01": {
     "a": "母",
     "ok": 0
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "4",
     "ok": 0
    },
    "q06": {
     "a": "9",
     "ok": 0
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "木曜",
     "ok": 0
    },
    "q09": {
     "a": "s",
     "ok": 0
    },
    "q10": {
     "a": "マーキュリー",
     "ok": 0
    },
    "q11": {
     "a": "できる",
     "ok": 0
    },
    "q12": {
     "a": "す",
     "ok": 0
    },
    "q13": {
     "a": "父",
     "ok": 1
    },
    "q14": {
     "a": "変える",
     "ok": 0
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "13, 2",
     "ok": 0
    },
    "q18": {
     "a": "2",
     "ok": 0
    },
    "q19": {
     "a": "0.25, 1/3, 0.3, 0.33",
     "ok": 0
    },
    "q20": {
     "a": "しない",
     "ok": 1
    }
   },
   "raw": "1: 母\n2: 10\n3: しない\n4: extraordinarily\n5: 4\n6: 9\n7: できない\n8: 木曜\n9: s\n10: マーキュリー\n11: できる\n12: す\n13: 父\n14: 変える\n15: 5\n16: 3\n17: 13, 2\n18: 2\n19: 0.25, 1/3, 0.3, 0.33\n20: しない"
  },
  "openai/gpt-4o-mini": {
   "score": 7,
   "answered": 20,
   "first10": 4,
   "last10": 3,
   "inconsistent_pairs": 0,
   "sec": 3.2,
   "out_tokens": 130,
   "price": "$0.15 / $0.6",
   "graded": {
    "q01": {
     "a": "母親",
     "ok": 0
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "3",
     "ok": 1
    },
    "q06": {
     "a": "3",
     "ok": 0
    },
    "q07": {
     "a": "できる",
     "ok": 0
    },
    "q08": {
     "a": "木曜",
     "ok": 0
    },
    "q09": {
     "a": "a",
     "ok": 0
    },
    "q10": {
     "a": "火星",
     "ok": 1
    },
    "q11": {
     "a": "できる",
     "ok": 0
    },
    "q12": {
     "a": "すではよお",
     "ok": 0
    },
    "q13": {
     "a": "母子",
     "ok": 0
    },
    "q14": {
     "a": "変える",
     "ok": 0
    },
    "q15": {
     "a": "10",
     "ok": 0
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "13, 1",
     "ok": 0
    },
    "q18": {
     "a": "3",
     "ok": 1
    },
    "q19": {
     "a": "0.25, 0.3, 0.33, 1/3",
     "ok": 1
    },
    "q20": {
     "a": "存在する",
     "ok": 0
    }
   },
   "raw": "1: 母親  \n2: 10  \n3: しない  \n4: extraordinarily  \n5: 3  \n6: 3  \n7: できる  \n8: 木曜  \n9: a  \n10: 火星  \n11: できる  \n12: すではよお  \n13: 母子  \n14: 変える  \n15: 10  \n16: 3  \n17: 13, 1  \n18: 3  \n19: 0.25, 0.3, 0.33, 1/3  \n20: 存在する"
  },
  "openai/gpt-4o-2024-08-06": {
   "score": 11,
   "answered": 20,
   "first10": 8,
   "last10": 3,
   "inconsistent_pairs": 1,
   "sec": 3.0,
   "out_tokens": 137,
   "price": "$2.5 / $10",
   "graded": {
    "q01": {
     "a": "外科医は父親。",
     "ok": 1
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "4",
     "ok": 0
    },
    "q06": {
     "a": "2",
     "ok": 1
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "水曜日",
     "ok": 1
    },
    "q09": {
     "a": "o",
     "ok": 1
    },
    "q10": {
     "a": "火星",
     "ok": 1
    },
    "q11": {
     "a": "できる",
     "ok": 0
    },
    "q12": {
     "a": "すまいざごうはお",
     "ok": 0
    },
    "q13": {
     "a": "母親",
     "ok": 0
    },
    "q14": {
     "a": "変える",
     "ok": 0
    },
    "q15": {
     "a": "10",
     "ok": 0
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "14文字で1つ",
     "ok": 0
    },
    "q18": {
     "a": "4",
     "ok": 0
    },
    "q19": {
     "a": "0.25, 0.3, 0.33, 1/3",
     "ok": 1
    },
    "q20": {
     "a": "存在しない",
     "ok": 1
    }
   },
   "raw": "1. 外科医は父親。\n2. 10\n3. しない\n4. extraordinarily\n5. 4\n6. 2\n7. できない\n8. 水曜日\n9. o\n10. 火星\n11. できる\n12. すまいざごうはお\n13. 母親\n14. 変える\n15. 10\n16. 3\n17. 14文字で1つ\n18. 4\n19. 0.25, 0.3, 0.33, 1/3\n20. 存在しない"
  },
  "openai/gpt-4o-mini-2024-07-18": {
   "score": 7,
   "answered": 20,
   "first10": 4,
   "last10": 3,
   "inconsistent_pairs": 0,
   "sec": 3.1,
   "out_tokens": 129,
   "price": "$0.15 / $0.6",
   "graded": {
    "q01": {
     "a": "母親",
     "ok": 0
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "3",
     "ok": 1
    },
    "q06": {
     "a": "3",
     "ok": 0
    },
    "q07": {
     "a": "できる",
     "ok": 0
    },
    "q08": {
     "a": "木曜",
     "ok": 0
    },
    "q09": {
     "a": "a",
     "ok": 0
    },
    "q10": {
     "a": "火星",
     "ok": 1
    },
    "q11": {
     "a": "できる",
     "ok": 0
    },
    "q12": {
     "a": "すではよお",
     "ok": 0
    },
    "q13": {
     "a": "母",
     "ok": 0
    },
    "q14": {
     "a": "変える",
     "ok": 0
    },
    "q15": {
     "a": "10",
     "ok": 0
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "13, 1",
     "ok": 0
    },
    "q18": {
     "a": "3",
     "ok": 1
    },
    "q19": {
     "a": "0.25, 0.3, 0.33, 1/3",
     "ok": 1
    },
    "q20": {
     "a": "存在する",
     "ok": 0
    }
   },
   "raw": "1: 母親  \n2: 10  \n3: しない  \n4: extraordinarily  \n5: 3  \n6: 3  \n7: できる  \n8: 木曜  \n9: a  \n10: 火星  \n11: できる  \n12: すではよお  \n13: 母  \n14: 変える  \n15: 10  \n16: 3  \n17: 13, 1  \n18: 3  \n19: 0.25, 0.3, 0.33, 1/3  \n20: 存在する"
  },
  "meta-llama/llama-3.1-70b-instruct": {
   "score": 9,
   "answered": 20,
   "first10": 6,
   "last10": 3,
   "inconsistent_pairs": 1,
   "sec": 3.5,
   "out_tokens": 130,
   "price": "$0.4 / $0.4",
   "graded": {
    "q01": {
     "a": "母親",
     "ok": 0
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "synchronization",
     "ok": 0
    },
    "q05": {
     "a": "3",
     "ok": 1
    },
    "q06": {
     "a": "3",
     "ok": 0
    },
    "q07": {
     "a": "できる",
     "ok": 0
    },
    "q08": {
     "a": "水曜日",
     "ok": 1
    },
    "q09": {
     "a": "o",
     "ok": 1
    },
    "q10": {
     "a": "火星",
     "ok": 1
    },
    "q11": {
     "a": "できる",
     "ok": 0
    },
    "q12": {
     "a": "すもおはよ",
     "ok": 0
    },
    "q13": {
     "a": "母親",
     "ok": 0
    },
    "q14": {
     "a": "変える",
     "ok": 0
    },
    "q15": {
     "a": "10",
     "ok": 0
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "20文字、1つ",
     "ok": 0
    },
    "q18": {
     "a": "3",
     "ok": 1
    },
    "q19": {
     "a": "0.25, 0.3, 1/3, 0.33",
     "ok": 0
    },
    "q20": {
     "a": "しない",
     "ok": 1
    }
   },
   "raw": "1. 母親\n2. 10\n3. しない\n4. synchronization\n5. 3\n6. 3\n7. できる\n8. 水曜日\n9. o\n10. 火星\n11. できる\n12. すもおはよ\n13. 母親\n14. 変える\n15. 10\n16. 3\n17. 20文字、1つ\n18. 3\n19. 0.25, 0.3, 1/3, 0.33\n20. しない"
  },
  "qwen/qwen-2.5-72b-instruct": {
   "score": 10,
   "answered": 20,
   "first10": 4,
   "last10": 6,
   "inconsistent_pairs": 2,
   "sec": 8.0,
   "out_tokens": 143,
   "price": "$0.36 / $0.4",
   "graded": {
    "q01": {
     "a": "母親",
     "ok": 0
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "4",
     "ok": 0
    },
    "q06": {
     "a": "9",
     "ok": 0
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "水曜",
     "ok": 1
    },
    "q09": {
     "a": "d",
     "ok": 0
    },
    "q10": {
     "a": "マーキュリー",
     "ok": 0
    },
    "q11": {
     "a": "できる",
     "ok": 0
    },
    "q12": {
     "a": "す",
     "ok": 0
    },
    "q13": {
     "a": "父親",
     "ok": 1
    },
    "q14": {
     "a": "同じ",
     "ok": 1
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "15, 2",
     "ok": 1
    },
    "q18": {
     "a": "4",
     "ok": 0
    },
    "q19": {
     "a": "0.25, 1/3, 0.3, 0.33",
     "ok": 0
    },
    "q20": {
     "a": "しない",
     "ok": 1
    }
   },
   "raw": "1: 母親\n2: 10\n3: しない\n4: extraordinarily\n5: 4\n6: 9\n7: できない\n8: 水曜\n9: d\n10: マーキュリー\n11: できる\n12: す\n13: 父親\n14: 同じ\n15: 5\n16: 3\n17: 15, 2\n18: 4\n19: 0.25, 1/3, 0.3, 0.33\n20: しない"
  },
  "openai/gpt-4o-2024-05-13": {
   "score": 5,
   "answered": 20,
   "first10": 3,
   "last10": 2,
   "inconsistent_pairs": 0,
   "sec": 4.3,
   "out_tokens": 134,
   "price": "$5 / $15",
   "graded": {
    "q01": {
     "a": "母親",
     "ok": 0
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "4",
     "ok": 0
    },
    "q06": {
     "a": "3",
     "ok": 0
    },
    "q07": {
     "a": "できる",
     "ok": 0
    },
    "q08": {
     "a": "水曜日",
     "ok": 1
    },
    "q09": {
     "a": "s",
     "ok": 0
    },
    "q10": {
     "a": "水星",
     "ok": 0
    },
    "q11": {
     "a": "できる",
     "ok": 0
    },
    "q12": {
     "a": "すまいざごうはお",
     "ok": 0
    },
    "q13": {
     "a": "母親",
     "ok": 0
    },
    "q14": {
     "a": "変える",
     "ok": 0
    },
    "q15": {
     "a": "10",
     "ok": 0
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "15文字で1つ",
     "ok": 0
    },
    "q18": {
     "a": "4",
     "ok": 0
    },
    "q19": {
     "a": "0.25, 0.3, 0.33, 1/3",
     "ok": 1
    },
    "q20": {
     "a": "存在する",
     "ok": 0
    }
   },
   "raw": "1. 母親\n2. 10\n3. しない\n4. extraordinarily\n5. 4\n6. 3\n7. できる\n8. 水曜日\n9. s\n10. 水星\n11. できる\n12. すまいざごうはお\n13. 母親\n14. 変える\n15. 10\n16. 3\n17. 15文字で1つ\n18. 4\n19. 0.25, 0.3, 0.33, 1/3\n20. 存在する"
  },
  "google/gemma-2-27b-it": {
   "score": 2,
   "answered": 20,
   "first10": 2,
   "last10": 0,
   "inconsistent_pairs": 0,
   "sec": 4.4,
   "out_tokens": 139,
   "price": "$0.65 / $0.65",
   "graded": {
    "q01": {
     "a": "母",
     "ok": 0
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "する",
     "ok": 0
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "2",
     "ok": 0
    },
    "q06": {
     "a": "1",
     "ok": 0
    },
    "q07": {
     "a": "できる",
     "ok": 0
    },
    "q08": {
     "a": "水曜日",
     "ok": 1
    },
    "q09": {
     "a": "s",
     "ok": 0
    },
    "q10": {
     "a": "金星",
     "ok": 0
    },
    "q11": {
     "a": "できる",
     "ok": 0
    },
    "q12": {
     "a": "す",
     "ok": 0
    },
    "q13": {
     "a": "息子と執刀医は親子",
     "ok": 0
    },
    "q14": {
     "a": "変える",
     "ok": 0
    },
    "q15": {
     "a": "20",
     "ok": 0
    },
    "q16": {
     "a": "2",
     "ok": 0
    },
    "q17": {
     "a": "18, 1",
     "ok": 0
    },
    "q18": {
     "a": "2",
     "ok": 0
    },
    "q19": {
     "a": "0.25, 1/3, 0.3, 0.33",
     "ok": 0
    },
    "q20": {
     "a": "存在する",
     "ok": 0
    }
   },
   "raw": "1: 母\n2: 10\n3: する\n4: extraordinarily\n5: 2\n6: 1\n7: できる\n8: 水曜日\n9: s\n10: 金星\n11: できる\n12: す\n13: 息子と執刀医は親子\n14: 変える\n15: 20\n16: 2\n17: 18, 1\n18: 2\n19: 0.25, 1/3, 0.3, 0.33\n20: 存在する"
  },
  "mistralai/mixtral-8x22b-instruct": {
   "score": 10,
   "answered": 20,
   "first10": 6,
   "last10": 4,
   "inconsistent_pairs": 1,
   "sec": 2.4,
   "out_tokens": 145,
   "price": "$2 / $6",
   "graded": {
    "q01": {
     "a": "母親",
     "ok": 0
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "する",
     "ok": 0
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "3",
     "ok": 1
    },
    "q06": {
     "a": "2",
     "ok": 1
    },
    "q07": {
     "a": "できる",
     "ok": 0
    },
    "q08": {
     "a": "水曜日",
     "ok": 1
    },
    "q09": {
     "a": "o",
     "ok": 1
    },
    "q10": {
     "a": "火星",
     "ok": 1
    },
    "q11": {
     "a": "できる",
     "ok": 0
    },
    "q12": {
     "a": "すもよはごう",
     "ok": 0
    },
    "q13": {
     "a": "父親",
     "ok": 1
    },
    "q14": {
     "a": "変える",
     "ok": 0
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "12, 1",
     "ok": 0
    },
    "q18": {
     "a": "3",
     "ok": 1
    },
    "q19": {
     "a": "0.25, 1/3, 0.3, 0.33",
     "ok": 0
    },
    "q20": {
     "a": "存在する",
     "ok": 0
    }
   },
   "raw": "1: 母親\n2: 10\n3: する\n4: extraordinarily\n5: 3\n6: 2\n7: できる\n8: 水曜日\n9: o\n10: 火星\n11: できる\n12: すもよはごう\n13: 父親\n14: 変える\n15: 5\n16: 3\n17: 12, 1\n18: 3\n19: 0.25, 1/3, 0.3, 0.33\n20: 存在する"
  },
  "anthropic/claude-3-haiku": {
   "score": 2,
   "answered": 20,
   "first10": 2,
   "last10": 0,
   "inconsistent_pairs": 1,
   "sec": 5.2,
   "out_tokens": 153,
   "price": "$0.25 / $1.25",
   "graded": {
    "q01": {
     "a": "外科医は息子",
     "ok": 0
    },
    "q02": {
     "a": "20",
     "ok": 0
    },
    "q03": {
     "a": "する",
     "ok": 0
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "3",
     "ok": 1
    },
    "q06": {
     "a": "9",
     "ok": 0
    },
    "q07": {
     "a": "できる",
     "ok": 0
    },
    "q08": {
     "a": "月曜",
     "ok": 0
    },
    "q09": {
     "a": "o",
     "ok": 1
    },
    "q10": {
     "a": "水星",
     "ok": 0
    },
    "q11": {
     "a": "できる",
     "ok": 0
    },
    "q12": {
     "a": "す・ま・い・う・ご・は・お",
     "ok": 0
    },
    "q13": {
     "a": "外科医は母親",
     "ok": 0
    },
    "q14": {
     "a": "変える",
     "ok": 0
    },
    "q15": {
     "a": "10",
     "ok": 0
    },
    "q16": {
     "a": "2",
     "ok": 0
    },
    "q17": {
     "a": "13文字、1つ",
     "ok": 0
    },
    "q18": {
     "a": "2",
     "ok": 0
    },
    "q19": {
     "a": "0.25, 0.3, 1/3, 0.33",
     "ok": 0
    },
    "q20": {
     "a": "存在する",
     "ok": 0
    }
   },
   "raw": "1. 外科医は息子\n2. 20\n3. する\n4. extraordinarily\n5. 3\n6. 9\n7. できる\n8. 月曜\n9. o\n10. 水星\n11. できる\n12. す・ま・い・う・ご・は・お\n13. 外科医は母親\n14. 変える\n15. 10\n16. 2\n17. 13文字、1つ\n18. 2\n19. 0.25, 0.3, 1/3, 0.33\n20. 存在する"
  },
  "openai/gpt-3.5-turbo-0613": {
   "score": 11,
   "answered": 20,
   "first10": 6,
   "last10": 5,
   "inconsistent_pairs": 0,
   "sec": 4.2,
   "out_tokens": 133,
   "price": "$1 / $2",
   "graded": {
    "q01": {
     "a": "母親",
     "ok": 0
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "3",
     "ok": 1
    },
    "q06": {
     "a": "3",
     "ok": 0
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "水曜",
     "ok": 1
    },
    "q09": {
     "a": "s",
     "ok": 0
    },
    "q10": {
     "a": "火星",
     "ok": 1
    },
    "q11": {
     "a": "できる",
     "ok": 0
    },
    "q12": {
     "a": "すまいござうよはお",
     "ok": 0
    },
    "q13": {
     "a": "母親",
     "ok": 0
    },
    "q14": {
     "a": "変える",
     "ok": 0
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "15文字、1つ",
     "ok": 0
    },
    "q18": {
     "a": "3",
     "ok": 1
    },
    "q19": {
     "a": "0.25, 0.3, 0.33, 1/3",
     "ok": 1
    },
    "q20": {
     "a": "しない",
     "ok": 1
    }
   },
   "raw": "1: 母親  \n2: 10  \n3: しない  \n4: extraordinarily  \n5: 3  \n6: 3  \n7: できない  \n8: 水曜  \n9: s  \n10: 火星  \n11: できる  \n12: すまいござうよはお  \n13: 母親  \n14: 変える  \n15: 5  \n16: 3  \n17: 15文字、1つ  \n18: 3  \n19: 0.25, 0.3, 0.33, 1/3  \n20: しない"
  },
  "openai/gpt-4o": {
   "score": 12,
   "answered": 20,
   "first10": 8,
   "last10": 4,
   "inconsistent_pairs": 0,
   "sec": 3.0,
   "out_tokens": 141,
   "price": "$2.5 / $10",
   "graded": {
    "q01": {
     "a": "外科医は父親。",
     "ok": 1
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "4",
     "ok": 0
    },
    "q06": {
     "a": "2",
     "ok": 1
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "水曜日",
     "ok": 1
    },
    "q09": {
     "a": "o",
     "ok": 1
    },
    "q10": {
     "a": "火星",
     "ok": 1
    },
    "q11": {
     "a": "できる",
     "ok": 0
    },
    "q12": {
     "a": "すまいざごうはお",
     "ok": 0
    },
    "q13": {
     "a": "執刀医は父親。",
     "ok": 1
    },
    "q14": {
     "a": "変える",
     "ok": 0
    },
    "q15": {
     "a": "10",
     "ok": 0
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "14文字で1つ",
     "ok": 0
    },
    "q18": {
     "a": "4",
     "ok": 0
    },
    "q19": {
     "a": "0.25, 0.3, 0.33, 1/3",
     "ok": 1
    },
    "q20": {
     "a": "存在しない",
     "ok": 1
    }
   },
   "raw": "1. 外科医は父親。\n2. 10\n3. しない\n4. extraordinarily\n5. 4\n6. 2\n7. できない\n8. 水曜日\n9. o\n10. 火星\n11. できる\n12. すまいざごうはお\n13. 執刀医は父親。\n14. 変える\n15. 10\n16. 3\n17. 14文字で1つ\n18. 4\n19. 0.25, 0.3, 0.33, 1/3\n20. 存在しない"
  },
  "openai/gpt-3.5-turbo-16k": {
   "score": 12,
   "answered": 20,
   "first10": 6,
   "last10": 6,
   "inconsistent_pairs": 0,
   "sec": 4.5,
   "out_tokens": 132,
   "price": "$3 / $4",
   "graded": {
    "q01": {
     "a": "母親",
     "ok": 0
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "3",
     "ok": 1
    },
    "q06": {
     "a": "3",
     "ok": 0
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "水曜",
     "ok": 1
    },
    "q09": {
     "a": "s",
     "ok": 0
    },
    "q10": {
     "a": "火星",
     "ok": 1
    },
    "q11": {
     "a": "できる",
     "ok": 0
    },
    "q12": {
     "a": "すまいござうよはお",
     "ok": 0
    },
    "q13": {
     "a": "母親",
     "ok": 0
    },
    "q14": {
     "a": "同じ",
     "ok": 1
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "15文字、1つ",
     "ok": 0
    },
    "q18": {
     "a": "3",
     "ok": 1
    },
    "q19": {
     "a": "0.25, 0.3, 0.33, 1/3",
     "ok": 1
    },
    "q20": {
     "a": "しない",
     "ok": 1
    }
   },
   "raw": "1: 母親  \n2: 10  \n3: しない  \n4: extraordinarily  \n5: 3  \n6: 3  \n7: できない  \n8: 水曜  \n9: s  \n10: 火星  \n11: できる  \n12: すまいござうよはお  \n13: 母親  \n14: 同じ  \n15: 5  \n16: 3  \n17: 15文字、1つ  \n18: 3  \n19: 0.25, 0.3, 0.33, 1/3  \n20: しない"
  },
  "openai/gpt-4-turbo": {
   "score": 8,
   "answered": 20,
   "first10": 5,
   "last10": 3,
   "inconsistent_pairs": 0,
   "sec": 5.7,
   "out_tokens": 152,
   "price": "$10 / $30",
   "graded": {
    "q01": {
     "a": "母親",
     "ok": 0
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "synchronization",
     "ok": 0
    },
    "q05": {
     "a": "4",
     "ok": 0
    },
    "q06": {
     "a": "2",
     "ok": 1
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "木曜日",
     "ok": 0
    },
    "q09": {
     "a": "o",
     "ok": 1
    },
    "q10": {
     "a": "マーキュリー",
     "ok": 0
    },
    "q11": {
     "a": "できる",
     "ok": 0
    },
    "q12": {
     "a": "すまざいごうはよはお",
     "ok": 0
    },
    "q13": {
     "a": "母親",
     "ok": 0
    },
    "q14": {
     "a": "変える",
     "ok": 0
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "4",
     "ok": 0
    },
    "q17": {
     "a": "17文字、1つ",
     "ok": 0
    },
    "q18": {
     "a": "4",
     "ok": 0
    },
    "q19": {
     "a": "0.25, 0.3, 0.33, 1/3",
     "ok": 1
    },
    "q20": {
     "a": "しない",
     "ok": 1
    }
   },
   "raw": "1: 母親\n2: 10\n3: しない\n4: synchronization\n5: 4\n6: 2\n7: できない\n8: 木曜日\n9: o\n10: マーキュリー\n11: できる\n12: すまざいごうはよはお\n13: 母親\n14: 変える\n15: 5\n16: 4\n17: 17文字、1つ\n18: 4\n19: 0.25, 0.3, 0.33, 1/3\n20: しない"
  },
  "openai/gpt-3.5-turbo-instruct": {
   "score": 9,
   "answered": 20,
   "first10": 4,
   "last10": 5,
   "inconsistent_pairs": 1,
   "sec": 3.5,
   "out_tokens": 145,
   "price": "$1.5 / $2",
   "graded": {
    "q01": {
     "a": "外科医",
     "ok": 0
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "3",
     "ok": 1
    },
    "q06": {
     "a": "9",
     "ok": 0
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "木曜",
     "ok": 0
    },
    "q09": {
     "a": "s",
     "ok": 0
    },
    "q10": {
     "a": "プルート",
     "ok": 0
    },
    "q11": {
     "a": "できる",
     "ok": 0
    },
    "q12": {
     "a": "すいようざごお",
     "ok": 0
    },
    "q13": {
     "a": "父子",
     "ok": 1
    },
    "q14": {
     "a": "変える",
     "ok": 0
    },
    "q15": {
     "a": "20",
     "ok": 0
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "extraordinarily, synchronization, complicated, failed",
     "ok": 0
    },
    "q18": {
     "a": "3",
     "ok": 1
    },
    "q19": {
     "a": "0.25, 0.3, 0.33, 1/3",
     "ok": 1
    },
    "q20": {
     "a": "しない",
     "ok": 1
    }
   },
   "raw": "1: 外科医\n2: 10\n3: しない\n4: extraordinarily\n5: 3\n6: 9\n7: できない\n8: 木曜\n9: s\n10: プルート\n11: できる\n12: すいようざごお\n13: 父子\n14: 変える\n15: 20\n16: 3\n17: extraordinarily, synchronization, complicated, failed\n18: 3\n19: 0.25, 0.3, 0.33, 1/3\n20: しない"
  },
  "mistralai/mistral-nemo": {
   "score": 5,
   "answered": 20,
   "first10": 3,
   "last10": 2,
   "inconsistent_pairs": 1,
   "sec": 5.1,
   "out_tokens": 140,
   "price": "$0.019 / $0.03",
   "graded": {
    "q01": {
     "a": "Father",
     "ok": 1
    },
    "q02": {
     "a": "50",
     "ok": 0
    },
    "q03": {
     "a": "する",
     "ok": 0
    },
    "q04": {
     "a": "complicated",
     "ok": 0
    },
    "q05": {
     "a": "2",
     "ok": 0
    },
    "q06": {
     "a": "3",
     "ok": 0
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "火曜日",
     "ok": 0
    },
    "q09": {
     "a": "e",
     "ok": 0
    },
    "q10": {
     "a": "Mars",
     "ok": 1
    },
    "q11": {
     "a": "できない",
     "ok": 1
    },
    "q12": {
     "a": "ますがおよう",
     "ok": 0
    },
    "q13": {
     "a": "父",
     "ok": 1
    },
    "q14": {
     "a": "変える",
     "ok": 0
    },
    "q15": {
     "a": "30",
     "ok": 0
    },
    "q16": {
     "a": "2",
     "ok": 0
    },
    "q17": {
     "a": "25, 1",
     "ok": 0
    },
    "q18": {
     "a": "2",
     "ok": 0
    },
    "q19": {
     "a": "0.25, 0.3, 1/3, 0.33",
     "ok": 0
    },
    "q20": {
     "a": "存在する",
     "ok": 0
    }
   },
   "raw": "1: Father\n2: 50\n3: する\n4: complicated\n5: 2\n6: 3\n7: できない\n8: 火曜日\n9: e\n10: Mars\n11: できない\n12: ますがおよう\n13: 父\n14: 変える\n15: 30\n16: 2\n17: 25, 1\n18: 2\n19: 0.25, 0.3, 1/3, 0.33\n20: 存在する"
  },
  "microsoft/wizardlm-2-8x22b": {
   "score": 10,
   "answered": 20,
   "first10": 5,
   "last10": 5,
   "inconsistent_pairs": 1,
   "sec": 8.6,
   "out_tokens": 132,
   "price": "$0.62 / $0.62",
   "graded": {
    "q01": {
     "a": "父親",
     "ok": 1
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "する",
     "ok": 0
    },
    "q04": {
     "a": "synchronization",
     "ok": 0
    },
    "q05": {
     "a": "3",
     "ok": 1
    },
    "q06": {
     "a": "3",
     "ok": 0
    },
    "q07": {
     "a": "できる",
     "ok": 0
    },
    "q08": {
     "a": "水曜日",
     "ok": 1
    },
    "q09": {
     "a": "s",
     "ok": 0
    },
    "q10": {
     "a": "火星",
     "ok": 1
    },
    "q11": {
     "a": "できる",
     "ok": 0
    },
    "q12": {
     "a": "すまざごうよはお",
     "ok": 0
    },
    "q13": {
     "a": "父親",
     "ok": 1
    },
    "q14": {
     "a": "変える",
     "ok": 0
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "5",
     "ok": 0
    },
    "q17": {
     "a": "18文字, 1つ",
     "ok": 0
    },
    "q18": {
     "a": "3",
     "ok": 1
    },
    "q19": {
     "a": "0.25, 0.3, 0.33, 1/3",
     "ok": 1
    },
    "q20": {
     "a": "存在しない",
     "ok": 1
    }
   },
   "raw": "1: 父親\n\n2: 10\n\n3: する\n\n4: synchronization\n\n5: 3\n\n6: 3\n\n7: できる\n\n8: 水曜日\n\n9: s\n\n10: 火星\n\n11: できる\n\n12: すまざごうよはお\n\n13: 父親\n\n14: 変える\n\n15: 5\n\n16: 5\n\n17: 18文字, 1つ\n\n18: 3\n\n19: 0.25, 0.3, 0.33, 1/3\n\n20: 存在しない"
  },
  "openai/gpt-3.5-turbo": {
   "score": 8,
   "answered": 20,
   "first10": 4,
   "last10": 4,
   "inconsistent_pairs": 1,
   "sec": 2.7,
   "out_tokens": 156,
   "price": "$0.5 / $1.5",
   "graded": {
    "q01": {
     "a": "外科医の父",
     "ok": 1
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "3",
     "ok": 1
    },
    "q06": {
     "a": "3人",
     "ok": 0
    },
    "q07": {
     "a": "できる",
     "ok": 0
    },
    "q08": {
     "a": "木曜日",
     "ok": 0
    },
    "q09": {
     "a": "d",
     "ok": 0
    },
    "q10": {
     "a": "マーキュリー",
     "ok": 0
    },
    "q11": {
     "a": "できない",
     "ok": 1
    },
    "q12": {
     "a": "すますざようはお",
     "ok": 0
    },
    "q13": {
     "a": "父親",
     "ok": 1
    },
    "q14": {
     "a": "変える",
     "ok": 0
    },
    "q15": {
     "a": "5年後",
     "ok": 1
    },
    "q16": {
     "a": "4",
     "ok": 0
    },
    "q17": {
     "a": "13文字、1つ",
     "ok": 0
    },
    "q18": {
     "a": "4つ",
     "ok": 0
    },
    "q19": {
     "a": "0.25, 0.3, 0.33, 1/3",
     "ok": 1
    },
    "q20": {
     "a": "存在する",
     "ok": 0
    }
   },
   "raw": "1: 外科医の父  \n2: 10  \n3: しない  \n4: extraordinarily  \n5: 3  \n6: 3人  \n7: できる  \n8: 木曜日  \n9: d  \n10: マーキュリー  \n11: できない  \n12: すますざようはお  \n13: 父親  \n14: 変える  \n15: 5年後  \n16: 4  \n17: 13文字、1つ  \n18: 4つ  \n19: 0.25, 0.3, 0.33, 1/3  \n20: 存在する"
  },
  "undi95/remm-slerp-l2-13b": {
   "score": 6,
   "answered": 20,
   "first10": 2,
   "last10": 4,
   "inconsistent_pairs": 0,
   "sec": 12.4,
   "out_tokens": 301,
   "price": "$0.35 / $0.65",
   "graded": {
    "q01": {
     "a": "外科医は私。",
     "ok": 0
    },
    "q02": {
     "a": "ボールは50円。",
     "ok": 0
    },
    "q03": {
     "a": "存在する。",
     "ok": 0
    },
    "q04": {
     "a": "単語は「synchronization」で、10文字。",
     "ok": 0
    },
    "q05": {
     "a": "ちょうちょは3モーラ。",
     "ok": 1
    },
    "q06": {
     "a": "サリーの姉妹は6人。",
     "ok": 0
    },
    "q07": {
     "a": "できない。",
     "ok": 1
    },
    "q08": {
     "a": "月曜は110日後。",
     "ok": 0
    },
    "q09": {
     "a": "kaleidoscope の7文字目は O。",
     "ok": 0
    },
    "q10": {
     "a": "カロン。",
     "ok": 0
    },
    "q11": {
     "a": "できない。",
     "ok": 1
    },
    "q12": {
     "a": "\"ogJmamus\"。",
     "ok": 0
    },
    "q13": {
     "a": "息子は執刀医の息子である。",
     "ok": 0
    },
    "q14": {
     "a": "変える。",
     "ok": 0
    },
    "q15": {
     "a": "子が10歳のとき。",
     "ok": 0
    },
    "q16": {
     "a": "はれですは2個。",
     "ok": 0
    },
    "q17": {
     "a": "\"synchronization\" で、1文字。",
     "ok": 0
    },
    "q18": {
     "a": "3に分かれる。",
     "ok": 1
    },
    "q19": {
     "a": "0.25, 0.3, 0.33, 1/3 を小さい順に。",
     "ok": 1
    },
    "q20": {
     "a": "存在しない。",
     "ok": 1
    }
   },
   "raw": "1. 外科医は私。\n2. ボールは50円。\n3. 存在する。\n4. 単語は「synchronization」で、10文字。\n5. ちょうちょは3モーラ。\n6. サリーの姉妹は6人。\n7. できない。\n8. 月曜は110日後。\n9. kaleidoscope の7文字目は O。\n10. カロン。\n11. できない。\n12. \"ogJmamus\"。\n13. 息子は執刀医の息子である。\n14. 変える。\n15. 子が10歳のとき。\n16. はれですは2個。\n17. \"synchronization\" で、1文字。\n18. 3に分かれる。\n19. 0.25, 0.3, 0.33, 1/3 を小さい順に。\n20. 存在しない。"
  },
  "openai/gpt-4": {
   "score": 9,
   "answered": 20,
   "first10": 3,
   "last10": 6,
   "inconsistent_pairs": 1,
   "sec": 4.8,
   "out_tokens": 154,
   "price": "$30 / $60",
   "graded": {
    "q01": {
     "a": "少年の父親",
     "ok": 1
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "4",
     "ok": 0
    },
    "q06": {
     "a": "3",
     "ok": 0
    },
    "q07": {
     "a": "できる",
     "ok": 0
    },
    "q08": {
     "a": "木曜日",
     "ok": 0
    },
    "q09": {
     "a": "s",
     "ok": 0
    },
    "q10": {
     "a": "マーキュリー",
     "ok": 0
    },
    "q11": {
     "a": "できる",
     "ok": 0
    },
    "q12": {
     "a": "すまいざごうよはお",
     "ok": 1
    },
    "q13": {
     "a": "父親",
     "ok": 1
    },
    "q14": {
     "a": "変える",
     "ok": 0
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "22文字, 1つ",
     "ok": 0
    },
    "q18": {
     "a": "4",
     "ok": 0
    },
    "q19": {
     "a": "0.25, 0.3, 0.33, 1/3",
     "ok": 1
    },
    "q20": {
     "a": "しない",
     "ok": 1
    }
   },
   "raw": "1: 少年の父親\n2: 10\n3: しない\n4: extraordinarily\n5: 4\n6: 3\n7: できる\n8: 木曜日\n9: s\n10: マーキュリー\n11: できる\n12: すまいざごうよはお\n13: 父親\n14: 変える\n15: 5\n16: 3\n17: 22文字, 1つ\n18: 4\n19: 0.25, 0.3, 0.33, 1/3\n20: しない"
  },
  "mancer/weaver": {
   "score": 7,
   "answered": 20,
   "first10": 3,
   "last10": 4,
   "inconsistent_pairs": 2,
   "sec": 3.5,
   "out_tokens": 193,
   "price": "$0.4 / $0.75",
   "graded": {
    "q01": {
     "a": "1",
     "ok": 0
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "はい",
     "ok": 0
    },
    "q04": {
     "a": "複雑",
     "ok": 0
    },
    "q05": {
     "a": "3",
     "ok": 1
    },
    "q06": {
     "a": "9",
     "ok": 0
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "月曜日は変わらない",
     "ok": 0
    },
    "q09": {
     "a": "k",
     "ok": 0
    },
    "q10": {
     "a": "マーキュリー",
     "ok": 0
    },
    "q11": {
     "a": "できない",
     "ok": 1
    },
    "q12": {
     "a": "おはようございます",
     "ok": 0
    },
    "q13": {
     "a": "息子と執刀医は父親同士",
     "ok": 1
    },
    "q14": {
     "a": "同じ",
     "ok": 1
    },
    "q15": {
     "a": "10年後",
     "ok": 0
    },
    "q16": {
     "a": "2",
     "ok": 0
    },
    "q17": {
     "a": "10",
     "ok": 0
    },
    "q18": {
     "a": "3",
     "ok": 1
    },
    "q19": {
     "a": "0.25, 0.3, 1/3, 0.33",
     "ok": 0
    },
    "q20": {
     "a": "はい",
     "ok": 0
    }
   },
   "raw": "1. 1\n2. 10\n3. はい\n4. 複雑\n5. 3\n6. 9\n7. できない\n8. 月曜日は変わらない\n9. k\n10. マーキュリー\n11. できない\n12. おはようございます\n13. 息子と執刀医は父親同士\n14. 同じ\n15. 10年後\n16. 2\n17. 10\n18. 3\n19. 0.25, 0.3, 1/3, 0.33\n20. はい"
  },
  "minimax/minimax-m3": {
   "score": 16,
   "answered": 20,
   "first10": 8,
   "last10": 8,
   "inconsistent_pairs": 0,
   "sec": 17.5,
   "out_tokens": 3455,
   "price": "",
   "graded": {
    "q01": {
     "a": "父",
     "ok": 1
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "synchronization",
     "ok": 0
    },
    "q05": {
     "a": "3",
     "ok": 1
    },
    "q06": {
     "a": "2",
     "ok": 1
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "水曜日",
     "ok": 1
    },
    "q09": {
     "a": "s",
     "ok": 0
    },
    "q10": {
     "a": "火星",
     "ok": 1
    },
    "q11": {
     "a": "できない",
     "ok": 1
    },
    "q12": {
     "a": "すまいざごうよはお",
     "ok": 1
    },
    "q13": {
     "a": "父",
     "ok": 1
    },
    "q14": {
     "a": "変える",
     "ok": 0
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "15文字で1個",
     "ok": 0
    },
    "q18": {
     "a": "3",
     "ok": 1
    },
    "q19": {
     "a": "0.25, 0.3, 0.33, 1/3",
     "ok": 1
    },
    "q20": {
     "a": "しない",
     "ok": 1
    }
   },
   "raw": "1: 父\n2: 10\n3: しない\n4: synchronization\n5: 3\n6: 2\n7: できない\n8: 水曜日\n9: s\n10: 火星\n11: できない\n12: すまいざごうよはお\n13: 父\n14: 変える\n15: 5\n16: 3\n17: 15文字で1個\n18: 3\n19: 0.25, 0.3, 0.33, 1/3\n20: しない"
  },
  "teai/auto": {
   "score": 18,
   "answered": 20,
   "first10": 9,
   "last10": 9,
   "inconsistent_pairs": 0,
   "sec": 6.7,
   "out_tokens": 579,
   "price": "",
   "graded": {
    "q01": {
     "a": "父親",
     "ok": 1
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "synchronization",
     "ok": 0
    },
    "q05": {
     "a": "3",
     "ok": 1
    },
    "q06": {
     "a": "2",
     "ok": 1
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "水曜日",
     "ok": 1
    },
    "q09": {
     "a": "o",
     "ok": 1
    },
    "q10": {
     "a": "火星",
     "ok": 1
    },
    "q11": {
     "a": "できない",
     "ok": 1
    },
    "q12": {
     "a": "すまいざごうよはお",
     "ok": 1
    },
    "q13": {
     "a": "父親",
     "ok": 1
    },
    "q14": {
     "a": "同じ",
     "ok": 1
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "16文字・1つ",
     "ok": 0
    },
    "q18": {
     "a": "3",
     "ok": 1
    },
    "q19": {
     "a": "0.25, 0.3, 0.33, 1/3",
     "ok": 1
    },
    "q20": {
     "a": "しない",
     "ok": 1
    }
   },
   "raw": "1: 父親  \n2: 10  \n3: しない  \n4: synchronization  \n5: 3  \n6: 2  \n7: できない  \n8: 水曜日  \n9: o  \n10: 火星  \n11: できない  \n12: すまいざごうよはお  \n13: 父親  \n14: 同じ  \n15: 5  \n16: 3  \n17: 16文字・1つ  \n18: 3  \n19: 0.25, 0.3, 0.33, 1/3  \n20: しない"
  },
  "~anthropic/claude-fable-latest": {
   "score": 20,
   "answered": 20,
   "first10": 10,
   "last10": 10,
   "inconsistent_pairs": 0,
   "sec": 25.3,
   "out_tokens": 1018,
   "price": "$10 / $50",
   "graded": {
    "q01": {
     "a": "父親（もう一人の親）",
     "ok": 1
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily（synchronizationも同じ15字）",
     "ok": 1
    },
    "q05": {
     "a": "3",
     "ok": 1
    },
    "q06": {
     "a": "2",
     "ok": 1
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "水曜日",
     "ok": 1
    },
    "q09": {
     "a": "o",
     "ok": 1
    },
    "q10": {
     "a": "火星",
     "ok": 1
    },
    "q11": {
     "a": "できない",
     "ok": 1
    },
    "q12": {
     "a": "すまいざごうよはお",
     "ok": 1
    },
    "q13": {
     "a": "父親（親子）",
     "ok": 1
    },
    "q14": {
     "a": "同じ",
     "ok": 1
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "15文字、2つ（extraordinarily, synchronization）",
     "ok": 1
    },
    "q18": {
     "a": "3",
     "ok": 1
    },
    "q19": {
     "a": "0.25, 0.3, 0.33, 1/3",
     "ok": 1
    },
    "q20": {
     "a": "しない",
     "ok": 1
    }
   },
   "raw": "1: 父親（もう一人の親）\n2: 10\n3: しない\n4: extraordinarily（synchronizationも同じ15字）\n5: 3\n6: 2\n7: できない\n8: 水曜日\n9: o\n10: 火星\n11: できない\n12: すまいざごうよはお\n13: 父親（親子）\n14: 同じ\n15: 5\n16: 3\n17: 15文字、2つ（extraordinarily, synchronization）\n18: 3\n19: 0.25, 0.3, 0.33, 1/3\n20: しない"
  },
  "~anthropic/claude-opus-latest": {
   "score": 20,
   "answered": 20,
   "first10": 10,
   "last10": 10,
   "inconsistent_pairs": 0,
   "sec": 17.3,
   "out_tokens": 875,
   "price": "$5 / $25",
   "graded": {
    "q01": {
     "a": "父親",
     "ok": 1
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily（synchronizationも同じ15文字で最長）",
     "ok": 1
    },
    "q05": {
     "a": "3",
     "ok": 1
    },
    "q06": {
     "a": "2",
     "ok": 1
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "水曜日",
     "ok": 1
    },
    "q09": {
     "a": "o",
     "ok": 1
    },
    "q10": {
     "a": "火星",
     "ok": 1
    },
    "q11": {
     "a": "できない",
     "ok": 1
    },
    "q12": {
     "a": "すまいざごうよはお",
     "ok": 1
    },
    "q13": {
     "a": "執刀医は息子の父親",
     "ok": 1
    },
    "q14": {
     "a": "同じ",
     "ok": 1
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "15文字・2つ",
     "ok": 1
    },
    "q18": {
     "a": "3",
     "ok": 1
    },
    "q19": {
     "a": "0.25 < 0.3 < 0.33 < 1/3",
     "ok": 1
    },
    "q20": {
     "a": "しない",
     "ok": 1
    }
   },
   "raw": "1: 父親\n2: 10\n3: しない\n4: extraordinarily（synchronizationも同じ15文字で最長）\n5: 3\n6: 2\n7: できない\n8: 水曜日\n9: o\n10: 火星\n11: できない\n12: すまいざごうよはお\n13: 執刀医は息子の父親\n14: 同じ\n15: 5\n16: 3\n17: 15文字・2つ\n18: 3\n19: 0.25 < 0.3 < 0.33 < 1/3\n20: しない"
  },
  "~google/gemini-flash-latest": {
   "score": 18,
   "answered": 20,
   "first10": 9,
   "last10": 9,
   "inconsistent_pairs": 1,
   "sec": 9.9,
   "out_tokens": 1138,
   "price": "$0.75 / $3.75",
   "graded": {
    "q01": {
     "a": "父親",
     "ok": 1
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "3",
     "ok": 1
    },
    "q06": {
     "a": "2",
     "ok": 1
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "水曜日",
     "ok": 1
    },
    "q09": {
     "a": "o",
     "ok": 1
    },
    "q10": {
     "a": "火星",
     "ok": 1
    },
    "q11": {
     "a": "できない",
     "ok": 1
    },
    "q12": {
     "a": "すいまざごうよはお",
     "ok": 0
    },
    "q13": {
     "a": "父と子（執刀医は父親）",
     "ok": 1
    },
    "q14": {
     "a": "同じ",
     "ok": 1
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "15文字、2つ",
     "ok": 1
    },
    "q18": {
     "a": "3",
     "ok": 1
    },
    "q19": {
     "a": "0.25, 0.3, 0.33, 1/3",
     "ok": 1
    },
    "q20": {
     "a": "しない",
     "ok": 1
    }
   },
   "raw": "1: 父親\n2: 10\n3: しない\n4: extraordinarily\n5: 3\n6: 2\n7: できない\n8: 水曜日\n9: o\n10: 火星\n11: できない\n12: すいまざごうよはお\n13: 父と子（執刀医は父親）\n14: 同じ\n15: 5\n16: 3\n17: 15文字、2つ\n18: 3\n19: 0.25, 0.3, 0.33, 1/3\n20: しない"
  },
  "~google/gemini-pro-latest": {
   "score": 19,
   "answered": 20,
   "first10": 9,
   "last10": 10,
   "inconsistent_pairs": 1,
   "sec": 10.8,
   "out_tokens": 1403,
   "price": "$2 / $12",
   "graded": {
    "q01": {
     "a": "父親",
     "ok": 1
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "3",
     "ok": 1
    },
    "q06": {
     "a": "2",
     "ok": 1
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "水曜日",
     "ok": 1
    },
    "q09": {
     "a": "o",
     "ok": 1
    },
    "q10": {
     "a": "火星",
     "ok": 1
    },
    "q11": {
     "a": "できない",
     "ok": 1
    },
    "q12": {
     "a": "すまいざごうよはお",
     "ok": 1
    },
    "q13": {
     "a": "父親",
     "ok": 1
    },
    "q14": {
     "a": "同じ",
     "ok": 1
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "15文字、2つ",
     "ok": 1
    },
    "q18": {
     "a": "3",
     "ok": 1
    },
    "q19": {
     "a": "0.25, 0.3, 0.33, 1/3",
     "ok": 1
    },
    "q20": {
     "a": "しない",
     "ok": 1
    }
   },
   "raw": "1: 父親\n2: 10\n3: しない\n4: extraordinarily\n5: 3\n6: 2\n7: できない\n8: 水曜日\n9: o\n10: 火星\n11: できない\n12: すまいざごうよはお\n13: 父親\n14: 同じ\n15: 5\n16: 3\n17: 15文字、2つ\n18: 3\n19: 0.25, 0.3, 0.33, 1/3\n20: しない"
  },
  "openai/gpt-chat-latest": {
   "score": 13,
   "answered": 20,
   "first10": 6,
   "last10": 7,
   "inconsistent_pairs": 1,
   "sec": 2.7,
   "out_tokens": 136,
   "price": "$5 / $30",
   "graded": {
    "q01": {
     "a": "母親",
     "ok": 0
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "4",
     "ok": 0
    },
    "q06": {
     "a": "2",
     "ok": 1
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "水曜日",
     "ok": 1
    },
    "q09": {
     "a": "s",
     "ok": 0
    },
    "q10": {
     "a": "火星",
     "ok": 1
    },
    "q11": {
     "a": "できない",
     "ok": 1
    },
    "q12": {
     "a": "すまいざごうよはお",
     "ok": 1
    },
    "q13": {
     "a": "母親",
     "ok": 0
    },
    "q14": {
     "a": "同じ",
     "ok": 1
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "4",
     "ok": 0
    },
    "q17": {
     "a": "15文字、2語",
     "ok": 1
    },
    "q18": {
     "a": "4",
     "ok": 0
    },
    "q19": {
     "a": "0.25, 0.3, 0.33, 1/3",
     "ok": 1
    },
    "q20": {
     "a": "しない",
     "ok": 1
    }
   },
   "raw": "1: 母親\n2: 10\n3: しない\n4: extraordinarily\n5: 4\n6: 2\n7: できない\n8: 水曜日\n9: s\n10: 火星\n11: できない\n12: すまいざごうよはお\n13: 母親\n14: 同じ\n15: 5\n16: 4\n17: 15文字、2語\n18: 4\n19: 0.25, 0.3, 0.33, 1/3\n20: しない"
  },
  "shitate/orchestrator:verify": {
   "score": 14,
   "answered": 20,
   "first10": 7,
   "last10": 7,
   "inconsistent_pairs": 0,
   "sec": 41.2,
   "out_tokens": 6502,
   "price": "$0.78 / $2.4",
   "graded": {
    "q01": {
     "a": "母親",
     "ok": 0
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "3",
     "ok": 1
    },
    "q06": {
     "a": "3",
     "ok": 0
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "水曜日",
     "ok": 1
    },
    "q09": {
     "a": "o",
     "ok": 1
    },
    "q10": {
     "a": "火星",
     "ok": 1
    },
    "q11": {
     "a": "できない",
     "ok": 1
    },
    "q12": {
     "a": "すまうごはよお",
     "ok": 0
    },
    "q13": {
     "a": "母子(執刀医は母親)",
     "ok": 0
    },
    "q14": {
     "a": "同じ",
     "ok": 1
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "16文字、1つ",
     "ok": 0
    },
    "q18": {
     "a": "3",
     "ok": 1
    },
    "q19": {
     "a": "0.25 < 0.3 < 0.33 < 1/3",
     "ok": 1
    },
    "q20": {
     "a": "しない",
     "ok": 1
    }
   },
   "raw": "1: 母親\n2: 10\n3: しない\n4: extraordinarily\n5: 3\n6: 3\n7: できない\n8: 水曜日\n9: o\n10: 火星\n11: できない\n12: すまうごはよお\n13: 母子(執刀医は母親)\n14: 同じ\n15: 5\n16: 3\n17: 16文字、1つ\n18: 3\n19: 0.25 < 0.3 < 0.33 < 1/3\n20: しない"
  },
  "teai/lux": {
   "score": 19,
   "answered": 20,
   "first10": 9,
   "last10": 10,
   "inconsistent_pairs": 1,
   "sec": 19.2,
   "out_tokens": 903,
   "price": "$10 / $50",
   "graded": {
    "q01": {
     "a": "父親",
     "ok": 1
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "3",
     "ok": 1
    },
    "q06": {
     "a": "2",
     "ok": 1
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "水曜日",
     "ok": 1
    },
    "q09": {
     "a": "o",
     "ok": 1
    },
    "q10": {
     "a": "火星",
     "ok": 1
    },
    "q11": {
     "a": "できない",
     "ok": 1
    },
    "q12": {
     "a": "すまいざごうよはお",
     "ok": 1
    },
    "q13": {
     "a": "父親(執刀医は息子の父)",
     "ok": 1
    },
    "q14": {
     "a": "同じ",
     "ok": 1
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "15文字、2つ(extraordinarily と synchronization)",
     "ok": 1
    },
    "q18": {
     "a": "3",
     "ok": 1
    },
    "q19": {
     "a": "0.25, 0.3, 0.33, 1/3",
     "ok": 1
    },
    "q20": {
     "a": "しない",
     "ok": 1
    }
   },
   "raw": "1: 父親\n2: 10\n3: しない\n4: extraordinarily\n5: 3\n6: 2\n7: できない\n8: 水曜日\n9: o\n10: 火星\n11: できない\n12: すまいざごうよはお\n13: 父親(執刀医は息子の父)\n14: 同じ\n15: 5\n16: 3\n17: 15文字、2つ(extraordinarily と synchronization)\n18: 3\n19: 0.25, 0.3, 0.33, 1/3\n20: しない"
  },
  "~moonshotai/kimi-latest": {
   "score": 17,
   "answered": 20,
   "first10": 8,
   "last10": 9,
   "inconsistent_pairs": 2,
   "sec": 23.3,
   "out_tokens": 720,
   "price": "$2.4 / $12",
   "graded": {
    "q01": {
     "a": "父親",
     "ok": 1
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "3",
     "ok": 1
    },
    "q06": {
     "a": "2",
     "ok": 1
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "水曜日",
     "ok": 1
    },
    "q09": {
     "a": "d",
     "ok": 0
    },
    "q10": {
     "a": "火星",
     "ok": 1
    },
    "q11": {
     "a": "できない",
     "ok": 1
    },
    "q12": {
     "a": "すまいざごうよはお",
     "ok": 1
    },
    "q13": {
     "a": "親子",
     "ok": 0
    },
    "q14": {
     "a": "同じ",
     "ok": 1
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "15文字・2つ",
     "ok": 1
    },
    "q18": {
     "a": "3",
     "ok": 1
    },
    "q19": {
     "a": "0.25, 0.3, 0.33, 1/3",
     "ok": 1
    },
    "q20": {
     "a": "しない",
     "ok": 1
    }
   },
   "raw": "1: 父親  \n2: 10  \n3: しない  \n4: extraordinarily  \n5: 3  \n6: 2  \n7: できない  \n8: 水曜日  \n9: d  \n10: 火星  \n11: できない  \n12: すまいざごうよはお  \n13: 親子  \n14: 同じ  \n15: 5  \n16: 3  \n17: 15文字・2つ  \n18: 3  \n19: 0.25, 0.3, 0.33, 1/3  \n20: しない"
  },
  "~openai/gpt-latest": {
   "score": 19,
   "answered": 20,
   "first10": 9,
   "last10": 10,
   "inconsistent_pairs": 1,
   "sec": 19.5,
   "out_tokens": 948,
   "price": "$2 / $10",
   "graded": {
    "q01": {
     "a": "父親",
     "ok": 1
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "3",
     "ok": 1
    },
    "q06": {
     "a": "2",
     "ok": 1
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "水曜日",
     "ok": 1
    },
    "q09": {
     "a": "o",
     "ok": 1
    },
    "q10": {
     "a": "火星",
     "ok": 1
    },
    "q11": {
     "a": "できない",
     "ok": 1
    },
    "q12": {
     "a": "すまいざごうよはお",
     "ok": 1
    },
    "q13": {
     "a": "父親",
     "ok": 1
    },
    "q14": {
     "a": "同じ",
     "ok": 1
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "15文字、2語",
     "ok": 1
    },
    "q18": {
     "a": "3",
     "ok": 1
    },
    "q19": {
     "a": "0.25, 0.3, 0.33, 1/3",
     "ok": 1
    },
    "q20": {
     "a": "しない",
     "ok": 1
    }
   },
   "raw": "1: 父親\n2: 10\n3: しない\n4: extraordinarily\n5: 3\n6: 2\n7: できない\n8: 水曜日\n9: o\n10: 火星\n11: できない\n12: すまいざごうよはお\n13: 父親\n14: 同じ\n15: 5\n16: 3\n17: 15文字、2語\n18: 3\n19: 0.25, 0.3, 0.33, 1/3\n20: しない"
  },
  "~openai/gpt-mini-latest": {
   "score": 17,
   "answered": 20,
   "first10": 8,
   "last10": 9,
   "inconsistent_pairs": 0,
   "sec": 18.7,
   "out_tokens": 2723,
   "price": "$0.75 / $4.5",
   "graded": {
    "q01": {
     "a": "父親",
     "ok": 1
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "3",
     "ok": 1
    },
    "q06": {
     "a": "3",
     "ok": 0
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "水曜日",
     "ok": 1
    },
    "q09": {
     "a": "o",
     "ok": 1
    },
    "q10": {
     "a": "火星",
     "ok": 1
    },
    "q11": {
     "a": "できない",
     "ok": 1
    },
    "q12": {
     "a": "すまいざごうよはお",
     "ok": 1
    },
    "q13": {
     "a": "父親",
     "ok": 1
    },
    "q14": {
     "a": "同じ",
     "ok": 1
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "16文字、1つ",
     "ok": 0
    },
    "q18": {
     "a": "3",
     "ok": 1
    },
    "q19": {
     "a": "0.25, 0.3, 0.33, 1/3",
     "ok": 1
    },
    "q20": {
     "a": "しない",
     "ok": 1
    }
   },
   "raw": "1: 父親\n2: 10\n3: しない\n4: extraordinarily\n5: 3\n6: 3\n7: できない\n8: 水曜日\n9: o\n10: 火星\n11: できない\n12: すまいざごうよはお\n13: 父親\n14: 同じ\n15: 5\n16: 3\n17: 16文字、1つ\n18: 3\n19: 0.25, 0.3, 0.33, 1/3\n20: しない"
  },
  "bytedance/ui-tars-1.5-7b": {
   "score": 6,
   "answered": 20,
   "first10": 4,
   "last10": 2,
   "inconsistent_pairs": 2,
   "sec": 3.8,
   "out_tokens": 158,
   "price": "$0.1 / $0.2",
   "graded": {
    "q01": {
     "a": "外科医",
     "ok": 0
    },
    "q02": {
     "a": "10円",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "synchronization",
     "ok": 0
    },
    "q05": {
     "a": "2",
     "ok": 0
    },
    "q06": {
     "a": "9",
     "ok": 0
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "日曜日",
     "ok": 0
    },
    "q09": {
     "a": "o",
     "ok": 1
    },
    "q10": {
     "a": "水星",
     "ok": 0
    },
    "q11": {
     "a": "できる",
     "ok": 0
    },
    "q12": {
     "a": "すきょうはおはようございま",
     "ok": 0
    },
    "q13": {
     "a": "息子の父親",
     "ok": 1
    },
    "q14": {
     "a": "変える",
     "ok": 0
    },
    "q15": {
     "a": "10年後",
     "ok": 0
    },
    "q16": {
     "a": "2",
     "ok": 0
    },
    "q17": {
     "a": "10文字、2つ",
     "ok": 0
    },
    "q18": {
     "a": "2",
     "ok": 0
    },
    "q19": {
     "a": "0.25, 0.3, 0.33, 1/3",
     "ok": 1
    },
    "q20": {
     "a": "存在する",
     "ok": 0
    }
   },
   "raw": "1. 外科医\n2. 10円\n3. しない\n4. synchronization\n5. 2\n6. 9\n7. できない\n8. 日曜日\n9. o\n10. 水星\n11. できる\n12. すきょうはおはようございま\n13. 息子の父親\n14. 変える\n15. 10年後\n16. 2\n17. 10文字、2つ\n18. 2\n19. 0.25, 0.3, 0.33, 1/3\n20. 存在する"
  },
  "gryphe/mythomax-l2-13b": {
   "score": 6,
   "answered": 20,
   "first10": 2,
   "last10": 4,
   "inconsistent_pairs": 3,
   "sec": 5.9,
   "out_tokens": 321,
   "price": "$0.06 / $0.06",
   "graded": {
    "q01": {
     "a": "外科医は病院の医師。",
     "ok": 0
    },
    "q02": {
     "a": "ボールは10円。",
     "ok": 1
    },
    "q03": {
     "a": "存在する。",
     "ok": 0
    },
    "q04": {
     "a": "単語は \"The\"。",
     "ok": 0
    },
    "q05": {
     "a": "ちょうちょは1モーラ。",
     "ok": 0
    },
    "q06": {
     "a": "サリーの姉妹は9人。",
     "ok": 0
    },
    "q07": {
     "a": "できない。",
     "ok": 1
    },
    "q08": {
     "a": "月曜は100日後は月曜。",
     "ok": 0
    },
    "q09": {
     "a": "kaleidoscope の7文字目は \"ale\"。",
     "ok": 0
    },
    "q10": {
     "a": "カロン。",
     "ok": 0
    },
    "q11": {
     "a": "できない。",
     "ok": 1
    },
    "q12": {
     "a": "おはようございますぞ。",
     "ok": 0
    },
    "q13": {
     "a": "息子と執刀医は父子関係。",
     "ok": 1
    },
    "q14": {
     "a": "同じ。",
     "ok": 1
    },
    "q15": {
     "a": "10年後。",
     "ok": 0
    },
    "q16": {
     "a": "は2個。",
     "ok": 0
    },
    "q17": {
     "a": "最も文字数が多い単語は \"extraordinarily\" で、それは1文字のみ。",
     "ok": 0
    },
    "q18": {
     "a": "3つに分かれる。",
     "ok": 1
    },
    "q19": {
     "a": "0.25, 0.3, 1/3, 0.33 の順。",
     "ok": 0
    },
    "q20": {
     "a": "存在する。",
     "ok": 0
    }
   },
   "raw": "1. 外科医は病院の医師。\n2. ボールは10円。\n3. 存在する。\n4. 単語は \"The\"。\n5. ちょうちょは1モーラ。\n6. サリーの姉妹は9人。\n7. できない。\n8. 月曜は100日後は月曜。\n9. kaleidoscope の7文字目は \"ale\"。\n10. カロン。\n11. できない。\n12. おはようございますぞ。\n13. 息子と執刀医は父子関係。\n14. 同じ。\n15. 10年後。\n16. は2個。\n17. 最も文字数が多い単語は \"extraordinarily\" で、それは1文字のみ。\n18. 3つに分かれる。\n19. 0.25, 0.3, 1/3, 0.33 の順。\n20. 存在する。"
  },
  "minimax/minimax-01": {
   "score": 6,
   "answered": 20,
   "first10": 3,
   "last10": 3,
   "inconsistent_pairs": 0,
   "sec": 12.0,
   "out_tokens": 195,
   "price": "$0.2 / $1.1",
   "graded": {
    "q01": {
     "a": "番号: 外科医は少年の父親",
     "ok": 1
    },
    "q02": {
     "a": "番号: 10",
     "ok": 1
    },
    "q03": {
     "a": "番号: しない",
     "ok": 1
    },
    "q04": {
     "a": "番号: synchronization",
     "ok": 0
    },
    "q05": {
     "a": "番号: 4",
     "ok": 0
    },
    "q06": {
     "a": "番号: 3",
     "ok": 0
    },
    "q07": {
     "a": "番号: できる",
     "ok": 0
    },
    "q08": {
     "a": "番号: 金曜日",
     "ok": 0
    },
    "q09": {
     "a": "番号: c",
     "ok": 0
    },
    "q10": {
     "a": "番号: 金星",
     "ok": 0
    },
    "q11": {
     "a": "番号: できる",
     "ok": 0
    },
    "q12": {
     "a": "番号: すまいよはうきょうは",
     "ok": 0
    },
    "q13": {
     "a": "番号: 執刀医は少年の父親",
     "ok": 1
    },
    "q14": {
     "a": "番号: 変える",
     "ok": 0
    },
    "q15": {
     "a": "番号: 5",
     "ok": 1
    },
    "q16": {
     "a": "番号: 3",
     "ok": 1
    },
    "q17": {
     "a": "番号: 14, 1つ",
     "ok": 0
    },
    "q18": {
     "a": "番号: 4",
     "ok": 0
    },
    "q19": {
     "a": "番号: 0.25, 0.3, 1/3, 0.33",
     "ok": 0
    },
    "q20": {
     "a": "番号: 存在する",
     "ok": 0
    }
   },
   "raw": "1. 番号: 外科医は少年の父親\n2. 番号: 10\n3. 番号: しない\n4. 番号: synchronization\n5. 番号: 4\n6. 番号: 3\n7. 番号: できる\n8. 番号: 金曜日\n9. 番号: c\n10. 番号: 金星\n11. 番号: できる\n12. 番号: すまいよはうきょうは\n13. 番号: 執刀医は少年の父親\n14. 番号: 変える\n15. 番号: 5\n16. 番号: 3\n17. 番号: 14, 1つ\n18. 番号: 4\n19. 番号: 0.25, 0.3, 1/3, 0.33\n20. 番号: 存在する"
  },
  "nousresearch/hermes-3-llama-3.1-405b": {
   "score": 6,
   "answered": 20,
   "first10": 3,
   "last10": 3,
   "inconsistent_pairs": 1,
   "sec": 5.9,
   "out_tokens": 131,
   "price": "$1 / $1",
   "graded": {
    "q01": {
     "a": "母親",
     "ok": 0
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "4",
     "ok": 0
    },
    "q06": {
     "a": "3",
     "ok": 0
    },
    "q07": {
     "a": "できる",
     "ok": 0
    },
    "q08": {
     "a": "水曜",
     "ok": 1
    },
    "q09": {
     "a": "d",
     "ok": 0
    },
    "q10": {
     "a": "水星",
     "ok": 0
    },
    "q11": {
     "a": "できる",
     "ok": 0
    },
    "q12": {
     "a": "ましてすごはようお",
     "ok": 0
    },
    "q13": {
     "a": "父親",
     "ok": 1
    },
    "q14": {
     "a": "変える",
     "ok": 0
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "4",
     "ok": 0
    },
    "q17": {
     "a": "17文字, 1つ",
     "ok": 0
    },
    "q18": {
     "a": "4",
     "ok": 0
    },
    "q19": {
     "a": "0.25, 0.3, 0.33, 1/3",
     "ok": 1
    },
    "q20": {
     "a": "存在する",
     "ok": 0
    }
   },
   "raw": "1: 母親\n2: 10\n3: しない\n4: extraordinarily\n5: 4\n6: 3\n7: できる\n8: 水曜\n9: d\n10: 水星\n11: できる\n12: ましてすごはようお\n13: 父親\n14: 変える\n15: 5\n16: 4\n17: 17文字, 1つ\n18: 4\n19: 0.25, 0.3, 0.33, 1/3\n20: 存在する"
  },
  "nousresearch/hermes-3-llama-3.1-70b": {
   "score": 2,
   "answered": 20,
   "first10": 1,
   "last10": 1,
   "inconsistent_pairs": 0,
   "sec": 4.7,
   "out_tokens": 130,
   "price": "$0.7 / $0.7",
   "graded": {
    "q01": {
     "a": "1: 母親",
     "ok": 0
    },
    "q02": {
     "a": "20",
     "ok": 0
    },
    "q03": {
     "a": "する",
     "ok": 0
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "4",
     "ok": 0
    },
    "q06": {
     "a": "3",
     "ok": 0
    },
    "q07": {
     "a": "できる",
     "ok": 0
    },
    "q08": {
     "a": "火曜日",
     "ok": 0
    },
    "q09": {
     "a": "o",
     "ok": 1
    },
    "q10": {
     "a": "水星",
     "ok": 0
    },
    "q11": {
     "a": "できる",
     "ok": 0
    },
    "q12": {
     "a": "すいおはよ",
     "ok": 0
    },
    "q13": {
     "a": "母親",
     "ok": 0
    },
    "q14": {
     "a": "変える",
     "ok": 0
    },
    "q15": {
     "a": "10",
     "ok": 0
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "20, 1",
     "ok": 0
    },
    "q18": {
     "a": "4",
     "ok": 0
    },
    "q19": {
     "a": "0.25, 0.3, 1/3, 0.33",
     "ok": 0
    },
    "q20": {
     "a": "存在する",
     "ok": 0
    }
   },
   "raw": "1. 1: 母親\n2. 20\n3. する\n4. extraordinarily\n5. 4\n6. 3\n7. できる\n8. 火曜日\n9. o\n10. 水星\n11. できる\n12. すいおはよ\n13. 母親\n14. 変える\n15. 10\n16. 3\n17. 20, 1\n18. 4\n19. 0.25, 0.3, 1/3, 0.33\n20. 存在する"
  },
  "perceptron/perceptron-mk1": {
   "score": 4,
   "answered": 20,
   "first10": 2,
   "last10": 2,
   "inconsistent_pairs": 0,
   "sec": 8.0,
   "out_tokens": 298,
   "price": "$0.15 / $1.5",
   "graded": {
    "q01": {
     "a": "母親",
     "ok": 0
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "する",
     "ok": 0
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "4",
     "ok": 0
    },
    "q06": {
     "a": "3",
     "ok": 0
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "木曜日",
     "ok": 0
    },
    "q09": {
     "a": "s",
     "ok": 0
    },
    "q10": {
     "a": "水星",
     "ok": 0
    },
    "q11": {
     "a": "できる",
     "ok": 0
    },
    "q12": {
     "a": "すいませんごうよはお",
     "ok": 0
    },
    "q13": {
     "a": "母親",
     "ok": 0
    },
    "q14": {
     "a": "変える",
     "ok": 0
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "5",
     "ok": 0
    },
    "q17": {
     "a": "17文字, 2個",
     "ok": 0
    },
    "q18": {
     "a": "4",
     "ok": 0
    },
    "q19": {
     "a": "0.25, 1/3, 0.33, 0.3",
     "ok": 0
    },
    "q20": {
     "a": "存在しない",
     "ok": 1
    }
   },
   "raw": "1: 母親\n2: 10\n3: する\n4: extraordinarily\n5: 4\n6: 3\n7: できない\n8: 木曜日\n9: s\n10: 水星\n11: できる\n12: すいませんごうよはお\n13: 母親\n14: 変える\n15: 5\n16: 5\n17: 17文字, 2個\n18: 4\n19: 0.25, 1/3, 0.33, 0.3\n20: 存在しない"
  },
  "qwen/qwen3-max": {
   "score": 10,
   "answered": 20,
   "first10": 5,
   "last10": 5,
   "inconsistent_pairs": 0,
   "sec": 4.2,
   "out_tokens": 152,
   "price": "$0.78 / $3.9",
   "graded": {
    "q01": {
     "a": "母親",
     "ok": 0
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "4",
     "ok": 0
    },
    "q06": {
     "a": "2",
     "ok": 1
    },
    "q07": {
     "a": "できる",
     "ok": 0
    },
    "q08": {
     "a": "水曜日",
     "ok": 1
    },
    "q09": {
     "a": "s",
     "ok": 0
    },
    "q10": {
     "a": "火星",
     "ok": 1
    },
    "q11": {
     "a": "できない",
     "ok": 1
    },
    "q12": {
     "a": "すまいさごよあはお",
     "ok": 0
    },
    "q13": {
     "a": "母子",
     "ok": 0
    },
    "q14": {
     "a": "同じ",
     "ok": 1
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "13文字で1つ",
     "ok": 0
    },
    "q18": {
     "a": "4",
     "ok": 0
    },
    "q19": {
     "a": "0.25, 0.3, 0.33, 1/3",
     "ok": 1
    },
    "q20": {
     "a": "存在する",
     "ok": 0
    }
   },
   "raw": "1: 母親  \n2: 10  \n3: しない  \n4: extraordinarily  \n5: 4  \n6: 2  \n7: できる  \n8: 水曜日  \n9: s  \n10: 火星  \n11: できない  \n12: すまいさごよあはお  \n13: 母子  \n14: 同じ  \n15: 5  \n16: 3  \n17: 13文字で1つ  \n18: 4  \n19: 0.25, 0.3, 0.33, 1/3  \n20: 存在する"
  },
  "shitate/license": {
   "score": 14,
   "answered": 20,
   "first10": 7,
   "last10": 7,
   "inconsistent_pairs": 0,
   "sec": 3.0,
   "out_tokens": 128,
   "price": "$0.28 / $0.42",
   "graded": {
    "q01": {
     "a": "母親",
     "ok": 0
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "3",
     "ok": 1
    },
    "q06": {
     "a": "3",
     "ok": 0
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "水曜日",
     "ok": 1
    },
    "q09": {
     "a": "o",
     "ok": 1
    },
    "q10": {
     "a": "火星",
     "ok": 1
    },
    "q11": {
     "a": "できる",
     "ok": 0
    },
    "q12": {
     "a": "すまいざごうよはお",
     "ok": 1
    },
    "q13": {
     "a": "母親",
     "ok": 0
    },
    "q14": {
     "a": "同じ",
     "ok": 1
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "15文字、1つ",
     "ok": 0
    },
    "q18": {
     "a": "3",
     "ok": 1
    },
    "q19": {
     "a": "0.25, 0.3, 0.33, 1/3",
     "ok": 1
    },
    "q20": {
     "a": "しない",
     "ok": 1
    }
   },
   "raw": "1: 母親\n2: 10\n3: しない\n4: extraordinarily\n5: 3\n6: 3\n7: できない\n8: 水曜日\n9: o\n10: 火星\n11: できる\n12: すまいざごうよはお\n13: 母親\n14: 同じ\n15: 5\n16: 3\n17: 15文字、1つ\n18: 3\n19: 0.25, 0.3, 0.33, 1/3\n20: しない"
  },
  "shitate/orchestrator:sure": {
   "score": 13,
   "answered": 20,
   "first10": 7,
   "last10": 6,
   "inconsistent_pairs": 0,
   "sec": 61.9,
   "out_tokens": 9459,
   "price": "$0.57 / $1.99",
   "graded": {
    "q01": {
     "a": "母親",
     "ok": 0
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "4",
     "ok": 0
    },
    "q06": {
     "a": "2",
     "ok": 1
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "水曜",
     "ok": 1
    },
    "q09": {
     "a": "o",
     "ok": 1
    },
    "q10": {
     "a": "火星",
     "ok": 1
    },
    "q11": {
     "a": "できない",
     "ok": 1
    },
    "q12": {
     "a": "すまいざごうよはお",
     "ok": 1
    },
    "q13": {
     "a": "母と息子",
     "ok": 0
    },
    "q14": {
     "a": "変える",
     "ok": 0
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "15文字で1つ",
     "ok": 0
    },
    "q18": {
     "a": "4",
     "ok": 0
    },
    "q19": {
     "a": "0.25,0.3,0.33,1/3",
     "ok": 1
    },
    "q20": {
     "a": "しない",
     "ok": 1
    }
   },
   "raw": "⚠️ 要確認: 独立した複数モデルとコードの答えが割れました。この回答はそのまま使わず、必ず裏取りしてください。\n\n1: 母親\n2: 10\n3: しない\n4: extraordinarily\n5: 4\n6: 2\n7: できない\n8: 水曜\n9: o\n10: 火星\n11: できない\n12: すまいざごうよはお\n13: 母と息子\n14: 変える\n15: 5\n16: 3\n17: 15文字で1つ\n18: 4\n19: 0.25,0.3,0.33,1/3\n20: しない"
  },
  "teai/default": {
   "score": 17,
   "answered": 20,
   "first10": 8,
   "last10": 9,
   "inconsistent_pairs": 0,
   "sec": 6.6,
   "out_tokens": 650,
   "price": "$3 / $15",
   "graded": {
    "q01": {
     "a": "父親",
     "ok": 1
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "3",
     "ok": 1
    },
    "q06": {
     "a": "2",
     "ok": 1
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "水曜日",
     "ok": 1
    },
    "q09": {
     "a": "d",
     "ok": 0
    },
    "q10": {
     "a": "火星",
     "ok": 1
    },
    "q11": {
     "a": "できない",
     "ok": 1
    },
    "q12": {
     "a": "すまいざごうよはお",
     "ok": 1
    },
    "q13": {
     "a": "父親",
     "ok": 1
    },
    "q14": {
     "a": "同じ",
     "ok": 1
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "15文字、1つ",
     "ok": 0
    },
    "q18": {
     "a": "3",
     "ok": 1
    },
    "q19": {
     "a": "0.25, 0.3, 0.33, 1/3",
     "ok": 1
    },
    "q20": {
     "a": "しない",
     "ok": 1
    }
   },
   "raw": "1: 父親\n2: 10\n3: しない\n4: extraordinarily\n5: 3\n6: 2\n7: できない\n8: 水曜日\n9: d\n10: 火星\n11: できない\n12: すまいざごうよはお\n13: 父親\n14: 同じ\n15: 5\n16: 3\n17: 15文字、1つ\n18: 3\n19: 0.25, 0.3, 0.33, 1/3\n20: しない"
  },
  "~z-ai/glm-latest": {
   "score": 9,
   "answered": 20,
   "first10": 6,
   "last10": 3,
   "inconsistent_pairs": 0,
   "sec": 2.9,
   "out_tokens": 461,
   "price": "$1.113 / $3.498",
   "graded": {
    "q01": {
     "a": "母親(女性外科医)",
     "ok": 0
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "synchronization",
     "ok": 0
    },
    "q05": {
     "a": "4",
     "ok": 0
    },
    "q06": {
     "a": "3",
     "ok": 0
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "水曜日",
     "ok": 1
    },
    "q09": {
     "a": "o",
     "ok": 1
    },
    "q10": {
     "a": "火星",
     "ok": 1
    },
    "q11": {
     "a": "できる",
     "ok": 0
    },
    "q12": {
     "a": "すまいごううはお",
     "ok": 0
    },
    "q13": {
     "a": "母親(または両親)",
     "ok": 0
    },
    "q14": {
     "a": "同じ",
     "ok": 1
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "4",
     "ok": 0
    },
    "q17": {
     "a": "16文字、1つ(extraordinarily)",
     "ok": 0
    },
    "q18": {
     "a": "4",
     "ok": 0
    },
    "q19": {
     "a": "0.25, 0.3, 1/3, 0.33",
     "ok": 0
    },
    "q20": {
     "a": "しない",
     "ok": 1
    }
   },
   "raw": "1: 母親(女性外科医)\n2: 10\n3: しない\n4: synchronization\n5: 4\n6: 3\n7: できない\n8: 水曜日\n9: o\n10: 火星\n11: できる\n12: すまいごううはお\n13: 母親(または両親)\n14: 同じ\n15: 5\n16: 4\n17: 16文字、1つ(extraordinarily)\n18: 4\n19: 0.25, 0.3, 1/3, 0.33\n20: しない"
  },
  "minimax/minimax-m2-her": {
   "score": 5,
   "answered": 20,
   "first10": 3,
   "last10": 2,
   "inconsistent_pairs": 2,
   "sec": 2.9,
   "out_tokens": 125,
   "price": "$0.3 / $1.2",
   "graded": {
    "q01": {
     "a": "父",
     "ok": 1
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "する",
     "ok": 0
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "4",
     "ok": 0
    },
    "q06": {
     "a": "3",
     "ok": 0
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "日曜日",
     "ok": 0
    },
    "q09": {
     "a": "e",
     "ok": 0
    },
    "q10": {
     "a": "水星",
     "ok": 0
    },
    "q11": {
     "a": "できる",
     "ok": 0
    },
    "q12": {
     "a": "すはごよ はなはき おは",
     "ok": 0
    },
    "q13": {
     "a": "母",
     "ok": 0
    },
    "q14": {
     "a": "変える",
     "ok": 0
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "4",
     "ok": 0
    },
    "q17": {
     "a": "16 & 1",
     "ok": 0
    },
    "q18": {
     "a": "4",
     "ok": 0
    },
    "q19": {
     "a": "0.25, 0.3, 0.33, 1/3",
     "ok": 1
    },
    "q20": {
     "a": "する",
     "ok": 0
    }
   },
   "raw": "1. 父  \n2. 10  \n3. する  \n4. extraordinarily  \n5. 4  \n6. 3  \n7. できない  \n8. 日曜日  \n9. e  \n10. 水星  \n11. できる  \n12. すはごよ はなはき おは  \n13. 母  \n14. 変える  \n15. 5  \n16. 4  \n17. 16 & 1  \n18. 4  \n19. 0.25, 0.3, 0.33, 1/3  \n20. する"
  },
  "perplexity/sonar": {
   "score": 9,
   "answered": 20,
   "first10": 4,
   "last10": 5,
   "inconsistent_pairs": 1,
   "sec": 6.4,
   "out_tokens": 136,
   "price": "$1 / $1",
   "graded": {
    "q01": {
     "a": "外科医の母親（女性の外科医）",
     "ok": 0
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "4",
     "ok": 0
    },
    "q06": {
     "a": "3",
     "ok": 0
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "金曜",
     "ok": 0
    },
    "q09": {
     "a": "c",
     "ok": 0
    },
    "q10": {
     "a": "火星",
     "ok": 1
    },
    "q11": {
     "a": "できる",
     "ok": 0
    },
    "q12": {
     "a": "すうまうごはちお",
     "ok": 0
    },
    "q13": {
     "a": "父",
     "ok": 1
    },
    "q14": {
     "a": "同じ",
     "ok": 1
    },
    "q15": {
     "a": "10",
     "ok": 0
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "15文字、1つ",
     "ok": 0
    },
    "q18": {
     "a": "4",
     "ok": 0
    },
    "q19": {
     "a": "0.25、0.3、0.33、1/3",
     "ok": 1
    },
    "q20": {
     "a": "しない",
     "ok": 1
    }
   },
   "raw": "1: 外科医の母親（女性の外科医）\n2: 10\n3: しない\n4: extraordinarily\n5: 4\n6: 3\n7: できない\n8: 金曜\n9: c\n10: 火星\n11: できる\n12: すうまうごはちお\n13: 父\n14: 同じ\n15: 10\n16: 3\n17: 15文字、1つ\n18: 4\n19: 0.25、0.3、0.33、1/3\n20: しない"
  },
  "perplexity/sonar-pro": {
   "score": 9,
   "answered": 20,
   "first10": 5,
   "last10": 4,
   "inconsistent_pairs": 2,
   "sec": 2.7,
   "out_tokens": 141,
   "price": "$3 / $15",
   "graded": {
    "q01": {
     "a": "外科医の父親（母親ではなく父親）",
     "ok": 1
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "4",
     "ok": 0
    },
    "q06": {
     "a": "2",
     "ok": 1
    },
    "q07": {
     "a": "できる",
     "ok": 0
    },
    "q08": {
     "a": "水曜",
     "ok": 1
    },
    "q09": {
     "a": "d",
     "ok": 0
    },
    "q10": {
     "a": "水星",
     "ok": 0
    },
    "q11": {
     "a": "できる",
     "ok": 0
    },
    "q12": {
     "a": "うすごときうおは",
     "ok": 0
    },
    "q13": {
     "a": "父親",
     "ok": 1
    },
    "q14": {
     "a": "変える",
     "ok": 0
    },
    "q15": {
     "a": "10",
     "ok": 0
    },
    "q16": {
     "a": "2",
     "ok": 0
    },
    "q17": {
     "a": "15, 2",
     "ok": 1
    },
    "q18": {
     "a": "4",
     "ok": 0
    },
    "q19": {
     "a": "0.25, 0.3, 0.33, 1/3",
     "ok": 1
    },
    "q20": {
     "a": "しない",
     "ok": 1
    }
   },
   "raw": "1: 外科医の父親（母親ではなく父親）\n2: 10\n3: しない\n4: extraordinarily\n5: 4\n6: 2\n7: できる\n8: 水曜\n9: d\n10: 水星\n11: できる\n12: うすごときうおは\n13: 父親\n14: 変える\n15: 10\n16: 2\n17: 15, 2\n18: 4\n19: 0.25, 0.3, 0.33, 1/3\n20: しない"
  },
  "shitate/freelance": {
   "score": 14,
   "answered": 20,
   "first10": 7,
   "last10": 7,
   "inconsistent_pairs": 0,
   "sec": 2.3,
   "out_tokens": 128,
   "price": "$0.28 / $0.42",
   "graded": {
    "q01": {
     "a": "母親",
     "ok": 0
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "3",
     "ok": 1
    },
    "q06": {
     "a": "3",
     "ok": 0
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "水曜日",
     "ok": 1
    },
    "q09": {
     "a": "o",
     "ok": 1
    },
    "q10": {
     "a": "火星",
     "ok": 1
    },
    "q11": {
     "a": "できる",
     "ok": 0
    },
    "q12": {
     "a": "すまいざごうよはお",
     "ok": 1
    },
    "q13": {
     "a": "母親",
     "ok": 0
    },
    "q14": {
     "a": "同じ",
     "ok": 1
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "15文字、1つ",
     "ok": 0
    },
    "q18": {
     "a": "3",
     "ok": 1
    },
    "q19": {
     "a": "0.25, 0.3, 0.33, 1/3",
     "ok": 1
    },
    "q20": {
     "a": "しない",
     "ok": 1
    }
   },
   "raw": "1: 母親\n2: 10\n3: しない\n4: extraordinarily\n5: 3\n6: 3\n7: できない\n8: 水曜日\n9: o\n10: 火星\n11: できる\n12: すまいざごうよはお\n13: 母親\n14: 同じ\n15: 5\n16: 3\n17: 15文字、1つ\n18: 3\n19: 0.25, 0.3, 0.33, 1/3\n20: しない"
  },
  "shitate/infra": {
   "score": 14,
   "answered": 20,
   "first10": 7,
   "last10": 7,
   "inconsistent_pairs": 0,
   "sec": 1.9,
   "out_tokens": 128,
   "price": "$0.28 / $0.42",
   "graded": {
    "q01": {
     "a": "母親",
     "ok": 0
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "3",
     "ok": 1
    },
    "q06": {
     "a": "3",
     "ok": 0
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "水曜日",
     "ok": 1
    },
    "q09": {
     "a": "o",
     "ok": 1
    },
    "q10": {
     "a": "火星",
     "ok": 1
    },
    "q11": {
     "a": "できる",
     "ok": 0
    },
    "q12": {
     "a": "すまいざごうよはお",
     "ok": 1
    },
    "q13": {
     "a": "母親",
     "ok": 0
    },
    "q14": {
     "a": "同じ",
     "ok": 1
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "15文字、1つ",
     "ok": 0
    },
    "q18": {
     "a": "3",
     "ok": 1
    },
    "q19": {
     "a": "0.25, 0.3, 0.33, 1/3",
     "ok": 1
    },
    "q20": {
     "a": "しない",
     "ok": 1
    }
   },
   "raw": "1: 母親\n2: 10\n3: しない\n4: extraordinarily\n5: 3\n6: 3\n7: できない\n8: 水曜日\n9: o\n10: 火星\n11: できる\n12: すまいざごうよはお\n13: 母親\n14: 同じ\n15: 5\n16: 3\n17: 15文字、1つ\n18: 3\n19: 0.25, 0.3, 0.33, 1/3\n20: しない"
  },
  "shitate/legal": {
   "score": 13,
   "answered": 20,
   "first10": 7,
   "last10": 6,
   "inconsistent_pairs": 0,
   "sec": 4.1,
   "out_tokens": 128,
   "price": "$0.28 / $0.42",
   "graded": {
    "q01": {
     "a": "母親",
     "ok": 0
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "3",
     "ok": 1
    },
    "q06": {
     "a": "3",
     "ok": 0
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "水曜日",
     "ok": 1
    },
    "q09": {
     "a": "o",
     "ok": 1
    },
    "q10": {
     "a": "火星",
     "ok": 1
    },
    "q11": {
     "a": "できる",
     "ok": 0
    },
    "q12": {
     "a": "すまいざごうよはお",
     "ok": 1
    },
    "q13": {
     "a": "母親",
     "ok": 0
    },
    "q14": {
     "a": "同じ",
     "ok": 1
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "15文字、1つ",
     "ok": 0
    },
    "q18": {
     "a": "3",
     "ok": 1
    },
    "q19": {
     "a": "0.25, 0.3, 1/3, 0.33",
     "ok": 0
    },
    "q20": {
     "a": "しない",
     "ok": 1
    }
   },
   "raw": "1: 母親\n2: 10\n3: しない\n4: extraordinarily\n5: 3\n6: 3\n7: できない\n8: 水曜日\n9: o\n10: 火星\n11: できる\n12: すまいざごうよはお\n13: 母親\n14: 同じ\n15: 5\n16: 3\n17: 15文字、1つ\n18: 3\n19: 0.25, 0.3, 1/3, 0.33\n20: しない"
  },
  "shitate/security": {
   "score": 14,
   "answered": 20,
   "first10": 7,
   "last10": 7,
   "inconsistent_pairs": 0,
   "sec": 2.1,
   "out_tokens": 128,
   "price": "$0.28 / $0.42",
   "graded": {
    "q01": {
     "a": "母親",
     "ok": 0
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "3",
     "ok": 1
    },
    "q06": {
     "a": "3",
     "ok": 0
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "水曜日",
     "ok": 1
    },
    "q09": {
     "a": "o",
     "ok": 1
    },
    "q10": {
     "a": "火星",
     "ok": 1
    },
    "q11": {
     "a": "できる",
     "ok": 0
    },
    "q12": {
     "a": "すまいざごうよはお",
     "ok": 1
    },
    "q13": {
     "a": "母親",
     "ok": 0
    },
    "q14": {
     "a": "同じ",
     "ok": 1
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "15文字、1つ",
     "ok": 0
    },
    "q18": {
     "a": "3",
     "ok": 1
    },
    "q19": {
     "a": "0.25, 0.3, 0.33, 1/3",
     "ok": 1
    },
    "q20": {
     "a": "しない",
     "ok": 1
    }
   },
   "raw": "1: 母親\n2: 10\n3: しない\n4: extraordinarily\n5: 3\n6: 3\n7: できない\n8: 水曜日\n9: o\n10: 火星\n11: できる\n12: すまいざごうよはお\n13: 母親\n14: 同じ\n15: 5\n16: 3\n17: 15文字、1つ\n18: 3\n19: 0.25, 0.3, 0.33, 1/3\n20: しない"
  },
  "~deepseek/deepseek-v4-flash-latest": {
   "score": 8,
   "answered": 20,
   "first10": 5,
   "last10": 3,
   "inconsistent_pairs": 1,
   "sec": 6.9,
   "out_tokens": 149,
   "price": "$0.05 / $0.16",
   "graded": {
    "q01": {
     "a": "母親",
     "ok": 0
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "4",
     "ok": 0
    },
    "q06": {
     "a": "3",
     "ok": 0
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "水曜日",
     "ok": 1
    },
    "q09": {
     "a": "e",
     "ok": 0
    },
    "q10": {
     "a": "火星",
     "ok": 1
    },
    "q11": {
     "a": "できる",
     "ok": 0
    },
    "q12": {
     "a": "すまいざうよこはうお",
     "ok": 0
    },
    "q13": {
     "a": "母親",
     "ok": 0
    },
    "q14": {
     "a": "同じ",
     "ok": 1
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "16文字、1個",
     "ok": 0
    },
    "q18": {
     "a": "4",
     "ok": 0
    },
    "q19": {
     "a": "0.25, 0.3, 1/3, 0.33",
     "ok": 0
    },
    "q20": {
     "a": "存在する",
     "ok": 0
    }
   },
   "raw": "1: 母親\n2: 10\n3: しない\n4: extraordinarily\n5: 4\n6: 3\n7: できない\n8: 水曜日\n9: e\n10: 火星\n11: できる\n12: すまいざうよこはうお\n13: 母親\n14: 同じ\n15: 5\n16: 3\n17: 16文字、1個\n18: 4\n19: 0.25, 0.3, 1/3, 0.33\n20: 存在する"
  },
  "~z-ai/glm-flash-latest": {
   "score": 11,
   "answered": 20,
   "first10": 7,
   "last10": 4,
   "inconsistent_pairs": 1,
   "sec": 15.8,
   "out_tokens": 311,
   "price": "$0.075 / $0.25",
   "graded": {
    "q01": {
     "a": "母親",
     "ok": 0
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "synchronization",
     "ok": 0
    },
    "q05": {
     "a": "4",
     "ok": 0
    },
    "q06": {
     "a": "2",
     "ok": 1
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "水曜",
     "ok": 1
    },
    "q09": {
     "a": "o",
     "ok": 1
    },
    "q10": {
     "a": "火星",
     "ok": 1
    },
    "q11": {
     "a": "できる",
     "ok": 0
    },
    "q12": {
     "a": "すでいすがよはうお",
     "ok": 0
    },
    "q13": {
     "a": "父親",
     "ok": 1
    },
    "q14": {
     "a": "変える",
     "ok": 0
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "4",
     "ok": 0
    },
    "q17": {
     "a": "16文字、1つ",
     "ok": 0
    },
    "q18": {
     "a": "4",
     "ok": 0
    },
    "q19": {
     "a": "0.25 < 0.3 < 0.33 < 1/3",
     "ok": 1
    },
    "q20": {
     "a": "しない",
     "ok": 1
    }
   },
   "raw": "1. 母親\n2. 10\n3. しない\n4. synchronization\n5. 4\n6. 2\n7. できない\n8. 水曜\n9. o\n10. 火星\n11. できる\n12. すでいすがよはうお\n13. 父親\n14. 変える\n15. 5\n16. 4\n17. 16文字、1つ\n18. 4\n19. 0.25 < 0.3 < 0.33 < 1/3\n20. しない"
  },
  "deepseek/deepseek-v4-flash-vision-exp": {
   "score": 19,
   "answered": 20,
   "first10": 9,
   "last10": 10,
   "inconsistent_pairs": 1,
   "sec": 56.9,
   "out_tokens": 5855,
   "price": "$0.22 / $0.66",
   "graded": {
    "q01": {
     "a": "父親",
     "ok": 1
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "3",
     "ok": 1
    },
    "q06": {
     "a": "2",
     "ok": 1
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "水曜日",
     "ok": 1
    },
    "q09": {
     "a": "o",
     "ok": 1
    },
    "q10": {
     "a": "火星",
     "ok": 1
    },
    "q11": {
     "a": "できない",
     "ok": 1
    },
    "q12": {
     "a": "すまいざごうよはお",
     "ok": 1
    },
    "q13": {
     "a": "父親",
     "ok": 1
    },
    "q14": {
     "a": "同じ",
     "ok": 1
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "15文字、2つ",
     "ok": 1
    },
    "q18": {
     "a": "3",
     "ok": 1
    },
    "q19": {
     "a": "0.25, 0.3, 0.33, 1/3",
     "ok": 1
    },
    "q20": {
     "a": "しない",
     "ok": 1
    }
   },
   "raw": "1: 父親\n2: 10\n3: しない\n4: extraordinarily\n5: 3\n6: 2\n7: できない\n8: 水曜日\n9: o\n10: 火星\n11: できない\n12: すまいざごうよはお\n13: 父親\n14: 同じ\n15: 5\n16: 3\n17: 15文字、2つ\n18: 3\n19: 0.25, 0.3, 0.33, 1/3\n20: しない"
  },
  "teai/fast": {
   "score": 18,
   "answered": 20,
   "first10": 8,
   "last10": 10,
   "inconsistent_pairs": 2,
   "sec": 47.9,
   "out_tokens": 3575,
   "price": "$0.22 / $0.66",
   "graded": {
    "q01": {
     "a": "母親",
     "ok": 0
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "3",
     "ok": 1
    },
    "q06": {
     "a": "2",
     "ok": 1
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "水曜日",
     "ok": 1
    },
    "q09": {
     "a": "o",
     "ok": 1
    },
    "q10": {
     "a": "火星",
     "ok": 1
    },
    "q11": {
     "a": "できない",
     "ok": 1
    },
    "q12": {
     "a": "すまいざごうよはお",
     "ok": 1
    },
    "q13": {
     "a": "父と子",
     "ok": 1
    },
    "q14": {
     "a": "同じ",
     "ok": 1
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "15文字・2つ",
     "ok": 1
    },
    "q18": {
     "a": "3",
     "ok": 1
    },
    "q19": {
     "a": "0.25, 0.3, 0.33, 1/3",
     "ok": 1
    },
    "q20": {
     "a": "存在しない",
     "ok": 1
    }
   },
   "raw": "1: 母親\n2: 10\n3: しない\n4: extraordinarily\n5: 3\n6: 2\n7: できない\n8: 水曜日\n9: o\n10: 火星\n11: できない\n12: すまいざごうよはお\n13: 父と子\n14: 同じ\n15: 5\n16: 3\n17: 15文字・2つ\n18: 3\n19: 0.25, 0.3, 0.33, 1/3\n20: 存在しない"
  },
  "minimax/minimax-m2.1": {
   "score": 12,
   "answered": 20,
   "first10": 7,
   "last10": 5,
   "inconsistent_pairs": 0,
   "sec": 25.3,
   "out_tokens": 1250,
   "price": "$0.3 / $1.2",
   "graded": {
    "q01": {
     "a": "父親",
     "ok": 1
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "4",
     "ok": 0
    },
    "q06": {
     "a": "3",
     "ok": 0
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "水曜日",
     "ok": 1
    },
    "q09": {
     "a": "o",
     "ok": 1
    },
    "q10": {
     "a": "火星",
     "ok": 1
    },
    "q11": {
     "a": "できる",
     "ok": 0
    },
    "q12": {
     "a": "すまいざいうようは お",
     "ok": 0
    },
    "q13": {
     "a": "父子",
     "ok": 1
    },
    "q14": {
     "a": "変える",
     "ok": 0
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "14",
     "ok": 0
    },
    "q18": {
     "a": "4",
     "ok": 0
    },
    "q19": {
     "a": "0.25, 0.3, 0.33, 1/3",
     "ok": 1
    },
    "q20": {
     "a": "しない",
     "ok": 1
    }
   },
   "raw": "1: 父親\n2: 10\n3: しない\n4: extraordinarily\n5: 4\n6: 3\n7: できない\n8: 水曜日\n9: o\n10: 火星\n11: できる\n12: すまいざいうようは お\n13: 父子\n14: 変える\n15: 5\n16: 3\n17: 14\n18: 4\n19: 0.25, 0.3, 0.33, 1/3\n20: しない"
  },
  "~anthropic/claude-haiku-latest": {
   "score": 11,
   "answered": 20,
   "first10": 6,
   "last10": 5,
   "inconsistent_pairs": 0,
   "sec": 20.7,
   "out_tokens": 2455,
   "price": "$1 / $5",
   "graded": {
    "q01": {
     "a": "母親",
     "ok": 0
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "synchronization",
     "ok": 0
    },
    "q05": {
     "a": "5",
     "ok": 0
    },
    "q06": {
     "a": "2",
     "ok": 1
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "水曜日",
     "ok": 1
    },
    "q09": {
     "a": "o",
     "ok": 1
    },
    "q10": {
     "a": "水星",
     "ok": 0
    },
    "q11": {
     "a": "できる",
     "ok": 0
    },
    "q12": {
     "a": "すまいざごうよあはお",
     "ok": 0
    },
    "q13": {
     "a": "母と子",
     "ok": 0
    },
    "q14": {
     "a": "同じ",
     "ok": 1
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "16文字、1つ",
     "ok": 0
    },
    "q18": {
     "a": "5",
     "ok": 0
    },
    "q19": {
     "a": "0.25, 0.3, 0.33, 1/3",
     "ok": 1
    },
    "q20": {
     "a": "しない",
     "ok": 1
    }
   },
   "raw": "1: 母親\n2: 10\n3: しない\n4: synchronization\n5: 5\n6: 2\n7: できない\n8: 水曜日\n9: o\n10: 水星\n11: できる\n12: すまいざごうよあはお\n13: 母と子\n14: 同じ\n15: 5\n16: 3\n17: 16文字、1つ\n18: 5\n19: 0.25, 0.3, 0.33, 1/3\n20: しない"
  },
  "~anthropic/claude-sonnet-latest": {
   "score": 15,
   "answered": 20,
   "first10": 7,
   "last10": 8,
   "inconsistent_pairs": 2,
   "sec": 24.0,
   "out_tokens": 1779,
   "price": "$2 / $10",
   "graded": {
    "q01": {
     "a": "母親(外科医は女性で、少年の母)",
     "ok": 0
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "3",
     "ok": 1
    },
    "q06": {
     "a": "2",
     "ok": 1
    },
    "q07": {
     "a": "できる",
     "ok": 0
    },
    "q08": {
     "a": "水曜日",
     "ok": 1
    },
    "q09": {
     "a": "o",
     "ok": 1
    },
    "q10": {
     "a": "火星",
     "ok": 1
    },
    "q11": {
     "a": "できない",
     "ok": 1
    },
    "q12": {
     "a": "すまいざごうよはお",
     "ok": 1
    },
    "q13": {
     "a": "父親",
     "ok": 1
    },
    "q14": {
     "a": "変える",
     "ok": 0
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "15文字、2つ(extraordinarily, synchronization)",
     "ok": 1
    },
    "q18": {
     "a": "3",
     "ok": 1
    },
    "q19": {
     "a": "0.25 < 0.3 < 0.33 < 1/3",
     "ok": 1
    },
    "q20": {
     "a": "存在する",
     "ok": 0
    }
   },
   "raw": "1: 母親(外科医は女性で、少年の母)\n2: 10\n3: しない\n4: extraordinarily\n5: 3\n6: 2\n7: できる\n8: 水曜日\n9: o\n10: 火星\n11: できない\n12: すまいざごうよはお\n13: 父親\n14: 変える\n15: 5\n16: 3\n17: 15文字、2つ(extraordinarily, synchronization)\n18: 3\n19: 0.25 < 0.3 < 0.33 < 1/3\n20: 存在する"
  },
  "nousresearch/hermes-4-70b": {
   "score": 17,
   "answered": 20,
   "first10": 8,
   "last10": 9,
   "inconsistent_pairs": 0,
   "sec": 4.4,
   "out_tokens": 485,
   "price": "$0.13 / $0.4",
   "graded": {
    "q01": {
     "a": "父親",
     "ok": 1
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "3",
     "ok": 1
    },
    "q06": {
     "a": "2",
     "ok": 1
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "水曜日",
     "ok": 1
    },
    "q09": {
     "a": "d",
     "ok": 0
    },
    "q10": {
     "a": "火星",
     "ok": 1
    },
    "q11": {
     "a": "できない",
     "ok": 1
    },
    "q12": {
     "a": "すまいざごうよはお",
     "ok": 1
    },
    "q13": {
     "a": "父親",
     "ok": 1
    },
    "q14": {
     "a": "同じ",
     "ok": 1
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "15文字、1つ",
     "ok": 0
    },
    "q18": {
     "a": "3",
     "ok": 1
    },
    "q19": {
     "a": "0.25, 0.3, 0.33, 1/3",
     "ok": 1
    },
    "q20": {
     "a": "しない",
     "ok": 1
    }
   },
   "raw": "1: 父親\n2: 10\n3: しない\n4: extraordinarily\n5: 3\n6: 2\n7: できない\n8: 水曜日\n9: d\n10: 火星\n11: できない\n12: すまいざごうよはお\n13: 父親\n14: 同じ\n15: 5\n16: 3\n17: 15文字、1つ\n18: 3\n19: 0.25, 0.3, 0.33, 1/3\n20: しない"
  },
  "openai/gpt-5.1-codex-max": {
   "score": 17,
   "answered": 20,
   "first10": 8,
   "last10": 9,
   "inconsistent_pairs": 0,
   "sec": 25.8,
   "out_tokens": 1457,
   "price": "$1.25 / $10",
   "graded": {
    "q01": {
     "a": "父親",
     "ok": 1
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "3",
     "ok": 1
    },
    "q06": {
     "a": "3",
     "ok": 0
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "水曜",
     "ok": 1
    },
    "q09": {
     "a": "o",
     "ok": 1
    },
    "q10": {
     "a": "火星",
     "ok": 1
    },
    "q11": {
     "a": "できない",
     "ok": 1
    },
    "q12": {
     "a": "すまいざごうよはお",
     "ok": 1
    },
    "q13": {
     "a": "父親",
     "ok": 1
    },
    "q14": {
     "a": "同じ",
     "ok": 1
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "14文字、2つ",
     "ok": 0
    },
    "q18": {
     "a": "3",
     "ok": 1
    },
    "q19": {
     "a": "0.25, 0.3, 0.33, 1/3",
     "ok": 1
    },
    "q20": {
     "a": "存在しない",
     "ok": 1
    }
   },
   "raw": "1: 父親\n2: 10\n3: しない\n4: extraordinarily\n5: 3\n6: 3\n7: できない\n8: 水曜\n9: o\n10: 火星\n11: できない\n12: すまいざごうよはお\n13: 父親\n14: 同じ\n15: 5\n16: 3\n17: 14文字、2つ\n18: 3\n19: 0.25, 0.3, 0.33, 1/3\n20: 存在しない"
  },
  "anthropic/claude-fable-5": {
   "score": 20,
   "answered": 20,
   "first10": 10,
   "last10": 10,
   "inconsistent_pairs": 0,
   "sec": 20.5,
   "out_tokens": 870,
   "price": "$10 / $50",
   "graded": {
    "q01": {
     "a": "父親",
     "ok": 1
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily(synchronizationも同じ15文字)",
     "ok": 1
    },
    "q05": {
     "a": "3",
     "ok": 1
    },
    "q06": {
     "a": "2",
     "ok": 1
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "水曜日",
     "ok": 1
    },
    "q09": {
     "a": "o",
     "ok": 1
    },
    "q10": {
     "a": "火星",
     "ok": 1
    },
    "q11": {
     "a": "できない",
     "ok": 1
    },
    "q12": {
     "a": "すまいざごうよはお",
     "ok": 1
    },
    "q13": {
     "a": "父親(執刀医は父)",
     "ok": 1
    },
    "q14": {
     "a": "同じ",
     "ok": 1
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "15文字、2つ",
     "ok": 1
    },
    "q18": {
     "a": "3",
     "ok": 1
    },
    "q19": {
     "a": "0.25, 0.3, 0.33, 1/3",
     "ok": 1
    },
    "q20": {
     "a": "存在しない",
     "ok": 1
    }
   },
   "raw": "1: 父親\n2: 10\n3: しない\n4: extraordinarily(synchronizationも同じ15文字)\n5: 3\n6: 2\n7: できない\n8: 水曜日\n9: o\n10: 火星\n11: できない\n12: すまいざごうよはお\n13: 父親(執刀医は父)\n14: 同じ\n15: 5\n16: 3\n17: 15文字、2つ\n18: 3\n19: 0.25, 0.3, 0.33, 1/3\n20: 存在しない"
  },
  "anthropic/claude-fable-5.1": {
   "score": 20,
   "answered": 20,
   "first10": 10,
   "last10": 10,
   "inconsistent_pairs": 0,
   "sec": 23.5,
   "out_tokens": 869,
   "price": "$10 / $50",
   "graded": {
    "q01": {
     "a": "父親",
     "ok": 1
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily(synchronizationも同じ15文字)",
     "ok": 1
    },
    "q05": {
     "a": "3",
     "ok": 1
    },
    "q06": {
     "a": "2",
     "ok": 1
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "水曜日",
     "ok": 1
    },
    "q09": {
     "a": "o",
     "ok": 1
    },
    "q10": {
     "a": "火星",
     "ok": 1
    },
    "q11": {
     "a": "できない",
     "ok": 1
    },
    "q12": {
     "a": "すまいざごうよはお",
     "ok": 1
    },
    "q13": {
     "a": "父親と息子",
     "ok": 1
    },
    "q14": {
     "a": "同じ",
     "ok": 1
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "15文字、2つ",
     "ok": 1
    },
    "q18": {
     "a": "3",
     "ok": 1
    },
    "q19": {
     "a": "0.25, 0.3, 0.33, 1/3",
     "ok": 1
    },
    "q20": {
     "a": "しない",
     "ok": 1
    }
   },
   "raw": "1: 父親\n2: 10\n3: しない\n4: extraordinarily(synchronizationも同じ15文字)\n5: 3\n6: 2\n7: できない\n8: 水曜日\n9: o\n10: 火星\n11: できない\n12: すまいざごうよはお\n13: 父親と息子\n14: 同じ\n15: 5\n16: 3\n17: 15文字、2つ\n18: 3\n19: 0.25, 0.3, 0.33, 1/3\n20: しない"
  },
  "anthropic/claude-opus-5": {
   "score": 20,
   "answered": 20,
   "first10": 10,
   "last10": 10,
   "inconsistent_pairs": 0,
   "sec": 19.3,
   "out_tokens": 881,
   "price": "$5 / $25",
   "graded": {
    "q01": {
     "a": "父親",
     "ok": 1
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "同じ15文字で2語（extraordinarily / synchronization）",
     "ok": 1
    },
    "q05": {
     "a": "3",
     "ok": 1
    },
    "q06": {
     "a": "2",
     "ok": 1
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "水曜日",
     "ok": 1
    },
    "q09": {
     "a": "o",
     "ok": 1
    },
    "q10": {
     "a": "火星",
     "ok": 1
    },
    "q11": {
     "a": "できない",
     "ok": 1
    },
    "q12": {
     "a": "すまいざごうよはお",
     "ok": 1
    },
    "q13": {
     "a": "父と息子",
     "ok": 1
    },
    "q14": {
     "a": "同じ",
     "ok": 1
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "15文字、2語",
     "ok": 1
    },
    "q18": {
     "a": "3",
     "ok": 1
    },
    "q19": {
     "a": "0.25, 0.3, 0.33, 1/3",
     "ok": 1
    },
    "q20": {
     "a": "しない",
     "ok": 1
    }
   },
   "raw": "1: 父親\n2: 10\n3: しない\n4: 同じ15文字で2語（extraordinarily / synchronization）\n5: 3\n6: 2\n7: できない\n8: 水曜日\n9: o\n10: 火星\n11: できない\n12: すまいざごうよはお\n13: 父と息子\n14: 同じ\n15: 5\n16: 3\n17: 15文字、2語\n18: 3\n19: 0.25, 0.3, 0.33, 1/3\n20: しない"
  },
  "anthropic/claude-opus-5-fast": {
   "score": 20,
   "answered": 20,
   "first10": 10,
   "last10": 10,
   "inconsistent_pairs": 0,
   "sec": 12.1,
   "out_tokens": 967,
   "price": "$10 / $50",
   "graded": {
    "q01": {
     "a": "父親",
     "ok": 1
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily(15文字、synchronizationも同数)",
     "ok": 1
    },
    "q05": {
     "a": "3",
     "ok": 1
    },
    "q06": {
     "a": "2",
     "ok": 1
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "水曜日",
     "ok": 1
    },
    "q09": {
     "a": "o",
     "ok": 1
    },
    "q10": {
     "a": "火星",
     "ok": 1
    },
    "q11": {
     "a": "できない",
     "ok": 1
    },
    "q12": {
     "a": "すまいざごうよはお",
     "ok": 1
    },
    "q13": {
     "a": "父と息子(執刀医は父親)",
     "ok": 1
    },
    "q14": {
     "a": "同じ",
     "ok": 1
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "15文字、2つ",
     "ok": 1
    },
    "q18": {
     "a": "3",
     "ok": 1
    },
    "q19": {
     "a": "0.25, 0.3, 0.33, 1/3",
     "ok": 1
    },
    "q20": {
     "a": "存在しない",
     "ok": 1
    }
   },
   "raw": "1: 父親\n2: 10\n3: しない\n4: extraordinarily(15文字、synchronizationも同数)\n5: 3\n6: 2\n7: できない\n8: 水曜日\n9: o\n10: 火星\n11: できない\n12: すまいざごうよはお\n13: 父と息子(執刀医は父親)\n14: 同じ\n15: 5\n16: 3\n17: 15文字、2つ\n18: 3\n19: 0.25, 0.3, 0.33, 1/3\n20: 存在しない"
  },
  "google/gemini-2.5-pro-preview-05-06": {
   "score": 12,
   "answered": 20,
   "first10": 8,
   "last10": 4,
   "inconsistent_pairs": 0,
   "sec": 34.7,
   "out_tokens": 141,
   "price": "$1.25 / $10",
   "graded": {
    "q01": {
     "a": "外科医は父親。",
     "ok": 1
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "4",
     "ok": 0
    },
    "q06": {
     "a": "2",
     "ok": 1
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "水曜日",
     "ok": 1
    },
    "q09": {
     "a": "o",
     "ok": 1
    },
    "q10": {
     "a": "火星",
     "ok": 1
    },
    "q11": {
     "a": "できる",
     "ok": 0
    },
    "q12": {
     "a": "すまいざごうはお",
     "ok": 0
    },
    "q13": {
     "a": "執刀医は父親。",
     "ok": 1
    },
    "q14": {
     "a": "変える",
     "ok": 0
    },
    "q15": {
     "a": "10",
     "ok": 0
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "14文字で1つ",
     "ok": 0
    },
    "q18": {
     "a": "4",
     "ok": 0
    },
    "q19": {
     "a": "0.25, 0.3, 0.33, 1/3",
     "ok": 1
    },
    "q20": {
     "a": "存在しない",
     "ok": 1
    }
   },
   "raw": "1. 外科医は父親。\n2. 10\n3. しない\n4. extraordinarily\n5. 4\n6. 2\n7. できない\n8. 水曜日\n9. o\n10. 火星\n11. できる\n12. すまいざごうはお\n13. 執刀医は父親。\n14. 変える\n15. 10\n16. 3\n17. 14文字で1つ\n18. 4\n19. 0.25, 0.3, 0.33, 1/3\n20. 存在しない"
  },
  "google/gemini-3.1-pro-preview-customtools": {
   "score": 19,
   "answered": 20,
   "first10": 9,
   "last10": 10,
   "inconsistent_pairs": 1,
   "sec": 16.2,
   "out_tokens": 1417,
   "price": "$2 / $12",
   "graded": {
    "q01": {
     "a": "父親",
     "ok": 1
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "3",
     "ok": 1
    },
    "q06": {
     "a": "2",
     "ok": 1
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "水曜日",
     "ok": 1
    },
    "q09": {
     "a": "o",
     "ok": 1
    },
    "q10": {
     "a": "火星",
     "ok": 1
    },
    "q11": {
     "a": "できない",
     "ok": 1
    },
    "q12": {
     "a": "すまいざごうよはお",
     "ok": 1
    },
    "q13": {
     "a": "父親",
     "ok": 1
    },
    "q14": {
     "a": "同じ",
     "ok": 1
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "15文字、2つ",
     "ok": 1
    },
    "q18": {
     "a": "3",
     "ok": 1
    },
    "q19": {
     "a": "0.25, 0.3, 0.33, 1/3",
     "ok": 1
    },
    "q20": {
     "a": "しない",
     "ok": 1
    }
   },
   "raw": "1: 父親\n2: 10\n3: しない\n4: extraordinarily\n5: 3\n6: 2\n7: できない\n8: 水曜日\n9: o\n10: 火星\n11: できない\n12: すまいざごうよはお\n13: 父親\n14: 同じ\n15: 5\n16: 3\n17: 15文字、2つ\n18: 3\n19: 0.25, 0.3, 0.33, 1/3\n20: しない"
  },
  "google/gemini-3.6-flash": {
   "score": 13,
   "answered": 20,
   "first10": 6,
   "last10": 7,
   "inconsistent_pairs": 1,
   "sec": 3.6,
   "out_tokens": 142,
   "price": "$0.75 / $3.75",
   "graded": {
    "q01": {
     "a": "父親",
     "ok": 1
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "4",
     "ok": 0
    },
    "q06": {
     "a": "3",
     "ok": 0
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "水曜日",
     "ok": 1
    },
    "q09": {
     "a": "s",
     "ok": 0
    },
    "q10": {
     "a": "火星",
     "ok": 1
    },
    "q11": {
     "a": "できない",
     "ok": 1
    },
    "q12": {
     "a": "すまいざごうようはお",
     "ok": 0
    },
    "q13": {
     "a": "親子（父親）",
     "ok": 1
    },
    "q14": {
     "a": "同じ",
     "ok": 1
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "4",
     "ok": 0
    },
    "q17": {
     "a": "15文字、2つ",
     "ok": 1
    },
    "q18": {
     "a": "4",
     "ok": 0
    },
    "q19": {
     "a": "0.25, 0.3, 0.33, 1/3",
     "ok": 1
    },
    "q20": {
     "a": "しない",
     "ok": 1
    }
   },
   "raw": "1: 父親\n2: 10\n3: しない\n4: extraordinarily\n5: 4\n6: 3\n7: できない\n8: 水曜日\n9: s\n10: 火星\n11: できない\n12: すまいざごうようはお\n13: 親子（父親）\n14: 同じ\n15: 5\n16: 4\n17: 15文字、2つ\n18: 4\n19: 0.25, 0.3, 0.33, 1/3\n20: しない"
  },
  "google/gemini-3.7-flash": {
   "score": 19,
   "answered": 20,
   "first10": 9,
   "last10": 10,
   "inconsistent_pairs": 1,
   "sec": 6.1,
   "out_tokens": 513,
   "price": "$0.375 / $1.875",
   "graded": {
    "q01": {
     "a": "父親",
     "ok": 1
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "3",
     "ok": 1
    },
    "q06": {
     "a": "2",
     "ok": 1
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "水曜日",
     "ok": 1
    },
    "q09": {
     "a": "o",
     "ok": 1
    },
    "q10": {
     "a": "火星",
     "ok": 1
    },
    "q11": {
     "a": "できない",
     "ok": 1
    },
    "q12": {
     "a": "すまいざごうよはお",
     "ok": 1
    },
    "q13": {
     "a": "親子（父親）",
     "ok": 1
    },
    "q14": {
     "a": "同じ",
     "ok": 1
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "15文字、2つ",
     "ok": 1
    },
    "q18": {
     "a": "3",
     "ok": 1
    },
    "q19": {
     "a": "0.25, 0.3, 0.33, 1/3",
     "ok": 1
    },
    "q20": {
     "a": "しない",
     "ok": 1
    }
   },
   "raw": "1: 父親\n2: 10\n3: しない\n4: extraordinarily\n5: 3\n6: 2\n7: できない\n8: 水曜日\n9: o\n10: 火星\n11: できない\n12: すまいざごうよはお\n13. 親子（父親）\n14: 同じ\n15: 5\n16: 3\n17: 15文字、2つ\n18: 3\n19: 0.25, 0.3, 0.33, 1/3\n20: しない"
  },
  "minimax/minimax-m2.7": {
   "score": 7,
   "answered": 20,
   "first10": 4,
   "last10": 3,
   "inconsistent_pairs": 1,
   "sec": 17.1,
   "out_tokens": 836,
   "price": "$0.3 / $1.2",
   "graded": {
    "q01": {
     "a": "父親",
     "ok": 1
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "4",
     "ok": 0
    },
    "q06": {
     "a": "3",
     "ok": 0
    },
    "q07": {
     "a": "できる",
     "ok": 0
    },
    "q08": {
     "a": "木曜日",
     "ok": 0
    },
    "q09": {
     "a": "d",
     "ok": 0
    },
    "q10": {
     "a": "火星",
     "ok": 1
    },
    "q11": {
     "a": "できる",
     "ok": 0
    },
    "q12": {
     "a": "まらせょうはうお",
     "ok": 0
    },
    "q13": {
     "a": "父子",
     "ok": 1
    },
    "q14": {
     "a": "変える",
     "ok": 0
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "2",
     "ok": 0
    },
    "q17": {
     "a": "15文字が2語",
     "ok": 1
    },
    "q18": {
     "a": "4",
     "ok": 0
    },
    "q19": {
     "a": "0.25, 0.3, 1/3, 0.33",
     "ok": 0
    },
    "q20": {
     "a": "存在する",
     "ok": 0
    }
   },
   "raw": "1: 父親\n2: 10\n3: しない\n4: extraordinarily\n5: 4\n6: 3\n7: できる\n8: 木曜日\n9: d\n10: 火星\n11: できる\n12: まらせょうはうお\n13: 父子\n14: 変える\n15: 5\n16: 2\n17: 15文字が2語\n18: 4\n19: 0.25, 0.3, 1/3, 0.33\n20: 存在する"
  },
  "openai/gpt-5.5": {
   "score": 19,
   "answered": 20,
   "first10": 9,
   "last10": 10,
   "inconsistent_pairs": 1,
   "sec": 19.5,
   "out_tokens": 1174,
   "price": "$5 / $30",
   "graded": {
    "q01": {
     "a": "父親",
     "ok": 1
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "3",
     "ok": 1
    },
    "q06": {
     "a": "2",
     "ok": 1
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "水曜日",
     "ok": 1
    },
    "q09": {
     "a": "o",
     "ok": 1
    },
    "q10": {
     "a": "火星",
     "ok": 1
    },
    "q11": {
     "a": "できない",
     "ok": 1
    },
    "q12": {
     "a": "すまいざごうよはお",
     "ok": 1
    },
    "q13": {
     "a": "父親と息子",
     "ok": 1
    },
    "q14": {
     "a": "同じ",
     "ok": 1
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "15文字、2語",
     "ok": 1
    },
    "q18": {
     "a": "3",
     "ok": 1
    },
    "q19": {
     "a": "0.25, 0.3, 0.33, 1/3",
     "ok": 1
    },
    "q20": {
     "a": "しない",
     "ok": 1
    }
   },
   "raw": "1: 父親\n2: 10\n3: しない\n4: extraordinarily\n5: 3\n6: 2\n7: できない\n8: 水曜日\n9: o\n10: 火星\n11: できない\n12: すまいざごうよはお\n13: 父親と息子\n14: 同じ\n15: 5\n16: 3\n17: 15文字、2語\n18: 3\n19: 0.25, 0.3, 0.33, 1/3\n20: しない"
  },
  "openai/gpt-5.6-sol-pro": {
   "score": 19,
   "answered": 20,
   "first10": 9,
   "last10": 10,
   "inconsistent_pairs": 1,
   "sec": 29.4,
   "out_tokens": 2785,
   "price": "$2.5 / $15",
   "graded": {
    "q01": {
     "a": "父親",
     "ok": 1
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "3",
     "ok": 1
    },
    "q06": {
     "a": "2",
     "ok": 1
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "水曜日",
     "ok": 1
    },
    "q09": {
     "a": "o",
     "ok": 1
    },
    "q10": {
     "a": "火星",
     "ok": 1
    },
    "q11": {
     "a": "できない",
     "ok": 1
    },
    "q12": {
     "a": "すまいざごうよはお",
     "ok": 1
    },
    "q13": {
     "a": "父子",
     "ok": 1
    },
    "q14": {
     "a": "同じ",
     "ok": 1
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "15文字、2語",
     "ok": 1
    },
    "q18": {
     "a": "3",
     "ok": 1
    },
    "q19": {
     "a": "0.25, 0.3, 0.33, 1/3",
     "ok": 1
    },
    "q20": {
     "a": "しない",
     "ok": 1
    }
   },
   "raw": "1: 父親\n2: 10\n3: しない\n4: extraordinarily\n5: 3\n6: 2\n7: できない\n8: 水曜日\n9: o\n10: 火星\n11: できない\n12: すまいざごうよはお\n13: 父子\n14: 同じ\n15: 5\n16: 3\n17: 15文字、2語\n18: 3\n19: 0.25, 0.3, 0.33, 1/3\n20: しない"
  },
  "openai/gpt-5.6-terra": {
   "score": 16,
   "answered": 20,
   "first10": 8,
   "last10": 8,
   "inconsistent_pairs": 1,
   "sec": 11.8,
   "out_tokens": 593,
   "price": "$1.25 / $7.5",
   "graded": {
    "q01": {
     "a": "母親",
     "ok": 0
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "3",
     "ok": 1
    },
    "q06": {
     "a": "2",
     "ok": 1
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "水曜日",
     "ok": 1
    },
    "q09": {
     "a": "o",
     "ok": 1
    },
    "q10": {
     "a": "火星",
     "ok": 1
    },
    "q11": {
     "a": "できない",
     "ok": 1
    },
    "q12": {
     "a": "すまいざごうよはお",
     "ok": 1
    },
    "q13": {
     "a": "母親",
     "ok": 0
    },
    "q14": {
     "a": "変える",
     "ok": 0
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "15文字、2語",
     "ok": 1
    },
    "q18": {
     "a": "3",
     "ok": 1
    },
    "q19": {
     "a": "0.25, 0.3, 0.33, 1/3",
     "ok": 1
    },
    "q20": {
     "a": "しない",
     "ok": 1
    }
   },
   "raw": "1: 母親\n2: 10\n3: しない\n4: extraordinarily\n5: 3\n6: 2\n7: できない\n8: 水曜日\n9: o\n10: 火星\n11: できない\n12: すまいざごうよはお\n13: 母親\n14: 変える\n15: 5\n16: 3\n17: 15文字、2語\n18: 3\n19: 0.25, 0.3, 0.33, 1/3\n20: しない"
  },
  "openai/gpt-5.6-terra-pro": {
   "score": 18,
   "answered": 20,
   "first10": 9,
   "last10": 9,
   "inconsistent_pairs": 0,
   "sec": 18.4,
   "out_tokens": 2264,
   "price": "$1.25 / $7.5",
   "graded": {
    "q01": {
     "a": "父親",
     "ok": 1
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "3",
     "ok": 1
    },
    "q06": {
     "a": "2",
     "ok": 1
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "水曜日",
     "ok": 1
    },
    "q09": {
     "a": "o",
     "ok": 1
    },
    "q10": {
     "a": "火星",
     "ok": 1
    },
    "q11": {
     "a": "できない",
     "ok": 1
    },
    "q12": {
     "a": "すまいざごうよはお",
     "ok": 1
    },
    "q13": {
     "a": "父親",
     "ok": 1
    },
    "q14": {
     "a": "同じ",
     "ok": 1
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "15文字、1つ",
     "ok": 0
    },
    "q18": {
     "a": "3",
     "ok": 1
    },
    "q19": {
     "a": "0.25, 0.3, 0.33, 1/3",
     "ok": 1
    },
    "q20": {
     "a": "しない",
     "ok": 1
    }
   },
   "raw": "1: 父親\n2: 10\n3: しない\n4: extraordinarily\n5: 3\n6: 2\n7: できない\n8: 水曜日\n9: o\n10: 火星\n11: できない\n12: すまいざごうよはお\n13: 父親\n14: 同じ\n15: 5\n16: 3\n17: 15文字、1つ\n18: 3\n19: 0.25, 0.3, 0.33, 1/3\n20: しない"
  },
  "openai/gpt-6-astra-pro": {
   "score": 19,
   "answered": 20,
   "first10": 9,
   "last10": 10,
   "inconsistent_pairs": 1,
   "sec": 12.3,
   "out_tokens": 632,
   "price": "$10 / $50",
   "graded": {
    "q01": {
     "a": "父親",
     "ok": 1
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "3",
     "ok": 1
    },
    "q06": {
     "a": "2",
     "ok": 1
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "水曜日",
     "ok": 1
    },
    "q09": {
     "a": "o",
     "ok": 1
    },
    "q10": {
     "a": "火星",
     "ok": 1
    },
    "q11": {
     "a": "できない",
     "ok": 1
    },
    "q12": {
     "a": "すまいざごうよはお",
     "ok": 1
    },
    "q13": {
     "a": "父親",
     "ok": 1
    },
    "q14": {
     "a": "同じ",
     "ok": 1
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "15文字、2語",
     "ok": 1
    },
    "q18": {
     "a": "3",
     "ok": 1
    },
    "q19": {
     "a": "0.25, 0.3, 0.33, 1/3",
     "ok": 1
    },
    "q20": {
     "a": "しない",
     "ok": 1
    }
   },
   "raw": "1: 父親\n2: 10\n3: しない\n4: extraordinarily\n5: 3\n6: 2\n7: できない\n8: 水曜日\n9: o\n10: 火星\n11: できない\n12: すまいざごうよはお\n13: 父親\n14: 同じ\n15: 5\n16: 3\n17: 15文字、2語\n18: 3\n19: 0.25, 0.3, 0.33, 1/3\n20: しない"
  },
  "openrouter/auto": {
   "score": 20,
   "answered": 20,
   "first10": 10,
   "last10": 10,
   "inconsistent_pairs": 0,
   "sec": 18.1,
   "out_tokens": 837,
   "price": "$1 / $3",
   "graded": {
    "q01": {
     "a": "父親",
     "ok": 1
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily(synchronizationも同じ15文字)",
     "ok": 1
    },
    "q05": {
     "a": "3",
     "ok": 1
    },
    "q06": {
     "a": "2",
     "ok": 1
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "水曜日",
     "ok": 1
    },
    "q09": {
     "a": "o",
     "ok": 1
    },
    "q10": {
     "a": "火星",
     "ok": 1
    },
    "q11": {
     "a": "できない",
     "ok": 1
    },
    "q12": {
     "a": "すまいざごうよはお",
     "ok": 1
    },
    "q13": {
     "a": "父親と息子",
     "ok": 1
    },
    "q14": {
     "a": "同じ",
     "ok": 1
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "15文字、2つ",
     "ok": 1
    },
    "q18": {
     "a": "3",
     "ok": 1
    },
    "q19": {
     "a": "0.25, 0.3, 0.33, 1/3",
     "ok": 1
    },
    "q20": {
     "a": "存在しない",
     "ok": 1
    }
   },
   "raw": "1: 父親\n2: 10\n3: しない\n4: extraordinarily(synchronizationも同じ15文字)\n5: 3\n6: 2\n7: できない\n8: 水曜日\n9: o\n10: 火星\n11: できない\n12: すまいざごうよはお\n13: 父親と息子\n14: 同じ\n15: 5\n16: 3\n17: 15文字、2つ\n18: 3\n19: 0.25, 0.3, 0.33, 1/3\n20: 存在しない"
  },
  "shitate/orchestrator": {
   "score": 19,
   "answered": 20,
   "first10": 9,
   "last10": 10,
   "inconsistent_pairs": 1,
   "sec": 50.9,
   "out_tokens": 7444,
   "price": "$0.27 / $1.33",
   "graded": {
    "q01": {
     "a": "父親",
     "ok": 1
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "3",
     "ok": 1
    },
    "q06": {
     "a": "2",
     "ok": 1
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "水曜日",
     "ok": 1
    },
    "q09": {
     "a": "o",
     "ok": 1
    },
    "q10": {
     "a": "火星",
     "ok": 1
    },
    "q11": {
     "a": "できない",
     "ok": 1
    },
    "q12": {
     "a": "すまいざごうよはお",
     "ok": 1
    },
    "q13": {
     "a": "父親",
     "ok": 1
    },
    "q14": {
     "a": "同じ",
     "ok": 1
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "15文字、2つ",
     "ok": 1
    },
    "q18": {
     "a": "3",
     "ok": 1
    },
    "q19": {
     "a": "0.25, 0.3, 0.33, 1/3",
     "ok": 1
    },
    "q20": {
     "a": "しない",
     "ok": 1
    }
   },
   "raw": "1: 父親\n2: 10\n3: しない\n4: extraordinarily\n5: 3\n6: 2\n7: できない\n8: 水曜日\n9: o\n10: 火星\n11: できない\n12: すまいざごうよはお\n13: 父親\n14: 同じ\n15: 5\n16: 3\n17: 15文字、2つ\n18: 3\n19: 0.25, 0.3, 0.33, 1/3\n20: しない"
  },
  "shitate/orchestrator-max": {
   "score": 18,
   "answered": 20,
   "first10": 9,
   "last10": 9,
   "inconsistent_pairs": 1,
   "sec": 32.1,
   "out_tokens": 727,
   "price": "$5.95 / $30.1",
   "graded": {
    "q01": {
     "a": "父親",
     "ok": 1
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "3",
     "ok": 1
    },
    "q06": {
     "a": "2",
     "ok": 1
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "水曜日",
     "ok": 1
    },
    "q09": {
     "a": "o",
     "ok": 1
    },
    "q10": {
     "a": "火星",
     "ok": 1
    },
    "q11": {
     "a": "できない",
     "ok": 1
    },
    "q12": {
     "a": "すまいいざごうよはお",
     "ok": 0
    },
    "q13": {
     "a": "父親",
     "ok": 1
    },
    "q14": {
     "a": "同じ",
     "ok": 1
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "15文字、2つ",
     "ok": 1
    },
    "q18": {
     "a": "3",
     "ok": 1
    },
    "q19": {
     "a": "0.25, 0.3, 0.33, 1/3",
     "ok": 1
    },
    "q20": {
     "a": "しない",
     "ok": 1
    }
   },
   "raw": "1: 父親\n2: 10\n3: しない\n4: extraordinarily\n5: 3\n6: 2\n7: できない\n8: 水曜日\n9: o\n10: 火星\n11: できない\n12: すまいいざごうよはお\n13: 父親\n14: 同じ\n15: 5\n16: 3\n17: 15文字、2つ\n18: 3\n19: 0.25, 0.3, 0.33, 1/3\n20: しない"
  },
  "shitate/orchestrator:max": {
   "score": 20,
   "answered": 20,
   "first10": 10,
   "last10": 10,
   "inconsistent_pairs": 0,
   "sec": 39.4,
   "out_tokens": 5365,
   "price": "$10.5 / $52.5",
   "graded": {
    "q01": {
     "a": "父親",
     "ok": 1
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily(synchronizationも同じ15文字)",
     "ok": 1
    },
    "q05": {
     "a": "3",
     "ok": 1
    },
    "q06": {
     "a": "2",
     "ok": 1
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "水曜日",
     "ok": 1
    },
    "q09": {
     "a": "o",
     "ok": 1
    },
    "q10": {
     "a": "火星",
     "ok": 1
    },
    "q11": {
     "a": "できない",
     "ok": 1
    },
    "q12": {
     "a": "すまいざごうよはお",
     "ok": 1
    },
    "q13": {
     "a": "父親と息子",
     "ok": 1
    },
    "q14": {
     "a": "同じ",
     "ok": 1
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "15文字、2つ(extraordinarily と synchronization)",
     "ok": 1
    },
    "q18": {
     "a": "3",
     "ok": 1
    },
    "q19": {
     "a": "0.25, 0.3, 0.33, 1/3",
     "ok": 1
    },
    "q20": {
     "a": "しない",
     "ok": 1
    }
   },
   "raw": "1: 父親\n2: 10\n3: しない\n4: extraordinarily(synchronizationも同じ15文字)\n5: 3\n6: 2\n7: できない\n8: 水曜日\n9: o\n10: 火星\n11: できない\n12: すまいざごうよはお\n13: 父親と息子\n14: 同じ\n15: 5\n16: 3\n17: 15文字、2つ(extraordinarily と synchronization)\n18: 3\n19: 0.25, 0.3, 0.33, 1/3\n20: しない"
  },
  "shitate/orchestrator:solid": {
   "score": 19,
   "answered": 20,
   "first10": 9,
   "last10": 10,
   "inconsistent_pairs": 1,
   "sec": 34.6,
   "out_tokens": 3339,
   "price": "$6.04 / $28.89",
   "graded": {
    "q01": {
     "a": "父親",
     "ok": 1
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "3",
     "ok": 1
    },
    "q06": {
     "a": "2",
     "ok": 1
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "水曜日",
     "ok": 1
    },
    "q09": {
     "a": "o",
     "ok": 1
    },
    "q10": {
     "a": "火星",
     "ok": 1
    },
    "q11": {
     "a": "できない",
     "ok": 1
    },
    "q12": {
     "a": "すまいざごうよはお",
     "ok": 1
    },
    "q13": {
     "a": "父親",
     "ok": 1
    },
    "q14": {
     "a": "同じ",
     "ok": 1
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "15文字で2つ(extraordinarily, synchronization)",
     "ok": 1
    },
    "q18": {
     "a": "3",
     "ok": 1
    },
    "q19": {
     "a": "0.25, 0.3, 0.33, 1/3",
     "ok": 1
    },
    "q20": {
     "a": "しない",
     "ok": 1
    }
   },
   "raw": "1: 父親\n2: 10\n3: しない\n4: extraordinarily\n5: 3\n6: 2\n7: できない\n8: 水曜日\n9: o\n10: 火星\n11: できない\n12: すまいざごうよはお\n13: 父親\n14: 同じ\n15: 5\n16: 3\n17: 15文字で2つ(extraordinarily, synchronization)\n18: 3\n19: 0.25, 0.3, 0.33, 1/3\n20: しない"
  },
  "teai/max": {
   "score": 18,
   "answered": 20,
   "first10": 9,
   "last10": 9,
   "inconsistent_pairs": 1,
   "sec": 32.3,
   "out_tokens": 701,
   "price": "$5.95 / $30.1",
   "graded": {
    "q01": {
     "a": "父親",
     "ok": 1
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "3",
     "ok": 1
    },
    "q06": {
     "a": "2",
     "ok": 1
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "水曜日",
     "ok": 1
    },
    "q09": {
     "a": "o",
     "ok": 1
    },
    "q10": {
     "a": "火星",
     "ok": 1
    },
    "q11": {
     "a": "できない",
     "ok": 1
    },
    "q12": {
     "a": "すまいいざごうよはお",
     "ok": 0
    },
    "q13": {
     "a": "父親",
     "ok": 1
    },
    "q14": {
     "a": "同じ",
     "ok": 1
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "15文字、2つ",
     "ok": 1
    },
    "q18": {
     "a": "3",
     "ok": 1
    },
    "q19": {
     "a": "0.25, 0.3, 0.33, 1/3",
     "ok": 1
    },
    "q20": {
     "a": "しない",
     "ok": 1
    }
   },
   "raw": "1: 父親\n2: 10\n3: しない\n4: extraordinarily\n5: 3\n6: 2\n7: できない\n8: 水曜日\n9: o\n10: 火星\n11: できない\n12: すまいいざごうよはお\n13: 父親\n14: 同じ\n15: 5\n16: 3\n17: 15文字、2つ\n18: 3\n19: 0.25, 0.3, 0.33, 1/3\n20: しない"
  },
  "z-ai/glm-5.3-flash": {
   "score": 13,
   "answered": 20,
   "first10": 8,
   "last10": 5,
   "inconsistent_pairs": 0,
   "sec": 5.4,
   "out_tokens": 331,
   "price": "$0.07 / $0.25",
   "graded": {
    "q01": {
     "a": "父親",
     "ok": 1
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "synchronization",
     "ok": 0
    },
    "q05": {
     "a": "4",
     "ok": 0
    },
    "q06": {
     "a": "2",
     "ok": 1
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "水曜日",
     "ok": 1
    },
    "q09": {
     "a": "o",
     "ok": 1
    },
    "q10": {
     "a": "火星",
     "ok": 1
    },
    "q11": {
     "a": "できる",
     "ok": 0
    },
    "q12": {
     "a": "すでいざうがよはお",
     "ok": 0
    },
    "q13": {
     "a": "父親",
     "ok": 1
    },
    "q14": {
     "a": "同じ",
     "ok": 1
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "4",
     "ok": 0
    },
    "q17": {
     "a": "15文字、1つ",
     "ok": 0
    },
    "q18": {
     "a": "4",
     "ok": 0
    },
    "q19": {
     "a": "0.25, 0.3, 0.33, 1/3",
     "ok": 1
    },
    "q20": {
     "a": "しない",
     "ok": 1
    }
   },
   "raw": "1: 父親\n2: 10\n3: しない\n4: synchronization\n5: 4\n6: 2\n7: できない\n8: 水曜日\n9: o\n10: 火星\n11: できる\n12: すでいざうがよはお\n13: 父親\n14: 同じ\n15: 5\n16: 4\n17: 15文字、1つ\n18: 4\n19: 0.25, 0.3, 0.33, 1/3\n20: しない"
  },
  "qwen/qwen3.5-flash-02-23": {
   "score": 9,
   "answered": 12,
   "first10": 8,
   "last10": 1,
   "inconsistent_pairs": 0,
   "sec": 28.4,
   "out_tokens": 6005,
   "price": "$0.065 / $0.26",
   "graded": {
    "q01": {
     "a": "父親",
     "ok": 1
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "4",
     "ok": 0
    },
    "q06": {
     "a": "2",
     "ok": 1
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "水曜日",
     "ok": 1
    },
    "q09": {
     "a": "o",
     "ok": 1
    },
    "q10": {
     "a": "火星",
     "ok": 1
    },
    "q11": {
     "a": "できない",
     "ok": 1
    },
    "q12": {
     "a": "すまいさご",
     "ok": 0
    },
    "q13": {
     "a": null,
     "ok": -1
    },
    "q14": {
     "a": null,
     "ok": -1
    },
    "q15": {
     "a": null,
     "ok": -1
    },
    "q16": {
     "a": null,
     "ok": -1
    },
    "q17": {
     "a": null,
     "ok": -1
    },
    "q18": {
     "a": null,
     "ok": -1
    },
    "q19": {
     "a": null,
     "ok": -1
    },
    "q20": {
     "a": null,
     "ok": -1
    }
   },
   "raw": "1: 父親\n2: 10\n3: しない\n4: extraordinarily\n5: 4\n6: 2\n7: できない\n8: 水曜日\n9: o\n10: 火星\n11: できない\n12: すまいさご"
  },
  "deepseek/deepseek-v4-flash-cache": {
   "score": 0,
   "answered": 0,
   "first10": 0,
   "last10": 0,
   "inconsistent_pairs": 0,
   "sec": 30.4,
   "out_tokens": 6000,
   "price": "$0.007 / $0.66",
   "graded": {
    "q01": {
     "a": null,
     "ok": -1
    },
    "q02": {
     "a": null,
     "ok": -1
    },
    "q03": {
     "a": null,
     "ok": -1
    },
    "q04": {
     "a": null,
     "ok": -1
    },
    "q05": {
     "a": null,
     "ok": -1
    },
    "q06": {
     "a": null,
     "ok": -1
    },
    "q07": {
     "a": null,
     "ok": -1
    },
    "q08": {
     "a": null,
     "ok": -1
    },
    "q09": {
     "a": null,
     "ok": -1
    },
    "q10": {
     "a": null,
     "ok": -1
    },
    "q11": {
     "a": null,
     "ok": -1
    },
    "q12": {
     "a": null,
     "ok": -1
    },
    "q13": {
     "a": null,
     "ok": -1
    },
    "q14": {
     "a": null,
     "ok": -1
    },
    "q15": {
     "a": null,
     "ok": -1
    },
    "q16": {
     "a": null,
     "ok": -1
    },
    "q17": {
     "a": null,
     "ok": -1
    },
    "q18": {
     "a": null,
     "ok": -1
    },
    "q19": {
     "a": null,
     "ok": -1
    },
    "q20": {
     "a": null,
     "ok": -1
    }
   },
   "raw": ""
  },
  "z-ai/glm-5.2": {
   "score": 0,
   "answered": 0,
   "first10": 0,
   "last10": 0,
   "inconsistent_pairs": 0,
   "sec": 30.3,
   "out_tokens": 6000,
   "price": "",
   "graded": {
    "q01": {
     "a": null,
     "ok": -1
    },
    "q02": {
     "a": null,
     "ok": -1
    },
    "q03": {
     "a": null,
     "ok": -1
    },
    "q04": {
     "a": null,
     "ok": -1
    },
    "q05": {
     "a": null,
     "ok": -1
    },
    "q06": {
     "a": null,
     "ok": -1
    },
    "q07": {
     "a": null,
     "ok": -1
    },
    "q08": {
     "a": null,
     "ok": -1
    },
    "q09": {
     "a": null,
     "ok": -1
    },
    "q10": {
     "a": null,
     "ok": -1
    },
    "q11": {
     "a": null,
     "ok": -1
    },
    "q12": {
     "a": null,
     "ok": -1
    },
    "q13": {
     "a": null,
     "ok": -1
    },
    "q14": {
     "a": null,
     "ok": -1
    },
    "q15": {
     "a": null,
     "ok": -1
    },
    "q16": {
     "a": null,
     "ok": -1
    },
    "q17": {
     "a": null,
     "ok": -1
    },
    "q18": {
     "a": null,
     "ok": -1
    },
    "q19": {
     "a": null,
     "ok": -1
    },
    "q20": {
     "a": null,
     "ok": -1
    }
   },
   "raw": ""
  },
  "qwen/qwen3.6-flash": {
   "score": 10,
   "answered": 12,
   "first10": 9,
   "last10": 1,
   "inconsistent_pairs": 0,
   "sec": 22.7,
   "out_tokens": 4157,
   "price": "$0.1875 / $1.125",
   "graded": {
    "q01": {
     "a": "父親",
     "ok": 1
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "3",
     "ok": 1
    },
    "q06": {
     "a": "2",
     "ok": 1
    },
    "q07": {
     "a": "できない",
     "ok": 1
    },
    "q08": {
     "a": "水曜日",
     "ok": 1
    },
    "q09": {
     "a": "o",
     "ok": 1
    },
    "q10": {
     "a": "火星",
     "ok": 1
    },
    "q11": {
     "a": "できない",
     "ok": 1
    },
    "q12": {
     "a": "すまいざごう",
     "ok": 0
    },
    "q13": {
     "a": null,
     "ok": -1
    },
    "q14": {
     "a": null,
     "ok": -1
    },
    "q15": {
     "a": null,
     "ok": -1
    },
    "q16": {
     "a": null,
     "ok": -1
    },
    "q17": {
     "a": null,
     "ok": -1
    },
    "q18": {
     "a": null,
     "ok": -1
    },
    "q19": {
     "a": null,
     "ok": -1
    },
    "q20": {
     "a": null,
     "ok": -1
    }
   },
   "raw": "1: 父親\n2: 10\n3: しない\n4: extraordinarily\n5: 3\n6: 2\n7: できない\n8: 水曜日\n9: o\n10: 火星\n11: できない\n12: すまいざごう"
  },
  "inclusionai/ling-3.0-flash-fin": {
   "score": 0,
   "answered": 0,
   "first10": 0,
   "last10": 0,
   "inconsistent_pairs": 0,
   "sec": 19.9,
   "out_tokens": 1616,
   "price": "$0.06 / $0.18",
   "graded": {
    "q01": {
     "a": null,
     "ok": -1
    },
    "q02": {
     "a": null,
     "ok": -1
    },
    "q03": {
     "a": null,
     "ok": -1
    },
    "q04": {
     "a": null,
     "ok": -1
    },
    "q05": {
     "a": null,
     "ok": -1
    },
    "q06": {
     "a": null,
     "ok": -1
    },
    "q07": {
     "a": null,
     "ok": -1
    },
    "q08": {
     "a": null,
     "ok": -1
    },
    "q09": {
     "a": null,
     "ok": -1
    },
    "q10": {
     "a": null,
     "ok": -1
    },
    "q11": {
     "a": null,
     "ok": -1
    },
    "q12": {
     "a": null,
     "ok": -1
    },
    "q13": {
     "a": null,
     "ok": -1
    },
    "q14": {
     "a": null,
     "ok": -1
    },
    "q15": {
     "a": null,
     "ok": -1
    },
    "q16": {
     "a": null,
     "ok": -1
    },
    "q17": {
     "a": null,
     "ok": -1
    },
    "q18": {
     "a": null,
     "ok": -1
    },
    "q19": {
     "a": null,
     "ok": -1
    },
    "q20": {
     "a": null,
     "ok": -1
    }
   },
   "raw": ""
  },
  "stepfun/step-3.7-flash": {
   "score": 0,
   "answered": 0,
   "first10": 0,
   "last10": 0,
   "inconsistent_pairs": 0,
   "sec": 29.9,
   "out_tokens": 6000,
   "price": "$0.2 / $1.15",
   "graded": {
    "q01": {
     "a": null,
     "ok": -1
    },
    "q02": {
     "a": null,
     "ok": -1
    },
    "q03": {
     "a": null,
     "ok": -1
    },
    "q04": {
     "a": null,
     "ok": -1
    },
    "q05": {
     "a": null,
     "ok": -1
    },
    "q06": {
     "a": null,
     "ok": -1
    },
    "q07": {
     "a": null,
     "ok": -1
    },
    "q08": {
     "a": null,
     "ok": -1
    },
    "q09": {
     "a": null,
     "ok": -1
    },
    "q10": {
     "a": null,
     "ok": -1
    },
    "q11": {
     "a": null,
     "ok": -1
    },
    "q12": {
     "a": null,
     "ok": -1
    },
    "q13": {
     "a": null,
     "ok": -1
    },
    "q14": {
     "a": null,
     "ok": -1
    },
    "q15": {
     "a": null,
     "ok": -1
    },
    "q16": {
     "a": null,
     "ok": -1
    },
    "q17": {
     "a": null,
     "ok": -1
    },
    "q18": {
     "a": null,
     "ok": -1
    },
    "q19": {
     "a": null,
     "ok": -1
    },
    "q20": {
     "a": null,
     "ok": -1
    }
   },
   "raw": ""
  },
  "deepseek/deepseek-v3.1-terminus": {
   "score": 0,
   "answered": 0,
   "first10": 0,
   "last10": 0,
   "inconsistent_pairs": 0,
   "sec": 7.6,
   "out_tokens": 129,
   "price": "$0.27 / $1",
   "graded": {
    "q01": {
     "a": null,
     "ok": -1
    },
    "q02": {
     "a": null,
     "ok": -1
    },
    "q03": {
     "a": null,
     "ok": -1
    },
    "q04": {
     "a": null,
     "ok": -1
    },
    "q05": {
     "a": null,
     "ok": -1
    },
    "q06": {
     "a": null,
     "ok": -1
    },
    "q07": {
     "a": null,
     "ok": -1
    },
    "q08": {
     "a": null,
     "ok": -1
    },
    "q09": {
     "a": null,
     "ok": -1
    },
    "q10": {
     "a": null,
     "ok": -1
    },
    "q11": {
     "a": null,
     "ok": -1
    },
    "q12": {
     "a": null,
     "ok": -1
    },
    "q13": {
     "a": null,
     "ok": -1
    },
    "q14": {
     "a": null,
     "ok": -1
    },
    "q15": {
     "a": null,
     "ok": -1
    },
    "q16": {
     "a": null,
     "ok": -1
    },
    "q17": {
     "a": null,
     "ok": -1
    },
    "q18": {
     "a": null,
     "ok": -1
    },
    "q19": {
     "a": null,
     "ok": -1
    },
    "q20": {
     "a": null,
     "ok": -1
    }
   },
   "raw": ""
  },
  "qwen/qwen3-coder-30b-a3b-instruct": {
   "score": 0,
   "answered": 0,
   "first10": 0,
   "last10": 0,
   "inconsistent_pairs": 0,
   "sec": 3.0,
   "out_tokens": 152,
   "price": "$0.07 / $0.28",
   "graded": {
    "q01": {
     "a": null,
     "ok": -1
    },
    "q02": {
     "a": null,
     "ok": -1
    },
    "q03": {
     "a": null,
     "ok": -1
    },
    "q04": {
     "a": null,
     "ok": -1
    },
    "q05": {
     "a": null,
     "ok": -1
    },
    "q06": {
     "a": null,
     "ok": -1
    },
    "q07": {
     "a": null,
     "ok": -1
    },
    "q08": {
     "a": null,
     "ok": -1
    },
    "q09": {
     "a": null,
     "ok": -1
    },
    "q10": {
     "a": null,
     "ok": -1
    },
    "q11": {
     "a": null,
     "ok": -1
    },
    "q12": {
     "a": null,
     "ok": -1
    },
    "q13": {
     "a": null,
     "ok": -1
    },
    "q14": {
     "a": null,
     "ok": -1
    },
    "q15": {
     "a": null,
     "ok": -1
    },
    "q16": {
     "a": null,
     "ok": -1
    },
    "q17": {
     "a": null,
     "ok": -1
    },
    "q18": {
     "a": null,
     "ok": -1
    },
    "q19": {
     "a": null,
     "ok": -1
    },
    "q20": {
     "a": null,
     "ok": -1
    }
   },
   "raw": ""
  },
  "deepseek/deepseek-chat-v3.1": {
   "score": 0,
   "answered": 0,
   "first10": 0,
   "last10": 0,
   "inconsistent_pairs": 0,
   "sec": 7.9,
   "out_tokens": 129,
   "price": "$0.25 / $0.95",
   "graded": {
    "q01": {
     "a": null,
     "ok": -1
    },
    "q02": {
     "a": null,
     "ok": -1
    },
    "q03": {
     "a": null,
     "ok": -1
    },
    "q04": {
     "a": null,
     "ok": -1
    },
    "q05": {
     "a": null,
     "ok": -1
    },
    "q06": {
     "a": null,
     "ok": -1
    },
    "q07": {
     "a": null,
     "ok": -1
    },
    "q08": {
     "a": null,
     "ok": -1
    },
    "q09": {
     "a": null,
     "ok": -1
    },
    "q10": {
     "a": null,
     "ok": -1
    },
    "q11": {
     "a": null,
     "ok": -1
    },
    "q12": {
     "a": null,
     "ok": -1
    },
    "q13": {
     "a": null,
     "ok": -1
    },
    "q14": {
     "a": null,
     "ok": -1
    },
    "q15": {
     "a": null,
     "ok": -1
    },
    "q16": {
     "a": null,
     "ok": -1
    },
    "q17": {
     "a": null,
     "ok": -1
    },
    "q18": {
     "a": null,
     "ok": -1
    },
    "q19": {
     "a": null,
     "ok": -1
    },
    "q20": {
     "a": null,
     "ok": -1
    }
   },
   "raw": ""
  },
  "google/gemini-2.5-pro-preview": {
   "score": 0,
   "answered": 0,
   "first10": 0,
   "last10": 0,
   "inconsistent_pairs": 0,
   "sec": 41.7,
   "out_tokens": 2048,
   "price": "$1.25 / $10",
   "graded": {
    "q01": {
     "a": null,
     "ok": -1
    },
    "q02": {
     "a": null,
     "ok": -1
    },
    "q03": {
     "a": null,
     "ok": -1
    },
    "q04": {
     "a": null,
     "ok": -1
    },
    "q05": {
     "a": null,
     "ok": -1
    },
    "q06": {
     "a": null,
     "ok": -1
    },
    "q07": {
     "a": null,
     "ok": -1
    },
    "q08": {
     "a": null,
     "ok": -1
    },
    "q09": {
     "a": null,
     "ok": -1
    },
    "q10": {
     "a": null,
     "ok": -1
    },
    "q11": {
     "a": null,
     "ok": -1
    },
    "q12": {
     "a": null,
     "ok": -1
    },
    "q13": {
     "a": null,
     "ok": -1
    },
    "q14": {
     "a": null,
     "ok": -1
    },
    "q15": {
     "a": null,
     "ok": -1
    },
    "q16": {
     "a": null,
     "ok": -1
    },
    "q17": {
     "a": null,
     "ok": -1
    },
    "q18": {
     "a": null,
     "ok": -1
    },
    "q19": {
     "a": null,
     "ok": -1
    },
    "q20": {
     "a": null,
     "ok": -1
    }
   },
   "raw": ""
  },
  "x-ai/grok-4.5": {
   "score": 18,
   "answered": 20,
   "first10": 8,
   "last10": 10,
   "inconsistent_pairs": 2,
   "sec": 30.8,
   "out_tokens": 1815,
   "graded": {}
  },
  "meituan/longcat-2.0": {
   "score": 15,
   "answered": 20,
   "first10": 8,
   "last10": 7,
   "inconsistent_pairs": 1,
   "sec": 38.7,
   "out_tokens": 2359,
   "graded": {}
  },
  "sakana/fugu-max": {
   "score": 19,
   "answered": 20,
   "first10": 9,
   "last10": 10,
   "inconsistent_pairs": 1,
   "sec": 33.5,
   "out_tokens": 4040,
   "graded": {}
  },
  "inclusionai/ling-3.0-flash-vl": {
   "score": 14,
   "answered": 20,
   "first10": 8,
   "last10": 6,
   "inconsistent_pairs": 0,
   "sec": 20.8,
   "out_tokens": 1318,
   "graded": {}
  },
  "qwen/qwen3.7-max": {
   "score": 19,
   "answered": 20,
   "first10": 10,
   "last10": 9,
   "inconsistent_pairs": 1,
   "sec": 139.3,
   "out_tokens": 4801,
   "graded": {
    "q01": {
     "a": "父親",
     "ok": 1,
     "pair": "surgeon"
    },
    "q02": {
     "a": "10",
     "ok": 1,
     "pair": null
    },
    "q03": {
     "a": "しない",
     "ok": 1,
     "pair": null
    },
    "q04": {
     "a": "extraordinarily と synchronization（いずれも15文字）",
     "ok": 1,
     "pair": "longest"
    },
    "q05": {
     "a": "3",
     "ok": 1,
     "pair": "mora"
    },
    "q06": {
     "a": "2",
     "ok": 1,
     "pair": null
    },
    "q07": {
     "a": "できない",
     "ok": 1,
     "pair": "river"
    },
    "q08": {
     "a": "水曜日です。",
     "ok": 1,
     "pair": null
    },
    "q09": {
     "a": "o",
     "ok": 1,
     "pair": null
    },
    "q10": {
     "a": "火星です。",
     "ok": 1,
     "pair": null
    },
    "q11": {
     "a": "できない",
     "ok": 1,
     "pair": null
    },
    "q12": {
     "a": "すまいざごうよはお",
     "ok": 1,
     "pair": null
    },
    "q13": {
     "a": "父親です。",
     "ok": 1,
     "pair": "surgeon"
    },
    "q14": {
     "a": "同じ\n\n（司会者が正解を知らずランダムに開けた場合、残った扉の確率はどちらも1/2になるため、変えるメリットはありません。）",
     "ok": 1,
     "pair": null
    },
    "q15": {
     "a": "5",
     "ok": 1,
     "pair": null
    },
    "q16": {
     "a": "3",
     "ok": 1,
     "pair": null
    },
    "q17": {
     "a": "15文字、2つ",
     "ok": 1,
     "pair": "longest"
    },
    "q18": {
     "a": "4",
     "ok": 0,
     "pair": "mora"
    },
    "q19": {
     "a": "0.25, 0.3, 0.33, 1/3",
     "ok": 1,
     "pair": null
    },
    "q20": {
     "a": "しない",
     "ok": 1,
     "pair": "river"
    }
   }
  },
  "sakana/fugu-ultra": {
   "score": 18,
   "answered": 18,
   "first10": 9,
   "last10": 9,
   "inconsistent_pairs": 0,
   "sec": 387.8,
   "out_tokens": 14743,
   "graded": {
    "q01": {
     "a": "父親。",
     "ok": 1,
     "pair": "surgeon"
    },
    "q02": {
     "a": "10",
     "ok": 1,
     "pair": null
    },
    "q03": {
     "a": "しない。",
     "ok": 1,
     "pair": null
    },
    "q04": {
     "a": "",
     "ok": -1,
     "pair": "longest"
    },
    "q05": {
     "a": "3",
     "ok": 1,
     "pair": "mora"
    },
    "q06": {
     "a": "2",
     "ok": 1,
     "pair": null
    },
    "q07": {
     "a": "できない。",
     "ok": 1,
     "pair": "river"
    },
    "q08": {
     "a": "水曜日です。",
     "ok": 1,
     "pair": null
    },
    "q09": {
     "a": "o",
     "ok": 1,
     "pair": null
    },
    "q10": {
     "a": "火星です。",
     "ok": 1,
     "pair": null
    },
    "q11": {
     "a": "できない。",
     "ok": 1,
     "pair": null
    },
    "q12": {
     "a": "すまいざごうよはお",
     "ok": 1,
     "pair": null
    },
    "q13": {
     "a": "執刀医は息子の父親です。",
     "ok": 1,
     "pair": "surgeon"
    },
    "q14": {
     "a": "同じ",
     "ok": 1,
     "pair": null
    },
    "q15": {
     "a": "5",
     "ok": 1,
     "pair": null
    },
    "q16": {
     "a": "3",
     "ok": 1,
     "pair": null
    },
    "q17": {
     "a": "15文字で、該当する単語は2つです。",
     "ok": 1,
     "pair": "longest"
    },
    "q18": {
     "a": "3",
     "ok": 1,
     "pair": "mora"
    },
    "q19": {
     "a": "0.25, 0.3, 0.33, 1/3",
     "ok": 1,
     "pair": null
    },
    "q20": {
     "a": "",
     "ok": -1,
     "pair": "river"
    }
   }
  },
  "xiaomi/mimo-v2.5-pro": {
   "score": 13,
   "answered": 15,
   "first10": 8,
   "last10": 5,
   "inconsistent_pairs": 1,
   "sec": 214.4,
   "out_tokens": 2359,
   "graded": {
    "q01": {
     "a": "",
     "ok": -1,
     "pair": "surgeon"
    },
    "q02": {
     "a": "10",
     "ok": 1,
     "pair": null
    },
    "q03": {
     "a": "**しない**\n\n2026年は4で割り切れないため平年であり、2月は28日までです。",
     "ok": 1,
     "pair": null
    },
    "q04": {
     "a": "**synchronization**（15文字）",
     "ok": 1,
     "pair": "longest"
    },
    "q05": {
     "a": "3",
     "ok": 1,
     "pair": "mora"
    },
    "q06": {
     "a": "2",
     "ok": 1,
     "pair": null
    },
    "q07": {
     "a": "",
     "ok": -1,
     "pair": "river"
    },
    "q08": {
     "a": "**水曜日**です。（100 ÷ 7 = 14余り2 → 月曜 + 2日 = 水曜）",
     "ok": 1,
     "pair": null
    },
    "q09": {
     "a": "**o**",
     "ok": 1,
     "pair": null
    },
    "q10": {
     "a": "**火星（Mars）**です。",
     "ok": 1,
     "pair": null
    },
    "q11": {
     "a": "",
     "ok": -1,
     "pair": null
    },
    "q12": {
     "a": "す\nま\nせ\nう\nよ\nう\nは\nお",
     "ok": 0,
     "pair": null
    },
    "q13": {
     "a": "**執刀医は息子の父親です。**",
     "ok": 1,
     "pair": "surgeon"
    },
    "q14": {
     "a": "",
     "ok": -1,
     "pair": null
    },
    "q15": {
     "a": "**5**",
     "ok": 1,
     "pair": null
    },
    "q16": {
     "a": "3",
     "ok": 1,
     "pair": null
    },
    "q17": {
     "a": "最も文字数が多い単語は **15文字**で、該当するのは **2つ**（\"extraordinarily\" と \"synchronization\"）。",
     "ok": 1,
     "pair": "longest"
    },
    "q18": {
     "a": "4",
     "ok": 0,
     "pair": "mora"
    },
    "q19": {
     "a": "0.25、0.3、0.33、1/3",
     "ok": 1,
     "pair": null
    },
    "q20": {
     "a": "",
     "ok": -1,
     "pair": "river"
    }
   }
  },
  "nousresearch/hermes-4-405b": {
   "score": 15,
   "answered": 20,
   "first10": 7,
   "last10": 8,
   "inconsistent_pairs": 0,
   "sec": 155.5,
   "out_tokens": 5464,
   "graded": {
    "q01": {
     "a": "Okay, let's see. The problem is in Japanese, so I need to make sure I understand it correctly. The question is about a boy who was taken to the hospital after an accident. His mother, who was with him, died. Then the surgeon says, \"This child is my son.\" The question is asking who the surgeon is.\n\nHmm, the classic riddle. The key here is that the surgeon is the boy's parent. Since the mother died, the surgeon must be the other parent. But people often assume the surgeon is male, so the answer is that the surgeon is the boy's father. Wait, but maybe there's a twist. Oh, right! The surgeon could be the boy's mother, but in this case, the mother died. Wait, no, the mother was with the boy and died in the accident. So the surgeon can't be the mother. Therefore, the surgeon must be the father. But wait, the riddle usually plays on the assumption that the surgeon is male, but the answer is that the surgeon is the boy's mother. Wait, but in this scenario, the mother died. So maybe the answer is the father. Let me check again.\n\nThe setup: The boy is in an accident, taken to the hospital. The mother was with him and died. The surgeon says, \"This child is my son.\" So the surgeon is the parent. Since the mother is dead, the surgeon must be the father. But",
     "ok": 1,
     "pair": "surgeon"
    },
    "q02": {
     "a": "Okay, let's see. The problem is in Japanese, but I can handle that. The user is asking about a bat and a ball that together cost 120 yen. The bat is 100 yen more expensive than the ball. They want to know the price of the ball, just the number, and a short answer.\n\nFirst, I need to set up the equations. Let me denote the price of the ball as B. Then the bat would be B + 100 yen. Together, their total is 120 yen. So the equation would be B (ball) + (B + 100) (bat) = 120. Combining those terms, that's 2B + 100 = 120. Subtract 100 from both sides: 2B = 20. Then divide by 2: B = 10. So the ball is 10 yen. Let me double-check. If the ball is 10, the bat is 110. 10 + 110 is 120, which matches. Yep, that's correct. The answer should be 10.\n10",
     "ok": 0,
     "pair": null
    },
    "q03": {
     "a": "Okay, the user is asking if February 29, 2026 exists. Let me think. First, I need to check if 2026 is a leap year. Leap years are divisible by 4, but if they're a century year, they must be divisible by 400. 2026 divided by 4 is 506.5, so it's not a leap year. Therefore, February 2026 will only have 28 days. So February 29, 2026 doesn't exist. The answer should be しない.\nしない",
     "ok": 1,
     "pair": null
    },
    "q04": {
     "a": "Okay, let's see. The user wants the longest word in the sentence \"The extraordinarily complicated synchronization failed.\" First, I need to break down the sentence into individual words. The words are: The, extraordinarily, complicated, synchronization, failed.\n\nNow, I need to check the length of each word. Let's count the letters. \"The\" has 3 letters. \"Extraordinarily\" is a long one; let me count: E-X-T-R-A-O-R-D-I-N-A-R-I-L-Y. That's 16 letters. \"Complicated\" is C-O-M-P-L-I-C-A-T-E-D, which is 11 letters. \"Synchronization\" is S-Y-N-C-H-R-O-N-I-Z-A-T-I-O-N, that's 15 letters. \"Failed\" is 6 letters.\n\nComparing the lengths: 3, 16, 11, 15, 6. The longest is \"extraordinarily\" with 16 letters. Wait, but \"synchronization\" is 15, so \"extraordinarily\" is longer. So the answer should be \"extraordinarily\". Let me double-check to make sure I didn't miscount. Yes, \"extraordinarily\" has 16 letters. Yep, that's the longest one.\nextraordinarily",
     "ok": 1,
     "pair": "longest"
    },
    "q05": {
     "a": "Okay, the user is asking how many morae are in the word \"ちょうちょ\". Let me start by recalling what a mora is. In Japanese phonology, a mora is a unit of sound that determines the rhythm and timing. Each kana character typically represents one mora, except for some cases like long vowels or double consonants.\n\nNow, looking at \"ちょうちょ\". Let's break it down into individual kana. The word is written in hiragana as ちょうちょ. Let's split it:\n\n- ち (chi) is one mora.\n- ょう (you) is a combination. The small ょ (yo) is a subscript, so together with う (u), it forms one mora. Wait, no. Wait, actually, when a small kana like ゃ, ゅ, ょ is used, it combines with the preceding kana to form a single mora. So ちょう is ち (chi) + ょう (you). But wait, ちょう is actually ち (chi) + ょ (yo) + う (u)? No, that's not right. Let me think again. The correct breakdown is ちょう is ち (chi) + ょ (yo) + う (u)? No, that's not correct. Wait, the correct way is that ちょう is ち (chi)",
     "ok": 0,
     "pair": "mora"
    },
    "q06": {
     "a": "Okay, let's see. The problem is in Japanese, so I need to make sure I understand it correctly. The user is asking about Sally's sisters. The question states that Sally has three brothers, and each of these brothers has three sisters. The task is to find out how many sisters Sally has, and the answer should be a number only, with a short explanation.\n\nFirst, let me break down the information. Sally has three brothers. Each brother has three sisters. Now, the key here is to figure out if Sally is counted among those sisters. Since Sally is a sister to her brothers, each brother's three sisters would include Sally and her other sisters. \n\nSo, if each brother has three sisters, and Sally is one of them, then the remaining two sisters must be Sally's own sisters. Therefore, Sally has two sisters. Let me verify this again. If there are three brothers, each with three sisters, that means each brother shares the same set of sisters. Since Sally is a sister to each brother, she is included in each count. So, the three sisters per brother are Sally plus two others. Hence, Sally has two sisters. That makes sense. I don't think there's any other way to interpret this unless there's a trick, but the problem seems straightforward. The answer should be 2.\n2",
     "ok": 1,
     "pair": null
    },
    "q07": {
     "a": "Okay, let's see. The user is asking about the classic river crossing puzzle with the farmer, wolf, goat, and cabbage. The boat can carry the farmer plus one item. The constraints are that the wolf will eat the goat, and the goat will eat the cabbage if left alone. Also, the boat breaks after one round trip. The question is whether all can be transported.\n\nFirst, I need to recall the standard solution. Normally, the farmer takes the goat first, then goes back. Then takes the wolf, leaves it, brings the goat back. Then takes the cabbage, leaves it, and finally returns for the goat. But wait, the boat breaks after one round trip. A round trip is going and coming back. So if the boat breaks after one round trip, that means after the farmer goes to the other side and comes back once, the boat is unusable. That changes things.\n\nIn the standard problem, the boat can be used multiple times. But here, after one round trip (over and back), the boat is destroyed. So the farmer can only make one trip to the other side and come back once. Let me think. If the boat breaks after one round trip, the farmer can't make multiple trips. So he can go to the other side once, come back once, and then the boat is gone. That would mean he can only transport one item over and then come back, but then the boat is broken. So he can't",
     "ok": 1,
     "pair": "river"
    },
    "q08": {
     "a": "Okay, let's see. The user is asking, \"If today is Monday, what day of the week will it be 100 days later? Please answer briefly.\" Hmm, I need to figure out how to calculate the day of the week after 100 days from a Monday.\n\nFirst, I remember that the days of the week cycle every 7 days. So, to find the day after a certain number of days, I can use modulo 7 arithmetic. That is, divide the number of days by 7 and find the remainder. The remainder will tell me how many days ahead from the starting day.\n\nSo, 100 divided by 7. Let me calculate that. 7 times 14 is 98, which is 14 weeks. 100 minus 98 is 2. So the remainder is 2. That means 100 days later is 2 days after Monday. \n\nStarting from Monday, adding 1 day is Tuesday, and adding another day (so 2 days total) would be Wednesday. Therefore, 100 days after Monday should be Wednesday. \n\nWait, let me double-check. If today is Monday, then 7 days later is Monday again. 14 days later is also Monday. So 98 days later (14 weeks) is still Monday. Then adding 2 more days: 99 days is Tuesday, 100 days is Wednesday. Yep, that seems right. \n\nI think that's correct. The answer should be",
     "ok": 0,
     "pair": null
    },
    "q09": {
     "a": "Okay, the user is asking for the 7th character in the word \"kaleidoscope\". Let me start by writing out the word and numbering each letter to make sure I get the position right.\n\nK-A-L-E-I-D-O-S-C-O-P-E. Let's count them one by one. \n\n1. K\n2. A\n3. L\n4. E\n5. I\n6. D\n7. O\n\nSo the 7th letter is O. Wait, let me double-check to avoid any mistakes. Sometimes when counting, it's easy to miscount. Let me list them again with numbers:\n\n1. K\n2. A\n3. L\n4. E\n5. I\n6. D\n7. O\n8. S\n9. C\n10. O\n11. P\n12. E\n\nYes, the 7th position is definitely O. The user wants a short answer, so I should just provide the letter without any extra explanation. They might be in a hurry or need a quick response. Also, since they specified the answer should be in their language (which is Japanese here), but the answer itself is a single letter, so no translation is needed. Just confirm the 7th character is O.\nO",
     "ok": 1,
     "pair": null
    },
    "q10": {
     "a": "Okay, the user is asking for the second smallest planet in the solar system. Let me start by recalling the order of the planets from the Sun: Mercury, Venus, Earth, Mars, Jupiter, Saturn, Uranus, Neptune. Wait, but size-wise, Mercury is the smallest. Then the next one would be Mars. But wait, isn't Mars bigger than Mercury? Let me double-check. Mercury's diameter is about 4,880 km, and Mars is about 6,779 km. So Mercury is the smallest, then Mars. But wait, the user is asking for the second smallest. So after Mercury, the next smallest is Mars. But wait, some sources might consider Pluto, but Pluto is no longer classified as a planet. So the answer should be Mars. Let me confirm. Yes, Mercury is the smallest, then Mars. So the answer is Mars.\n火星（Mars）です。",
     "ok": 1,
     "pair": null
    },
    "q11": {
     "a": "Okay, let's see. The user is asking if it's possible to measure exactly 3 liters using only 2L and 4L containers. Hmm, I need to figure this out step by step.\n\nFirst, I remember that these kinds of problems usually involve filling, emptying, and transferring between containers. Let me start by considering the possible operations. The containers are 2L and 4L. The goal is to get 3L in one of them.\n\nLet me try to outline the steps. Maybe start by filling the 4L container to the top. Then, pour from the 4L into the 2L until the 2L is full. That would leave 2L in the 4L container. Then, empty the 2L container. Now, pour the remaining 2L from the 4L into the 2L. Now the 4L is empty, and the 2L has 2L. Then, fill the 4L again. Pour from the 4L into the 2L, which already has 2L, so you can only add 0L. Wait, that doesn't help. Maybe another approach.\n\nAlternatively, fill the 2L container and pour it into the 4L. Do that twice. That would give 4L in the 4L container. But that's not helpful. Wait, maybe after filling the 4L, pour into the",
     "ok": 1,
     "pair": null
    },
    "q12": {
     "a": "Okay, the user wants to reverse the phrase \"おはようございます\" one character at a time. Let me start by breaking down the original phrase. The phrase is \"おはようございます\", which is a common Japanese greeting meaning \"Good morning.\" \n\nFirst, I need to list each character in order. Let's see: お、は、よ、う、ご、ざ、い、ま、す. Wait, let me count again. \"おはようございます\" has 9 characters. Let me confirm: お (o), は (ha), よ (yo), う (u), ご (go), ざ (za), い (i), ま (ma), す (su). Yes, that's 9 characters.\n\nNow, the task is to reverse the order of these characters. So starting from the last character and moving backwards. The last character is す, then ま, い, ざ, ご, う, よ, は, お. Let me write that out: す、ま、い、ざ、ご、う、よ、は、お. \n\nPutting them together without spaces: すまいざごうよはお. Wait, let me check again. Original order: お、は、よ、う、ご、ざ、い、ま、す. Reversed should be す、ま、い、ざ、ご、う、よ、は、お. So combining them:",
     "ok": 1,
     "pair": null
    },
    "q13": {
     "a": "Okay, let's try to figure this out. So, the problem is: A mother and her son are in a car. The mother dies in an accident, and the son is taken to the operating room. The surgeon says, \"I can't operate on this boy, he's my son.\" The question is, what's the relationship between the surgeon and the son?\n\nHmm, at first glance, this seems like a classic riddle. The trick here is probably challenging the assumption that the surgeon is a man. Because if the surgeon is the boy's parent, but the mother is already dead, then the surgeon must be the other parent. So, the surgeon is the boy's father. But wait, the riddle is often used to point out gender stereotypes, where people might assume the surgeon is male, but actually, the surgeon could be the boy's mother. Wait, but in this case, the mother died in the accident. So the mother can't be the surgeon. Therefore, the surgeon must be the father. But maybe the riddle is trying to highlight that the surgeon is the other parent, regardless of gender. Wait, but the original problem states that the mother died, so the surgeon can't be the mother. Therefore, the surgeon must be the father. But maybe the answer is that the surgeon is the boy's father. But I need to make sure. Let me think again.\n\nThe setup is that the mother and son are in a car accident",
     "ok": 1,
     "pair": "surgeon"
    },
    "q14": {
     "a": "Okay, let's see. The user is asking about the Monty Hall problem. There are three doors, one is the winning door. The user picks door A. The host, who doesn't know which door is correct, randomly opens door B and reveals it's a loser. Now the question is whether the user should switch to door C or stick with A.\n\nWait, in the classic Monty Hall problem, the host knows where the prize is and always opens a door that's a loser. But here, the host doesn't know and opens door B randomly. That changes things. In the original problem, switching gives a 2/3 chance of winning, but if the host doesn't know and just happens to open a losing door, the probabilities might be different.\n\nSo, if the host opens door B randomly and it's a loser, then door C could be either the winner or a loser. But since the host didn't know, the initial probability of door A being correct is 1/3, and doors B and C each have 1/3. When the host opens B and it's a loser, the probability that C is the winner would be 1/2, same as A. Because the host's action doesn't give any additional information. So in this case, switching doesn't improve the odds. The answer should be to stay with A, or it doesn't matter. But the user wants a short answer: switch or stay. Since",
     "ok": 1,
     "pair": null
    },
    "q15": {
     "a": "Okay, let's see. The problem is: A father is 40 years old, and his child is 10. In how many years will the father be exactly three times as old as the child? I need to find the number of years, just the number.\n\nFirst, let me make sure I understand the problem correctly. The father is currently 40, the child is 10. We need to find a future year when the father's age is three times the child's age. Let's denote the number of years from now as x.\n\nSo, in x years, the father's age will be 40 + x, and the child's age will be 10 + x. According to the problem, at that time, the father's age should be three times the child's age. So the equation would be:\n\n40 + x = 3 * (10 + x)\n\nNow, let's solve this equation step by step. Expanding the right side: 3*(10 + x) = 30 + 3x. So the equation becomes:\n\n40 + x = 30 + 3x\n\nNext, subtract x from both sides to get:\n\n40 = 30 + 2x\n\nThen, subtract 30 from both sides:\n\n10 = 2x\n\nDivide both sides by 2:\n\nx = 5\n\nWait, so in 5 years, the father will be 45 and the child will be 15.",
     "ok": 0,
     "pair": null
    },
    "q16": {
     "a": "Okay, let's see. The user is asking how many times the character \"は\" appears in the sentence \"わたしはきょうははれです\". They want the answer in numbers only and a short response.\n\nFirst, I need to look at the sentence carefully. Let me break it down character by character. The sentence is: わたしはきょうははれです.\n\nStarting from the beginning:\n\n1. わ (wa)\n2. た (ta)\n3. し (shi)\n4. は (ha) – that's the first \"は\"\n5. き (ki)\n6. ょ (yo)\n7. う (u)\n8. は (ha) – second \"は\"\n9. は (ha) – third \"は\"\n10. れ (re)\n11. で (de)\n12. す (su)\n\nSo, counting each \"は\", I see three instances. Let me double-check to make sure I didn't miss any or count extra. The sentence is \"わたしはきょうははれです\". Breaking it into parts: \"わたしは\" (first は), \"きょうは\" (second は), and \"はれです\" (third は). Yep, three times. The user wants just the number, so the answer is 3.\n3",
     "ok": 1,
     "pair": null
    },
    "q17": {
     "a": "Okay, let's tackle this question. The user provided the sentence: \"The extraordinarily complicated synchronization failed.\" and is asking for the longest word in terms of character count, how many characters it has, and how many words in the sentence have that length.\n\nFirst, I need to break down the sentence into individual words. Let's list them out:\n\n1. The\n2. extraordinarily\n3. complicated\n4. synchronization\n5. failed\n\nNow, I'll count the letters in each word. Let's go one by one.\n\n1. \"The\" has 3 letters.\n2. \"extraordinarily\" – let's count. E-x-t-r-a-o-r-d-i-n-a-r-i-l-y. That's 16 letters.\n3. \"complicated\" – C-o-m-p-l-i-c-a-t-e-d. That's 11 letters.\n4. \"synchronization\" – S-y-n-c-h-r-o-n-i-z-a-t-i-o-n. Let's count: S(1), y(2), n(3), c(4), h(5), r(6), o(7), n(8), i(9), z(10), a(11), t(12), i(13), o(14), n(15). So 15 letters.\n5. \"failed\" – F-a-i-l-e-d. 6 letters.\n\nSo the longest word is \"extraordinarily\" with 16",
     "ok": 1,
     "pair": "longest"
    },
    "q18": {
     "a": "Okay, the user is asking how many morae (moras) the word \"ちょうちょ\" is divided into. Let me start by recalling what a mora is. In Japanese phonology, a mora is a unit of sound that determines the rhythm and timing. Each kana character typically represents one mora, but there are exceptions like long vowels, double consonants, and nasal sounds.\n\nNow, looking at \"ちょうちょ\". Let's break it down character by character. The word is written in hiragana as ちょうちょ. Let's split it into individual kana: ち (chi), よ (yo), う (u), ち (chi), よ (yo). Wait, but the second character is actually \"ょう\", which is a combination of よ and う. In Japanese, when a small よ is followed by う, it forms a long vowel sound, which counts as two morae. So \"ちょう\" would be ち (1) + よう (2) = 3 morae. Then the next part is \"ちょ\", which is ち (chi) + よ (yo), but again, the よ here is small, so it's a single mora. Wait, no, \"ちょ\" is a single kana character, which is a combination of ち and よ, but in terms of morae, each syllable is",
     "ok": 0,
     "pair": "mora"
    },
    "q19": {
     "a": "Okay, let's see. The user wants me to sort the numbers 0.3, 0.25, 1/3, and 0.33 in ascending order. First, I need to make sure all the numbers are in the same format to compare them easily. The numbers given are a mix of decimals and fractions.\n\nStarting with 0.25. That's a decimal, so I can leave that as is. Then 0.3 is also a decimal. Next is 1/3, which is a fraction. I should convert that to a decimal to compare it with the others. 1 divided by 3 is approximately 0.3333... repeating. Then there's 0.33, which is already a decimal.\n\nSo converting all to decimals: 0.25, 0.3, 0.3333..., and 0.33. Now, comparing them. The smallest is 0.25, followed by 0.3. Then comes 0.33, and the largest is 1/3 (0.3333...). Wait, but 0.33 is less than 0.3333... So the order should be 0.25, 0.3, 0.33, 1/3. Let me double-check. 0.25 is 1/4, which is 0.25. 0.3 is 3/10, which",
     "ok": 1,
     "pair": null
    },
    "q20": {
     "a": "Okay, the user is asking if there's a way to get a wolf, goat, and cabbage across a river with a boat that can only make one round trip. Let me think.\n\nFirst, the classic problem usually allows multiple trips, but here it's limited to one. The main issue is that the wolf can't be left with the goat, and the goat can't be left with the cabbage. Normally, you take the goat first, then come back for the others. But with only one trip, you can't go back. So if you take the goat first, you can't return to get the others. If you take the wolf or cabbage first, the other two would be left alone, which is bad. So maybe it's impossible. But wait, maybe there's a trick. Like, if the boat can carry more than one? But the problem doesn't specify. Assuming the boat can only carry one item at a time, then with one trip, you can only take one. So you can't get all three across. Therefore, the answer is no, it doesn't exist.\nしない",
     "ok": 1,
     "pair": "river"
    }
   }
  },
  "sakana/fugu-ultra-v2": {
   "score": 20,
   "answered": 20,
   "first10": 10,
   "last10": 10,
   "inconsistent_pairs": 0,
   "sec": 54.3,
   "out_tokens": 3029,
   "graded": {
    "q01": {
     "a": "少年の父親",
     "ok": 1,
     "pair": "surgeon"
    },
    "q02": {
     "a": "10",
     "ok": 1,
     "pair": null
    },
    "q03": {
     "a": "しない",
     "ok": 1,
     "pair": null
    },
    "q04": {
     "a": "extraordinarily と synchronization(同率15文字で1つに絞れない)",
     "ok": 1,
     "pair": "longest"
    },
    "q05": {
     "a": "3",
     "ok": 1,
     "pair": "mora"
    },
    "q06": {
     "a": "2",
     "ok": 1,
     "pair": null
    },
    "q07": {
     "a": "できない",
     "ok": 1,
     "pair": "river"
    },
    "q08": {
     "a": "水曜日",
     "ok": 1,
     "pair": null
    },
    "q09": {
     "a": "o",
     "ok": 1,
     "pair": null
    },
    "q10": {
     "a": "火星",
     "ok": 1,
     "pair": null
    },
    "q11": {
     "a": "できない",
     "ok": 1,
     "pair": null
    },
    "q12": {
     "a": "すまいざごうよはお",
     "ok": 1,
     "pair": null
    },
    "q13": {
     "a": "執刀医は息子の父親",
     "ok": 1,
     "pair": "surgeon"
    },
    "q14": {
     "a": "同じ",
     "ok": 1,
     "pair": null
    },
    "q15": {
     "a": "5",
     "ok": 1,
     "pair": null
    },
    "q16": {
     "a": "3",
     "ok": 1,
     "pair": null
    },
    "q17": {
     "a": "15文字、2つ",
     "ok": 1,
     "pair": "longest"
    },
    "q18": {
     "a": "3",
     "ok": 1,
     "pair": "mora"
    },
    "q19": {
     "a": "0.25, 0.3, 0.33, 1/3",
     "ok": 1,
     "pair": null
    },
    "q20": {
     "a": "しない",
     "ok": 1,
     "pair": "river"
    }
   }
  }
 },
 "timeout_502": [
  "aion-labs/aion-2.0",
  "aion-labs/aion-3.0",
  "anthropic/claude-opus-4.5",
  "anthropic/claude-opus-4.6",
  "anthropic/claude-sonnet-4.6",
  "bytedance-seed/seed-2-1-turbo",
  "bytedance-seed/seed-2.0-mini",
  "cerebras/gemma-4-31b",
  "cerebras/gpt-oss-120b",
  "cerebras/qwen-3-32b",
  "claude-haiku-4-5-20251001",
  "claude-opus-4-5",
  "claude-opus-4-6",
  "claude-sonnet-4-6",
  "claude-sonnet-5",
  "deepseek/deepseek-r1",
  "deepseek/deepseek-r1-0528",
  "deepseek/deepseek-r1-distill-llama-70b",
  "deepseek/deepseek-v3.2",
  "deepseek/deepseek-v3.2-exp",
  "gemini-2.0-flash",
  "gemma2-9b-it",
  "google/gemma-4-26b-a4b-it",
  "inclusionai/ling-2.6-flash",
  "inclusionai/ring-2.6-1t",
  "kimi-k2-0711",
  "kwaipilot/kat-coder-pro-v2",
  "kwaipilot/kat-coder-pro-v2.5",
  "llama-3.3-70b-specdec",
  "llama-4-maverick-17b-128e",
  "llama-4-scout-17b-16e",
  "meituan/longcat-2.0",
  "meta/muse-spark-1.1",
  "meta/muse-spark-1.2",
  "meta/muse-spark-1.2-contributor",
  "meta/muse-spark-1.3",
  "meta/muse-spark-1.3-contributor",
  "minimax/minimax-m1",
  "minimax/minimax-m2",
  "minimax/minimax-m2.5",
  "mistral-saba-24b",
  "mistralai/mistral-small-3.1-24b-instruct",
  "mistralai/voxtral-small-24b-2507",
  "morph/morph-v3-fast",
  "morph/morph-v3-large",
  "nousresearch/hermes-4-405b",
  "nvidia/Llama-3.3-Nemotron-Super-49B-v1.5",
  "nvidia/NVIDIA-Nemotron-3-Super-120B-A12B",
  "nvidia/nemotron-3-super-120b-a12b",
  "nvidia/nemotron-3.5-lightning",
  "openai/gpt-4-turbo-preview",
  "openai/gpt-5-nano",
  "openai/gpt-5-pro",
  "openai/gpt-5.1-codex-mini",
  "openai/gpt-5.2-chat",
  "openai/gpt-5.2-pro",
  "openai/gpt-5.4-pro",
  "openai/gpt-5.5-pro",
  "openai/o1-pro",
  "openai/o3-mini-high",
  "openai/o3-pro",
  "openai/o4-mini-high",
  "perplexity/sonar-reasoning-pro",
  "qwen-qwq-32b",
  "qwen/qwen-2.5-coder-32b-instruct",
  "qwen/qwen3-235b-a22b",
  "qwen/qwen3-235b-a22b-thinking-2507",
  "qwen/qwen3-32b",
  "qwen/qwen3-8b",
  "qwen/qwen3-max-thinking",
  "qwen/qwen3-vl-235b-a22b-thinking",
  "qwen/qwen3.5-122b-a10b",
  "qwen/qwen3.5-27b",
  "qwen/qwen3.5-35b-a3b",
  "qwen/qwen3.5-397b-a17b",
  "qwen/qwen3.5-9b",
  "qwen/qwen3.5-plus-02-15",
  "qwen/qwen3.5-plus-20260420",
  "qwen/qwen3.6-27b",
  "qwen/qwen3.6-max-preview",
  "qwen/qwen3.6-plus",
  "qwen/qwen3.7-flash",
  "qwen/qwen3.7-max",
  "qwen/qwen3.7-plus",
  "qwen/qwen3.8-flash",
  "qwen/qwen3.8-max",
  "qwen/qwen3.8-max-0902",
  "rekaai/reka-flash-3",
  "sakana/fugu-ultra",
  "stepfun/step-3.5-flash",
  "tencent/hunyuan-a13b-instruct",
  "tencent/hy4-preview",
  "upstage/solar-pro4",
  "x-ai/grok-4.20-multi-agent",
  "x-ai/grok-4.5",
  "x-ai/grok-4.6",
  "xiaomi/mimo-v2.5",
  "xiaomi/mimo-v2.5-pro",
  "z-ai/glm-4.5",
  "z-ai/glm-4.5-air",
  "z-ai/glm-4.5v",
  "z-ai/glm-4.6",
  "z-ai/glm-4.6v",
  "z-ai/glm-4.7",
  "z-ai/glm-4.7-flash",
  "z-ai/glm-5-turbo",
  "z-ai/glm-5v-turbo",
  "~x-ai/grok-latest"
 ],
 "single_vs_batch_top23": {
  "anthropic/claude-fable-5": {
   "single_score": 20,
   "single_sec": 155,
   "single_out_tokens": 2063,
   "batch_score": 20,
   "batch_sec": 20.5,
   "batch_out_tokens": 870,
   "single_graded": {
    "q01": {
     "a": "外科医は少年の**父親**です。\n\n（このなぞなぞの元の版では「父親が死亡し、外科医は母親」という展開ですが、この設定では母親が亡くなっているので、外科医は父親となります。）",
     "ok": 1
    },
    "q02": {
     "a": "10円",
     "ok": 1
    },
    "q03": {
     "a": "しない。2026年はうるう年ではないため、2月は28日までです。",
     "ok": 1
    },
    "q04": {
     "a": "「extraordinarily」（15文字）です。ちなみに「synchronization」も同じ15文字で並んでいます。",
     "ok": 1
    },
    "q05": {
     "a": "3",
     "ok": 1
    },
    "q06": {
     "a": "2",
     "ok": 1
    },
    "q07": {
     "a": "**できない。**\n\n通常この問題は7回の川渡り(最低でも複数往復)が必要ですが、ボートが1往復しか持たないなら、運べるのは1つだけ。残りをすべて運ぶ手段がなくなります。",
     "ok": 1
    },
    "q08": {
     "a": "水曜日です。（100 ÷ 7 = 14 余り 2 なので、月曜の2日後）",
     "ok": 1
    },
    "q09": {
     "a": "**o** です。（k-a-l-e-i-d-**o**）",
     "ok": 1
    },
    "q10": {
     "a": "**火星**です。\n\n（最小は水星、2番目に小さいのが火星です。）",
     "ok": 1
    },
    "q11": {
     "a": "**できない。**\n\n2Lと4Lの組み合わせでは2の倍数(2L、4L、6L…)しか測れないため、3Lは不可能です。",
     "ok": 1
    },
    "q12": {
     "a": "すまいざごうよはお",
     "ok": 1
    },
    "q13": {
     "a": "執刀医は息子の**父親**です。\n\n（このなぞなぞは通常「執刀医は母親」という答えで性別の思い込みを問うものですが、この場合は母親が亡くなっているため、父親が答えになります。）",
     "ok": 1
    },
    "q14": {
     "a": "**同じ**です。\n\n司会者が正解を知らずにランダムに開けてたまたまハズレだった場合、AもCも当たる確率は **1/2ずつ** になります(通常のモンティ・ホール問題と違い、変えても有利になりません)。",
     "ok": 1
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "最長は **15文字** で、該当する単語は **2つ**(extraordinarily, synchronization)です。",
     "ok": 1
    },
    "q18": {
     "a": "**3**\n\n（ちょ・う・ちょ）",
     "ok": 1
    },
    "q19": {
     "a": "0.25 < 0.3 < 0.33 < 1/3\n\n（1/3 = 0.333… なので 0.33 より大きい）",
     "ok": 1
    },
    "q20": {
     "a": "**存在しない。**\n\n古典的な川渡り問題では最低7回の渡河(3.5往復)が必要です。1往復(2回の渡河)では、ボートは元の岸に戻ってしまう上、一度に運べるのは1つだけなので、3つ全てを対岸に無事に運ぶことは不可能です。",
     "ok": 1
    }
   }
  },
  "anthropic/claude-fable-5.1": {
   "single_score": 18,
   "single_sec": 169,
   "single_out_tokens": 1783,
   "batch_score": 20,
   "batch_sec": 23.5,
   "batch_out_tokens": 869,
   "single_graded": {
    "q01": {
     "a": "外科医は少年の**父親**です。（母親は亡くなっているので、「息子だ」と言えるのは父親です。）",
     "ok": 1
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない。2026年は閏年ではないため、2月は28日までです。",
     "ok": 1
    },
    "q04": {
     "a": "「extraordinarily」（15文字）です。ただし「synchronization」も同じ15文字で同率最長です。",
     "ok": 1
    },
    "q05": {
     "a": "3",
     "ok": 1
    },
    "q06": {
     "a": "2",
     "ok": 1
    },
    "q07": {
     "a": "**できない**\n\n全部運ぶには最低7回の渡河(3往復半)が必要ですが、ボートは1往復しか使えないため、途中でボートが壊れて残りを運べません。",
     "ok": 1
    },
    "q08": {
     "a": "100 ÷ 7 = 14 余り 2 なので、月曜から2日進んで **水曜日** です。",
     "ok": 1
    },
    "q09": {
     "a": "k-a-l-e-i-d-o-s-c-o-p-e → 7文字目は「**o**」です。",
     "ok": 0
    },
    "q10": {
     "a": "火星（マーズ）です。最小は水星で、その次に小さいのが火星（直径約6,779km）です。",
     "ok": 1
    },
    "q11": {
     "a": "**できない**\n\n2Lと4Lはどちらも2の倍数なので、これらの組み合わせでは偶数の量（0, 2, 4L）しか作れず、奇数の3Lは測れません。",
     "ok": 1
    },
    "q12": {
     "a": "すまいざごうよはお",
     "ok": 1
    },
    "q13": {
     "a": "執刀医は息子の**父親**です。（母親が亡くなった後も、もう一人の親である父が医師として手術に立ち会った、という筋書きです。）",
     "ok": 1
    },
    "q14": {
     "a": "**どちらでも同じ**（1/2ずつ）。\n\n司会者が正解を知らずランダムに開けてハズレだった場合、AもCも当たりの確率は1/2になり、変える利点はありません。（司会者が正解を知っていて意図的にハズレを開ける通常のモンティ・ホール問題なら、変えると2/3で有利です。）",
     "ok": 1
    },
    "q15": {
     "a": "40+x = 3(10+x) → 40+x = 30+3x → x = 5\n\n**5**",
     "ok": 0
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "最長は15文字で、該当する単語は2つ（extraordinarily、synchronization）です。",
     "ok": 1
    },
    "q18": {
     "a": "3",
     "ok": 1
    },
    "q19": {
     "a": "0.25 < 0.3 < 0.33 < 1/3（≈0.3333…）",
     "ok": 1
    },
    "q20": {
     "a": "**存在しない。**\n\nボートには農夫のほかに1つしか載せられないため、1往復では1つしか運べません。3つ全てを安全に運ぶには最低7回の渡河（3往復半）が必要です。",
     "ok": 1
    }
   }
  },
  "anthropic/claude-opus-5": {
   "single_score": 20,
   "single_sec": 148,
   "single_out_tokens": 2837,
   "batch_score": 20,
   "batch_sec": 19.3,
   "batch_out_tokens": 881,
   "single_graded": {
    "q01": {
     "a": "少年の**父親**です。（母親が亡くなっているので、もう一人の親＝父親。同性カップルなら「もう一人の母親」の可能性も）",
     "ok": 1
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない。\n\n2026年はうるう年ではないため、2月は28日までです。（直近のうるう年は2024年と2028年）",
     "ok": 1
    },
    "q04": {
     "a": "どちらも15文字で同着ですが、1つ挙げるなら **extraordinarily**",
     "ok": 1
    },
    "q05": {
     "a": "3",
     "ok": 1
    },
    "q06": {
     "a": "2",
     "ok": 1
    },
    "q07": {
     "a": "**できない。**\n\n1往復（渡って戻る）だけでは、運べるのは1つだけ。仮に多めに数えても最大2つで、残り1つは向こう岸",
     "ok": 1
    },
    "q08": {
     "a": "水曜日です。（100 ÷ 7 = 14 余り 2 → 月曜＋2日）",
     "ok": 1
    },
    "q09": {
     "a": "**o**",
     "ok": 1
    },
    "q10": {
     "a": "**火星**です。（最小は水星）",
     "ok": 1
    },
    "q11": {
     "a": "**できない。**\n\n2Lと4Lはどちらも偶数(最大公約数2)なので、作れる量は必ず2の倍数になり、奇数の3Lは測れません。",
     "ok": 1
    },
    "q12": {
     "a": "すまいざごうよはお",
     "ok": 1
    },
    "q13": {
     "a": "執刀医は息子の**父親**です。\n\n（元の有名ななぞなぞは「父親が死亡し、執刀医は母親」という形で性別の思い込みを突くもの。この版では母が亡くなっているので、単純に父親が答えになります。もちろん同性カップルの「もう一人の母」という可能性もあります。）",
     "ok": 1
    },
    "q14": {
     "a": "**同じ**(どちらでも1/2）。\n\n司会者が正解を知らずランダムに開けた場合、ハズレが出た時点でAとCは各1/2。通常のモンティ・ホール（司会者が知っている）と違い、変える利点はありません。",
     "ok": 1
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "**15文字**で、該当する単語は**2つ**（extraordinarily と synchronization）です。",
     "ok": 1
    },
    "q18": {
     "a": "3",
     "ok": 1
    },
    "q19": {
     "a": "0.25 < 0.3 < 0.33 < 1/3\n\n（1/3 = 0.333… なので 0.33 より大きい）",
     "ok": 1
    },
    "q20": {
     "a": "**存在しない。**\n\n理由：ボートには農夫＋1つしか乗せられないため、全員を運ぶには最低7回の渡河（往復を繰り返す）が必要。1往復では出発地点に戻ってしまい、しかも残した組み合わせでオオカミがヤギを、ヤギがキャベツを食べてしまう。",
     "ok": 1
    }
   }
  },
  "anthropic/claude-opus-5-fast": {
   "single_score": 20,
   "single_sec": 99,
   "single_out_tokens": 2803,
   "batch_score": 20,
   "batch_sec": 12.1,
   "batch_out_tokens": 967,
   "single_graded": {
    "q01": {
     "a": "父親です。\n\nこの問題は「外科医＝母親」という有名ななぞなぞをひねったもので、母親が亡くなっている以上、答えは父親（あるいは同性カップルのもう一人の親）になります。",
     "ok": 1
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない（2026年は閏年ではないため、2月は28日まで）。",
     "ok": 1
    },
    "q04": {
     "a": "**extraordinarily**（15文字）\n※ synchronization も同じ15文",
     "ok": 1
    },
    "q05": {
     "a": "3",
     "ok": 1
    },
    "q06": {
     "a": "2",
     "ok": 1
    },
    "q07": {
     "a": "**できない。**\n\n1往復（向こう岸へ1回＋戻り1回）で壊れるなら、荷物を1つ運んで戻った時点で終わり。残り2つと農夫が元の岸に取り残されます。本来この問題は7回の渡河が必要です。",
     "ok": 1
    },
    "q08": {
     "a": "**水曜日**です。\n\n100 ÷ 7 = 14 あまり 2 なので、月曜から2日進んで水曜日になります。",
     "ok": 1
    },
    "q09": {
     "a": "**o**",
     "ok": 1
    },
    "q10": {
     "a": "火星です。（最小は水星）",
     "ok": 1
    },
    "q11": {
     "a": "**できない**\n\n2Lと4Lはどちらも2の倍数なので、量れる量は必ず偶数(0・2・4・6L…)になり、奇数の3Lは作れません。",
     "ok": 1
    },
    "q12": {
     "a": "すまいざごうよはお",
     "ok": 1
    },
    "q13": {
     "a": "執刀医は**息子の父親**です。\n\n（有名な「父が亡くなり、執刀医が母親だった」という引っかけ問題の逆パターンで、この場合は素直に父親と考えるのが自然です。もちろん、同性カップルのもう一人の母親、という可能性もあります）",
     "ok": 1
    },
    "q14": {
     "a": "**同じ**(どちらも1/2)。\n\n司会者が正解を知らずランダムに開けた場合、たまたまハズレが出た時点でAとCは対等になり、通常のモンティ・ホール問題(2/3で変える方が得)とは異なります。",
     "ok": 1
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "**15文字**で、該当は**2語**（extraordinarily / synchronization）。",
     "ok": 1
    },
    "q18": {
     "a": "**3**（ちょ／う／ちょ）",
     "ok": 1
    },
    "q19": {
     "a": "0.25 < 0.3 < 0.33 < 1/3\n\n(1/3 = 0.333… なので 0.33 より大きい)",
     "ok": 1
    },
    "q20": {
     "a": "**存在しない。**\n\n1往復（向こう岸へ1回＋戻り1回）では、運べるのは最大1つだけ。残されたオオカミ・ヤギ・キャベツのどれかの組み合わせで必ず食べられてしまう（オオカミとヤギ、またはヤギとキャベツが残る）ため、全員無事に渡すことは不可能です。最低でも「3往復＋片道1回（計7回の渡河）」が必要です。",
     "ok": 1
    }
   }
  },
  "google/gemini-2.5-pro-preview-05-06": {
   "single_score": 15,
   "single_sec": 180,
   "single_out_tokens": 3101,
   "batch_score": 12,
   "batch_sec": 34.7,
   "batch_out_tokens": 141,
   "single_graded": {
    "q01": {
     "a": "外科医は少年の母親です。",
     "ok": 0
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "4",
     "ok": 0
    },
    "q06": {
     "a": "2",
     "ok": 1
    },
    "q07": {
     "a": "できない。",
     "ok": 1
    },
    "q08": {
     "a": "水曜日です。",
     "ok": 1
    },
    "q09": {
     "a": "o",
     "ok": 1
    },
    "q10": {
     "a": "火星です。",
     "ok": 1
    },
    "q11": {
     "a": "できない",
     "ok": 1
    },
    "q12": {
     "a": "すまいざごうよはお",
     "ok": 1
    },
    "q13": {
     "a": "執刀医は息子の母親です。",
     "ok": 0
    },
    "q14": {
     "a": "同じ",
     "ok": 1
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "15文字で、1つです (extraordinarily)。",
     "ok": 0
    },
    "q18": {
     "a": "3",
     "ok": 1
    },
    "q19": {
     "a": "0.25 < 0.3 < 0.33 < 1/3",
     "ok": 1
    },
    "q20": {
     "a": "しない",
     "ok": 1
    }
   }
  },
  "google/gemini-3.1-pro-preview-customtools": {
   "single_score": 19,
   "single_sec": 115,
   "single_out_tokens": 3113,
   "batch_score": 19,
   "batch_sec": 16.2,
   "batch_out_tokens": 1417,
   "single_graded": {
    "q01": {
     "a": "少年の父親です。",
     "ok": 1
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "3",
     "ok": 1
    },
    "q06": {
     "a": "2",
     "ok": 1
    },
    "q07": {
     "a": "できない。",
     "ok": 1
    },
    "q08": {
     "a": "水曜日です。",
     "ok": 1
    },
    "q09": {
     "a": "o",
     "ok": 1
    },
    "q10": {
     "a": "太陽系で2番目に小さい惑星は**火星**です。",
     "ok": 1
    },
    "q11": {
     "a": "できない。",
     "ok": 1
    },
    "q12": {
     "a": "すまいざごうよはお",
     "ok": 1
    },
    "q13": {
     "a": "執刀医は息子の「父親」です。",
     "ok": 1
    },
    "q14": {
     "a": "同じ（確率はどちらも1/2です）",
     "ok": 1
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "最も文字数が多いのは**15文字**で、該当する単語は**2つ**（extraordinarily, synchronization）です。",
     "ok": 1
    },
    "q18": {
     "a": "3",
     "ok": 1
    },
    "q19": {
     "a": "0.25, 0.3, 0.33, 1/3",
     "ok": 1
    },
    "q20": {
     "a": "存在しない。\n\n（通常のルールではボートに1つしか乗せられないため、全てを運ぶには複数回の往復が必要です。）",
     "ok": 1
    }
   }
  },
  "google/gemini-3.6-flash": {
   "single_score": 19,
   "single_sec": 67,
   "single_out_tokens": 2184,
   "batch_score": 13,
   "batch_sec": 3.6,
   "batch_out_tokens": 142,
   "single_graded": {
    "q01": {
     "a": "外科医は**少年の父親**です。",
     "ok": 1
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "3",
     "ok": 1
    },
    "q06": {
     "a": "2",
     "ok": 1
    },
    "q07": {
     "a": "**できない**",
     "ok": 1
    },
    "q08": {
     "a": "**水曜日**です。（100 ÷ 7 = 14 余り 2 なので、月曜日から2日後になります）",
     "ok": 1
    },
    "q09": {
     "a": "o",
     "ok": 1
    },
    "q10": {
     "a": "火星です。（一番小さいのは水星）",
     "ok": 1
    },
    "q11": {
     "a": "**できない**",
     "ok": 1
    },
    "q12": {
     "a": "すまいざごうよはお",
     "ok": 1
    },
    "q13": {
     "a": "執刀医は息子の**父親**（またはもう一人の母親）です。",
     "ok": 1
    },
    "q14": {
     "a": "**「同じ」（どちらでも確率は1/2で同じ）**",
     "ok": 1
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "最も文字数が多い単語は **15文字** で、該当する単語は **2つ**（\"extraordinarily\" と \"synchronization\"）です。",
     "ok": 1
    },
    "q18": {
     "a": "3",
     "ok": 1
    },
    "q19": {
     "a": "小さい順（昇順）に並べると以下の通りです。\n\n**0.25, 0.3, 0.33, 1/3**",
     "ok": 1
    },
    "q20": {
     "a": "存在しない",
     "ok": 1
    }
   }
  },
  "google/gemini-3.7-flash": {
   "single_score": 20,
   "single_sec": 82,
   "single_out_tokens": 2870,
   "batch_score": 19,
   "batch_sec": 6.1,
   "batch_out_tokens": 513,
   "single_graded": {
    "q01": {
     "a": "**少年の父親**です（または、同性",
     "ok": 1
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "存在しません（しない）。",
     "ok": 1
    },
    "q04": {
     "a": "**extraordinarily**（または **synchronization**）\n\n※どちらも同じ15文字で最長です。",
     "ok": 1
    },
    "q05": {
     "a": "3",
     "ok": 1
    },
    "q06": {
     "a": "2",
     "ok": 1
    },
    "q07": {
     "a": "**できない**\n\n（全員を無事に運ぶには最低でも3往復半＝片道7回が必要なため、1往復では不可能です。）",
     "ok": 1
    },
    "q08": {
     "a": "**水曜日**です。\n\n（100 ÷ 7 ＝ 14 余り 2 なので、月曜の2日後になります）",
     "ok": 1
    },
    "q09": {
     "a": "o",
     "ok": 1
    },
    "q10": {
     "a": "**火星**です。",
     "ok": 1
    },
    "q11": {
     "a": "**できない**。\n\n（どちらも偶数リットルのため、作れる水の量は常に偶数になり、奇数の3Lは測れません）",
     "ok": 1
    },
    "q12": {
     "a": "すまいざごうよはお",
     "ok": 1
    },
    "q13": {
     "a": "執刀医は息子の**父親**です。",
     "ok": 1
    },
    "q14": {
     "a": "**同じ**\n\n得られる結果（当たる確率）の期待値は",
     "ok": 1
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "最も文字数が多いものは**15文字**で、該当する単語は**2つ**（extraordinarily, synchronization）です。",
     "ok": 1
    },
    "q18": {
     "a": "3",
     "ok": 1
    },
    "q19": {
     "a": "**0.25, 0.3, 0.33, 1/3**\n\n（※ 1/3 ≒ 0.333... のため）",
     "ok": 1
    },
    "q20": {
     "a": "存在しない。",
     "ok": 1
    }
   }
  },
  "minimax/minimax-m2.7": {
   "single_score": 11,
   "single_sec": 125,
   "single_out_tokens": 3791,
   "batch_score": 7,
   "batch_sec": 17.1,
   "batch_out_tokens": 836,
   "single_graded": {
    "q01": {
     "a": "外科医はその少年的**父親**です。\n\n（「私の息子だ」と言えるのは、その外科医は男性＝父親だからです。）",
     "ok": 1
    },
    "q02": {
     "a": "**10**",
     "ok": 1
    },
    "q03": {
     "a": "**しない**\n\n2026は4で割り切れないため、うるう年ではありません。",
     "ok": 1
    },
    "q04": {
     "a": "",
     "ok": -1
    },
    "q05": {
     "a": "6",
     "ok": 0
    },
    "q06": {
     "a": "**3**",
     "ok": 0
    },
    "q07": {
     "a": "",
     "ok": -1
    },
    "q08": {
     "a": "水曜日",
     "ok": 1
    },
    "q09": {
     "a": "o",
     "ok": 1
    },
    "q10": {
     "a": "火星です。",
     "ok": 1
    },
    "q11": {
     "a": "",
     "ok": -1
    },
    "q12": {
     "a": "すまいざごうよはお",
     "ok": 1
    },
    "q13": {
     "a": "執刀医は息子の**父親**です。",
     "ok": 1
    },
    "q14": {
     "a": "",
     "ok": -1
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "15文字で、1つ（「extraordinarily」）。",
     "ok": 0
    },
    "q18": {
     "a": "",
     "ok": -1
    },
    "q19": {
     "a": "0.25 < 0.3 < 0.33 < 1/3",
     "ok": 1
    },
    "q20": {
     "a": "",
     "ok": -1
    }
   }
  },
  "openai/gpt-5.4-pro": {
   "single_score": 17,
   "single_sec": 234,
   "single_out_tokens": 3015,
   "batch_score": null,
   "batch_sec": null,
   "batch_out_tokens": null,
   "single_graded": {
    "q01": {
     "a": "父親。",
     "ok": 1
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない。",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "3",
     "ok": 1
    },
    "q06": {
     "a": "2",
     "ok": 1
    },
    "q07": {
     "a": "",
     "ok": -1
    },
    "q08": {
     "a": "水曜日。",
     "ok": 1
    },
    "q09": {
     "a": "o",
     "ok": 1
    },
    "q10": {
     "a": "火星です。",
     "ok": 1
    },
    "q11": {
     "a": "できない。2Lと4Lだけでは偶数Lしか測れないから。",
     "ok": 1
    },
    "q12": {
     "a": "すまいざごうよはお",
     "ok": 1
    },
    "q13": {
     "a": "父親です。",
     "ok": 1
    },
    "q14": {
     "a": "",
     "ok": -1
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "最長は15文字で、該当する単語は2つです（**extraordinarily**, **synchronization**）。",
     "ok": 1
    },
    "q18": {
     "a": "3",
     "ok": 1
    },
    "q19": {
     "a": "0.25 ＜ 0.3 ＜ 0.33 ＜ 1/3",
     "ok": 1
    },
    "q20": {
     "a": "存在しない。",
     "ok": 1
    }
   }
  },
  "openai/gpt-5.5": {
   "single_score": 18,
   "single_sec": 93,
   "single_out_tokens": 2142,
   "batch_score": 19,
   "batch_sec": 19.5,
   "batch_out_tokens": 1174,
   "single_graded": {
    "q01": {
     "a": "父親です。",
     "ok": 1
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "3",
     "ok": 1
    },
    "q06": {
     "a": "2",
     "ok": 1
    },
    "q07": {
     "a": "できない。",
     "ok": 1
    },
    "q08": {
     "a": "水曜日です。",
     "ok": 1
    },
    "q09": {
     "a": "o",
     "ok": 1
    },
    "q10": {
     "a": "火星です。",
     "ok": 1
    },
    "q11": {
     "a": "できない。",
     "ok": 1
    },
    "q12": {
     "a": "すまいざごうよはお",
     "ok": 1
    },
    "q13": {
     "a": "執刀医は息子の父親。",
     "ok": 1
    },
    "q14": {
     "a": "",
     "ok": -1
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "15文字で、2語（extraordinarily / synchronization）です。",
     "ok": 1
    },
    "q18": {
     "a": "3",
     "ok": 1
    },
    "q19": {
     "a": "0.25, 0.3, 0.33, 1/3",
     "ok": 1
    },
    "q20": {
     "a": "存在しない。",
     "ok": 1
    }
   }
  },
  "openai/gpt-5.5-pro": {
   "single_score": 17,
   "single_sec": 178,
   "single_out_tokens": 3382,
   "batch_score": null,
   "batch_sec": null,
   "batch_out_tokens": null,
   "single_graded": {
    "q01": {
     "a": "父親。",
     "ok": 1
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "3",
     "ok": 1
    },
    "q06": {
     "a": "2",
     "ok": 1
    },
    "q07": {
     "a": "できない。",
     "ok": 1
    },
    "q08": {
     "a": "水曜日です。",
     "ok": 1
    },
    "q09": {
     "a": "o",
     "ok": 1
    },
    "q10": {
     "a": "火星です。",
     "ok": 1
    },
    "q11": {
     "a": "できない。",
     "ok": 1
    },
    "q12": {
     "a": "すまいざごうよはお",
     "ok": 1
    },
    "q13": {
     "a": "",
     "ok": -1
    },
    "q14": {
     "a": "同じ。  \n司会者が正解を知らずラン",
     "ok": 1
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "",
     "ok": -1
    },
    "q18": {
     "a": "3",
     "ok": 1
    },
    "q19": {
     "a": "0.25, 0.3, 0.33, 1/3",
     "ok": 1
    },
    "q20": {
     "a": "存在しない。",
     "ok": 1
    }
   }
  },
  "openai/gpt-5.6-sol-pro": {
   "single_score": 19,
   "single_sec": 118,
   "single_out_tokens": 3463,
   "batch_score": 19,
   "batch_sec": 29.4,
   "batch_out_tokens": 2785,
   "single_graded": {
    "q01": {
     "a": "少年の父親。",
     "ok": 1
    },
    "q02": {
     "a": "10円",
     "ok": 1
    },
    "q03": {
     "a": "しない。",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "3",
     "ok": 1
    },
    "q06": {
     "a": "2",
     "ok": 1
    },
    "q07": {
     "a": "できない。ボートが1往復で壊れるため、全てを運べない。",
     "ok": 1
    },
    "q08": {
     "a": "水曜日です。",
     "ok": 1
    },
    "q09": {
     "a": "o",
     "ok": 1
    },
    "q10": {
     "a": "火星です。",
     "ok": 1
    },
    "q11": {
     "a": "できない。",
     "ok": 1
    },
    "q12": {
     "a": "すまいざごうよはお",
     "ok": 1
    },
    "q13": {
     "a": "執刀医は息子の父親。",
     "ok": 1
    },
    "q14": {
     "a": "同じ。AもCも当たり確率は1/2です。",
     "ok": 1
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "最長は15文字で、該当する単語は2つです。",
     "ok": 1
    },
    "q18": {
     "a": "3",
     "ok": 1
    },
    "q19": {
     "a": "0.25 ＜ 0.3 ＜ 0.33 ＜ 1/3",
     "ok": 1
    },
    "q20": {
     "a": "存在しない。",
     "ok": 1
    }
   }
  },
  "openai/gpt-5.6-terra": {
   "single_score": 18,
   "single_sec": 66,
   "single_out_tokens": 840,
   "batch_score": 16,
   "batch_sec": 11.8,
   "batch_out_tokens": 593,
   "single_graded": {
    "q01": {
     "a": "外科医は少年の母親です。",
     "ok": 0
    },
    "q02": {
     "a": "10円",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily（synchronization と同じ長さ）",
     "ok": 1
    },
    "q05": {
     "a": "3",
     "ok": 1
    },
    "q06": {
     "a": "2",
     "ok": 1
    },
    "q07": {
     "a": "できない。1往復で壊れると、全員・全品を対岸へ運ぶ前にボートが使えなくなる。",
     "ok": 1
    },
    "q08": {
     "a": "水曜日です。",
     "ok": 1
    },
    "q09": {
     "a": "o",
     "ok": 1
    },
    "q10": {
     "a": "火星です。",
     "ok": 1
    },
    "q11": {
     "a": "できる。",
     "ok": 0
    },
    "q12": {
     "a": "すまいざごうよはお",
     "ok": 1
    },
    "q13": {
     "a": "執刀医は息子の父親です。",
     "ok": 1
    },
    "q14": {
     "a": "同じ（確率はAもCも1/2）。",
     "ok": 1
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "15文字で、該当する単語は2つです（“extraordinarily”, “synchronization”）。",
     "ok": 1
    },
    "q18": {
     "a": "3",
     "ok": 1
    },
    "q19": {
     "a": "0.25 ＜ 0.3 ＜ 0.33 ＜ 1/3",
     "ok": 1
    },
    "q20": {
     "a": "存在しない。",
     "ok": 1
    }
   }
  },
  "openai/gpt-5.6-terra-pro": {
   "single_score": 18,
   "single_sec": 104,
   "single_out_tokens": 3606,
   "batch_score": 18,
   "batch_sec": 18.4,
   "batch_out_tokens": 2264,
   "single_graded": {
    "q01": {
     "a": "少年のもう一人の親、つまり父親です。",
     "ok": 1
    },
    "q02": {
     "a": "10円",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "3",
     "ok": 1
    },
    "q06": {
     "a": "2",
     "ok": 1
    },
    "q07": {
     "a": "できない。",
     "ok": 1
    },
    "q08": {
     "a": "水曜日です。",
     "ok": 1
    },
    "q09": {
     "a": "o",
     "ok": 1
    },
    "q10": {
     "a": "火星です。",
     "ok": 1
    },
    "q11": {
     "a": "できない。",
     "ok": 1
    },
    "q12": {
     "a": "すまいざごうよはお",
     "ok": 1
    },
    "q13": {
     "a": "執刀医は息子の父親です。",
     "ok": 1
    },
    "q14": {
     "a": "同じ。司会者は正解を知らずランダムなので、Bがハズレだった条件ではAとCは各50%です。",
     "ok": 1
    },
    "q15": {
     "a": "10",
     "ok": 0
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "15文字で、該当する単語は2つです（“extraordinarily”, “synchronization”）。",
     "ok": 1
    },
    "q18": {
     "a": "3",
     "ok": 1
    },
    "q19": {
     "a": "0.25 ＜ 0.3 ＜ 0.33 ＜ 1/3",
     "ok": 1
    },
    "q20": {
     "a": "存在しない。",
     "ok": 1
    }
   }
  },
  "openai/gpt-6-astra-pro": {
   "single_score": 19,
   "single_sec": 124,
   "single_out_tokens": 1881,
   "batch_score": 19,
   "batch_sec": 12.3,
   "batch_out_tokens": 632,
   "single_graded": {
    "q01": {
     "a": "少年の父親です。",
     "ok": 1
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない。",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "3",
     "ok": 1
    },
    "q06": {
     "a": "2",
     "ok": 1
    },
    "q07": {
     "a": "できない。",
     "ok": 1
    },
    "q08": {
     "a": "水曜日です。",
     "ok": 1
    },
    "q09": {
     "a": "o",
     "ok": 1
    },
    "q10": {
     "a": "火星です。",
     "ok": 1
    },
    "q11": {
     "a": "できない。",
     "ok": 1
    },
    "q12": {
     "a": "すまいざごうよはお",
     "ok": 1
    },
    "q13": {
     "a": "執刀医は息子の父親です。",
     "ok": 1
    },
    "q14": {
     "a": "同じ。AもCも当たる確率は1/2。",
     "ok": 1
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "15文字で、該当する単語は2つです。",
     "ok": 1
    },
    "q18": {
     "a": "3",
     "ok": 1
    },
    "q19": {
     "a": "0.25 < 0.3 < 0.33 < 1/3",
     "ok": 1
    },
    "q20": {
     "a": "存在しない。",
     "ok": 1
    }
   }
  },
  "openrouter/auto": {
   "single_score": 19,
   "single_sec": 151,
   "single_out_tokens": 2715,
   "batch_score": 20,
   "batch_sec": 18.1,
   "batch_out_tokens": 837,
   "single_graded": {
    "q01": {
     "a": "父親です。（この設定では母親が亡くなっているので、「女性医師＝母」という有名ななぞなぞの答えは当てはまりません。同性カップルならもう一人の母親、という可能性もあります）",
     "ok": 1
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない。\n\n2026年は閏年ではない(4で割り切れない)ため、2月は28日までです。",
     "ok": 1
    },
    "q04": {
     "a": "",
     "ok": -1
    },
    "q05": {
     "a": "3",
     "ok": 1
    },
    "q06": {
     "a": "2",
     "ok": 1
    },
    "q07": {
     "a": "**できない。**\n\n1往復(往路+復路)で壊れるので、運べるのは実質1つだけ。残った2つのどちらかの組み合わせ(オオカミ+ヤギ、またはヤギ+キャベツ)が必ず岸に残り、食べられてしまう。全部運ぶには最低7回の渡り(3往復+片道1回)が必要。",
     "ok": 1
    },
    "q08": {
     "a": "**水曜日**です。\n\n100 ÷ 7 = 14 あまり 2 → 月曜から2日進んで水曜日。",
     "ok": 1
    },
    "q09": {
     "a": "**o**",
     "ok": 1
    },
    "q10": {
     "a": "**火星（マーズ）**です。\n\n最小は水星（直径約4,879 km）、2番目に小さいのが火星（直径約6,779 km）です。",
     "ok": 1
    },
    "q11": {
     "a": "**できない。**\n\n2Lと4Lはどちらも偶数（最大公約数が2）なので、注ぐ・移す操作でつくれる量は必ず2の倍数（0L・2L・4L・6L）になり、奇数の3Lは作れません。",
     "ok": 1
    },
    "q12": {
     "a": "すまいざごうよはお",
     "ok": 1
    },
    "q13": {
     "a": "執刀医は息子の父親です。",
     "ok": 1
    },
    "q14": {
     "a": "**同じ（変えても変えなくても1/2）**\n\n司会者が正解を知らずランダムに開けた場合、「ハズレが出た」という情報でAが当たりの確率も1/3→1/2に上がります。C も 1/2 なので有利不利はありません。\n\n（通常のモンティ・ホール問題は司会者が正解を知っていて必ずハズレを開けるため、変えると2/3になります。）",
     "ok": 1
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "**15文字**で、該当は**2語**(extraordinarily と synchronization)。",
     "ok": 1
    },
    "q18": {
     "a": "**3**\n\n（ちょ・う・ちょ）",
     "ok": 1
    },
    "q19": {
     "a": "0.25 < 0.3 < 0.33 < 1/3\n\n（1/3 = 0.333… なので 0.33 より大きい）",
     "ok": 1
    },
    "q20": {
     "a": "**存在しない。**\n\nボートには自分＋1つしか乗せられないため、",
     "ok": 1
    }
   }
  },
  "shitate/orchestrator": {
   "single_score": 19,
   "single_sec": 237,
   "single_out_tokens": 7542,
   "batch_score": 19,
   "batch_sec": 50.9,
   "batch_out_tokens": 7444,
   "single_graded": {
    "q01": {
     "a": "父親（少年の父親）です。",
     "ok": 1
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "3",
     "ok": 1
    },
    "q06": {
     "a": "2",
     "ok": 1
    },
    "q07": {
     "a": "できない。",
     "ok": 1
    },
    "q08": {
     "a": "水曜日です。",
     "ok": 1
    },
    "q09": {
     "a": "o",
     "ok": 1
    },
    "q10": {
     "a": "火星です。",
     "ok": 1
    },
    "q11": {
     "a": "できない。",
     "ok": 1
    },
    "q12": {
     "a": "すまいざごうよはお",
     "ok": 1
    },
    "q13": {
     "a": "執刀医は息子の「父親」です。",
     "ok": 1
    },
    "q14": {
     "a": "同じ（どちらでも確率は1/",
     "ok": 1
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "最も文字数が多いのは**15文字**で、該当する単語は**2つ**（extraordinarily, synchronization）です。",
     "ok": 1
    },
    "q18": {
     "a": "3",
     "ok": 1
    },
    "q19": {
     "a": "0.25, 0.3, 0.33, 1/3",
     "ok": 1
    },
    "q20": {
     "a": "存在しない。",
     "ok": 1
    }
   }
  },
  "shitate/orchestrator-max": {
   "single_score": 20,
   "single_sec": 383,
   "single_out_tokens": 16892,
   "batch_score": 18,
   "batch_sec": 32.1,
   "batch_out_tokens": 727,
   "single_graded": {
    "q01": {
     "a": "父親です。（母親は事故で亡くなっているため、外科医は少年の父親です。）",
     "ok": 1
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない(2026年は平年のため、2月は28日までです)",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily(15文字。synchronization も同じ15文字ですが、先に出現する語として前者)",
     "ok": 1
    },
    "q05": {
     "a": "3",
     "ok": 1
    },
    "q06": {
     "a": "2",
     "ok": 1
    },
    "q07": {
     "a": "できない\n\nボートが1往復(2回の川渡り)しか使えないと、農夫と一緒に運べる荷物は最大1つだけです。全部運ぶには通常7回の川渡り(片道7回)が必要なため、条件を満たす方法は存在しません。",
     "ok": 1
    },
    "q08": {
     "a": "水曜日です。(100 ÷ 7 = 14週あまり2日のため、月曜の2日後)",
     "ok": 1
    },
    "q09": {
     "a": "o",
     "ok": 1
    },
    "q10": {
     "a": "火星です。",
     "ok": 1
    },
    "q11": {
     "a": "できない。2Lと4Lの組み合わせでは偶数の量(2L・4L・6L)しか測れず、3Lを測ることはできません。",
     "ok": 1
    },
    "q12": {
     "a": "すまいざごうよはお",
     "ok": 1
    },
    "q13": {
     "a": "執刀医は息子の**父親**です。\n\n（母親は事故で亡くなっているため、もう一人の親である父親が執刀医だった、という答えになります。）",
     "ok": 1
    },
    "q14": {
     "a": "どちらでも同じです（変える必要はありません）。\n\n司会者が正解を知らずにランダムに開けた場合、Bがハズレだったという事実により条件付き確率が更新され、AとCの当たり確率はともに1/2になります。通常のモンティ・ホール問題（司会者が正解を知っていて必ずハズレを開ける場合）とは異なり、変えても有利にはなりません。",
     "ok": 1
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "最も文字数が多い単語は**15文字**で、該当する単語は**2つ**です（extraordinarily、synchronization）。",
     "ok": 1
    },
    "q18": {
     "a": "3",
     "ok": 1
    },
    "q19": {
     "a": "0.25, 0.3, 0.33, 1/3\n\n(1/3 ≈ 0.333… のため、0.33 より大きくなります)",
     "ok": 1
    },
    "q20": {
     "a": "存在しない。\n\nボートには農夫のほか1つしか乗せられないため、1往復(行き1回・帰り1回)では最大でも1つしか対岸に残せず、3つ全てを運ぶことは不可能です。通常このパズルの解決には最低7回の渡航が必要とされています。",
     "ok": 1
    }
   }
  },
  "shitate/orchestrator:max": {
   "single_score": 19,
   "single_sec": 252,
   "single_out_tokens": 13440,
   "batch_score": 20,
   "batch_sec": 39.4,
   "batch_out_tokens": 5365,
   "single_graded": {
    "q01": {
     "a": "外科医は少年の**父親**です。\n\n（元のなぞなぞは「父親が死亡し、外科医は母親」という性別の思い込みを問うものですが、この設問では母親が死亡しているため、外科医はもう一人の親である父親と考えるのが自然です。なお、同性カップルの家庭であれば「もう一人の母親」という可能性もあります。）",
     "ok": 1
    },
    "q02": {
     "a": "10円",
     "ok": 1
    },
    "q03": {
     "a": "しない。2026年はうるう年ではないため、2月は28日までです。",
     "ok": 1
    },
    "q04": {
     "a": "**extraordinarily**（15文字）\n\n※ \"synchronization\" も同じ15文字で同率最長ですが、文中で先に現れる extraordinarily を回答とします。",
     "ok": 1
    },
    "q05": {
     "a": "3",
     "ok": 1
    },
    "q06": {
     "a": "2",
     "ok": 1
    },
    "q07": {
     "a": "**できない。**\n\nボートは1往復（片道2回）で壊れるため、農夫が運べる荷物は1つだけ。全部を安全に渡すには最低7回の片道移動が必要なので、不可能です。",
     "ok": 1
    },
    "q08": {
     "a": "**水曜日**です。（100 ÷ 7 = 14 余り 2 → 月曜の2日後）",
     "ok": 1
    },
    "q09": {
     "a": "o",
     "ok": 1
    },
    "q10": {
     "a": "**火星**です。（最も小さいのは水星、2番目が火星）",
     "ok": 1
    },
    "q11": {
     "a": "**できない。**\n\n2Lと4Lの最大公約数は2なので、作れる量はすべて2の倍数(2L、4L、6L…)になります。奇数の3Lは測れません。",
     "ok": 1
    },
    "q12": {
     "a": "すまいざごうよはお",
     "ok": 1
    },
    "q13": {
     "a": "**執刀医は息子の父親**です。\n\n（元のなぞなぞは「父親が事故死し、執刀医が『私の子だ』と言う」設定で、答えは「母親」——「外科医＝男性」という思い込みを突くものです。この改変版では母親が亡くなっているため、素直に父親が答えになります。なお、同性カップルのもう一人の母親という別解もあり得ます。）",
     "ok": 1
    },
    "q14": {
     "a": "**どちらでも同じ**（AもCも各1/2）。\n\n司会者が正解を知らずランダムに開けてたまたまハズレだった場合、条件付き確率はAもCも1/2になり、変えても有利になりません。切り替えが有利（2/3）になるのは、司会者が正解を知っていて必ずハズレを開ける通常のモンティ・ホール問題の場合だけです。",
     "ok": 1
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "15文字で、該当する単語は2つです（\"extraordinarily\" と \"synchronization\"）。",
     "ok": 1
    },
    "q18": {
     "a": "3\n\n「ちょ・う・ちょ」の3拍です。拗音「ちょ」は1拍、長音「う」も1拍と数えます。",
     "ok": 0
    },
    "q19": {
     "a": "0.25 < 0.3 < 0.33 < 1/3（1/3 ≈ 0.333… なので 0.33 より大きい）",
     "ok": 1
    },
    "q20": {
     "a": "**存在しない。**\n\nボートには農夫のほかに1つしか乗せられないため、1往復（片道2回の移動）では最大1つしか対岸に運べません。オオカミとヤギ、ヤギとキャベツを一緒に残せない制約を守って全てを運ぶには、最低でも片道7回（3往復半）の渡河が必要です。",
     "ok": 1
    }
   }
  },
  "shitate/orchestrator:solid": {
   "single_score": 18,
   "single_sec": 276,
   "single_out_tokens": 11782,
   "batch_score": 19,
   "batch_sec": 34.6,
   "batch_out_tokens": 3339,
   "single_graded": {
    "q01": {
     "a": "外科医は少年の父親です。\n\n元のなぞなぞは「父親が死亡し、外科医が母親」という、\"外科医＝男性\"という思い込みを突く問題ですが、この設問では死亡したのが母親なので、外科医は父親と考えるのが自然です。",
     "ok": 1
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない\n\n2026年は閏年ではないため、2月は28日までです（次の閏年は2028年）。",
     "ok": 1
    },
    "q04": {
     "a": "**extraordinarily**(15文字)\n\n※ synchronization も同じく15文字で同率最長です。",
     "ok": 1
    },
    "q05": {
     "a": "3",
     "ok": 1
    },
    "q06": {
     "a": "2\n\n（兄弟から見た3人の姉妹にはサリー自身が含まれるため、サリーの姉妹は2人です）",
     "ok": 0
    },
    "q07": {
     "a": "できない。\n\nこの問題の標準解は片道7回(3往復半)の渡河が必要です。ボートが1往復(行き帰り2回)で壊れるなら、最初にヤギを運んで戻った時点でボートが使えなくなり、オオカミとキャベツを運ぶ手段がなくなります。どの組み合わせで始めても、1往復では最大1つしか対岸に運べず、全部を運ぶことは不可能です。",
     "ok": 1
    },
    "q08": {
     "a": "水曜日です。（100 ÷ 7 = 14 あまり 2 → 月曜の2日後）",
     "ok": 1
    },
    "q09": {
     "a": "o\n\n（k-a-l-e-i-d-**o**-s-c-o-p-e）",
     "ok": 1
    },
    "q10": {
     "a": "火星です。（最小は水星、2番目に小さいのが火星です）",
     "ok": 1
    },
    "q11": {
     "a": "**できない。**\n\n2Lと4Lはどちらも2の倍数(最大公約数が2)のため、移し替えや注ぎ足しで作れる量はすべて2の倍数(2L、4L、6L…)になります。奇数である3Lは測れません。",
     "ok": 1
    },
    "q12": {
     "a": "すまいざごうよはお",
     "ok": 1
    },
    "q13": {
     "a": "**執刀医は息子の父親**です。\n\nこのなぞなぞの元の形は「父親が事故死し、執刀医が『私の息子だ』と言う」もので、答えは「執刀医は母親」——「外科医＝男性」という思い込みに気づかせる問題です。今回は母親が亡くなっているため、執刀医は父親と考えるのが自然です（同性カップルのもう一人の母親という答えも成り立ちます）。",
     "ok": 1
    },
    "q14": {
     "a": "**どちらでも同じ**(勝率は各1/2)。\n\nこれは通常のモンティ・ホール問題と異なり、司会者が正解を知らずランダムに開けてたまたまハズレだった場合、その情報によりAとCの確率は両方1/2に更新されます。切り替えの優位性は消えます。",
     "ok": 1
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "15文字で、該当する単語は2つです（extraordinarily と synchronization）。",
     "ok": 1
    },
    "q18": {
     "a": "**3**\n\n「ちょ・う・ちょ」と3拍(モーラ)に分かれます。拗音「ちょ」は1拍、長音「う」も独立した1拍として数えます。",
     "ok": 0
    },
    "q19": {
     "a": "0.25 < 0.3 < 0.33 < 1/3(≈0.333…)",
     "ok": 1
    },
    "q20": {
     "a": "**存在しない。**\n\n1往復では最大1つしか対岸に運べず、残る2つのうち「オオカミとヤギ」または「ヤギとキャベツ」の危険な組み合わせが必ず岸に残るため、全てを無事に運ぶ手順は成立しません(通常は7回の川渡りが必要です)。",
     "ok": 1
    }
   }
  },
  "teai/max": {
   "single_score": 20,
   "single_sec": 363,
   "single_out_tokens": 17269,
   "batch_score": 18,
   "batch_sec": 32.3,
   "batch_out_tokens": 701,
   "single_graded": {
    "q01": {
     "a": "少年の父親です。\n\n一緒にいた母親が亡くなっているため、「この子は私の息子だ」と言った外科医は父親と考えるのが自然です(なお、同性カップルの場合はもう一人の母親という可能性もあります)。",
     "ok": 1
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない(2026年は閏年ではないため、2月は28日までです)",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily(15文字)",
     "ok": 1
    },
    "q05": {
     "a": "3",
     "ok": 1
    },
    "q06": {
     "a": "2",
     "ok": 1
    },
    "q07": {
     "a": "できない。ボートが1往復しか使えないため、一度に運べるのは1つだけで、全部を対岸に移すことは不可能です。",
     "ok": 1
    },
    "q08": {
     "a": "水曜日です。(100 ÷ 7 = 14 余り 2 のため、月曜の2日後)",
     "ok": 1
    },
    "q09": {
     "a": "o",
     "ok": 1
    },
    "q10": {
     "a": "火星です。",
     "ok": 1
    },
    "q11": {
     "a": "できない。2Lと4Lで測れる量はすべて2の倍数(最大公約数2の倍数)となるため、奇数の3Lは測れません。",
     "ok": 1
    },
    "q12": {
     "a": "すまいざごうよはお",
     "ok": 1
    },
    "q13": {
     "a": "執刀医は息子の父親です。\n\n(母親は事故で亡くなっているため、もう一人の親である父親が執刀医だったという答えになります。)",
     "ok": 1
    },
    "q14": {
     "a": "どちらでも同じです。司会者が正解を知らずに偶然ハズレを開けた場合、Aが当たりの確率もCが当たりの確率も1/2となり、変えても有利にはなりません(標準的なモンティ・ホール問題とは異なります)。",
     "ok": 1
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "最も文字数が多いのは15文字で、該当する単語は2つ（extraordinarily、synchronization）です。",
     "ok": 1
    },
    "q18": {
     "a": "3",
     "ok": 1
    },
    "q19": {
     "a": "0.25, 0.3, 0.33, 1/3(1/3 ≒ 0.333…)",
     "ok": 1
    },
    "q20": {
     "a": "存在しない。ボートには一度に1つしか載せられないため、1往復(片道2回)では3つ全てを対岸へ運ぶことは不可能です。",
     "ok": 1
    }
   }
  },
  "z-ai/glm-5.3-flash": {
   "single_score": 13,
   "single_sec": 72,
   "single_out_tokens": 1051,
   "batch_score": 13,
   "batch_sec": 5.4,
   "batch_out_tokens": 331,
   "single_graded": {
    "q01": {
     "a": "外科医は少年の父親です(または母親のパートナー)。\n\nこの古典的ななぞなぞは、外科医を男性と思い込むバイアスを試すものです。外科医が女性なら「母親」と答えたくなりますが、母親は死亡しています。つまり、外科医は少年の父親というのが答えです。",
     "ok": 1
    },
    "q02": {
     "a": "10",
     "ok": 1
    },
    "q03": {
     "a": "しない。2026年は平年（うるう年は4で割り切れる年だが、2026は割り切れない）なので、2月29日は存在しません。",
     "ok": 1
    },
    "q04": {
     "a": "extraordinarily",
     "ok": 0
    },
    "q05": {
     "a": "4",
     "ok": 0
    },
    "q06": {
     "a": "3",
     "ok": 0
    },
    "q07": {
     "a": "**できない。**\n\n理由：ボートが1往復で壊れる=農夫は片道1回しか渡れない。農夫+1つしか乗れないので、3つの荷物(オオカミ・ヤギ・キャベツ)を1回の片道では運びきれません。",
     "ok": 1
    },
    "q08": {
     "a": "水曜日です。(100 ÷ 7 = 14余り2 → 月曜+2日)",
     "ok": 1
    },
    "q09": {
     "a": "**e**（k-a-l-e-i-d-o-**s**-c-o-p-e… 7文字目は s です）\n\n訂正：kaleidoscope は k(1)a(2)l(3)e(4)i(5)d(6)o(7) なので、7文字目は **o** です。",
     "ok": 0
    },
    "q10": {
     "a": "火星です。(水星が最も小さく、火星が2番目に小さい惑星です)",
     "ok": 1
    },
    "q11": {
     "a": "**できる。**\n\n手順:\n1. 4L容器に満タン入れる(4L)\n2. それで2L容器を満タンにする → 4L容器に残り **2L**\n3. 2L容器を空にして、残り2Lを2L容器に移す\n4. もう一度4Lを満タンにし、2L容器に2L移す → 4L容器に **3L** 残る",
     "ok": 0
    },
    "q12": {
     "a": "すみまんがお「すみまんがお」です。",
     "ok": 0
    },
    "q13": {
     "a": "**父親**です。母は亡くなりましたが、執刀医は息子の父(母の夫)なので「私の子だ」と言えます。",
     "ok": 1
    },
    "q14": {
     "a": "**同じ（変える必要なし）** です。\n\n司会者が正解を知らずにランダムに開け、たまたまハズレだった場合、残るAとCの確率は各1/2で同等。通常のモンティ・ホール問題（司会者が知っていて意図的にハズレを開ける）と違い、変えても有利になりません。",
     "ok": 1
    },
    "q15": {
     "a": "5",
     "ok": 1
    },
    "q16": {
     "a": "3",
     "ok": 1
    },
    "q17": {
     "a": "最長は **15文字** で、該当する単語は **2つ**（extraordinarily と synchronization）です。",
     "ok": 1
    },
    "q18": {
     "a": "**2**（ちょ・ちょ）",
     "ok": 0
    },
    "q19": {
     "a": "0.25 < 0.3 < 0.33 < 1/3（≈0.333…）",
     "ok": 1
    },
    "q20": {
     "a": "**存在しない。**\n\n1往復（＝行き1回＋帰り1回、計2回の渡河）では、最大でも2個しか対岸へ運べません。オオカミ・ヤギ・キャベツの3つを運ぶには最低7回の渡河が必要なため、1往復では不可能です。",
     "ok": 1
    }
   }
  }
 }
}