ChatGPT, an artificial intelligence model developed by OpenAI, was defeated in a game against a simple chess engine running on an emulated version of the 1977 Atari 2600 game console. The experiment was conducted by Citrix engineer Robert Jr. Caruso. The software it played against was Atari’s classic chess game called “Video Chess.”
ChatGPT officially failed in chess
The Video Chess engine, which runs at 1 MHz, clearly won the game against modern AI. ChatGPT had difficulty recognizing the pieces on the chessboard. It confused rooks with bishops, ignored the existence of pawns, and could not understand the basic flow of the game.
Caruso had to re-explain the situation on the board to the AI with each move. The model could not understand the course of the game and constantly expected user guidance. The engineer described this situation as “lack of piece awareness.”
What happened throughout the match revealed the weaknesses of large language models in strategy-based games. ChatGPT made incorrect evaluations at many points in the game. In some cases, the game had to be restarted from scratch.
ChatGPT, which blamed its errors on Video Chess’ iconic interface, failed against a simple emulated game engine. The Atari chess engine, with 8-bit graphics and very limited hardware capabilities, clearly defeated an AI system with millions of parameters.
After this experiment, Caruso began preparations for his next test. The next plan is to play the game “Battle Chess” with ChatGPT on an emulated 80286 processor running at 16 MHz. The results of the new experiment have the potential to provide more data on the game-based capabilities of large language models.
