Why did Vijay Singh conclude that the ghost would not be a worthy opponent to him? Was he fair in his judgement?

As the ghost failed to match the skills of Vijay Singh in breaking the rock, he proclaimed that the ghost was not a worthy opponent to fight with.

This was very smart on part of Vijay Singh as the ghost might have killed him in combat. This judgement doesn’t seem fair as Vijay Singh has tricked in both the exercises with the rock.