The length of the mercury column is balanced by the pressure and can be changed only with a pressure change. This is the principle of barometer. In case of an open column attached to a vacuum pump, we are decreasing the pressure within the column. Hence, the mercury will try to balance the pressure until the pressure inside equals the pressure outside (atmospheric pressure). This happens when we achieve vacuum like conditions in the tube. As now there is no pressure difference, the mercury cannot rise further in the tube.