The move leaves the government with 64 votes in the 100-member legislature.
The New party said in a letter to Berzins that the government's work in the areas of education, finance management and state administration required immediate improvement.
In the letter, New party Chairwoman Ingrida Udre said the party as early as last October brought to the attention of the ruling coalition the necessity to work on amending the pension law, which was planned when the coalition was formed.
She pointed to the vote in the Parliament last week that rejected amendments to the pension law and imposed a tax on church property.
Udre said the New Party had on several occasions objected to the politicization of state trustees and councils of companies with state-owned capital shares.
Berzins was supported by Latvia's Way and the People's Party in the expulsion of the New Party. But For Fatherland and Freedom, another ruling coalition member, insisted the coalition agreement did not allow him to expel parties and asked that the New Party remain.