Team manager Trevor Francis smacked Kolinko in the head during a recent game because the goalie, a reserve with the Premiere League club, laughed when opponent Bradford scored a goal during a recent match. Kolinko was sitting on the bench at the time.
British police and the English Football Association began an investigation after Francis said he had "cuffed him around the ear."
In an interview with London's Evening Standard newspaper, Kolinko said Francis should be held accountable for his actions.
"I like Palace as a club but this situation is a real problem for me," Kolinko was quoted as saying. "I want to stay, but it would have to be without Trevor.
"It's a very strange situation for me," he said. "It is just unbelievable. I would have been forced out of the club if I had punched him. I'd have spent the next two days sitting back home in Latvia watching TV. My career in English football would have been finished.
Writing on the Crystal Palace official Web site, Francis said he had struck Kolinko after Bradford scored a goal, forcing a 1-1 draw in a recent game.
"I was very disappointed by their goal and when I turned around, I saw Kolinko laughing, so I cuffed him around the ear," Francis said. "It was only a bit of fun."
Paul Armstrong, the referee during the match, sent Francis off after his attention was drawn to the incident by another official.
Kolinko is the Latvian national team's top goalie.