The minister's remarks come as polls show a divide in people's views on unrestricted abortion up to 12 weeks
A member of the Independent Alliance has called for common sense on the abortion debate.
John Halligan was asked about an apparent slip in support for the liberalisation of abortion laws in weekend polls.
The Sunday Independent poll found that 63% of respondents were in favour of a referendum on repealing the 8th amendment.
However, a combined total of 52% were either against (33%) or unsure (19%) of the proposal for unrestricted abortion up to 12 weeks, compared to 48% in favour.
Minister Halligan said comments by some politicians on term limits are not helpful.
He observed: "I'm taken aback with some of the comments by politicians, who argue about 12 weeks, 10 weeks and so on.
"I think we are doing a terrible injustice to human rights for women, if after next year thousands and thousands more women are going to have to get on buses, boats, trains and planes and take lonely journeys over to England for their right to have control of their own body."
Earlier, Senator Catherine Noone said she agrees with the Social Protection Minister Regina Doherty that the public may not vote for abortion.
Speaking over the weekend Ms Doherty said the Government needs to give people more evidence before they go to the polls.
Critics claim her comments show a lack of confidence in Fine Gael, but Senator Noone - who chaired the Oireachtas committee on the 8th amendment - says her party colleague is being realistic.
Senator Noone told The Pat Kenny Show: "Every day those of us who understand and believe that the position should be changed legally in this country have a job to do to communicate that.
"Even the children's rights referendum, where it was a really positive thing, it wasn't won by any major majority. So it's a fair comment that the Minister made."
The Government is currently planning for a referendum on the 8th amendment to be held in May or June.
Reporting by Stephen McNeice and Juliette Gash