The former Taoiseach also ruled out reunification
In an interview on Sky News' All Out Politics, Mr Ahern also said he expected the UK Government to honour its financial obligations and pay a bill to the leave the EU - an idea backed by current Taoiseach Enda Kenny.
It has been estimated that the exit bill could be as high as £50bn (€57m).
Mr Ahern said the impact of Brexit on Ireland would be "enormous", and predicted the food sector especially would suffer because about 45% goes to the UK.
"They will take a big hit," he said.
Mr Ahern also said that "the whole border issue" is "a nightmare for us".
He said he had hoped that the UK would not pull out of the customs union - an agreement that reduces administrative and financial trade barriers such as customs checks and charges and boosts economic cooperation.
"The border between north and south, between the Republic of Ireland and Northern Ireland, becomes the natural border between UK and EU," he said.
"I don't know where that's going to end up but it is a big worry."
Asked if Ireland could choose to leave the European Union as a result of Brexit, he ruled it out.
"Sometimes we are mad in Ireland but we're not that mad to leave the European Union," he joked. "We're not going to do it now, tomorrow or in the future."
He also ruled out the prospect of unification, saying: "I am a person who would love to see a united Ireland someday but it's not going to happen as part of these discussions."
Additional reporting from IRN