He wrote a piece for Britain's Daily Telegraph on Monday
The British Prime Minister Theresa May has backed calls for Boris Johnson to apologise, for saying Muslim women in burkas look like "bank robbers".
The former UK foreign secretary has so far defied mounting calls for him to apologise for the comments.
Mrs May said the remarks "clearly caused offence" as she told him to say sorry.
Mrs May said: "Some of the terms Boris used describing people's appearance obviously have offended.
"So I agree with Brandon Lewis.
"What's important is do we believe people should have the right to practise their religion and, in the case of women and the burka and niqab, to choose how they dress.
"I believe women should be able to choose how they dress."
A source close to Mr Johnson said it was "ridiculous" his views were being "attacked".
They said Britain must not "fall into the trap of shutting down the debate on serious issues".
"We have to call it out," the source added. "If we fail to speak up for liberal values then we are simply yielding ground to reactionaries and extremists."
The controversy sprung from a piece Mr Johnson wrote for Britain's Daily Telegraph on Monday, in which he argued against following Denmark's lead in banning the burka.
But he made several comparisons between the covering worn by some women, saying it made them look like a "bank robber" or a "letter box".
Mr Johnson also revealed he would "feel entitled" to ask a woman wearing the burka to remove it if she came to his constituency office.
He concluded: "In Britain today there is only a tiny, tiny minority of women who wear these odd bits of headgear.
"One day, I am sure, they will go."
The comment sparked a backlash from senior UK Conservatives, including Brandon Lewis, the party's chair.
He revealed on Tuesday he had asked Mr Johnson to apologise.