Jamie Mearns, 35, from Aberdeen, said he was arrested at a checkpoint just outside the capital Sana'a when soldiers looked at his passport.
He said that despite repeated requests he was not allowed to contact the British embassy. He spent the whole month of May in jail without being allowed phone calls.
"My parents didn't know where I was for almost five weeks," he told The Daily Telegraph. "When I finally did meet a representative from the British embassy, they told me they had asked about my whereabouts but the Yemenis denied having any British citizens in prison."
After studying Arabic and working as an English language teacher in Yemen, Mr Mearns, who converted to Islam three years ago, had considered flying to Dubai to renew his visa, but was told by his Arabic Language institute that he would only need to pay a fine at the airport when he left.
On May 1, Mearns and another friend studying Arabic took a day trip out of the city but passed a checkpoint on the return trip and were arrested.
He said some of his fellow inmates were members of al-Qaeda, others Houthis, members of a Shia sect who have been fighting for more autonomy in northern Yemen.
"I couldn't believe where I was," he said. "My only crime was that my visa had expired and now I found myself sharing a room with dangerous people. Some told me they had killed and beheaded people."
Mr Mearns said he shared a cell with the former bodyguard of Abu Musab al-Zarqawi, a Jordanian militant who was killed while leading al-Qaeda in Iraq.
He also said he frequently met Jamal Ahmed Mohammed al-Badawi, a member of the al-Qaeda cell who helped plan the USS Cole bombing in 2000 that killed 17 American sailors in the port city of Aden.
Despite being held for so long, he said he was only interrogated for a total of two hours, mostly over his religious views.
The authorities have been under pressure from America to take on al-Qaeda in Yemen since Umar Farouk Abdulmutallab, the so-called Detroit "underpants bomber", was revealed to have been trained by a cell in the country.
In the last few days of May, Mr Mearns went on a four-day hunger strike. On May 31 he received his first visit from a British embassy official. The next day he was handcuffed, driven to the airport and deported.
The British embassy in Sana'a said they did not comment on individual cases. Embassy officials believe there are no longer any Britons in custody