Oakes Barn was built on the land that originally formed part of the town's medieval defence ditch, dug in the 12th century. The town wall ran along the boundary of the building and Guildhall Street to the rear. The ditch was filled in the 16th Century and it is thought that the land may have been used for its water wells after this. It is known from the 1791 Warren map that' Oakes Barn was a part of a long building that was to the rear of 80-82 Guildhall Street, and was probably used by James Oakes as wool sheds. As the yarn industry declined, Oakes became a local banker, and it is thought at this stage, most of the wool shed was demolished and given given over to gardens. Oakes Barn remained and was converted to be a stable by Henry Oakes in the 19th Century. The Barn then became a garage and the entrance to Guildhall Street was closed in the 20th Century.
Oakes was born in 1742. In 1757, at 16 after his education at the King Edward VI Grammar School, he was apprenticed to his Uncle Orbell Ray into the yarn industry.
In 1768, aged 27, Oakes inherited Orbell's share of the Ray yarn-making business, one of the largest in England.
In 1778, Oakes began his diaries which he kept for the next 49 years. These provide an important history of the town, politics, business and trade. The diaries are written in a very factual style highlighting Oakes' social and business dealings. He recorded the cost of his undertakings, including the cost of meals and journeys to London by coach. They span half a century covering the American Revolution and the Great Wars with revolutionary and Napoleonic France.
In the late 1700's Oakes was appointed as Receiver General of the Land Tax for West Suffolk during the time of the decline and ending of his yarn business. Given his then availability of cash, he became a banker. Oakes' money notes include his family crest and motto. The oak tree pictured on the note inspired the Oakes Barn Logo.
In 1827, James Oakes became unable to continue to write his diaries as he was nearly 86, arthritic and deaf, and in 1829 Oakes died. He had been five times Alderman of the town, Deputy Lieutenant of Suffolk, Justice of the Peace, Turnpike Trustee for Thetford and Sudbury roads, a Guildhall Feoffment and Governor of King Edward VI Grammar School.