University dress includes an academic cap, or mortarboard, with a black tassel. At Oxford and Cambridge from around the 1600s the titled young undergraduates began to wear gold tassels, known as tufts, as a mark of their superior status.
As often happens with language, the word’s usage was extended to the young men themselves. It gradually shifted to tofts and by the mid-1800s was toffs. It became a slang word used by the working class for the upper class, or for someone who dressed smartly, as an aristocrat might.
The nowadays rarely-heard tuft hunter, for a sycophant or toady, has its origins here too. A tuft hunter was someone who followed, flattered and fawned before these young noblemen.