One in eight Brits worry about debts all the time and 20% often worry about the money they owe on loans and credit cards.
Only 6.5% of people who owe money said they never worry about it, according to a new study the Debt Advisory Centre.
More than half of borrowers who worry about debts say it keeps them awake at night, while more than a third say it's putting a strain on their relationships and their friendships.
More worrying is how money trouble affects people's well-being, with a 31.7% of British adults believing it has had a detrimental effect on their mental health and a similar number (28.7%) considering it to have affected their physical health.
Younger people tended to be more concerned about debts, with 17.9% of those aged 25 to 34 most likely to constantly worry about debt.
There was no significant drop until people reached their mid-50s. In this age group, only 5.9% worried about debt.
Ian Williams of the Debt Advisory Centre said: "It's clear a large number of us feel under stress because of our debts. While it's natural to worry about money from time to time, if this has become a regular occurrence it's a real concern.
"It is particularly worrying that many of those with debts feel their health and well-being is being affected by it. If you can't sleep at night because the stress of debt is keeping you awake or if you think it's causing your relationships to suffer, then it is probably time to seek expert debt help."