The centre, which is situated in Charles Street, is set to close its doors next January.
The decision to close the centre comes at a time when councillors have voiced their concerns over cuts being made to facilities in the town.
Town mayor Ian Plant said: "The county council has not been taking referrals for people to go to the day centre, so how can they say people are not coming in.
"There has been no money put into Cheadle Day Centre. This is a sad day as we are losing yet another service and where will people go? The building will also become virtually empty."
Councillor Ron Locker also said that the county council had blamed dwindling numbers but they had not been taking referrals for some time.
He said: "This is another sad loss for Cheadle.
The Staffordshire Moorlands seems to be taking the brunt of the cuts.
"As taxpayer we are getting a poor deal. These facilities are so important to people and their carers.
"It is easy to close down premises, but we never hear what is taking their place, so what will happen to the people using Cheadle Day Centre?"
Councillor Stephen Ellis said: "I'm saddened the Cheadle Day Centre is going to close as it's provided many years of excellent service to Cheadle.
"Changing care patterns and supporting people in their own homes has reduced numbers and appears to have made the centre unviable. It's a really sad loss for our town."
The county council stated the closure was due to the poor buildings and the dwindling number of people attending.
Robbie Marshall, Staffordshire County Council's cabinet member for health and wellbeing, said: "As a county council we are committed to providing the best service possible for older people and the reality is that the Leek and Cheadle centres are underused and no longer fulfil the requirements of residents.
"Both buildings are in poor condition and as attendance has dwindled we feel they no longer provide the best environment to support people.
"Cheadle has a capacity for 25, but on average only nine people attend and in Leek there is room for 30 people, but on average it is less than half full.
"As demonstrated by the falling attendance many people feel the centre no longer benefits them and feel more tailored, flexible support, better suits their requirements in 2013.
"Over the coming weeks we will be working with those who use the day centres and their families to find alternative support in the community."
County councillor for the town, Mark Deaville, said at the last county council meeting he had asked for suitable places to be found for the users before the day centre closed.
He said: "I was assured that the county council would honour it's long standing commitment that no service would cease until all people currently using a service have found suitable alternative.
"I was also told that two reputable organisations, Moorlands Home Link and Approach already offer high quality day support in Cheadle and both were keen to support more people.
"As a local member I will be banging the drums for the people and their carers.
"We have got to find better alternatives before closure as we place importance on the older people of Cheadle."