The Black Cats saw their hopes of a second successive extended Capital One Cup run dashed on Tuesday night when Stoke came from behind to win 2-1 at the Stadium of Light and book their place in round four at the expense of last season's beaten finalists.
Gus Poyet's men have now played seven games in all competitions so far this season, but have won only one of them, at Sky Bet Championship Birmingham in the previous round of the League Cup, and while that does not represent a crisis, Altidore and his team-mates know it cannot continue if they are to prosper.
The United States international said: "I didn't think we deserved to lose and I don't think we played that badly, but this has haunted us the whole of last season and a bit this year.
"Games we should win, we're not winning
It's something that we need to address now, or else it's inevitable that what happened last season - and the pressure - will come back.
"If it's close and you're going to lose the game, you want to look back and be applauding the other team for a great strike
We're still losing games where we're leaving ourselves short.
"We deserve better - and it's not just saying, 'This guy is playing badly'
We need to be collectively better
We understand that, and we've got home games coming up where we've got the chance to do that."
No-one on Wearside needs reminding of the plight in which Sunderland found themselves last season, and while a remarkable run-in - which saw them win at Chelsea and Manchester United and draw at Manchester City to eventually finish in 14th place in the Barclays Premier League - was hugely creditable, it was simply too close for comfort.
Poyet, along with sporting director Lee Congerton, spent the summer investing owner Ellis Short's rebuilding fund carefully, and while he was not entirely satisfied by the business he was able to do, he is confident that a workable plan is in place.
That will not now, however, include a repeat of last season's glorious League Cup run, which only ended in heroic defeat at the hands of Manuel Pellegrini's international all-stars at Wembley.
A disappointed Altidore said: "The cup is a big deal and I enjoy playing in these games because you have the chance to win something.
"It also keeps the squad fresh, which is important
We have a squad where everybody is challenging for places.
"When you've only got one game a week, it's difficult for the manager and difficult for the players to stay fit, so it's a bit of a shame
But it makes training much more important and we can look forward to the FA Cup."
For Altidore himself, there was a chink of light after he opened the scoring with his first goal since December and just his third in total for the club since his £6.5million switch from AZ Alkmaar in July last year.
The 24-year-old said: "It's good to get it away, but it's difficult
We have three strikers and we're having trouble getting those chances to score."
Altidore's efforts, however, were ultimately in vain and it was Stoke's Marc Muniesa who proved the unlikely match-winner with a double, his first goals for the club since moving from Barcelona.
Manager Mark Hughes said: "He has done well, he is progressing well
He's still a young man in terms of football and his age, and he is just gaining experience of the Premier League and what it asks of you as a player.
"We are delighted how he is progressing
He was excellent towards the back end of last year, so we are pleased with him."
Source : PA
Source: PA