The Coventry City striker - subject of at least three bids from Birmingham boss Steve Bruce - struck with 12 minutes remaining to send Niall Quinn to defeat in his first game as Sunderland manager.
Coventry had already come from behind through a goal from Stern John after Daryl Murphy had given Sunderland the lead in the 52nd minute, and probably would have settled for a point.
But McSheffrey had other ideas. He collected a quickly taken free-kick from Don Hutchison, cut into the area and unleashed a shot which took a deflection on the way into the net.
It had looked as if Republic of Ireland international Murphy would make it a victorious start for the relegated Black Cats.
The 23-year-old striker capitalised on some poor home defending to slide in a goal which might have brought his reluctant manager three welcome points.
Quinn, who headed a consortium which bought Sunderland in the summer, kept a typically low profile in the dugout as his side took the lead.
The goal may not have been eye-catching but it was in keeping with the quality of the game.
Grant Leadbitter's 52nd-minute corner bobbled about inside the Coventry area before it was hooked back across the box allowing Murphy to get in front of his marker and beat Andy Marshall in the Coventry goal.
City were left appealing for handball but they had failed to deal with the flag-kick and could not have any complaints.
The Sky Blues' equalising goal underlined the class of John.
The Trinidad & Tobago international was expected to move on from the Ricoh after his exploits at the World Cup but no offer was forthcoming.
And how glad, on this evidence, City manager Micky Adams will be about that.
There was little on when John collected the ball on the edge of the area, but he turned and curled a sublime shot in past Ben Alnwick.
John could have added to his tally, but failed to convert two good chances, but at the other end Coventry coped comfortably and held on to take the points following McSheffrey's strike.