The damage was done in the early stages when the Terriers raced into a two-goal lead, thanks to well-taken strikes by Tony Carss and Jon Stead.
Sunderland clawed their way back into it with a powerful header from Kevin Kyle in the 25th minute and wasted a number of chances to level before Peter Jackson's side clinched it in dramatic fashion.
Jackson said: "No-one expected us to win it and I'm delighted. It was a great performance and one of the best cup wins I've experienced."
Andy Booth's header was handled on the line by Ben Clark before Andy Holdsworth converted from close range. Clark was then shown the red card.
It was a strange decision by Walsall official Mark Cooper who had both sides up in arms with some of his officiating.
The sending off ended any hopes Sunderland had of completing a comeback and they will now miss out on the chance to take on one of the big guns in round three.
Carss fired Town ahead in the second minute from a spectacular free-kick, curling a shot past Black Cats keeper Michael Ingham from just outside the box.
Then dangerman Stead made it 2-0 with a superb curling effort in the 20th minute, although McCarthy will feel he was given too much time from Sunderland's back four.
The Black Cats threatened to turn it round, but there was no way back once Clark was sent off. Kyle grabbed a consolation goal in injury time but it was too little, too late for Sunderland as Booth had already added a fourth for the visitors.
The young Huddersfield side deserved the win with a plucky display and can now look forward to a possible round three meeting with a Premiership side.
Sunderland had chances to hit back through Sean Thornton and Thomas Butler when the score was 2-1 and Huddersfield had keeper Ian Gray to thank for keeping them out.
Once Clark was sent off it was fairly easy for Town who could have won by a greater margin.
A disappointed McCarthy commented: "It was my decision to change things and maybe I found out one or two things about my squad."
Sunderland: Ingham, James, McCartney, Bjorklund, Clark, Butler, Whitley, Thornton, Oster, Kyle, Proctor. Subs: Poom, Stewart, Williams, Babb, Leadbitter.
Huddersfield: Gray, Schofield, Mirfin, Scott, Yates, Carss, Worthington, Fowler, Holdsworth, Booth, Stead. Subs: Booty, Senior, Brown, Mattis, Newby.
Referee: M Cooper (West Midlands).