Lewis Grabban returned to haunt his former club, with the former Canaries striker finding the net twice either side of summer signing Aiden McGeady's fantastic first-time effort.
The Black Cats moved onto four points after two Championship games, following their opening day draw with Derby last week, but manager Grayson has already set his sights on their next challenge.
He said: "This is a tough place to come to and we knew they would be putting us under pressure in their first home game of the season, but I thought the the lads stuck to the game plan really well
"Obviously when you defend well and then take your chances really well to go 3-0 up, you have got to be absolutely delighted.
"We were disciplined and well organised, sitting back and letting them have the ball in certain areas and then moving in when we had to.
"I was also really pleased with the finishing
Lewis is always going to cause sides problems and at this level, while Aiden is probably the most talented player I have ever worked with and is looking good value at only £250,000.
"It's a great win and it has been a strong-ish start for us - but that's all it is, a good start, and no-one is getting carried away.
"What has happened here is in the past - it's all about what happens in the future now and what I will say is that I have got a good, hard-working group here - it's a working class city and that's what the people demand."
Sunderland had to withstand plenty of pressure before taking the lead against the run of play after 27 minutes, with Grabban latching onto a flick from another former Norwich player James Vaughan before guiding the ball past Angus Gunn
The lead was extended on the hour mark when McGeady blasted home a loose ball from the edge of the box, and 11 minutes later the points were in the bag as Grabban headed home McGeady's superb cross at the back post.
Grabban later guided the ball into his own net while straining to clear a dangerous ball into the six-yard box - but by then the damage had been done.
Norwich boss Daniel Farke was disappointed with the result - but not the performance of his side.
"I thought it was a really good performance; we had 75 per cent of possession in the first half, just over 70 in the second, it was 13-1 in corners and we had a lot more chances than them.
"But in this game it doesn't matter about your philosophy or your performance, it is about the results and it is down to the small things that we lost this game.
"For the goals, we defended sloppily and that cost us
For the first goal we failed to deal with their goal-kick and for the third we should have stopped the cross coming in.
"It is very frustrating but the fact that we dominated a game against a side who have only just come down from the Premier League is a good sign.
"They were well organised and sat deep and it's not easy to score goals against a team like that
This just showed that you can always lose a game when you are not awake defensively for the full 90 minutes."
Source : PA
Source: PA