Rovers' second-half fightback proved in vain as they bowed out of the Checkatrade Trophy.
Manager Grant McCann rotated his side, with Shaun Cummings making his debut at right-back after signing a two-month deal earlier in the week.
Rovers started well, with Matty Blair having the first effort on goal in the opening minute, but the home side went ahead through Kristian Dennis after 16 minutes.
That prompted a response from Rovers, with Ross Fitzsimmons having to be alert to deny Alfie May then Will Longbottom.
Danny Amos almost got his first goal for the club when he hit the crossbar from Blair's cross,
Fitzsimmons was quick to rush out and block May's effort when the forward nipped in to get a toe on a loose header from defender Richard Duffy.
But Rovers were hit by a sucker punch when Dennis turned provider for Enzo Boldewijn to score and put Notts 2-0 up.
Rovers were still creating all the chances, with Ali Crawford heading just wide from a Cummings cross then stinging the palms of Fitzsimmons after controlling the ball well.
But Dennis flicked home a header just before the break to give Notts a 3-0 lead in a match that could easily have been level.
Rovers made two changes at the break, bringing Niall Mason for Cummings and Mallik Wilks for Jermaine Anderson.
They needed an early goal to get back into the game – knowing a single-goal defeat would still be enough to progress to the knockout stages on goal difference - and got that five minutes after the restart when Alfie May smashed home a cross from Amos.
May and Mason both went close to getting a second for Rovers before the latter was judged to have pulled down a player when defending a corner.
Dennis stepped up to take the penalty, and calmly slotted the ball home into a bottom corner to complete the hat-trick and show the form that saw him claim 19 goals for Chesterfield the previous season.
That left Rovers with a mountain to climb, but May grabbed his second of the game with 20 minutes to go when he headed home after Wilks' cross - the on-loan Leeds man had done brilliantly to create space on the left wing.
Rovers pushed for the strike that would have sent them through, with Wilks bringing a great save out of Fitzsimmons after running the length of the pitch on a counter-attack.
Long bottom and Anderson also went close, but in the end Rovers could not find a crucial third goal.