Rovers registered a victory on the road thanks to a brace from Billy Bodin and a goal from defender Tom Lockyer.
On his 200th appearance for the club, Tom Lockyer made it a day to remember by opening scoring at Gigg Lane.
In an equal first half, both sides had chances to break the deadlock, as a fine save from Sam Slocombe denied Harry Bunn, as Tom Nichols' fine first-time effort at the other end forced Joe Murphy into making a vital stop.
It was Lockyer's first goal since February 2015 that allowed Rovers to take the lead, as defender produced a clean header to take advantage of Lee Brown's set-piece.
With just over fifteen minutes left on the clock, Billy Bodin leaped in a congested area to double Rovers lead to head home a Joe Partington cross, scoring his third goal of the season in the process.
Bodin added Rovers' third goal in the final moments, as two late goals from hosts pulled the goal deficit back to just one-goal, as Rovers registered a victory on the road.
Heading into the game, Darrell Clarke included Sam Slocombe in goal, as the 'keeper became the 900th player to make his debut for Rovers. The game also saw another milestone, with Ollie Clarke starting his 100th fixture for the club.
On the bench, development 'keeper Alexis André Jr was joined by Ryan Broom, Jonny Burn and loanees Dominic Telford and Marc Bola.
In the opening moments, the game went through a frantic period, as both teams have the chance to open the scoring in the opening minute. A dangerous cross from the right for the hosts almost found Greg Leigh at the far-post, who took a tumble when attempting to apply a close-range finish, as the referee quickly signaled for play to resume. Rovers broke immediately from that move, with Tom Nichols deflecting the ball into the path of Liam Sercombe, who was able to take numerous touches inside the area, but the ball never sat kindly for the midfielder to hit and effort either side of the advancing Joe Murphy.
It was Billy Bodin who registered the first real chance of the game for Rovers, as his trademark footwork on the edge of the area opened up a path to goal for the forward, who unleashed a powerful effort that bounced narrowly past the post.
A little after ten minutes, the hosts thought that they had broken through the Rovers defence, as a quick counter-attack from a Rovers corner saw Nicky Ajose played through on goal, but the striker was correctly flagged for offside, after an anxious few moment for the Rovers back-four.
On his debut for the club, Sam Slocombe's first save came just before the thirty-minute mark, as a cross from the right found youngster Harry Bunn open at the far-post. The former Manchester City and Huddersfield player was able to take a touch and force a powerful effort on target, but the Rovers shot-stopper was alert to dive to his right and make a fine reaction save to tip the ball around the post.
Both 'keepers came out on top during the first-half, after a flowing move down the left by Rovers allowed Tom Nichols to direct a well-struck effort on goal that had to be deflected away by the outstretched legs of Joe Murphy, who advanced quickly off his line to close down the striker's vision of the goal.
In the final minutes of the opening forty-five minutes, a driven cross from Lee Brown wasn't dealt with by the hosts, who had to throw themselves at efforts from Chris Lines and Ollie Clarke, both of which looked destined to hit the target.
During the interval, the hosts made a double change, as Tsun Dai and Chris Humphrey were introduced for Craig Jones and Callum Styles, as Rovers remained unchanged.
It took just ten minutes of the second half for Rovers to open the scoring, as a floating corner from the left by Lee Brown was delivered into a dangerous area, with defender Tom Lockyer rising to send a thumping header past Joe Murphy, with the ball just clipping the post on its way into the back of the net.
The goal spurred Rovers into life, with Liam Sercombe taking advantage of being given space outside the area and connecting with a fierce shot that curled fractionally off target and wide of the post.
Rovers managed to double their lead, as a cross from the right by Joe Partington was delivered into a congested area, where Billy Bodin was able to make a well-timed run to attack the ball and direct a header past Joe Murphy.
Bodin added his second of the game just five minutes left on the clock, finishing a twisting run into the box with a finish that wrong-footed 'keeper Joe Murphy, burying the ball into the bottom corner.
Jermaine Beckford reacted the quickest in the area to head home for the hosts in the 89th minute, as a further late goal from Nicky Ajose led to a tense final few moments for Rovers, who were able to hold onto record the victory.
Next up for Rovers is a trip to Craven Cottage to take on Fulham in the Carabao Cup on Tuesday evening.
Bury F.C: J.Murphy, C.Jones (45' Humphrey), G.Leigh, J.Beckford (C), N.Ajose, C.Reilly, T.Aldred, A.Bruce, A.Whitmore, C.Styles (45 Dai), H.Bunn
Subs: A.Thompson, P.Edwards, Z.Ismail, L.Fasan (GK), R.Lowe
Rovers: S.Slocombe, L.Brown, T.Lockyer (C), R.Sweeney, J.Partington, O.Clarke, L.Sercombe, C.Lines (71' Sinclair), B.Bodin, T.Nichols (81' Telford), E.Harrison (71' Gaffney).
Subs: A.Andre (GK), J.Burn, M.Bola, R.Broom.
Attendance: 3,402 (546 away)