Rovers produced a great display at the Mem on Tuesday evening, but were dealt a cruel blow when Peterborough's Joe Ward thundered home a 30-yard equaliser in the 90th minute.
Tom Nichols got Rovers off to a great start when he drove into the box and won a penalty, before stepping up and dispatching it into the top left corner.
Perhaps against the run of play and amidst some offside controversy, United levelled when Ivan Toney rounded Bonham and found the empty net.
Just before the break, Ollie Clarke rounded off an exciting first half and restored the hosts lead with an exquisite 25-yard lob.
There was predictably waves of pressure from Boro as they sought to salvage a point, and Rovers defended valiantly but were powerless to resist a screamer from substitute Ward in the closing minutes.
Rovers made one change going into the game and it was in the shape of Liam Sercombe, who returned to the starting XI at the expense of Kyle Bennett.
New arrival Gabriel Osho started the game on the bench.
Graham Coughlan's side made a really positive start to the match, as they applied pressure to the Posh's defence from the off, with Nichols trying his luck and Sercombe seeing a dipping volley go just over the bar.
O.Clarke was in the thick of it from the get-go and a powerful run into the left side of the box saw him lay off a nice pass for Joe Partington on the edge of the box, his shot was on target but it was blocked.
Rovers' strong, early showing thankfully paid dividends with just eight minutes on the clock.
Ex-Boro man Nichols looked fired up as he faced his former employers, no more so when he carried the ball excellently some distance in the penalty area, and checked inside smartly before he was brought down and awarded a penalty.
The striker then took charge, placed the ball on the spot and rifled it home into the top left corner to give Rovers the lead.
The visitors were gently trying to grow into the game and did enjoy a bit more possession in Rovers half - there was a heart in mouth moment when Lee Tomlin's miss-hit cross flew just marginally past Jack Bonham's post.
Bonham had his first bit of work to do on 26 minutes, as Toney worked some space on the left in the area and shot across goal but the Gas stopper parried it behind.
Rovers were probably playing some of the best football at the Mem so far this season, and caused continual problems for Peterborough's backline with their link up and movement; typified with a lovely give and go between Rodman and Lines down the right that saw the former surge into the box, he had cut back options but perhaps a bit selfishly went for goal and could only find the side netting.
Despite that, a sucker-punch against the run of play was dealt with little less than 10 minutes left on the clock.
Ex-Bristol City Tomlin disguised a cute pass on the edge of the area through Toney, Bonham got out well to challenge at the forward's feet but the luck belonged to the Peterborough man as the ball fell kindly for him before he stroked it home into an empty net - huge protests of offside were ignored by the officials.
However, with the interval fast approaching there was to be another big moment and it went Rovers way.
After a bit of a mix up from Peterborough's defence forced by good pressure from Reilly, the ball popped out to O.Clarke about 25-yards out, he noticed the keeper off his line and calmly lifted the ball over him and into the back of the net to claim his sixth goal of the season.
Understandably after the break, the Posh looked to come straight out and get on the front foot and nearly found an equaliser when Matt Godden was put through on goal, he struck powerfully before Bonham produced a fine reflex save to deny him.
64 minutes gone and Rovers still held their slender lead, but it was not plain sailing as Boro maintained a threat, with Toney continuing to try and get in behind and almost doing so as he latched onto a lovely chip over the top, but his touch let him down and Bonham swept up.
Rodman had been excellent down the right flank all evening and was still causing issues in the 70th minute, as he marauded down the wing and fired an inviting cross across the face of goal that sadly could not be met by someone in blue and white.
Shortly after, Coughlan made his first change as he brought on Alex Jakubiak in place of Nichols.
Rodman again was proving an outlet for Rovers down the right side, this time spinning off his man onto his left foot before firing over the target.
A second substitution came on the 84th minute, as James Clarke came on in place of Reilly to sure things up at the back - fair play to the Scotsman who had ran his socks off all evening.
Rovers looked to have done enough to claim all three points, until Ward stepped off the bench to arrow home a 30-yard screamer in the 90th minute as Coughlan's men were dealt a real sucker punch.
Rovers XI: Jack Bonham (GK), Joe Partington, Tom Lockyer (c), Tony Craig, Tareiq Holmes-Dennis, Ollie Clarke, Chris Lines, Liam Sercombe, Alex Rodman, Tom Nichols (Alex Jakubiak, 70'), Gavin Reilly (James Clarke, 84').
Subs: Adam Smith (GK), Gabriel Osho, Ed Upson, Sam Matthews, Stefan Payne.
Peterborough United: Conor O'Malley, Jason Naismith, Alex Woodyard (C), Ben White, Matt Godden (Siriki Dembele, 74'), Callum Cooke, Rhys Bennett, Ivan Toney, Danny Lafferty, Lee Tomlin (Marcus Maddison 65'), Kyle Dempsey (Joe Ward 65').
Subs: Aaron Chapman, Louis Reed, George Cooper, Matthew Stevens.