Alex Jakubiak came off the bench to score a superb solo goal one minute from time to send Rovers through to the next round of the Checkatrade Trophy.
After a fairly even first half, Swansea took an early advantage in the second half, as Jordon Garrett fired home a precise finish to open the scoring.
Rovers responded well, as just five minutes later Tony Craig headed home an equaliser from an Ed Upson free-kick.
The game looked destined to end in a draw and penalties but it was settled by fine individual goal from substitute Jakubiak, who scored in the 89th minute to book Rovers a place in the third round of the comeptition.
Heading into the tie in South Wales, Rovers boss Darrell Clarke named a strong starting XI to take on the youthful Swansea side.
Jack Bonham started the encounter in goal, commanding a back-four that included James Clarke, Michael Kelly, Daniel Leadbitter and Tony Craig, with the captain's armband handed to Craig for the evening's proceedings.
The front line for Rovers resembled the Payne and Nichols partnership, while Rodman and Bennett were included to occupy a threat out wide.
Inside the early stages of the game, both sides enjoyed spells of possession to ease into the contest - with Rovers registering an early shot through Stefan Payne, but Benda in the Swansea goal wasn't troubled by the striker's attempt that arrived at the goalkeeper at a comfortable height.
At the other end, Jack Bonham produced a fine stop to push clear a close-range effort from Joel Asoro, after the frontman navigated his way into the Rovers area to make contact with a Jordan Carrick cross at the far-post.
With the game progressing into quite an open contest, Rovers started to enjoy the fair share of chances on the Swansea goal. An adventuring break forward by Daniel Leadbitter created the opportunity to launch a cross into the mix - as a smart delivery from the right-back was glanced over the crossbar by a powerful header from Stefan Payne.
Leadbitter continued to cause the hosts problems down the right, as the speedy right-back was hauled down for Rovers to gain a free-kick in a promising position. Ed Upson was tasked with taking the set-piece and curled a teasing ball into the crowded area, as James Clarke glanced a header wide of the upright after beating his marker in the air.
The flow of chances continued for Rovers inside the opening twenty-five minutes - a cushioned header back across the box from Alex Rodman fell to Tom Nichols, as the striker witnessed his volley at the near post deflected wide of the target.
There was a credible shout for a penalty for Rovers as the game approached the half-hour mark, as Daniel Leadbitter was tripped by Arran Lewis on his way into the Swansea penalty area. Casting judgement on the incident from a good vantage point, the referee deemed that the original contact was outside the area, awarding Rovers a free-kick on the very edge of the penalty box.
Perhaps the best chance of the first-half fell Rovers' way, as a cross from Leadbitter bounced around the Swansea area and was turned towards goal with a powerful volley from Alex Rodman, with Benda reaching high into the top corner to tip the effort from the winger wide.
As the game edged towards the break, a neat demonstration of footwork from both Sercombe and Bennett allowed the latter to carve out a chipped cross, which was directed over the target by a bullet header from Stefan Payne.
The start of the second half harboured a slight lull, as both teams took a period of time to ease back into the tie, with a slight knock for Alex Rodman causing brief pause to the tie as the winger was treated and patched up a bandage.
The game was sparked into life moments later, as Rodman broke deep into the final third and fizzed a low shot across the face of goal that was touched onto the upright by Tom Nichols sliding to meet the shot-turned-cross at the far-post, with the hosts able to steer away any chance of a Rovers player launching a rebound.
Swansea pushed forward and grasped the opening goal of the encounter shortly after Nichols' attempt. A quick break forward saw Yan Dhanda mask a neat backheel into the path of Jordon Garrick, who turned home a finish past Jack Bonham.
The lead for the hosts was shortlived, as just five minutes later Rovers were back on level terms as Tony Craig guided a delicate header into the Swansea net from a well-worked set-piece from Ed Upson - with the stand-in captain doing well to battle his way to the delivery inside a congested penalty area.
Rovers looked to take full advantage of the momentum that had swung their way, as a good show of skill from Tom Nichols drew in a foul twenty-five yards out from goal. Liam Sercombe stood over the resulting free-kick and opted to take the low option, with a driven strike flashing narrowly wide of the left-hand post.
Swansea did pose the odd threat on the Rovers goal as the game entered the final quarter, with Aaron Lewis firing into the side netting with a low effort from twenty yards out.
There was chances for a late deciding goal, as Rovers resumed control of the game - Nichols did well on the left to wriggle through challenges and into the box before setting his strike partner close in, but Payne's path was blocked by plenty of white shirts as his shot flew wide.
Shortly after that, D.Clarke made a double change, swapping his existing strike force of Payne and Nichols for the fresh legs of Gavin Reilly and Jakubiak.
The visitors continued to look the most likely to find a winner, Kelly drove a menacing half-volley that looked to be heading for goal, but it was blocked by a Swans defender and was followed by shouts for handball; dismissed by the referee.
With the game seemingly heading to penalties, up stepped Jakubiak.
The striker picked the ball up in the middle of the pitch and drove forward, he had Reilly to his left but used him as a decoy before surging past one, then another, into the box and one-on-one, he looked up and slotted underneath the onrushing keeper to send his side into the next stage of the competition.
Rovers XI: Jack Bonham (GK), Daniel Leadbitter, James Clarke, Tony Craig, Michael Kelly, Ed Upson, Liam Sercombe, Alex Rodman, Kyle Bennett (86' Sam Matthews), Tom Nichols (74' Gavin Reilly), Stefan Payne (74' Alex Jakubiak).
Subs: Adam Smith (GK), Rollin Menayese, Tareiq Holmes-Dennis, Luke Russe.
Swansea XI: Steven Benda (GK), Joel Asoro (79' Botti Biabi), George Byers, Adnan Maric, Yan Dhanda, Aaron Lewis, Jordon Garrick, Tyler Reid (62' Jordi Govea), Cian Harries, Brandon Cooper, Simon Paulet.
Subs: Gregor Zabret (GK), Liam Cullen, Oli Cooper, Adam King, Joe Lewis.
Attendance: 371 (82 away).