The home side, who play in the Blue Square South league, two divisions below League Two, were 3-0 up within 37 minutes and withstood a Swansea fightback to secure a dream trip to Anfield.
An own goal by Garry Monk put them ahead before Jamie Collins (25) and Rocky Baptiste (37) made it 3-0.
Swansea got one back through Guillem Bauza just two minutes later, and they should have been within one goal but for a Leon Britton penalty to be saved.
Jason Scotland reduced the deficit to 3-2 just three minutes after the break, but a header from Tom Jordan with 25 minutes to go secured a famous victory.
In other replays, Newcastle United, with new manager Kevin Keegan watching in the stands, booked their place in round four with a 4-1 win over Stoke City despite the 29th-minute sending-off of Emre.
Michael Owen put them ahead, and Claudio Cacapa, James Milner and Damien Duff completed the scoring for the home side, who announced the return of Keegan earlier Wednesday.
A header from Brazilian Elano 18 minutes from time gave Manchester City a 1-0 win over West Ham United, while Hereford United beat Tranmere Rovers 1-0.