On 18 December 2012, he was called up for the first time by Maidstone United, remaining an unused substitute in their 1–2 defeat away to FC Halifax Town in the FA Trophy second round at The Shay.
In Maidstone’s youth ranks, Pavey scored over 300 goals in three years, including 124 in his final season before on 22 May 2013 joining Millwall on a two-year academy scholarship. He was included in the squad for one match in that Championship campaign, called up by interim manager Neil Harris for their 1–3 home defeat to Leicester City at The New Den on 1 January 2014.
On 26 September 2014, Pavey joined Conference leaders Barnet on a month-long loan, saying “It should be a great experience for me playing against bigger, stronger players and working on my hold-up play”. The following day, he made his senior debut in their 2–1 win over Forest Green Rovers at The New Lawn, replacing Sam Hoskins for the final eleven minutes and earning a yellow card for a foul on David Pipe. He made two further appearances for Barnet, both again as a substitute.
Millwall ended the 2014–15 Championship season with relegation to League One. Pavey was included in their squad for the final game of the season, away to Wolverhampton Wanderers at the Molineux Stadium. He made his professional debut in the 2–4 defeat, coming on in the final minute for fellow debutant Jamie Philpot, who himself had come on as a substitute ten minutes earlier and scored.
On 8 January 2016, Pavey returned to the National League, joining Aldershot Town on a 28-day loan. The following day, he made his first career start and scored his first senior goal in a 3–1 home win over Chester. After two goals in 12 games, including seven starts, his loan was extended for the remainder of the season.
On 16 August, Pavey returned to the National League, this time on a three-month loan with Bromley. He made his debut on the same day, in a 2-2 draw with former club Aldershot Town.
On 9 March 2017, Pavey joined National League South side Dartford on a youth loan for a 28-day period. He scored his first goal for the club in his second match on 21 March 2017 in a 4-0 victory over Margate.
At the end of the 2016/17 season, Pavey was released by Millwall. On 15 June 2017 he joined Welling United.
He played two games for Welling United – without scoring – before returning to Dartford on 14 August 2017.
On 29 April 2018, Pavey was awarded the Vanarama National League South Player of the Season award.
On 25 November 2018, Pavey signed for Dover Athletic.
Alfie Pavey celebrates his 2nd goal for Dover Athletic in the 1-2 win over Chesterfield in the National League, 3rd August 2019. Picture John Hobson/AHPIX LTD, Proact Stadium, Chesterfield.