Gradi's father died when he was very young and Gradi, along with his English mother, moved to London after the end World War II. He had an undistinguished spell as a professional footballer, playing for Sutton United and Tooting & Mitcham United before turning to a career in management. His first managerial position was as assistant coach at Chelsea in 1971.
From 1971 to 1983, he coached at several other clubs, including Sutton United, Derby County, and Wimbledon as an assistant coach, then as a full manager at Wimbledon and Crystal Palace.
In 1983, he moved to Crewe Alexandra, a club that had spent the previous two decades toiling in the Football League's bottom division. He led them to promotion in 1989. They were relegated two years later, but were promoted again in 1994 and then advanced to the second-tier Division One in 1997, where they remained until 2002.
Gradi retired from management on 1 July 2007 after 24 years and 1,244 matches in charge of Crewe Alexandra. At the time of his retirement, he was the longest-serving manager in English football. In recognition of his contribution to the game, he was admitted in 2004 to the English Football Hall of Fame.
He is currently Crewe Alexandra's director of football.