His time at the Northern Railway was marked by conflict with the architect Frederick William Cumberland, with whom he started the Canadian Institute and who was general manager of the railway until 1855. Starting as assistant engineer in 1852, Fleming replaced Cumberland in 1855 but was in turn ousted by him in 1862. In 1863 he became the chief government surveyor of Nova Scotia charged with the construction of a line from Truro to Pictou. When he would not accept the tenders from contractors that he considered too high, he was asked to bid for the work himself and completed the line by 1867 with great savings to the government and at profit to himself.
By 1871, the strategy of a railway connection was being used to bring British Columbia into federation and Fleming was offered the chief engineer post on the Canadian Pacific Railway. Although he hesitated because of the amount of work he had, in 1872 he set off with a small party to survey the route, particularly through the Rocky Mountains, finding a practicable route through the Yellowhead Pass. One of his companions, George Monro Grant wrote an account of the trip, which became a best-seller. By 1880, with 600 miles completed, a change of government brought a desire for a private company to own the whole project and Fleming was dismissed by Sir Charles Tupper, with a $30,000 payoff. It was the hardest blow of Fleming's life, though he obtained a promise of monopoly, later revoked, on his next project, a trans-pacific telegraph cable. Nevertheless, in 1884 he became a director of the Canadian Pacific Railway and was present as the last spike was driven.