Although the introduction of the Bagnall locomotives and the traffic generated by the works at Waddesdon Manor had boosted the route's fortunes, it remained in serious financial difficulty. The only connection with the national railway network was by way of the turntable at Quainton Road. The 3rd Duke of Buckingham chaired the Aylesbury and Buckingham Railway but its management regarded the Tramway as a nuisance. In the 1870s it charged disproportionately high fees for through traffic between the Tramway and the main line with the intention of forcing the Tramway out of business. Relations deteriorated between Jones and J. G. Rowe, Secretary and Traffic Manager of the A&B. The A&B's trains at Quainton Road would miss connections with the Tramway, causing milk shipped to Quainton to become unsellable, to the extent that Jones began unloading milk at Waddesdon and shipping it to Aylesbury by road. Although Jones asked the Duke to intervene relations remained poor; in 1888 Rowe blocked the telegraph along the Tramway, and in one meeting Jones and Rowe threatened violence.
Jones sought legal advice and was told that the Duke would probably win a legal action against the A&BR. However, the A&BR was in such a precarious financial position that any successful legal action against them would likely have forced the line through Quainton Road to close, severing the Tramway's connection with the national network altogether. Meanwhile local dairy farmers began to switch to beef and butter, causing a drop in milk transport. From its peak of 20,994 tons carried in 1877, goods traffic fell in each of the next four years, dropping to 9,139 tons (9,286 t) in 1881.