Answer
Paper punched cards is the answer. These were the primary form of computer data storage well into the '60s.
Babages "Difference Engine" was programmed using punched paper cards. The cards were strung together to form a kind of chain or "tape". As Gary inferred, he got this idea from the Jacquard Loom.
Although Turing's work was entirely theoretical, his hypothetical machine assumed an infinite "tape" composed of cells which could be read or written.
Hollerith's cards were not attached. This allowed them to be sorted. Hollerith's Tabulating Machine Company built machines to tabulate census data. The company name was later changed to International Business Machines (IBM).