Elliott was born in 1931 in Brooklyn, New York, United States, the son of Florence (Rieger) and Abraham Adnopoz, an eminent doctor. His family was Jewish. He attended Midwood High School in Brooklyn and graduated in 1949. Elliott grew up inspired by the rodeos at Madison Square Garden, and wanted to be a cowboy. Encouraged instead to follow his father's example and become a surgeon, Elliott rebelled, running away from home at the age of 15 to join Col. Jim Eskew's Rodeo, the only rodeo east of the Mississippi. They traveled throughout the Mid-Atlantic states and New England. He was with them for only three months before his parents tracked him down and had him sent home, but Elliott was exposed to his first singing cowboy, Brahmer Rogers, a rodeo clown who played guitar and five-string banjo, sang songs, and recited poetry. Back home, Elliott taught himself guitar and started busking for a living. Eventually he got together with Woody Guthrie and stayed with him as an admirer and student.