James W. 'Red' Holloway was born in Helena, Arkansas, on May 31, 1927, to a mother who played piano and a father who played violin. At the age of five, he moved to Chicago, where he graduated from DuSable High School and attended the Conservatory of Music. During grade school, Red played banjo and harmonica in school bands.

His first musical instrument, a tenor, was given to him by his stepfather when he was twelve years old. During high school, he was in the DuSable big band where he sat next to fellow classmate and reedman Johnny Griffin. At sixteen and while still in school, he was hired for his first professional engagement by bassist Gene Wright, who later went on to become a member of the Dave Brubeck Quanet. Red played with Wright's Big Band for three years at the Parkway Ballroom and then, when he was nineteen, joined the Army, where he became Bandmaster for the U. S. Fifth Army Band. After completing his military service, Red returned to Chicago and played with Yusef Lateef and Dexter Gordon, among others. In 1948 he was asked by blues vocalist Roosevelt Sykes to join Sykes' U. S. Road Tour. During this time, other bluesmen heard him and subsequently hired him, including  Willie Dixon, Junior Parker, Bobby ~BIue~ Bland, Lloyd Price, John Mayall, and B. B. King. Because of these associations, Red became typecast, perhaps unfairly, as primarily a blues player. During the Fifties, Red continued playing in the Chicago area with such notables as Billie Holiday, Muddy Waters, Chuck Berry, Ben Webster, Jimmy Rushing, Arthur Prysock, Dakota Staton, Eddie ~Cleanhead~ Vinson, Wardell Gray, Sonny Rollins, Red Rodney, Lester Young, Joe Williams, Redd Foxx, Aretha Franklin, and many others. During this same period, he also played road tours with Danny Overbee, Sonny Stitt, Memphis Slim, Lefty Bates, and Lionel Hampton.