Born in Oneonta, New York, Sherman Fairchild was the only child of George Winthrop Fairchild (1854–1924) and Josephine Mills Sherman (1859–1924). His father was a Republican Congressman as well as a co-founder and the first Chairman of IBM. His mother was the daughter of William Sherman, of Davenport, Iowa.
His father died on December 31, 1924, and as an only child he inherited his father's multimillion-dollar estate. He also inherited his father's IBM stock, becoming IBM's largest individual stockholder until his death in 1971.
Known to be a particularly bright and naturally inquisitive child, Sherman Fairchild matriculated at Harvard University in 1915 where, in his freshman year, he invented the first synchronized camera shutter and flash. During these college years he also contracted tuberculosis and under the advice of his physician, moved to Arizona to recover in the drier climate. He then transferred to the University of Arizona, where he became increasingly interested in photography. He would later transfer to Columbia University in New York. Due to his ongoing medical problems, Fairchild did not earn a degree from any of these schools. Instead he pursued his desire to become an entrepreneur.
Having never been married or having any children, he spent his time exploring a variety of interests. Aside from maintaining his companies, he enjoyed architecture, cooking, jazz, dancing, philosophy, and tennis. He always kept an eye out for opportunities to create or improve upon existing technology or capabilities.