Men have been steadily dropping out of the workforce — especially men ages 25 to 54, who are considered to be in their prime working years. About 10.5% of men in their prime working years, or roughly 6.8 million men nationwide, are neither working nor looking for employment, compared with just 2.5% in 1954. So what exactly is driving men out of the workforce?