Polybrominated diphenyl ethers (PBDEs) can be transferred to infants through the ingestion of breast milk, resulting in potential health risk. In this study, PBDEs, hydroxylated polybrominated diphenyl ethers (OH-PBDEs) and 2,2′,4,4′,5,6′-hexachlorobiphenyl (CB-153) in human milk from women living adjacent to e-waste recycling sites of Wenling, China, were investigated. The median level of PBDEs in samples from residents living in the e-waste recycling environment > 20 years (R20 group, 19.5 ng/g lipid weight (lw)) was significantly higher than that for residents living in Wenling < 3 years (R3 group, 3.88 ng/g lw) (p < 0.05), likely ascribable to specific exposure to PBDEs from e-waste recycling activities. In the R20 group, most congeners (except for BDE-209) were correlated with each other (p < 0.05). Moreover, CB-153 showed significant association with most PBDE congeners, rather than BDE-209. The relationship indicated that most BDE congeners other than BDE-209 shared common sources and/or pathways with CB-153, e.g., dietary ingestion. The correlations between BDE-209 and other congeners were different in the two groups, likely suggesting their different exposure sources and/or pathways for PBDEs. Although estimated dietary intake of PBDEs for infants via breast milk was lower than the minimum value affecting human health, the PBDE exposure of infants should be of great concern because of their potential effect on the development of neonates over long-term exposure. OH-PBDEs were not detected in the collected samples, which is in accordance with reports in published literature, likely indicating that they were not apt to be accumulated in human milk.