Heavy metals, pharmaceuticals, and other wastes released into the environment can significantly influence environmental antibiotic resistance. We investigated the occurrence of 22 antimicrobial resistance genes (ARGs) and 10 heavy metal concentrations, and the relationship between ARGs and heavy metals in surface sediment from seven sites of Lake Taihu. The results showed significant correlations (p < 0.05) between sediment ARG levels, especially for tetracycline and sulfonamides (e.g., tet(A), tet(D), tet(E), tet(O), sul1, sul2 and int-1) and specific heavy metals (Fe, Mn, Cr, Cu, Zn, among others) in the Lake. In the surface sediments, heavy metals showed an interaction with resistance genes, but the strength of interaction was diminished with increasing depth. For most of the heavy metals, the concentration of elements in the top sediments was higher than that in other depths. Tetracycline resistance genes (tet(A), tet(B), tet(D), tet(E) and tet(O), β-lactam resistance genes (SHV, TEM, CTX, OXA and OXY) and sulfonamide resistance genes (sulA, sul1, sul2, sul3 and int-1) were detected. They showed a trend which inferred a statistically significant increase followed by decreases in the relative abundance of these ARGs (normalized to 16S rRNA genes) with increasing depth. This study revealed that tet(A), tet(O), TEM, OXY, int-1, sul1 and sul3 were widespread in surface sediments with high abundance, indicating that these genes deserve more attention in future work.