Osteogenesis is a complex biological process that includes synthesis of an organic matrix composed mainly of type I collagen and mineralization of the organic matrix by deposition of hydroxyapatite crystals. Mutations in the PHEX gene, a gene encoding a peptidase (formerly PEX; Phosphate regulating gene with homologies to endopeptidases on the X chromosome), are responsible for X-linked hypophosphatemic rickets; a genetic disease that is characterized by undermineralization of the bone extracellular matrix. Several experimental observations support a role for PHEX in mineralization. Here we report on the design, synthesis, and in vitro biological activity of mercaptoacyl dipeptide-based inhibitors of PHEX. A parallel solid phase peptide synthesis approach was used for producing focused compound libraries resulting in single digit nanomolar PHEX inhibitors. Structure activity relationships studies revealed that the P1’ aspartic acid residue is critical and its deletion or modification lead to a large decrease in activity. The stereochemistry of the aspartic acid residue at P1’ is also important. Replacing L-aspartic acid with its D enantiomer led to about a seven fold loss of potency. We explored multiple sites of diversity around the central aspartic acid and these results are also reported. In assessing selectivity for PHEX versus NEP, all the derivatives tested were highly selective for PHEX. Such compounds may have potential usage in regulating bone mineralization and/or as osteogenic agents.