Summary: Binary systems are quite common within the populations of near-Earth asteroids, main-belt asteroids, and Kuiper belt asteroids. The dynamics of binary systems, which can be modeled as the full two-body problem, is a fundamental problem for their evolution and the design of relevant space missions. This paper proposes a new shape-based model for the mutual gravitational potential of binary asteroids, differing from prior approaches such as inertia integrals, spherical harmonics, or symmetric trace-free tensors. One asteroid is modeled as a homogeneous polyhedron, while the other is modeled as an extended rigid body with arbitrary mass distribution. Since the potential of the polyhedron is precisely described in a closed form, the mutual gravitational potential can be formulated as a volume integral over the extended body. By using Taylor expansion, the mutual potential is then derived in terms of inertia integrals of the extended body, derivatives of the polyhedron’s potential, and the relative location and orientation between the two bodies. The gravitational forces and torques acting on the two bodies described in the body-fixed frame of the polyhedron are derived in the form of a second-order expansion. The gravitational model is then used to simulate the evolution of the binary asteroid (66391) 1999 KW4, and compared with previous results in the literature.