Summary: What are called secondary characteristic classes in Chern-Weil theory are a refinement of ordinary characteristic classes of principal bundles from cohomology to differential cohomology. We consider the problem of refining the construction of secondary characteristic classes from cohomology sets to cocycle spaces; and from Lie groups to higher connected covers of Lie groups by smooth \(\infty\)-groups, i.e., by smooth groupal \(A_{\infty}\)-spaces. Namely, we realize differential characteristic classes as morphisms from \(\infty\)-groupoids of smooth principal \(\infty\)-bundles with connections to \(\infty\)-groupoids of higher \(U(1)\)-gerbes with connections. This allows us to study the homotopy fibres of the differential characteristic maps thus obtained and to show how these describe differential obstruction problems. This applies in particular to the higher twisted differential spin structures called twisted differential string structures and twisted differential fivebrane structures.