Summary: The lack of feedback-loop design and uncontrollable state reconstruction, such as load demand and renewable generation, are key factors to hinder the development of energy control. Adaptive dynamic programming is an efficient method to break through the bottleneck. However, most current works assume that the evolution of uncontrollable state satisfies Markovian property. This paper proposes a time-delayed adaptive dynamic programming framework for optimal energy control, utilizing more comprehensive historical data for state reconstruction. The uncontrollable state is reconstructed by an attention mechanism-based Transformer network based on online measured data. Then, a self-learning iterative algorithm is developed to obtain the optimal state feedback control policy under a discounted performance index function. The problem of solving iterative control policy, which is a mixed-integer nonlinear programming problem with an enormous computational burden, is decomposed into three sub-problems, i.e. nonlinear programming with no discrete variable, which is solved by the Projected Newton method, mixed-integer quadratic programming, and quadratic programming. Numerical results illustrate that the developed control algorithm minimizes the electricity cost in the long term and avoids peak load.