Summary: In this paper, we present a novel control method, combining observer-based optimized backstepping (OB) control, reinforcement learning (RL) strategy, and adaptive neural networks (NN), for strict-feedback multi-agent systems with unmeasurable states. The primary objective is to enhance the overall system backstepping controller by optimizing both virtual and actual controllers for corresponding subsystems. To achieve this, we develop an observer-critic-actor RL approach based on NN approximation in all backstepping steps. The observers are utilized to estimate the unmeasurable system states, while the critic-actor algorithm assesses the control performance and executes control actions. Our optimized control method offers advantages such as not requiring the state observer to satisfy the Hurwitz equation. Additionally, our designed RL algorithm is simple due to the critic-actor adaptive laws obtained by means of the negative gradient of a simple positive function associated with the partial derivative of Hamilton-Jacobi-Bellman (HJB) equation. Consequently, our proposed control method ensures that all error states of the multi-agent systems are semi-globally uniformly ultimately bounded (SGUUB), and that all outputs can synchronously follow the reference signal with desired accuracy. Finally, we demonstrate the efficacy of our control strategy by means of theoretical analysis and simulation results.