The problem of passive localization using an unmanned aerial vehicle (UAV) swarm is studied. For multi-UAV localization systems with limited communication and observation range, the challenge is how to obtain accurate target state consistency estimates through local UAV communication. In this paper, an ant colony pheromone mechanism-based passive localization method using a UAV swarm is proposed. Different from traditional distributed fusion localization algorithms, the proposed method makes use of local interactions among individuals to process the observation data with UAVs, which greatly reduces the cost of the system. First, the UAVs that have detected the radiation source target estimate the rough target position based on the pseudo-linear estimation (PLE). Then, the ant colony pheromone mechanism is introduced to further improve localization accuracy. The ant colony pheromone mechanism consists of two stages: pheromone injection and pheromone transmission. In the pheromone injection mechanism, each UAV uses the maximum likelihood (ML) algorithm with the current observed target bearing information to correct the initial target position estimate. Then, the UAV swarm weights and fuses the target position information between individuals based on the pheromone transmission mechanism. Numerical results demonstrate that the accuracy of the proposed method is better than that of traditional localization algorithms and close to the Cramer–Rao lower bound (CRLB) for small measurement noise.