Neural activity in mammalian brains exhibits large spontaneous fluctuations whose structure reveals the intrinsic functional connectivity of the brain on many spatial and temporal scales. Between remote brain regions, spontaneous activity is organized into large-scale functional networks. To date, it has remained unclear whether the intrinsic functional connectivity between brain regions scales down to the fine detail of anatomical connections, for example the fine-grained topographic connectivity structure in visual cortex. Here, we show that fMRI signal fluctuations reveal a detailed retinotopically organized functional connectivity structure between the visual field maps of remote areas of the human visual cortex. The structured coherent fluctuations were even preserved in complete darkness when all visual input was removed. While the topographic connectivity structure was clearly visible in within hemisphere connections, the between hemisphere connectivity structure differs for representations along the vertical and horizontal meridian respectively. These results suggest a tight link between spontaneous neural activity and the fine-grained topographic connectivity pattern of the human brain. Thus, intrinsic functional connectivity reflects the detailed connectivity structure of the cortex at a fine spatial scale. It might thus be a valuable tool to complement anatomical studies of the human connectome, which is one of the keys to understand the functioning of the human brain.