Neural networks offer viable alternatives for energy versus accuracy tradeoffs, in particular with regards to the precision of the computational circuit. This paper explores use of progressive modular redundancy of intrinsically low energy, low precision circuits as an alternative to more complex networks yielding higher accuracy directly. Results indicate that a lower footprint temporal modular redundancy, which is applied progressively as needed, can have lower footprint and reduced energy consumption at comparable or slightly reduced accuracy as more complex neural networks. This provides an alternative to binarization and other model compression options for intelligence at the edge of the network. Our Progressive Modular Redundancy approach using varied activations implemented using a network shows a 3% improvement in accuracy compared to the baseline case of network with sigmoidal activation, at 86.1% and 87% reduction in power and weighted crossbar normalized area overhead, respectively, 87.5% reduction in power error product (PEP) at the cost of ~2.6x increased throughput latency.