Fast Loosely-Timed Deep Neural Network Models with Accurate Memory Contention

The term von Neumann bottleneck, widely known as the memory bottleneck, was coined by John Backus in 1978 [5]. Von Neumann computers are built around an inherent bottleneck that is “the word-at-a-time tube connecting the CPU to the memory” [5]. Since the birth of the first von Neumann computer in 1945, various innovations have developed to alleviate the memory bottleneck. Multi-level cache hierarchies, shared scratchpad memory, multi-channel memory architecture, Non-Uniform Memory Access (NUMA) architecture, and more recently, computation-in-memory [32] are only a few of the inventions to tackle the memory bottleneck in computer systems. Despite all these efforts, the memory bottleneck remains one of the grand challenges of computer science and engineering.

With the rapid growth in the complexity of electronic devices and the drastic reduction in time to market, Electronic System Level (ESL) methodology has been proposed for modeling systems at higher levels of abstraction [7, 13]. ESL ideas resulted in defining System-level Description Languages (SLDL), such as SpecC [14] and SystemC [15], that can model both hardware and software components.

ESL techniques focus on TLM, which separates computation from communication in the model [9]. TLM allows the refinement of computation and communication independently and on different abstraction levels. In this way, TLM can speed up simulation significantly by replacing many pin-level events in RTL simulation with an abstract function call. Generally, the higher the level of abstraction is, the faster the simulation runs. Naturally, this simulation speedup typically comes at the price of lower model accuracy.

In summary, ESL and TLM raise the design abstraction above RTL to overcome the challenges of designing today’s complex SoCs. In particular, TLM provides an agile hardware-software codesign framework for the early exploration of the wide range of design metrics and the evaluation of design candidates. Moreover, TLM provides a codesign environment wherein software can be developed in parallel with hardware. TLM is beneficial not only for earlier system integration but also for rapid feedback to system designers.

Deep Learning (DL) is a known technique in machine learning to extract useful features from input data, perform data transformations, and arrive at a final meaningful representation. One of the main application areas of DL is visual recognition, and in particular, image classification, which is the problem of assigning a descriptive label to an input image from a fixed set of categories. DL and convolutional neural networks (CNNs) have been shown to solve this challenging problem fast and with acceptable precision.

Early work on CNNs dates back to 1989 with the LeNet network for handwritten digit recognition [22]. However, the early 2010s started a new era for CNN applications with the introduction of AlexNet [20] for image classification. Growth of computing power, availability of massive datasets for training, and rapid innovation in DL architectures have paved the way for the success of DL techniques in recent years [33].

A CNN consists of alternating convolution layers and pooling (sub-sampling) layers. Each convolution layer extracts features from the input by applying trainable filters to the input. Later, the convolved feature is fed to an activation function, for example, a Rectifier Linear Unit (ReLU), to introduce non-linearity and obtain activation maps. Each pooling layer down-samples the activation maps to reduce computation and memory usage in the network. Features extracted from the previous convolution and pooling layers are fed to a fully connected layer to perform classification. Typically, a softmax activation function can be placed following the final fully connected layer to output the probability corresponding to each classification label.

Choosing state-of-the-art deep CNNs for TLM modeling enables our investigation of the memory bottleneck problem.

A large body of research exists on performance modeling and memory contention modeling and analysis. We can broadly categorize most system-level performance analysis methods into two main classes: analytical and simulation-based. In analytical approaches, we mathematically model the systems and analytically derive their performance as a function of workload and input parameters. Frank et al. [11] define an analytical contention model in parallel algorithms on a multiprocessor workstation. Chen et al. [10] use queueing theory to model contention in bus-based system design. Analytical models depend on the architecture described, and a new model must be developed for each new architecture or application [8]. Moreover, analytical modeling does not take into account the dynamic behavior of the system, and often use of more realistic assumptions makes meaningful analysis difficult [6].

Simulation-based approaches can capture many dynamic and complex interactions in a system. SpecC [14] and SystemC [15] are widely-used SLDL for modeling, simulation, and validation of complex SoC models. The SystemC C++ class library is an IEEE standard that enables system and TLM using discrete event simulation (DES) [16]. Simulation techniques often suffer from long simulator run-times at lower abstraction levels. Furthermore, there are high costs associated with manually building simulation models and debugging them.

A systematic and quantitative analysis of TLM’s speed/accuracy tradeoff has been studied in [30]. A method of overcoming this general tradeoff for the specific case of processor models is proposed in [31]. The study of speed and accuracy tradeoff in DES has also been carried out in other scientific fields, such as network simulation. Packet-level network simulators enable high-accuracy simulation but can lead to long simulation times. For example, SimGrid, developed by Legrand et al. [23], is a simulation framework that simulates networks at higher levels, thus enabling fast simulation but losing accuracy. This speed-accuracy tradeoff is quantitatively evaluated and confirmed by Fujiwara et al. [12]. Packet-level models are essentially the analog of non-blocking TLM transactions as both model the atomic elements circulating on the interconnect.

To overcome strictly simulation-based methods, a hybrid approach of analytical and simulation methodologies has been proposed. Lunzli et al. [21] offer a technique to combine SystemC-based simulation with formal analysis based on real-time calculus. Borbek et al. [8] also combine simulation with an analytical method focusing on the study of shared resource contention. While these mixed methodologies help to shorten simulator run-times, the coverage for corner cases in simulation remains difficult [35]. Furthermore, [8] operates at a much higher level of abstraction than TLM and thus sacrifices some accuracy for higher simulation speedup.

Aside from analytical and simulation-based modeling approaches, there are also experimental techniques to measure the effect of memory contention. More recently, DNN library profilers such as PyTorch Profiler [28] and performance profilers such as Intel VTune Profiler [17] provide some coarse-grain measures on memory usage and footprint. However, the results are valid only for a specific processor architecture and memory hierarchy. This hardware dependency is not helpful for DSE or refinement to lower-level abstraction.

Our proposed TLM framework is based on the well-defined SystemC methodology, which makes it easy to deploy. Our automatic model generation dramatically reduces the burden of constructing and debugging simulation models. Furthermore, memory contention is modeled accurately and simulated fast, enabling efficient early DSE.

A well-defined modeling strategy is essential to manage the system’s complexity and provide maximum flexibility. Our system modeling framework follows three criteria introduced in [2]:

We have used the Caffe (Convolutional Architecture for Fast Feature Embedding) model zoo to obtain pre-trained network parameters. Caffe is a DL framework originally developed at the University of California, Berkeley, and is available under BSD license [18]. Caffe models come with (1) a binary file .caffemodel that contains network parameters and (2) a text file .prototxt that specifies the network architecture. Class labels are also provided in a text file format that includes a synonym ring or synset of those labels.

Our SystemC models rely on efficient, optimized code inside OpenCV 3.4.1. OpenCV is a library of computer vision functions mainly aimed at real-time applications written in C/C++ [27]. The OpenCV library was originally developed by Intel and is now freely accessible under the open-source BSD license. OpenCV uses an internal data structure to represent an n-dimensional dense numerical single-channel or multi-channel array, a so-called Mat class. Therefore, our models use the Mat data type to store images, weight matrices, bias vectors, feature maps, and class scores. This design decision becomes practical while interacting with various OpenCV application programming interfaces (APIs).

Our previous study [2] shows that the multi-threaded OpenCV library delivers the highest level of parallelism for simulation speedup compared to existing thread-level parallelism at the SystemC level. Therefore, we rely on multi-threaded OpenCV with a sequential SystemC simulator instead of a parallel simulator such as the Recoding Infrastructure for SystemC (RISC) [24] for better simulation performance. Moreover, RISC still needs support for all language constructs for approximately-timed modeling.

Our system-level modeling framework follows the well-known Specify-Explore-Refine (SER) methodology [13], which is a successive, stepwise refinement of design models, as described in subsequent sections.

TLM-1 implements message-passing semantics primarily to separate communication from computation. Through well-defined TLM-1 interface method calls, any internal state changes in one SystemC module are hidden from other modules [16].

Following the TLM-1 coding style, each layer in the CNN is modeled as a sc_module with input and output ports. Ports in each module are defined as sc_port and are parameterized either by primitive or user-defined interface classes. The user-defined interfaces are derived from sc_interface and declare read and write access methods with a granularity of Mat. The choice of Mat for the granularity of port parameterization simplifies the design by focusing on the proper level of abstraction at this level of modeling.

Each module has a main thread that continuously reads its input port, computes results, and writes those to its output port. Data processing is handled by a run method that interacts with the OpenCV library. The run method creates an instance of OpenCV layer and calls its forward method by passing references to input Mat and output Mat objects.

Channels are modeled as queues with FIFO semantics, allowing to consume/produce data in a first-in, first-out discipline. These channels implement interface methods for read and write access. Encapsulating communication in channels allows various communication mechanisms and buffer sizes to be modeled independently from the module functionality. This exploratory approach provides early feedback on the amount of available parallelism and local communication interactions. More details on the characteristics of our four channel variants and simulation results for the corresponding TLM-1 models A, B, C, and D in Figure 1 can be found in [3, 4].

While TLM-1 provides early feedback on parallelism and local communication, it is not specifically intended for bus modeling or interoperability. TLM-2.0 introduces a generic payload and blocking/non-blocking transport interfaces for the abstract modeling of memory-mapped buses.

Instead of defining a strict taxonomy of abstraction levels, TLM-2.0 establishes a set of APIs and describes a set of appropriate coding styles for various use cases. For example, the LT coding style utilizes the blocking transport interface (b_transport) for the use cases of software development and performance optimization. LT models simulate fast and have sufficient timing details to boot an operating system. On the other hand, the approximately-timed (AT) coding style utilizes the non-blocking transport interface (nb_transport) for the use cases of architecture exploration and detailed performance analysis. Generally, AT models simulate slower but carry better timing accuracy than LT models [16].

In TLM-2.0, a socket is instantiated within each initiator and each target module for every transaction-level connection. The generic payload captures the information to pass on with each bus transaction between the initiator and the target. The initiator module instantiates the generic payload transaction object and sets its attributes before passing a reference to this object to the target module via its transport interface.

For an initial estimation of the specific DNN memory usage at the system level, we can extract the read/write accesses initiated from each module to a memory. At this stage of modeling, the granularity of accesses can be the size of an input or output buffer, and the latency of each access can be a delta-cycle delay.

In our proposed model, the initiator sockets are connected to target sockets of shared memory, as shown in Figure 4(a). Each module has a dedicated address space in the memory to read and write its buffers. This model uses a blocking transport interface (b_transport) to pass transactions between the initiator and the target memory. The transaction is a tlm_generic_payload object, and its data pointer points to the start address of the input/output buffer. For early estimation of memory usage and accessible visualizations at this stage of modeling, the data length of the generic payload is set to the entire buffer inside the shared memory. Since the model is untimed, the timing annotation argument of b_transport is set to a delta cycle.

Generally, the communication mechanism in a single-buffer scheme is as follows: each producer places its output into a buffer in the shared memory. Each consumer reads its input from the same shared buffer. To avoid race conditions, each consumer waits for an event notification from its producer. The arrows between the modules in Figure 4(a) illustrate this feed-forward notification mechanism. This design forms the TLM-2.0 untimed model with feed-forward events mechanism, model E in Figure 1.

Without a balanced graph topology, event synchronization for multiple producers or consumers requires delta-cycle delay compensation. Such behavior can be seen in every inception module in GoogLeNet, as shown in Figure 3(a). Note that the four parallel tracks contain 2, 4, 4, and 3 modules. Our proposed untimed model compensates for these irregularities to guarantee correct synchronization between the modules.

We also devise a back-pressure events mechanism to allow the untimed TLM-2.0 model to execute safely even in aggressive out-of-order parallel simulation for maximum speedup [3]. Event connections for the first convolution and ReLU layers in GoogLeNet are depicted in Figure 4(b). Each module has a set of two sc_events for each input and output. The stb event is notified once a module has valid data inside the memory to be read, and the ready event signals a module is ready to read new data. By connecting events between all subsequent modules, the model forms a robust back-pressure mechanism that safely controls the data flow inside the pipeline.

Support for the back-pressure events mechanism is extended to all neighboring modules in the TLM-2.0 model. The double-buffering scheme guarantees a continuous stream of data inside the design pipeline, maximizing model parallelism and model throughput with the minimum number of buffers in the memory. This design forms the TLM-2.0 untimed model with back-pressure events mechanism, model F in Figure 1.

Given that the TLM-2.0 untimed model F provides only causal ordering between processes, timing is introduced at the next lower abstraction level. Our LT approach models a transaction’s start and end times using the blocking transport interface with a timing annotation, providing a good tradeoff between timing accuracy and simulation speed.

While LT modeling generally aims at timed simulation as fast as possible, our goal of observing memory contention requires the explicit modeling of interconnect components and memory interfaces. Thus, we cannot use more aggressive TLM-2.0 abstraction techniques, such as Direct Memory Interface (DMI)¹ or temporal decoupling.² However, we maintain the LT assumption that memory transactions are complete in one function call and cannot overlap with others.³

The LT model adds three sources of latency: (1) memory, (2) computation, and (3) interconnect. Our step-wise approach incrementally refines the model, adding one source of latency at each step. We keep the system model topology as in the TLM-2.0 untimed model, i.e., the mapping of each layer to a separate module. We refine further the point-to-point connection between layers and memory with a generic interconnect. The interconnect can be considered a shared bus matrix at this modeling stage. With the interoperability mechanism offered by TLM-2.0, our framework can also support different interconnect topologies.

As described in Section 4.1, each transaction in an LT model covers a whole transaction in one function call. LT has two timing points, the start, and the end. It is possible to increase the number of timing points for each transaction to have a higher degree of timing accuracy. More timing points help to more accurately monitor throughput, latency, and bandwidth utilization and can support interleaving transactions and pipelined bus protocols. However, processes are more likely to run in lock step with the SystemC scheduler with more timing points. Hence, it is generally expected that AT models simulate significantly slower than their LT counterparts.

The AT coding style uses a non-blocking transport interface which, in addition to the timing annotation, supports multiple phases within the lifetime of a transaction. The base protocol for AT modeling defines four phases to represent four timing points for each transaction: the start and end of the request and the beginning and the end of the response. These four phases of the base protocol can model three timing parameters: (1) the request accept delay, (2) the latency of the target, and (3) the response accept delay [16, 19].

To build an AT model for a DNN, we first refine the communication part of the LT style initiator. The blocking transport interface is replaced with an implementation of the nb_transport_fw and nb_transport_bw functions. Following TLM-2.0 guidelines with regards to the usage of the generic payload in non-blocking interfaces, we instantiate a memory manager to acquire a generic payload transaction from a pool of transaction objects and release it to return to the same pool once the transaction is no longer in use. Furthermore, the logic for handling the base protocol call sequence is also added with the help of a Payload Event Queue (PEQ) and its callback method.

Next, the LT interconnect is replaced with an interconnect that supports the handling of non-blocking interfaces and the AT base protocol. Unlike its LT counterpart, the AT interconnect can queue incoming requests if there is already a request in progress and the interconnect has not completed the END_REQ phase for that request. AT interconnect can also queue up incoming responses from the memory to forward transactions later on the backward path to initiators. The sequence of phase transitions for each transaction helps to model the contention. Our AT interconnect supports two arbitration strategies, namely, FCFS and RR policy. Finally, we reuse the logic for address mapping and address translation from the LT interconnect.

As the final refinement step, we replace the LT memory with an AT counterpart. Our AT memory model implements the base protocol with four phases to provide the proper timing granularity for the AT coding style and accurate contention modeling. The bus width, size, request, and response delays for the AT memory are all configurable. To provide accurate timing for comparative analysis, we devise an estimation for read and write request and response delays for each transaction as follows:

Given TLM-2.0 AT compliance, we connect the multi_passthrough_initiator_socket of the AT interconnect to the target_socket of the AT memory module.

Transaction modeling using LT coding style simulates fast because transactions are complete in a single blocking transport method call, namely, a b_transport call. For the same reason, LT models are usually not used for contention analysis which typically requires a detailed sequence of interactions between the initiator and the target within the life of a transaction. On the contrary, AT coding style uses a non-blocking transport interface that supports multiple phases within the lifetime of a transaction. The AT modeling naturally enables resource contention modeling to find performance bottlenecks in the design. However, an AT model simulates slower than its LT counterpart because it can contain up to four function calls to complete a transaction.

AT modeling is one of the more complex aspects of TLM-2.0, which makes AT model development a non-trivial task. Therefore, the development of AT models is often postponed to later stages of the design flow, typically only after the LT model is in place. Moreover, when AT model development is disregarded due to a tight project schedule, RTL simulations are most likely used to find performance bottlenecks. However, chip-level RTL simulations suffer from an order of magnitude slower simulation speed compared to system-level AT models.

A cycle-accurate model is the closest abstraction to the final hardware and can exhibit the design’s most accurate estimation of memory contention. However, as mentioned earlier, the simulation performance of cycle-accurate and even AT models are typically orders of magnitude slower than an LT model. Importantly, techniques to tackle memory contention at the lower levels of abstraction usually have a sub-optimal impact on the overall performance. Hence, memory contention must already become visible in the design flow’s early stages. This new modeling approach enables system designers and chip architects to codesign both hardware and software to mitigate memory bottlenecks optimally.

Our LT-CA modeling supports two major arbitration policies, namely, FCFS and RR scheduling, which we detail in the following two sections.

Our fast and accurate LT-CA modeling enables the efficient exploration of alternative memory organizations. To demonstrate this, we describe an alternative architecture with local memories and interconnect components adjacent to computing units which improves the locality of data and thus minimizes contention. As illustrated in Figure 8, we can model each layer with two initiator sockets connected to two local memories that store layers’ input and output. The global interconnect also breaks into separate components that exclusively service two adjacent layers. In other words, the global shared memory transforms into many local memories which store the intermediate results.

For the case that layers have more than one input or output in the network, we can arrange multiple local memories for each input or output layer. In the case of a multi-consumer layer, we partition the output data into multiple smaller memories and devise a static scheduling scheme between the consumers to avoid contention in simultaneous read accesses. The sizes of these local memories equal the output buffer size divided by the number of consumers. This design leaves the total memory requirement of the application unchanged.

In the case of a multi-producer layer, each producer owns a local interconnect and memory. The dedicated memory and local interconnect for each input layer prevents the competition between producers to access the single local memory between all producers. In this organization, producers output data to local memory as soon as they complete their processing without any risk of contention by the other producers.

Since data is stored and processed locally in separate memories, there is no need for any extra logic to implement cache coherence protocols between multiple computing units. This memory organization also eliminates the performance penalty for cache coherency, high interconnect congestion, and high global memory contention.

We use SystemC 2.3.1 and OpenCV 3.4.1 built in the default release mode settings for simulation. For benchmarking, we measure the simulator run-time using Linux /usr/bin/time under CentOS 6.10. To have reproducible experiments, the Linux CPU scaling governor is set to “performance” mode to run all cores at the maximum frequency, and file I/O operations are minimized. An Intel Xeon E5-2680 CPU running at 2.7 GHz with eight physical cores, two threads per core, and two CPU sockets are used as our simulation platform.⁷ Lastly, the stimulus module is configured to feed 100 images with the size of 224 × 224 pixels to the model, which results in reasonable simulator run-times for our experiments.

For an estimation of the load on the memory from the DNN, we generate an untimed TLM-2.0 model (model F in Figure 1). The layers’ input and output data are stored in a single global memory, and each layer uses local storage to process its data. We use double-buffering in the memory so that the producer layer can write to the front buffer, the consumer layer can simultaneously read the data from the back buffer, and vice versa. Since the untimed model runs on a delta-cycle basis, we devise a specific buffer architecture for the concat layers at the bottom of the inception modules to compensate for the unbalanced graph topology. The concat layers require 4, 2, 2, and 3 buffers in the tracks to synchronize correctly in the double-buffering scheme. Figure 10 shows the total read and write memory accesses to the single memory architecture using the model running the classification of 100 images. As the figure shows, the memory accesses peak at over 90 million bytes per delta cycle once all layers are active and processing data. This simulation result confirms that GoogLeNet is a very memory-intensive application.

The TLM-2.0 untimed model also measures the memory usage of the DNN. For example, Table 3 lists the memory requirements for each layer in the GoogLeNet. The total required memory for each layer is the combination of its input buffers, output buffer, and, in the case of learned parameters, its weights and biases. To reduce the memory footprint of the DNN, adjacent modules can share their input/output buffers. Hence, a consumer module points to the output of the corresponding producer module.

As described in Section 3.3, to implement a double-buffering scheme, extra filler buffers are required to balance the graph architecture. Therefore, the total memory requirement of GoogLeNet in a double-buffering mode is double the size of the total output buffers: \(40.02 \times 2 = 80.04 \; MiB\). It is also worth mentioning that the memory footprint can be further reduced with a smart addressing scheme. For example, the consumers of the concat layers can simply point to the output buffers of the concat producers.

Our proposed netspec can automatically generate TLM-2.0 models for early software performance analysis and virtual prototyping. The generated LT models carry sufficient timing information to provide a coarse-grain estimation of the application execution time. However, the generic LT does not take into account any bus contention. In contrast, our proposed LT-CA counterpart is developed to analyze the effect of interconnect and memory contention. Additionally, AT models provide even better timing accuracy for memory contention analysis. Therefore, we also generate AT models of our application for comparison.

By choosing four different memory latencies⁸ (1 ns, 10 ns, 100 ns, 1,000 ns) and four different computational capacities⁹ (model G in Figure 1). We also instruct netspec to generate 16 LT-CA models (model I in Figure 1) and 16 AT models (model H in Figure 1) across the same parameters. The LT memory bus is configured to have a 64-bit width and the interconnect is configured to avoid extra accept latency. Furthermore, we set the burst length to 8, and the size of the generic payload for each transaction is configured to be the burst length multiplied by the memory data width (8*8 B = 64 B). The choice of generic payload size reflects the more realistic operation of memory in burst mode, resulting in a more accurate timing estimation of the memory accesses. Since the AT memory can model both request and response accept latencies, we configure the AT memory request accept delay equal to memory latency and the response accept delay identical to LT-CA and LT memory delays. Finally, we set the size of the generic payload for each transaction identical to LT-CA and LT models (64 B). Table 4 summarizes the total simulated time for all 48 models.

As shown in Table 4, the total simulated times of LT models (left box) are significantly less than their LT-CA and AT counterparts. In generic LT modeling, transactions that simultaneously access the shared memory complete their accesses at the same simulated time point. This lack of contention modeling incorrectly makes the total simulated time shorter than the other modeling styles that reflect contention. As shown in the middle box of Table 4, the total simulated times of the LT-CA models show a significant increase compared to LT models by considering the effect of contention. Finally, the right box shows the total simulated times of AT models that accurately model contention of both memory requests and memory responses. Comparing LT-CA and AT simulated times shows the high accuracy and high fidelity of our proposed LT-CA modeling.

Figure 11(a) and (b) visualizes the total simulated time of LT and LT-CA models reported in Table 4. As expected, lower computational capacities and higher memory latencies increase the simulated time. However, the impact of memory with higher latencies on performance is more significant. For example, the memory with 1,000 ns latency constantly performs over all computational capacities, indicating that an increase in computational power has no considerable effect on simulated time. Moreover, the impact of memory with higher latencies on the performance becomes more significant in higher computational powers. For example, having 1,000 GFLOPS computational capacity available, a 10× increase in the memory latency from 10 ns to 100 ns leads to a 10× decrease in performance. On the contrary, at the same interval in 1 GFLOPS, performance only decreases 3× . This result clearly shows the application is heavily memory-bound rather than compute-bound.

Figure 11(c) illustrates the impact of memory contention by showing the ratios of simulated times for LT-CA models over simulated times for LT models. The negative effect of memory contention is visible in every computation/memory configuration. However, the contention is more noticeable in higher computational capacities. For example, for 1,000 GFLOPS, the contention reduces the performance by 13× . Meanwhile, the contention has a lower impact on the lower computational capacities. For 1 GFLOPS, decreasing memory latency by 10× from 10 ns to 1 ns (which is quite an expensive design decision) has little effect on the performance.

To show further the generality of the LT-CA modeling approach, we change the interconnect FCFS scheduling to approximate RR scheduling. Table 5 shows the total simulated time for LT-CA and AT models using RR scheduling. Since LT modeling does not support contention and scheduling policies, we do not report the total simulated time for LT models. Similar to Table 4, it is evident that the LT-CA simulated times show high accuracy and high fidelity results compared to the reference AT models. Notably, AT simulated times for RR and FCFS scheduling policies match. This shows that arbitration policy does not play a big role in this given application.

We exhibit the accuracy of LT and LT-CA models compared to the reference AT model for each scheduling policy in Table 6. As shown in Table 6(a) for FCFS scheduling, LT models show a very low accuracy (7% for 1,000GFLOPS) compared to their AT model counterparts. At the same time, LT-CA models show almost complete accuracy. The same pattern applies for LT and LT-CA models in RR scheduling (Table 6(b)). While the LT-CA for RR shows a minor decrease in accuracy, it is still a very accurate model compared to the LT model.

In contrast to the simulated execution times, Table 7 lists the total simulator run-time for all LT, LT-CA, and AT models. As shown, LT models simulate faster than their LT-CA and AT counterparts because they use only a single function call to complete a transaction. LT-CA models simulate slightly slower than LT models (1.2×) by storing memory congestion status inside the interconnect. AT models have much longer simulator run-times as each transaction can have multiple phases and can use up to four function calls to complete a transaction. In particular, the simulator speed of the LT-CA models is an order of magnitude higher than their AT counterparts. Notably, the LT-CA models show an impressive total speedup of 46× in simulation while providing the same accuracy.

We also measure the total simulator run-time for LT-CA and AT models using RR scheduling in Table 8. As expected, AT models have long run-times (about 2 hours) while LT-CA models simulate much faster (about 20 minutes). As also expected, the AT reference models for both RR and FCFS policies are equally slow.

To signify the better speed/accuracy tradeoff, we quantify the simulator run-time for LT and LT-CA modeling styles and scheduling policies. Table 9 shows the speedup of LT and LT-CA models compared to the reference AT models for different scheduling policies. For both policies, LT models show the maximum simulation speedup (50× -60×). However, the simulator run-time for RR is about 9× longer than FCFS, which is expected as the complexity of the RR approximation Algorithm 3 is higher than the simple FCFS Algorithm 2.

By having the fast and accurate LT-CA models available, we can further analyze the memory access patterns and the effect of the interconnect/memory contention in the network. We simulate the application with 1,000 GFLOPS computational capacity and 1ns memory latency using the same setup in LT-CA modeling and the buffer size as the generic payload data length. Figure 12 shows the transaction-level timing diagram of the first inception module, inception_3a, in GoogLeNet. The left diagram shows the timing without contention (LT model) and the right one with contention (LT-CA). The x- and y-axes represent the simulated time and the names of the parallel tracks in the inception module, respectively. The elapsed times for memory write, memory read, computation, and contention are colored in blue, light green, dark green, and red, respectively.

Since the LT does not model the contention, the layers in parallel tracks access memory without blocking each other. However, when a layer accesses the memory in the LT-CA model, the other layers are blocked and wait until access becomes available again (red areas). For instance, at the beginning of the contention diagram in Figure 12, once the first layer in the 1x1 track issues a read transaction, all layers in the other tracks are blocked until the read transaction completes.

It is clear from the visual charts in Figure 12 that the LT-CA model accurately reflects the red idle waiting periods for the modules blocked by memory contention. For example, the first layers in all four tracks of inception_3a simultaneously initiate read accesses to the memory at 5.07 ms with a payload size of 602,112 bytes. The layer in the 1x1 track grants access and blocks the memory for approximately 0.07 ms (total delay to access 602,112 bytes of data). Therefore, the read transaction of the layer in 3x3 track experiences a delay of 0.07 ms. Subsequently, layers in 5x5 and pool tracks face contention of 0.15 ms, and 0.23 ms, respectively. Later, as soon as the layer in the 1x1 track completes its computation at 5.15 ms, it sends a transaction to write its result to the memory. Since there are already pending transactions, the layer in the 1x1 track must wait until the memory becomes available again at 5.37 ms once the layer in the pool track completes its read transaction. Hence, the timing annotation of the write transaction for the layer in the 1x1 track is updated with the waiting time for the next memory availability (5.37 ms - 5.15 ms = 0.22 ms). This schedules the write operation to start at 5.37 ms precisely after the memory has become available (start of the blue bar in the 1x1 track in Figure 12).

Figures 13 and 14 show the transaction-level timing diagrams for all inception modules in the GoogLeNet for one pass of an image without and with contention, respectively. As seen in Figure 14, layers in parallel tracks block each other, and contention is high. Furthermore, the track of 3× 3 has the highest elapsed time in all inceptions, making it the critical path of execution. That is valuable feedback to system architects on how to allocate computation resources.

The effect of contention becomes more significant once the DNN pipeline is full of images to process and layers frequently access the shared memory. We simulate the LT-CA model of GoogLeNet by feeding in 100 images to classify. Figure 15 shows the timing diagram for the 75th image. As clearly seen by almost all red coloring in the figure, the layers are mostly blocked due to contention. The effect of back-pressure is also quite visible, especially in the first inception module, where all its layers are mostly blocked for the subsequent inceptions to process.

We also build the LT-CA model of the proposed local memory organization discussed in Section 4.4 for the final experiment. Since all data is stored close to the compute cores and the possibility for contention is carefully looked after, memory contention is entirely resolved. Indeed, our measurements confirm that the simulated time of the network matches precisely the time of the LT model (model G in Figure 1) with global shared memory that does not consider contention (Table 4). Thus, there is actually no contention. This benefit of the local memory organization becomes more significant when the DNN is processing many images at once.

The high amount of contention on the shared memory suggests new architectures that place private local memories close to computing units (such as Figure 8). Memory organizations that rely on fine-grained data localities of the given application reduce the average bandwidth usage on the global shared memory. Such a memory organization eliminates the performance overhead of memory contention and complex cache coherency. However, new processor architectures with private local memories close to the computing units require us to rethink the conventional programming model, compilation flow, and run-time system support. For the same reason, the TLM of architectures with local memories requires a fast yet accurate estimation of the contention. Our contention awareness is attractive for enabling the efficient codesign of both hardware and software solutions in the future.