The Virtual Interface Architecture (VIA) specification has been developed to standardize user-level network interfaces that provide low latency, high bandwidth communications. Few hardware and software implementations of VIA exist. Since the VIA specification is flexible, different choices exist for implementing various components of VIA such as doorbells, address translation methods, and completion queues. Although previous studies have evaluated the overall performance of different VIA implementations, there has not been a comparative study on the performance of VIA components. In this paper, we evaluate and compare the performance of different implementations of essential VIA components. We discuss the pros and cons of each design approach and describe the required support for implementing each of them. The IBM SP Switch-Connected NT cluster is one the newest clustering platforms available. In this paper, we discuss an experimental implementation of the Virtual Interface Archite...

This paper presents Linux/SimOS, a Linux operating system port to SimOS, which is a complete machine simulator from Stanford. The motivation for Linux/SimOS is to alleviate the limitations of SimOS, which only supports proprietary operating systems. The contributions made in this paper are two-fold: First, the major modifications that were necessary to run Linux on SimOS are described. Second, a detailed analysis of the UDP/IP protocol and M-VIA is performed to demonstrate the capabilities of Linux/SimOS. The simulation study shows that Linux/SimOS is capable of capturing all aspects of communication performance, including the effects of the kernel, device drivers, and network interface.

We present the results of a detailed study of the Virtual Interface (VI) paradigm as a communication foundation for a distributed computing environment. Using Active Messages and the Split-C global memory model, we analyze the inherent costs of using VI primitives to implement these highlevel communication abstractions. We demonstrate a minimum mapping cost (i.e. the host processing required to map one abstraction to a lower abstraction) of 5.4 sec for both Active Messages and Split-C using 4-way 550 MHz Pentium III SMPs and the Myrinet network. We break down this cost to use of individual VI primitives in supporting flow control, buffer management and event processing and identify the completion queue as the source of the highest overhead. Bulk transfer performance plateaus at 44 Mbytes/sec for both implementations due to the addition of fragmentation requirements. Based on this analysis, we present the implications for the VI successor, Infiniband.

In this paper, we describe an implementation of MPI-IO on top of the Direct Access File System (DAFS) standard. The implementation is realized by porting ROMIO on top of DAFS. We identify one of the main mismatches between MPI-IO and DAFS is memory management. Three different design alternatives for memory management are proposed, implemented, and evaluated. We find that memory management in the ADIO layer performs better in situations where the DAFS Provider uses Direct data transfer to handle I/O requests. For the other case of Inline data transfer, it may hurt performance. We propose that the DAFS Provider can expose such implementation information for applications to take full advantage of Inline and Direct data transfers and memory management. Comparative analysis of MPI-IO performance over DAFS, network file system (NFS) and local file system (LFS) shows that MPI-IO on DAFS over VIA on cLAN performs 1.6-5.6 times better than on NFS over UDP/IP on cLAN. The performance of MPI-IO on DAFS is found to be comparable to the performance on local file system. Additional experiments show that MPI-IO nonblocking I/O primitives implemented by DAFS nonblocking operations can completely overlap I/O and computation. These results show that MPI-IO on DAFS can take full advantage of DAFS features to achieve high performance I/O over VI Architectures.

Distributed and high performance applications require a low latency, high bandwidth communication facility for exchanging data and for synchronization operations. The Virtual Interface Architecture (VIA) has been recently proposed to standardize dierent existing low-latency, high-bandwidth communication subsystems and provide various features of these communication subsystems all together. Since the introduction of VIA, software and hardware implementations of VIA have become available. VIA has dierent components (such as doorbells, completion queues, and virtual-to-physical address translation). Dierent implementations of VIA lead to dierent design strategies for eciently implementing higher level communication layers/libraries (such as Message Passing Interface (MPI)). It also has implication on the performance of applications. Currently, there is no framework for evaluating dierent design choices and for obtaining insight about the design choices made in a particular...

Abstract — Recent work in low-latency, high-bandwidth communication systems has resulted in building user–level Network Interface Controllers (NICs) and communication abstractions that support direct access from the NIC to applications virtual memory to avoid both data copies and operating system intervention. Such mechanisms require the ability to directly manipulate user–level communication buffers for delivering data and achieving protection. To provide such abilities, NICs must maintain appropriate translation data structures. Most user–level NICs manage these data structures statically which results both in high memory requirements for the NIC and limitations on the total size and number of communication buffers that a NIC can handle. In this paper, we categorize the types of data structures used by NICs and propose dynamic handle lookup as a mechanism to manage such data structures dynamically. We implement our approach in a modern, user–level communication system, we evaluate our design with both micro-benchmarks and real applications, and we study the impact of various cache parameters on system performance. In this work we focus mostly on the results of our work. We find that, with appropriate cache tuning, our approach reduces the amount of NIC memory required in our system by a factor of two for the total NIC memory and by more than 80 % for the lookup data structures. For larger system configurations the gains can be even more significant. Moreover, our approach eliminates the limitations imposed by current NICs on the amount of host memory that can be used for communication buffers. Our approach increases execution time by at most 3 % for all but one applications we examine. I.

Recent work in low-latency, high-bandwidth communication systems has resulted in building Network Interface Controllers (NIC) and communication abstractions that support direct access from the NIC to application virtual memory to avoid both data copies and operating system intervention. Such mechanisms require the ability to directly manipulate application buffers in host memory for protection and delivering data. Most modern NICs statically maintain address translation and protection information. However, this results both in high memory requirements for the NIC and limitations in the size of host memory. In this thesis, we categorize the types of data structures for managing communication buffers used in modern NICs, and propose mechanisms to dynamically manage such data structures to alleviate the related limitations. We implement our approach in a modern user–level communication system. The contributions of this thesis are: (i) The integrated approach for dynamic handle lookup that deals with all major lookup data structures reduces NIC memory requirements significantly and eliminates restrictions on