Networked heterogeneous systems pose a great challenge for parallel and distributed applications. Middleware provides a critical link between the vast resources and the application domain that simplifies development, provides robust and reliable access to resources, helps optimize resource utilization, and facilitates the generation of stable distributed software. In this paper, we discuss a layered framework for distributed systems middleware, called Delmon, which supports parallel and distributed programming models, tools and applications on heterogeneous systems. Distributed systems middleware provides multiple layers of abstractions to hide the details of the distributed system and facilitate the efficient utilization of such systems. From a software engineering point-of-view, such a layered middleware approach and the separation of concerns improve the development and management of parallel and distributed applications in many ways. The proposed model represented by Delmon is a general three-layer middleware that provides the services required to support parallel and distributed models and applications in clusters and heterogeneous systems. The layers are: the resource-specific services, the self-organized run-time environment, and the parallel and distributed programming models. We also demonstrate the benefits of this well-defined layered approach that offers different levels of services and functionalities.