The interaction between a robot and a human becomes meaningful when the robotic agent possesses some level of human-like cognition. This paper proposes an attention-based approach for multi-modal HRI. The core of the proposed approach is a bio-inspired artificial model of visual attention which enables a robot to focus on the visually salient and/or behaviorally relevant stimuli in the surrounding. The attention model provides the human partner with the opportunity to manipulate the attention behavior of the robot through natural speech command. Similarly the robot is able to manipulate the attention of the human partner using its actuators. Thus the bio-inspired visual attention mechanism along with the sensors and actuators enables the robot to establish joint attention with the human partner. Formation of this joint attention is the basis for further human-robot interaction. Experimental results validate different aspects of the proposed attention-based HRI framework.