This work proposes a geometric feature-based descriptor for efficient Facial Expression Recognition (FER) that can be used for better human-computer interaction. Although lots of research has been focused on descriptor-based FER still different problems have to be solved regarding noise, recognition rate, time and error rates. The Japanese Female Facial Expression (JAFFE) data sets help to make FER more reliable and efficient as pixels are distributed uniformly. The proposed system introduces novel geometric features to extract important features from the images and layered Hidden Markov Model (HMM) as a classifier. The layered HMM is used to recognised seven facial expressions i.e., anger, disgust, fear, joy, sadness, surprise and neutral. The proposed framework is compared with existing systems where the proposed framework proves its superiority with the recognition rate of 84.7% with the others 85%. Our proposed framework is also tested in terms of recognition rates, processing time and error rates and found its best accuracy with the other existing systems.