Summary: In this paper we study a class of stochastic individual-based models that describe the evolution of haploid populations where each individual is characterised by a phenotype and a genotype. The phenotype of an individual determines its natural birth- and death rates as well as the competition kernel, \(c(x,y)\) which describes the induced death rate that an individual of type \(x\) experiences due to the presence of an individual or type \(y\). When a new individual is born, with a small probability a mutation occurs, i.e. the offspring has different genotype as the parent. The novel aspect of the models we study is that an individual with a given genotype may expresses a certain set of different phenotypes, and during its lifetime it may switch between different phenotypes, with rates that are much larger then the mutation rates and that, moreover, may depend on the state of the entire population. The evolution of the population is described by a continuous-time, measure-valued Markov process. In [the first author et al., A stochastic model for immunotherapy of cancer. Sci. Rep. 6, 24169 (2016)], such a model was proposed to describe tumor evolution under immunotherapy. In the present paper we consider a large class of models which comprises the example studied in [the first author et al., loc. cit.] and analyse their scaling limits as the population size tends to infinity and the mutation rate tends to zero. Under suitable assumptions, we prove convergence to a Markov jump process that is a generalisation of the polymorphic evolution sequence (PES) as analysed in [N. Champagnat, Stochastic Processes Appl. 116, No. 8, 1127–1160 (2006; Zbl 1100.60055); with S. Méléard, Probab. Theory Relat. Fields 151, No. 1–2, 45–94 (2011; Zbl 1225.92040)].