Summary: This paper is devoted to the convergence analysis of stochastic approximation algorithms of the form \(\theta_{n+1} = \theta_n + \gamma_{n+1} H_{\theta_n}({X_{n+1}})\), where \({\left\{ {\theta}_n, n \in {\mathbb{N}} \right\}}\) is an \({\mathbb{R}}^d\)-valued sequence, \({\left\{ {\gamma}_n, n \in {\mathbb{N}} \right\}}\) is a deterministic stepsize sequence, and \({\left\{ {X}_n, n \in {\mathbb{N}} \right\}}\) is a controlled Markov chain. We study the convergence under weak assumptions on smoothness-in-\(\theta\) of the function \(\theta \mapsto H_{\theta}({x})\). It is usually assumed that this function is continuous for any \(x\); in this work, we relax this condition. Our results are illustrated by considering stochastic approximation algorithms for (adaptive) quantile estimation and a penalized version of the vector quantization.