Let \(\Omega\subset\mathbb{R}^ n\), \(n\geq 3\), be a bounded domain with \(C^ 4\) boundary \(\partial\Omega\). Let \(\Gamma_ 0\) and \(\Gamma_ 1\) be \((n- 1)\)-dimensional submanifolds of \(\partial\Omega\) such that \(\Gamma_ 0\cap \Gamma_ 1=\emptyset\) and \(\partial\Omega= \Gamma_ 0\cup \Gamma_ 1\). Define \(H(\Gamma_ 0)=\{u\in H^ 1(\Omega)\); \(u=0\) on \(\Gamma_ 0\}\). Let \(\alpha\in L^ \infty(\Omega)\) and \(\beta\in C^ 1(\Gamma_ 1)\) be such that \[ \| u\|^ 2= \int_ \Omega | \nabla u|^ 2 dx +\int_ \Omega \alpha u^ 2 dx+2^{-1} (n-2) \int_{\Gamma_ 1} \beta u^ 2 dy \] defines an equivalent norm on \(H(\Gamma_ 0)\). For \(u\in H(\Gamma_ 0)\) and \(q=n(n-2)^{-1}\), define \(Q(u)=\| u\|^ 2 (\int_{\partial \Omega} | u|^{q+1} dy)^{-2(q+1)^{-1}}\). Let \(S_ 1=\inf\{ \int_{\mathbb{R}^ n_ +} | \nabla u|^ 2 dx\); \(u\in H^ 1(\mathbb{R}^ n_ +)\), \(\int_{\mathbb{R}^{n-1}} | u|^{q+1} dx=1\}\), be the best constant in the trace imbedding for the half-space. For \(x\in\partial\Omega\) let \(\lambda_ i(x)\), \(1\leq i\leq n-1\), denote the principal curvatures and \(H(x)=(n-1)^{-1} \sum^{n-1}_{i=1} \lambda_ i(x)\) the mean curvature at \(x\) with respect to the unit outward normal.
Consider the following problem \[ \begin{cases} -\Delta u+ \alpha u=0 \text{ in }\Omega; \;u>0 \text{ in }\Omega;\;u=0 \text{ on } \Gamma_ 0,\\ \partial u/\partial\nu+ 2^{-1}(n-2) \beta u=u^{n(n-2)^{-1}} \text{ on } \Gamma_ 1.\end{cases} \tag{1} \] The main purpose of this paper is an existence result for solutions in \(H^ 1(\Omega)\) of the problem (1). To this end assume that there exists a point \(x_ 0\) in the interior of \(\Gamma_ 1\) and a neighbourhood \(U(x_ 0)\) of \(x_ 0\) such that \(\Omega\cap U(x_ 0)\) lies on one side of the tangent plane at \(x_ 0\). Further assume that either of the following holds:
\((\beta(x_ 0)<H(x_ 0)\), \(n\geq 3)\) or \((\beta(x_ 0)= H(x_ 0)\), \(n\geq 5\); \((n-2)(n-4) \sum_{i\neq j} (\lambda_ i(x_ 0)- \lambda_ j(x_ 0))^ 2+ 4(n-1)^ 2 \alpha(x_ 0)<0\); \(\alpha\) is \(C^ 1\) in a neighbourhood of \(x_ 0)\). Then the problem (1) admits a solution \(u\in H^ 1(\Omega)\) with \(Q(u)<S_ 1\).