This paper proposes semi-autonomous control that minimizes intervention considering driver's steering and braking intentions. The biggest challenge of this problem is how to fairly judge driver's intentions that appear differently in the lateral and longitudinal directions and how to minimize controller intervention. A decoupled model predictive control (MPC) and optimal intervention decision methods are proposed considering driver incompatibility. Several MPCs are designed first considering the fact that the driver can avoid obstacles either by braking or moving to the left or right lanes. The control input to avoid the collision is calculated for each MPC such that its intervention can be minimized reflecting the driver's intention. After driver incompatibility is formalized, the optimal input is selected to minimize the incompatibility among the paths that can avoid accidents. The proposed algorithm is validated in simulations where collision can be avoided while minimizing the intervention.