In this article, we consider a simple step-stress model for exponentially distributed lifetime units. As failure rate is lower at the initial stress level, therefore, it is important to pay more attention to the stress changing time. Here, we consider a simple step-stress model where the stress level changes either after a prefixed time or after a prefixed number of failures, whichever occurs later. It ensures a prefixed minimum number of failures at the first stress level and also sets up a control on the expected experimental time. We have obtained the maximum likelihood estimators of the model parameters along with their exact distributions. The monotonicity properties of the maximum likelihood estimators have been established here, and it can be used to construct the exact confidence intervals of the unknown parameters. We provide approximate and bias-corrected accelerated bootstrap confidence intervals of the model parameters. We also define an optimality criteria and based on that obtain an optimal stress changing time for a given sample size. Finally, an extensive simulation study has been performed to asses the performance of the proposed methods and provide the analyses of two data sets for illustrative purpose.