Since a mooring system design is designed by selecting many design parameters and considering various factors, a basic design is mainly done empirically. In this basic design process, studies have been conducted to improve mooring design using static analysis results as objective cost. Static analysis cannot capture all the effects of time domain analysis. In this study, among the mooring parameters in a specific environment condition, parameters that have a large influence on the tension on the line were selected and mooring system optimization was performed using bayesian optimization & ANN based GA that use line tension result in time domain as objective cost. In conclusion, both methods of the mooring system optimization suggested a mooring system with a tension that is 50% lower than the stability criterion for line breaking. In addition, when the two optimization methods were applied to the mooring system, the advantages and disadvantages of each optimization method were confirmed.