Abstract: The segmentation of knee joint is of great significance for diagnosis, guidance and treatment of knee osteoarthritis. However, manual delineation is time-consuming and labor-intensive since various anatomical structures are involved in the 3D MRI volume. Automatic segmentation of the whole knee joint requires no human effort, additionally can improve the quality of arthritis diagnosis and treatment by describing the details more accurately. Existing knee joint segmentation methods in the literature focus on only one or few structures out of many. In this paper, we study the feasibility of knee joint segmentation on MR images based on neural networks and deal with the following challenges: (1) end-to-end segmentation of 15 anatomical structures, including bone and soft tissue, of the whole knee on MR images; (2) robust segmentation of small structures such as the anterior cruciate ligament, accounting for about 0.036% of the volume data. Experiments on the knee joint MR images demonstrate that the average segmentation accuracy of our method achieves 92.92%. The Dice similarity coefficients of 9 structures were above 94%, five structures were between 87% and 90%, and the remaining one was about 76%.