Abstract: The successful development of shale oil and gas reservoirs is the biggest technological revolution in the oil and gas industry. Its key technologies are horizontal well drilling and fracturing, which are based on understanding the mechanical properties of reservoir rocks. Therefore, it is critical to obtain the reservoir mechanical parameters quickly, efficiently, and inexpensively. In this study, shale samples were collected from three basins in Southwest China, and the elastic modulus of shale in the indentation depth range of 0–5000 nm was obtained by nanoindentation experiments. Experimental results showed that different indentation depths had different physical characteristics. The shallower depths had the mechanical properties of single minerals, while the deeper depths had the mechanical properties of a multi-mineral composite. The difference between the two represented the cementation strength between the mineral particles. The error between the calculation results of the existing equivalent medium theoretical model and experimental data reached 324%. In this study, a weak cementation model was adopted, and three parameters obtained by nanoindentation experiments were considered: the soft component volume content, intergranular cementation strength, and mineral particle size. This solved the problem of assuming rather than calculating the values of some parameters in the existing model and realized the prediction of the macroscopic mechanical parameters of shale. The calculation error was reduced to less than 20%, and the test method and calculation model can be popularized and applied in engineering.