Enhanced oil recovery (EOR) processes are applied to recover trapped or residual oil in the reservoir rocks after primary and secondary recovery methods. Changing the wettability of the rock from oil-wet to water-wet is named wettability alteration. It is an important factor for EOR. Due to their unique properties, nanoparticles have gained great attention for improving oil recovery. Despite the promising results, the main challenges of applying nanoparticles are related to the colloidal stability of the nanofluids in the harsh conditions of the reservoirs. In recent years, polymer-grafted nanoparticles have been considered as novel promising materials for EOR. The obtained results showed that adding a hydrophobic agent trimethoxy (propyl) silane on the surface of modified silica nanoparticles with polyethylene glycol methyl ether has an effective role in improving retention and wettability alteration, especially in the oil-wet substrate due to hydrophobic interaction. The modified silica nanoparticle by mixed polyethylene glycol methyl ether (Mn ~ 5000) and trimethoxy (propyl) silane showed a proper performance at a concentration of 1000 ppm and a salinity range of 2000–40,000 ppm. The obtained findings can help for a better understanding of the silica nanofluid modification with both hydrophilic and hydrophobic agents for the EOR application of near-wellbore.