Ezekiel 11:18-20

When they return there, they will remove from it all (traces of) its detestable things and all its repulsive things (remnants of paganism). And I will give them one heart (a new heart), and put a new spirit within them. I will take from them then the heart of stone, and will give them a heart of flesh (that is responsive to My touch), that they may walk in My statues and keep My ordinances and do them. Then they shall be My people, and I will be their God.

God removes the hardened sinful heart and replaces it with a soft and responsive heart when He makes us new. New means the old ways are abolished and gone, and we can live life connected to God. The new connection helps us follow His statutes and ordinances, keeping us away from sin and our old ways.