Effective potential in density matrix functional theory