Solving nonlinear matrix equations arising in tree-like stochastic processes