On certain theoretical problems of transmission characteristics and nonlinear phenomena in optical fiber waveguides