While several approaches have been proposed to optimize the geometrical dimensions of multilayer photonic nanostructures with a given material composition, very few works have considered simultaneously optimizing the material composition and dimensions of such nanostructures. Here, we develop a hybrid optimization algorithm as a method to design optimal multilayer photonic structures. Leveraging recent progress in metaheuristic optimization, we develop an optimization method consisting of a Monte Carlo simulation, a continuous adaptive genetic algorithm, and a pattern search algorithm. We first perform a Monte Carlo simulation over the entire design space. Structures are ranked according to the chosen fitness function. We find that this method yields viable material compositions. The material compositions of the best structures are used to parameterize the genetic algorithm in the next stage. A number of genetic algorithm populations are generated, one for each material composition, to optimize the thicknesses. These populations are run in parallel for a number of generations, evaluating the structures of each generation and using the characteristics of those that best satisfy the fitness function to improve other structures. The resulting populations converge towards the optimum of their solution space typically after a few thousand generations. The genetic algorithm used is continuous because parameters are treated as real numbers rather than bit strings as in classical genetic algorithms, and adaptive because the algorithm uses characteristics of the population pool to guide optimization. Finally, we apply a pattern search local optimization algorithm to the best result from each population to find the exact optimum.