The rational planning of land around rail transit stations in cities can effectively improve the convenience of transportation and economic development of cities. This paper briefly introduced the transit-oriented development (TOD) mode of urban planning. We constructed a hierarchical structure for evaluating the quality of land planning of urban rail transit stations through the analytic hierarchy process (AHP) method. The structure started from three large aspects, i.e., traffic volume, regional environmental quality, and regional economic efficiency, and every large aspect was divided into three small aspects. Then, an optimization model was established for land planning of rail transit stations. The land planning scheme was optimized by a genetic algorithm (GA). To enhance the optimization performance of the GA, it was improved by coevolution, i.e., plural populations iterated independently, and every population replaced the poor chromosomes in the other populations with its excellent chromosomes in the previous process. Finally, the Jinzhonghe street station in Hebei District, Tianjin city, was taken as a subject for analysis. The results suggested that the improved GA obtained a set of non-inferior Pareto solutions when solving a multi-objective optimization problem. The distribution of solutions in the set also indicated that any two objectives among traffic volume, environmental quality, and economic efficiency was improved at the cost of the remaining objectives. The land planning schemes optimized by the particle swarm optimization (PSO) algorithm, the traditional GA, and the improved GA, respectively, were superior than the initial scheme, and the optimized scheme of the improved GA was more in line with the characteristics of the TOD mode than the traditional one and the PSO algorithm, and the fitness value was also higher. In conclusion, the GA can be used to optimize the planning design of land in rail transit areas under the TOD mode, and the optimization performance of the GA can be improved by means of coevolution.