Instability of a capillary wave and breakup of a square cylindrical liquid column during its relaxation have been investigated numerically by simulating three-dimensional Navier-Stokes equations. For this investigation a computer code based on the volume-of-fluid (VOF) method has been developed and validated with published experimental results. The result shows that the agreement of numerical simulation is quite good with the experimental data. The code is then used to study the capillary wave and breakup phenomena of the liquid column. The investigation shows the underlying physics during relaxation of the square cylindrical liquid column, illustrates the formation and propagation of the capillary wave and breakup processes. The breakup behavior for the present configuration of the liquid column shows some significant differences from those predicted by conventional jet atomization theories. The formation of the capillary wave is initiated by the surface tension on the sharp edge of the square end of the cylinder and the propagation of the wave occurs due to the effect of surface tension force on the motion of the fluid. The propagation of the capillary wave to the end of the liquid column causes a disturbance in the system and makes the waves unstable which initiates the breakup of the liquid column. The characteristics of the capillary wave show that the amplitude of the swell grows faster than the neck of the wave and that of the tip wave grows much faster than other waves. The velocity of the liquid particle is dominated by the pressure in the liquid column.