Apparently they needed the right equipment that is extremely sensitive:

Albert Einstein therefore thought it was impossible to detect gravitational waves. And yet a number of scientists nevertheless conjured up instruments that were expected to succeed here. The first generation of instruments consisted of aluminium cylinders, weighing several tons, equipped with sensors. Pulses of gravitational waves should cause them to oscillate like the clapper of a church bell. However, these resonance detectors produced no results despite their having highly sensitive amplifiers.

The researchers therefore designed receivers which were even much more sensitive. Their principle is based on the thought experiment with the rubber sheet. For this purpose, we replace the start/finish circle with a laser, the turning point with a mirror, and imagine the ants to be the wave crests of a light signal. In order to detect the tiny delays in the arrival time, a second beam path must be arranged perpendicular to the first one so that the light waves of these two arms superpose.

At the end of 2015, the USA will put into operation aLIGO at two locations 3000 kilometres apart – second generation interferometric detectors, each with an arm length of four kilometres, which use many of the measuring technologies developed at GEO600. Near the Italian city of Pisa, Virgo is being expanded to have measuring arms three kilometres in length, and Japanese scientists are currently building the subterranean detector KAGRA of the same size. A first successful reception of the messages from space is expected during the next few years.

This is awesome I can’t wait to see what more they discover about gravity!  Here is also a cool video titled “gravity visualized”.