THEY missed something and it is a rather big something. When the Pennsylvania Railroad demolished Penn Station in 1964, it was described by The New York Times ''a monumental act of vandalism.'' Many New Yorkers and architectural historians still regret the loss of the station, and aficionados search the lower depths of the present station for remnants of the old one - handrails, elevator doors, lighting fixtures and the like.

But there is one little-noted element of the old station left, a monumental building in its own right, which is itself vulnerable to demolition. It is the grimy 1908 Penn Station Service Building at 242 West 31st Street, across the street from the present station.

As the 19th century gave way to the 20th, the Pennsylvania Railroad was chafing under a peculiar burden - it had no station in New York City. Passengers had to disembark in Jersey City, crossing the Hudson to Manhattan on ferries. It was an embarrassingly awkward entrance to the nation's greatest city for one of the nation's greatest railroads, and it lost passengers bound for destinations like Chicago, who could go by a competing line, the New York Central, from Grand Central Terminal.

Tunneling under the Hudson was less expensive than a bridge, but steam locomotives could not regularly negotiate the length of the tunnel. By the turn of the century electric engines, like those on the new IRT subway designed in 1900, were being developed, even for the heavier intercity trains.

In 1902, the Pennsylvania decided to proceed with a huge new station between 31st and 33d Streets from Seventh to Eighth Avenues, a block away from the emerging Herald Square shopping district.

The station was completed in 1910, a giant pink granite Roman-style structure surrounded by old brownstones and tenaments. But construction photographs of the station site in 1908 - when it was just a giant hole in the ground - also show a gleaming new building on the south side of 31st Street - the station's service plant.

The service plant held the key to the railroad's new operation, for it provided the electric power for the engines in and out of New York. Research by the industrial archaeologist Thomas Flagg indicates that it was also used to supply heat, light, elevator hydraulics and refrigeration for the station as well as compressed air for braking and signaling. It even incinerated the station's garbage.

The mid-block building, 160 feet long and 86 feet high, is divided by a north-south fire wall with boilers for power generation on the west side and power distribution, offices and other elements on the east.

The station and the service plant were designed by McKim, Mead & White, specifically Charles McKim and partner William Symmes Richardson. Writing in Transactions of the American Society of Civil Engineers for October 1910, Richardson said that, on the station itself, ''all unnecessary detail of ornamentation was omitted.''

For the service building the architects assembled some of the simplest elements from the station in the Stony Creek pink granite.

The Roman Doric exterior, a row of severe pilasters bracketing ventilation windows covered with iron grills, is about as plain as a building can get and still have an identifiable style. Cleaned, it could be a post-modern historical society or a crematorium.

WHEN the old Penn Station was demolished the service plant was not jeopardized, for it was just another nearby building. But it gradually lost its critical function.

Now it only provides compressed air to move switches. It controls but does not generate power for the station. One of the station's engineers, Mark Wurple, said that ''what goes out from here is basically just telephone lines'' that could be simply relocated. The western half is sealed off because of asbestos contamination and the eastern half is only partly used.

In 1986, there were proposals to replace the present Madison Square Garden with a new office tower and talk of new bulidings on the surrounding areas - including the service-plant site, which is adjacent to a large parking garage and a one-story church.

''The whole Penn Station complex is in a sort of constant flux,'' said Clifford Black, a spokesman for Amtrak, which owns the service-plant building. ''The whole Penn Station area may be ripe for redevelopment.'' The current weak market in office space has dampened interest in any real development, but it may be just a matter of time until there is the opportunity for a second battle over Penn Station.

The Penn Station Service Building, West 31st St., in 1910. (Avery Library)