The call records theft - security of batch processing

Batch processing is most often overlooked during any security analysis. The main reason is that batch processing operates on millions upon millions of records at a time, and does that at a very fast rate. The second reason is that batch processing usually functions as a 'black box' with little input or intervention of the users, so it is easily forgotten by Security Officers.
But batch processing programs can contain a very dangerous covert code which if not investigated would go unnoticed for years

Example scenario

One of the largest batch processing systems in the world are telecoms billing systems. As an introduction, here is the generic process of billing in a Telco environment

  1. The call data is recorded by the Telco switches in specific records called Call Detail Records (CDRs). They can be sent online to billing but in most cases for speed and redundancy are simply saved in files
  2. The CDR files are then transferred to a Mediation System.
  3. The Mediation System is a conversion program that knows how to read the file formats of each switch manufacturer and version (Nokia, Siemens, Ericsson, Nortel...) and to convert the information in each CDR into a consistent and unified record format for all calls in the network.
  4. The converted unified CDR records are written to the central billing database.
  5. The billing software reads through the CDR records in the billing database, identifies each originating phone number and his owner and applier tariffs and discounts to the call.

Steps 3 and 5 are usually batch processing programs, that run at least once a month, but is usually run every week or every fortnight, to distribute the overall processing onto smaller chunks.

A programmer or engineer with malicious intent can insert a covert process in either the mediation or billing processes which can:
  1. Modify CDR's to reduce his or other costs or transfer costs to other owners
  2. Modify CDR's to erase records of calls being made
  3. Search for and collect specific information for calls made to and from a telephone number for blackmail, sale or publication purposes.

The points 1 and 2 are very well addressed in a Telco environment, since they impact the income of the operator. Therefore, fraud and internal audit departments are always hunting for any indication of billing data modification, and are constantly matching the resulting billing to CRM trends and reports from traffic analysis software.

Point 3 however is poorly addressed, since information leakage in such an environment is difficult to identify, and since the impact of information leakage is not directly visible on the bottom line, it can go on unnoticed for years.

Here is a detailed attack analysis

  1. A malicious attacker would install a covert process into the mediation or billing system.
  2. He would issue a command to the covert process using ping, e-mail, or simply send a patch with a change in config parameter as a command to sift through the processed data and to make simple copies or summaries of certain information, by originating or destination phone number.
  3. The copied information would be hidden in a file, possibly even encrypted, and can be sent out using any number of covert channels like:
  4. Encapsulating the information in slow rate ping payload and sending it to a Internet server
  5. Encapsulating the information in slow rate ping payload and sending it to a compromised client on the network, from where it can be sent further via ping, email or http.
  6. Burying the file somewhere on the computer drive to be copied during a patch or maintenance visit

There is an excellent market for this information, ranging from private investigators collecting information for a client to criminal groups preparing a blackmail.

Security Verification of Batch Processes

The process of verification of batch processes should be done at least once a year and at any time a possible breach of confidentiality is identified. The the process below is a generic one, and can be applied to any batch process.

  1. Enumerate batch processes and map them against manufacturer technical documentation to identify expected triggered processes.
  2. For each process use Process Explorer (for windows) and lsof and strace/truss (for unix/linux variants)
  3. Set a temporary full file access audit to review which files are being touched during the process - the results will probably be enormous, but can yield additional insight.
  4. Run a passive sniffer running on the LAN interface of the investigated server to identify where is the information travelling and in which types of packets
  5. Run the process both on test systems and on production, the covert code can be set-up to run only if operating on production to avoid detection - billing and mediation servers change rarely, so the covert process can be configured to enable itself ONLY if the MAC address of an interface is set to something, or the IP address is set to some value etc.

The verification process is very long and difficult, so be prepared for a lot of screen staring!

Related posts

8 Tips for Securing from the Security experts

Talckback and comments are most welcome


Michael Munoz said...

File access auditing is also highly efficient security practice.

In our company we use file system auditor that can audit and report in real-time mode on any file access attempt or operation.

Jody Dewey said...

Would it be possible to have a baseline hash value of each file of the process and periodically re-run the hash to see if the file was modified?

Designed by Posicionamiento Web