Take the 2-minute tour ×

I'm using a static dictionary file with some words and values for this words. This values are not fixed sized, for example the is 1, love is 01, kill is 101 etc. When I try to compress a group of words, I traverse every word and look up to dictionary if a value exists for that word. If one exists I change the word with the value, if it doesn't exist I encode the word as bytes. After compression I got a chunk of bits, and because these dictionary values and uncompressed words are not fixed sized I can not group the bits and decode them.

I have thought about using 1 bit flag for every group of bits to determine it is compressed or uncompressed, but I can't detect the flag bit because of this unknown length of a codeword or regular word.

If I use a 1 byte delimiter, it still has problems. Let's say my delimiter is 00000000, and before the delimiter I have 100 and after delimiter I have 001, so we have 10000000000001, how am I supposed to know that which group of these bits are my delimiter? Can I use some other method to group these compressed/uncompressed bits to decode them? Thank you.

share|improve this question
add comment

1 Answer

First off,what language and system are you intending to deploy this? Many languages provide their own libraries and tools for compression and may suite your needs without major low-level design effors.

The answer here is to establish some more rigorous bookkeeping and file formatting to be able to undo the compression. Most compression systems have some amount of overhead in their file format which is why when you compress something twice you don't necessarily save anything and can actually increase the size of the file.

Often files take advantage of header at the start of a file to provide key information. which would be a good place to define any rules that are specific to the compressed file.

  1. create fixed size delimiter to use between code words only. This can be determined after analyzing the file but before actually writing out the compressed data.
  2. If you generate your delimiter rather than a fixed known value, include this as one of your header items.
  3. keep your header a simple ascii format so that you can easily extract it with standard tools like sscanf and fscanf.
  4. if you want to have a header that can contain extra information you may need a consistent way to tell where the header ends and the data begins. Including something to the effect of "ENDHEADER" should be enough and still easily identifiable.
share|improve this answer
I should develop this for a specific project, so I can't use any library or tool. And the question about a fixed size delimeter is on the last paragraph. –  gmnnn Nov 18 '12 at 18:31
add comment

Your Answer


