Effective text compression with simultaneous digram and trigram encoding

Abstract

A new approach to data compression based on digram encoding is presented. Statistical analysis of the linguistic property of digram co-occurrence in textual material leads to a new, more effective simultaneous digram and trigram encoding technique. A combined list of digrams and trigrams eligible for encoding is constructed by an original scheme that replaces some of the frequently co-occurring encodable digrams with so-called master trigrams. Using Shannon's second order Markoff process approximation to English, conditions which the master trigrams should satisfy are formulated and then tested on four representative textual files. The performance of the proposed technique is experimentally examined and com pared with the performance of the conventional digram encod ing. It is found that the simultaneous digram and trigram encoding produces better results with regard to the measured compression ratios and use of processing time for encoding.