Struct Compressor 

pub struct Compressor {
    pub model: Arc<Model>,
    pub state: State,
    pub scratch: ScratchBuffers,
    pub pdf_buffer: Vec<f64>,
    pub cdf_buffer_ac: Vec<u32>,
    pub cdf_buffer_rans: Vec<u32>,
    pub rans_freq_buffer: Vec<i64>,
}

Main compressor/decompressor that combines RWKV7 inference with entropy coding.

The compressor maintains internal state and pre-allocated buffers to minimize allocations during the compression/decompression hot path.

Fields

model: Arc<Model>

RWKV7 model for generating probability distributions.

state: State

Model state (recurrent hidden states).

scratch: ScratchBuffers

pdf_buffer: Vec<f64>

Pre-allocated PDF buffer (eliminates allocations in compression loop).

cdf_buffer_ac: Vec<u32>

Reusable AC CDF buffer (vocab_size + 1 entries).

cdf_buffer_rans: Vec<u32>

Reusable rANS CDF buffer (vocab_size + 1 entries).

rans_freq_buffer: Vec<i64>

Scratch frequencies for rANS quantization.

Implementations

impl Compressor

pub fn new<P: AsRef<Path>>(model_path: P) -> Result<Self>

Create a new compressor with the given model.

Arguments

• model_path - Path to RWKV7 model weights (.safetensors format)

Returns

A new Compressor ready for compression/decompression operations.

pub fn load_model<P: AsRef<Path>>(model_path: P) -> Result<Arc<Model>>

pub fn new_from_model(model: Arc<Model>) -> Self

pub fn reset(&mut self)

Reset the model state to initial values.

Call this between independent compression/decompression operations to ensure a clean state.

pub fn vocab_size(&self) -> usize

Get the vocabulary size (should always be 256 for byte-level).

pub fn compress(&mut self, data: &[u8], coder: CoderType) -> Result<Vec<u8>>

Compress data using the specified entropy coder.

Arguments

• data - Raw bytes to compress
• coder - Entropy coder to use (AC or rANS)

Returns

Compressed data including header with checksum.

pub fn compress_into<W: Write>( &mut self, data: &[u8], coder: CoderType, w: &mut W, ) -> Result<()>

pub fn compress_chain_into<W: Write>( &mut self, parts: &[&[u8]], coder: CoderType, w: &mut W, ) -> Result<()>

pub fn compress_size(&mut self, data: &[u8], coder: CoderType) -> Result<u64>

pub fn compress_size_chain( &mut self, parts: &[&[u8]], coder: CoderType, ) -> Result<u64>

pub fn decompress(&mut self, data: &[u8]) -> Result<Vec<u8>>

Decompress data.

Arguments

• data - Compressed data (must include header)

Returns

Original decompressed data. Returns error if checksum doesn’t match.

pub fn cross_entropy(&mut self, data: &[u8]) -> Result<f64>

Calculate cross-entropy (bits per byte) for data without compression.

This measures how well the model predicts the data, giving a theoretical lower bound on achievable compression. Useful for evaluating model quality.

Arguments

• data - Data to analyze

Returns

Average bits per byte (lower is better, 8.0 means no compression possible).

pub fn cross_entropy_conditional_chain( &mut self, prefix_parts: &[&[u8]], data: &[u8], ) -> Result<f64>

pub fn cross_entropy_conditional( &mut self, prefix: &[u8], data: &[u8], ) -> Result<f64>

pub fn joint_cross_entropy_aligned_min( &mut self, x: &[u8], y: &[u8], ) -> Result<f64>

Trait Implementations

impl Clone for Compressor

fn clone(&self) -> Self

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.