[back] switch on

Tensors. Today:
TSAC on FPGA: Pressing 44.1k audio through an uart line at 9600 baud

This runs Fabrice Bellard's TSAC -- https://bellard.org/tsac/ -- the neural audio codec that compresses music to a few kilobits a second: a DAC autoencoder, residual vector quantisation and a Transformer that predicts every code for an arithmetic coder. The network, its trained weights (dac_stereo_q8.bin, tsac_stereo_q8.bin of the tsac-2024-04-08 release, MIT licence) and the coder's structure are his. What is new here is where it runs: the whole ENCODER, Transformer and range coder included, in the fabric of one Kintex-7 FPGA, in real time.

The hardware: an Enclustra Mercury+ KX2 FPGA module (AMD Kintex-7 XC7K160T-2, 2 GByte DDR3) on an Enclustra Mercury+ PE1 base board -- the board below, with the Ethernet cable in the left RJ45 and the Platform Cable on the JTAG header.

44.1 kHz stereo audio goes into the KX2 over Ethernet; the FPGA encodes it as it arrives; the compressed stream leaves over the UART at 7.21 kbit/s -- 196 times less than the PCM that went in. That fits a 9600 baud line, the slowest standard rate that carries it: 8N1 takes 10 bits on the wire a byte, so 9600 baud moves 7.68 kbit/s, and the stream's busiest second needs 7.96 kbit/s -- at 9600 baud at most 53 bytes would wait (this build runs its UART at 1 Mbaud). The host decodes the stream back to audio. The board's bytes are the reference encoder's, bit for bit: on 2026-11-03 a 12.6 s clip gave 11,045 bytes over the UART, every one equal to what the integer model on the host computes.

The datapath is the WALKER: one convolution engine of 512 multipliers, fed by a command list that the host computes beforehand -- every layer of TSAC, cut into tiles, one command a tile. Below: how to run it, how TSAC's layers work, and how the walker runs them.

Download: l8tsac-dist.tar.gz (13M): the prebuilt bitstream bit/top.bit, the design as Verilog with its constraints and Vivado flow (build/), the host tools and the example clip.

tar xzf l8tsac-dist.tar.gz && cd l8tsac
make prereq           # the Python venv and Bellard's model files
make run              # program the KX2, encode the clip on it, play the board's stream

How to

You need: the KX2 on the PE1, a Xilinx Platform Cable USB II on the PE1's JTAG header (J1700), an Ethernet cable from the host to the PE1's left RJ45 (Eth0), a USB cable to the PE1's micro USB (J603, the UART), and on the host Vivado's hardware server (Vivado or Vivado Lab), Python 3.10+, ffmpeg.

The network

The board's Ethernet is a direct link to one interface of the host, at 100 Mbit/s. The board is 192.168.77.3 and answers ARP; the host's interface needs a static address in 192.168.77.0/24 and nothing else -- no gateway, no DHCP (a DHCP client on that interface would wait forever and may take the address away). Find the interface (ip -br link: the one that goes UP when the cable is plugged in, often enx<mac> for a USB adapter), then either for this session:

sudo ip addr add 192.168.77.1/24 dev enx001122334455
sudo ip link set enx001122334455 up

or once, with NetworkManager:

nmcli con add type ethernet ifname enx001122334455 con-name kx2 \
      ipv4.method manual ipv4.addresses 192.168.77.1/24 ipv6.method disabled
nmcli con up kx2

make net IFACE=enx001122334455 does the first (with sudo). Check: after make prog, make status prints the walker's status words (all zero before a run). The UART appears as /dev/ttyUSB0 / /dev/ttyUSB1 when the micro USB cable is in (your user needs the dialout group); this build uses the second one.

Run

tar xzf l8tsac-dist.tar.gz && cd l8tsac
make prereq          # a Python venv (torch CPU, descript-audio-codec, soundfile, pyserial) and Bellard's
                     # model files from bellard.org/tsac into ~/.cache/tsac
make run             # program the FPGA, encode the clip on it, and play the board's stream on ffplay

make run first checks what it needs (make check: the venv's packages load, the model files, ffplay, vivado, the UART) and prepares the DDR3 image of examples/clip.wav once (run/image.bin: the weights, tables and the walker's command list, under a minute). Then it prints five steps:

== 1/5 program the FPGA: bit/top.bit over JTAG (Vivado; log in run/prog.log)
== 2/5 the image into the DDR3 over Ethernet (192.168.77.3): zeros, then the weights and the walker's program
   image  [####################]  ... MByte  ... MByte/s
== 3/5 the encoder starts ('G' on /dev/ttyUSB1); the clip (12.6 s) streams into its DDR3 ring at 44.1 kHz
== 4/5 the board's coded bytes on the UART (9600 baud would carry them)
   clip 12.6 / 12.6 s [################]   UART 11045 bytes  7.0 kbit/s
== 5/5 decode the 11045 bytes (1056 frames) on the host and play them (ffplay)
   bytes 11045  frames  412/1056 [######..........]  BUFFERING 4.8 s / ~... s [###.................]

1. program: the bitstream over JTAG (VIVADO=/path/to/vivado); the walker keeps its state, so every run programs it again;
2. upload: zero the DDR3 from the image's start to its top, write the image, read it back;
3. start: G on the UART starts the walker, the clip goes over Ethernet into the DDR3 ring at real time -- the board encodes as it comes;
4. UART: the board's range-coded bytes as they arrive, against the clip sent;
5. decode and play: the bytes go to the decoder as they arrive -- the range decoder and the integer Transformer frame by frame, the DAC decoder 32 frames at a time -- into a buffer, and ffplay plays the buffer as a stream on the speakers. Playback starts as soon as the buffered audio outlasts the decoding still to do, so it runs to the end without a gap. On a desktop CPU the integer decoder is ~26 times slower than real time, so for this clip most of it is buffered before the first sound; the board's encoder itself runs at 0.78 of real time. run/stream.wav keeps what was played.

make prep CLIP=your.wav SEC=10 codes your own clip (44.1 kHz stereo), make send runs the board without playing, make decode decodes run/got.bin again in one piece (run/out.wav), make status reads the walker's status words. kx2_host.py prep DIR --ref also runs the executor (~15 min): DIR/bytes.txt, the bytes the board must send, which the run then compares.

Building the bitstream yourself: build/ holds the emitted design (top.v, tsac_kx2.v, the constraints, the MIG's project file, the Vivado script): make -C build synth VIVADO=... -- about 40 minutes.

Listen

The first 12.6 s of the test clip, as sent to the board, and as decoded from the board's 11,045 bytes:

(The page's players are MP3 for size; examples/ in the archive holds the WAVs.)

The codec

TSAC is three networks and a coder. The DAC ENCODER turns the waveform into a latent: 1024 numbers every 512 samples (86.13 frames a second). The RESIDUAL VECTOR QUANTISER replaces each frame's latent by 12 codebook indices of 10 bits -- raw, 10.34 kbit/s. The TRANSFORMER predicts the probability of every code from everything before it, and the RANGE CODER spends -log2 p bits on each code: likely codes cost almost nothing, which takes the stream from 10.34 to 7.21 kbit/s on this clip.

The decoder (on the host) runs the same Transformer to recover the same probabilities, decodes the codes, and turns them back into audio with the DAC decoder. For that the board and the host must compute the probabilities identically, to the last bit -- so every stage on the board is integer arithmetic, and the host's decoder runs the same integer model (lm_emul.py).

How the layers work

The encoder: 29 convolutions

The encoder is a stack of 1-D convolutions over time. A first convolution (kernel 7) lifts the 2 audio channels to 64. Then four BLOCKS each do the same thing at a coarser time scale: three RESIDUAL UNITS look at the signal with growing reach (a kernel of 7 taps spread out by a dilation of 1, 3 and 9, so 7, 19 and 55 samples wide), and a DOWN-SAMPLER -- a convolution with stride 2, 4, 8 and 8 -- halves the time resolution by that factor while it doubles the channels. After the fourth block a frame is one row of 1024 channels, 86 rows a second, and a last convolution (kernel 3) gives the latent.

The activation is SNAKE, x + sin^2(a x) / a with a learned a per channel: a periodic nonlinearity that suits audio. Every residual unit adds its input back (x + f(x)), so a unit only has to learn a correction. The cost is in blocks 2 and 3: 10.1 G multiply-adds a second each, 30.7 G in all.

How the walker runs a residual unit: the epilogue after the multiply-add array can add a residual and apply a Snake on the way out, so the six steps of a unit become two TILE KINDS -- the kernel-7 convolution with its Snake, then the kernel-1 convolution with the residual add and the next unit's Snake. Each tile writes up to two results per row: o, the raw value the next residual needs, and n, the Snake of it that the next convolution reads.

The quantiser and the Transformer

The RVQ codes a frame in 12 rounds. Each round projects the residual (1024 values) down to 8 (in_proj), compares it with the codebook's 1024 entries of unit length -- a matrix product, so the conv engine computes the 1024 scores as 32 tiles -- and sp_amaxr keeps the best. That entry, projected back to 1024 values (a table in DDR3), is subtracted from the residual, and the next round codes what is left. Coarse codebooks first, fine ones last.

The main Transformer sees the frames: its input at frame t is the latent the codes of frame t - 1 stand for (the sum of their entries), and 12 GPT-style blocks of width 512 look back over up to 1024 frames (11.9 s). A block is a LayerNorm, the q, k, v projections, rotary positions on q and k, attention in 4 heads of 128, an output projection with a residual add, another LayerNorm and an MLP (512 -> 2048 -> 512, GELU) with a residual add. On the walker every projection is a tile kind (a linear layer is a convolution of kernel 1), the keys and values of every frame are kept in DDR3 packed as weight words, so the scores q . K and the weighted sum p . V are tiles too; LayerNorm, softmax and the rotary positions are ROW OPS beside the engine. At frame 1024 TSAC resets to the last 256 frames, and the walker runs those again at positions 0..255.

The codebook decoder then predicts the 12 codes of the frame one by one: 3 more blocks over 12 positions, the input at position q the sum of the entries of codes 0..q-1 plus which codebook comes next, and proj_f turns the result into 1024 logits -- one per possible code. sp_coder takes the logits and the code the RVQ chose and emits the bits: ten binary decisions down a tree of the 1024 codes, each probability from an exp table.

The walker

The walker (sp_walk) is the datapath and its sequencer in one. It is not a processor -- it has no registers, no branches but the batch loop, and no arithmetic of its own; it reads a COMMAND LIST that the host computed (kx2_host.py prep), and each command names one piece of work and where its data lies in DDR3.

One engine, many layers. Every convolution, linear layer and attention product of TSAC is the same operation -- output row = sum over taps and input channels of weights x inputs -- so one engine runs them all:

• sp_window holds a window of input rows, 16 channels (int16) to a chunk, and reads it in im2col order: for each output row, tap by tap, chunk by chunk.
• sp_wbuf holds the weights in two banks of 512 words, a word the 16 x 32 int8 weights of one chunk; one bank is filled from DDR3 while the other computes, and a layer whose weights fit stays RESIDENT for the whole batch.
• sp_dot is 32 columns of 16 DSP48 slices, chained through their cascade ports: each clock it multiplies one chunk of 16 inputs by 16 x 32 weights and adds them into 32 running sums -- one per output channel. A row of a kernel-k layer with cin inputs takes k x cin / 16 clocks.
• sp_epi turns each 32-bit sum into the layer's output, four lanes at a time over 26 pipeline stages: bias, requantisation to int16, the residual add, Snake (a sin^2 table) or GELU.

A tile is one command: up to a window's worth of output rows times 32 output channels. The command says which input ring, the convolution's kernel, dilation and stride, the weights' address, the output rings. The walker FETCHES it (1024 bits, two DDR3 lines), works out this batch's row numbers, FILLS the window with only the input rows it does not hold yet, loads WEIGHTS and PARAMETERS if this layer's are not resident, streams the SKIP rows of a residual, and RUNS the rows through the engine; the results go back to DDR3 as one line a row.

Memory is all rings. Every tensor -- the audio, each layer's output, the Transformer's K and V -- is a ring of rows in DDR3, 32 channels of int16 to a 512-bit line, so a tile's rows of one channel group are one burst. A ring is four batches deep, so a layer may run behind the one before by up to that much.

The batch is 16 384 samples, 32 frames. op 2 waits until the audio ring holds them, then come that batch's tiles of every encoder layer, the RVQ of its frames, the Transformer's block of 32 frames and the coder, and op 0 closes the batch. The commands in one batch of this clip (from the image that ran):

The encoder is 1 692 of the tiles every batch; the rest is the Transformer, and it grows with the context (the attention looks at more frames) until the reset. The whole 12.6 s clip is 351,344 commands, 155.8 M clocks for a second of audio: a real-time factor of 0.78 at 200 MHz.

Integers throughout: weights int8 with a scale per output channel, activations int16, sums 32 bit; LayerNorm in Q10, attention scores in 1/64 nat, softmax to Q8, rotary in Q14. The scales are fixed when the image is made (the encoder's and the RVQ's from the clip's first 3 s, the Transformer's from a fixed calibration clip), and the host's Python model (hw_emul.py, lm_emul.py) does the same integer operations -- that is why the board's bytes can be compared byte for byte.

The system

The host prepares an IMAGE of the DDR3 (kx2_host.py prep): the weights and parameters, the RVQ's tables, and the command list. kx2_host.py run writes the image over Ethernet, starts the walker with a G on the UART, then sends the PCM at real time and reads the coder's bytes back from the UART.

The link (dbg_link) is a debug link of this project over UDP, at 100 Mbit/s on the PE1's left RJ45 (Eth0): bus 0 is a memory master on the DDR3 (the image, at ~7.7 MByte/s), pipe 0 a held stream for the PCM (each datagram acknowledged), bus 1 the status words (kx2_host.py status).

The memory: 2 GByte of DDR3 behind Xilinx's MIG, one 512-bit port at 200 MHz, shared by the link, the PCM writer and the walker through a round-robin merge. An adapter between the merge and the MIG XORs the row into the bank bits (the walker's streams sat in one bank and paid a page miss a line) and registers both directions. The PCM writer (sp_pcm) stores a sample a line into a ring of 64 K lines (1.5 s), at most one store every 1024 clocks, so its lines never break up the walker's bursts.

On the KX2

The routed design (Vivado 2026.1, XC7K160T-2, 200 MHz), and inside the walker part by part:

Timing: WNS -0.060 ns, TNS -0.707 ns over 28 of 261855 endpoints; hold WHS 0.035 ns. The DSP slices are nearly all spent (596 of 600): the encoder's 30.7 G multiply-adds a second need the 16 x 32 array busy most of the time.

Later: the tensor unit behind RISC-V AME

The walker is built for this one network. The project's general tensor unit, shape3 -- a TPU-style unit that a RISC-V core drives through the Attached Matrix Extension (AME), with C kernels -- runs in simulation; at the width TSAC needs it does not fit the 160T beside the DDR3 controller and the link. It is the next step, on a larger part, with stream engines like this one beside it.

Files

Measured 2026-11-03. Fabrice Bellard's TSAC: https://bellard.org/tsac/.

[back]