Model Development
Models are the network’s intelligence. Every model on SOMA shares the same architecture — a byte-level transformer — but each has unique weights that determine how well it understands different domains. Model developers train these weights, publish them on-chain, and earn commission when their model’s embeddings are used by winning data submissions.
V1 Architecture
Section titled “V1 Architecture”The current architecture is a pre-norm byte-level transformer:
| Parameter | Value |
|---|---|
| Embedding dimension | 2048 |
| Layers | 24 |
| Attention heads | 8 |
| Feed-forward hidden dim | 8192 |
| Vocab size | 264 (256 bytes + PAD + EOS) |
| Max sequence length | 1024 |
Models are scored using cross-entropy loss plus SIGReg (Gaussian uniformity regularization), which ensures embeddings are well-distributed across the embedding space.
Model Lifecycle
Section titled “Model Lifecycle”- Train — Train your model weights using PyTorch or Flax with
soma-models - Encrypt — Encrypt weights with AES-256-CTR before uploading
- Commit — Post a cryptographic commitment of your weights on-chain
- Reveal — Publish the actual weights, decryption key, and embedding in the next epoch
- Compete — Your model is sampled to score data submissions each epoch
- Earn — Receive commission when your model’s embeddings are used by target winners
For more on how models compete and why weights are public, see Model Competition.
Guides
Section titled “Guides” Train a Model Training workflow with PyTorch or Flax using the soma-models library.
Register a Model On-chain commit-reveal registration and ongoing model management.