# Models

Models are neural networks that compete on the SOMA network. They are trained independently and submitted to compete against one another.

## What Models Do

A model on SOMA predicts the next byte in a sequence. Lower loss means better understanding.

## Architecture

Every model on SOMA uses the same architecture. Validators define this architecture and can upgrade it network-wide. The architecture is shared, but the weights are different for each model.

The current network architecture is a byte-level pre-norm transformer:

- **Layers**: 24
- **Embedding dimension**: 2048
- **Attention heads**: 8 (256 head dim)
- **FFN hidden dimension**: 8192 (4× expansion)
- **Vocabulary**: 264 tokens (256 byte values + 8 special tokens: PAD, EOS, etc.)
- **Max sequence length**: 1024 bytes
- **Positional encoding**: RoPE
- **Serialization**: safetensors (PyTorch and Flax implementations available)

For full architecture details, constants, and code, see the [Model Reference](https://docs.soma.org/reference/models/overview/).

## Loss

Loss is the network's universal scoring metric. SOMA uses [cross-entropy](https://en.wikipedia.org/wiki/Cross-entropy#Cross-entropy_loss_function_and_logistic_regression) loss to measure the average next-byte prediction error across a sequence. Lower loss means the model understands the data better, and the lowest-loss model in a competition earns rewards.

A [SIGReg](https://arxiv.org/abs/2511.08544) regularization term is added to prevent embedding collapse.

## Model Embedding

Each model also registers an embedding representing its own specialization, the domain it understands best. The network uses this embedding for [routing](https://docs.soma.org/concepts/targets/#model-assignment), determining which targets a model is assigned to. See [Your model's embedding](https://docs.soma.org/guides/model-strategies/#your-models-embedding) for how to compute, update, and strategically position it.

## Stake

Models [stake](https://docs.soma.org/concepts/economics) SOMA to participate in the network. The amount of stake a model holds directly determines its routing priority.

More stake means more routing priority via [stake-weighted KNN](https://docs.soma.org/concepts/targets/#model-assignment). Models that win more can re-stake for higher priority.

Model owners can set a commission rate for stakers who delegate to their model. For full details on staking, see [Economics](https://docs.soma.org/concepts/economics/#staking).

## Publishing Weights

SOMA models become active through a create-commit-reveal protocol:

1. **Create**: Register a new model with your stake amount and commission rate
2. **Commit**: Post a URL to your encrypted weights, a hash of your weights, and a hash of your embedding
3. **Reveal**: Publish the decryption key and your embedding in the next epoch
4. **Compete**: In the following epoch, you're eligible for competition

Models commit before they can see competitors' weights, then reveal simultaneously.

## How Models Compete

When [targets](https://docs.soma.org/concepts/targets) are created each epoch, the network assigns active models to each target via [stake-weighted KNN](https://docs.soma.org/concepts/targets/#model-assignment). From there:

1. [Data submitters](https://docs.soma.org/concepts/submitting) download those models' weights and run them locally on their data
2. The model with the lowest loss wins
3. The winning model's embedding of the data becomes the submission's embedding
4. If the embedding is within the target's threshold, the submitter can hit the target for a [reward](https://docs.soma.org/concepts/economics)

A model earns 50% of the target reward when its weights produce the winning submission.

## Updating Weights

Active models use the same commit and reveal steps to update weights:

1. **Commit**: Post your new encrypted weights and hashes with your model ID
2. **Reveal**: Publish the decryption key the next epoch
3. **Compete**: The epoch after