PyPI page
Home page
Author:
FiveAttn Team
Summary:
100M-parameter Transformer comparing 5-Component Attention (GAU) and MHA, with SmolLM tokenizer, Cut-Cross-Entropy, SwiGLU, and RMSNorm
Latest version:
0.1.1
Required dependencies:
cut-cross-entropy
|
datasets
|
huggingface-hub
|
numpy
|
pyarrow
|
tokenizers
|
torch
|
transformers
Optional dependencies:
build
|
pytest
|
twine
Downloads last day:
0
Downloads last week:
6
Downloads last month:
261
7-day smoothing