PollikAI Empirical language model pretraining from token zero.
An independent research project training a 498M-parameter Transformer from completely random initialization on the FineWeb-Edu corpus. Engineered with Grouped-Query Attention, SwiGLU, and 8-bit optimizer state optimization to investigate pretraining efficiency and scaling boundaries on constrained hardware.
Independent Research Grounded in First Principles
Exploring pretraining stability, loss dynamics, and memory conservation without relying on third-party pre-trained weights.
Pretraining From Scratch vs. Fine-Tuning
Most contemporary AI applications fine-tune or apply lightweight adapters (LoRA) on top of existing commercial foundations. While practical for product deployment, this obfuscates the core mechanics of foundational learning.
PollikAI was initiated to study pretraining from absolute ground zero. Every weight is initialized randomly. Every token processed by the model contributes to its initial cross-entropy entropy reduction. By designing and orchestrating our own PyTorch training pipeline, we directly measure the impact of attention pooling, non-linear activation shapes, and numerical precision on early pretraining dynamics.
Initial stage 1 pretraining validates algorithmic stability and memory frugality on consumer hardware. Further scaling of dataset volume and sequence length is architected for distributed cloud execution, designed to leverage accelerated AWS EC2 compute instances (via AWS Activate credit sponsorship).
Scientific Transparency
We maintain complete transparency regarding project structure and research scope:
Model Parameters & Technical Dimensions
Quantitative breakdown of the 498M parameter autoregressive architecture currently running in pretraining.
| Parameter Attribute | Configured Value | Design Rationale & Technical Context |
|---|---|---|
| Total Trainable Parameters | 498,293,760 | Exact parameter count calculated across all 26 decoder blocks and vocabulary projection. |
| Model Family | Decoder-Only Transformer | Causal autoregressive language modeling with next-token prediction cross-entropy loss. |
| Hidden Dimension (d_model) | 1024 | Internal vector representation width across all attention heads and feed-forward residual streams. |
| Transformer Layers | 26 Layers | Uniformly stacked blocks combining pre-layer normalization, self-attention, and gated MLPs. |
| Attention Mechanism | GQA (16 Q / 4 KV Heads) | Grouped-Query Attention with a 4:1 query-to-key-value ratio, reducing KV cache footprint by 75%. |
| Feed-Forward Activation | SwiGLU | Gated linear unit with intermediate dimensional scaling for improved non-linear representational capacity. |
| Layer Normalization | RMSNorm | Root Mean Square pre-normalization applied to attention and MLP inputs for strict gradient stability. |
| Positional Embeddings | RoPE (Rotary Position) | Algebraic relative position encoding applied directly to query and key vectors in complex subspace. |
| Context Window Length | 1,024 Tokens | Configured sequence length for stage 1 pretraining; expandable to 2,048+ tokens with cloud compute. |
| Tokenizer & Vocabulary | GPT-2 BPE (50,257 Tokens) | Byte-pair encoding tokenizer ensuring stable subword coverage across general text and code. |
| Numerical Precision | Native PyTorch BF16 Mixed | Bfloat16 precision provides identical dynamic range to FP32, preventing underflow without dynamic loss scaling. |
| Memory Optimization | AdamW8bit + Grad Checkpointing | Reduces optimizer memory footprint from 16 bytes/param to 2 bytes/param, fitting 498M parameters into 8GB VRAM. |
GQA Attention Efficiency
By pairing 16 query heads with 4 key-value heads, the model preserves high-capacity multi-head attention expressiveness while slashing memory bandwidth bottlenecks during inference and training.
SwiGLU Non-Linearity
Replaces standard ReLU or GeLU activations with a gated Swish product, accelerating learning rate absorption and yielding consistently lower validation perplexity per training step.
Rotary Positional Embeddings
Rotary position encoding (RoPE) preserves relative token distance semantics algebraically, simplifying subsequent context window expansion (e.g. via NTK-aware scaling).
8-Bit Optimizer Footprint
Blockwise 8-bit quantization of AdamW first and second momentum states keeps optimizer VRAM within 1.0 GB, allowing batch sizes that maintain gradient variance stability on consumer hardware.
Validation Loss Trajectory
Continuous empirical validation on held-out splits of the FineWeb-Edu dataset over the first stage.
Pretraining Observations & Verification
The empirical curve verifies that the custom PyTorch implementation (including the custom RoPE rotary kernels, GQA attention masking, and SwiGLU forward pass) behaves strictly according to standard autoregressive scaling trajectories. Loss decays smoothly without oscillation, indicating appropriate learning rate scheduling (cosine decay with warm-up) and weight initialization scaling.
Decoder Block Topology
Pre-RMSNorm architectural pipeline designed for numerical stability at scale.
Transformer Block (Layer ℓ ∈ [1, 26])
from dataclasses import dataclass
@dataclass
class PollikAIConfig:
# Architectural Dimensions
model_name: str = "PollikAI-498M"
total_params: int = 498_293_760
n_layers: int = 26
d_model: int = 1024
# Attention Configuration (GQA 4:1)
n_query_heads: int = 16
n_kv_heads: int = 4
head_dim: int = 64
max_seq_len: int = 1024
vocab_size: int = 50257
# Non-Linearities & Normalization
activation: str = "SwiGLU"
norm_type: str = "RMSNorm"
pos_embedding: str = "RoPE"
# Precision & Hardware Bounds
precision: str = "bfloat16"
gradient_checkpointing: bool = True
optimizer: str = "AdamW8bit"
dataset: str = "FineWeb-Edu"Current Workstation Infrastructure
Sustaining pretraining on consumer silicon through aggressive memory management.
Consumer Silicon Baseline
All 900M+ tokens have been pre-trained locally on a single desktop workstation. This serves as an empirical proof-of-concept for the numerical stability and memory frugality of our pipeline.
The Compute Scaling Analysis
Mathematical comparison of pretraining duration across compute tiers. Cloud access compresses months of training into days.
Pretraining Wall-Clock Duration
Select pretraining token budget to compare turnaround time across hardware setups:
Compute Sponsorship & Contact
Inviting cloud compute sponsors, AWS credit reviewers, and independent research partners to scale this work.
Direct Inquiry
We are actively seeking compute sponsorship and cloud credit support, specifically through the AWS Activate program, to scale our upcoming multi-billion token pretraining runs across accelerated EC2 clusters. We welcome discussions with program evaluators, mentors, and prospective research sponsors.
Deliverables Available Upon Review
- 01. Complete PyTorch pretraining codebase, tokenization scripts, and configuration manifests.
- 02. Verifiable evaluation loss checkpoints, token logs, and convergence monitoring data.
- 03. Technical write-ups documenting consumer hardware optimization techniques and scaling bounds.