Machine learning server requirements

Machine learning needs a server sized for the job: strong CPU for data prep, enough RAM to hold batches, fast storage for checkpoints, and GPU capacity when training or serving models would outgrow ordinary compute. Train on rented GPUs; infer close to users.

Start with the workload, not the brand

The bad purchase usually begins with a tidy sentence: we need a machine learning server. It sounds sensible and hides the bill. Training burns matrix maths, memory bandwidth, storage writes, and retry time. Inference is a different bill: latency, predictable throughput, model size, and user geography.

For a small tabular model, a CPU server can be the honest answer. For image, audio, recommendation, and language workloads, the GPU becomes the billable bottleneck. A provider logo cannot rescue a dataset that misses memory or a queue stuck behind slow storage.

Use this order before you compare hosts:

• Dataset shape: rows, files, feature width, and whether preprocessing expands the data in memory.
• Model phase: training, fine-tuning, batch inference, or live inference.
• Memory pressure: GPU VRAM first, then system RAM, then disk spill.
• Failure cost: checkpoint size, restore time, and whether a stopped job wastes paid accelerator time.
• Serving geography: user latency, data residency expectations, and traffic leaving the region.

CPU still matters before the GPU wakes up

GPU marketing makes CPU look secondary. That is only true after the data is already clean, tokenised, batched, and waiting in the right format. ETL, decompression, feature generation, validation, and small-model experiments still lean on CPU and RAM.

HostScout hardware records show the spread. AMD EPYC 7763 is a many-core server CPU with 64 cores and 128 threads. Intel Xeon E-2388G is a smaller server CPU with 8 cores and 16 threads. They are not substitutes; they sit in different bottlenecks.

GPU choice is mostly memory, queue time, and locality

The first GPU question is not which chip has the loudest benchmark. It is whether the model, batch, and framework overhead fit in VRAM without constant offload. After that, inspect memory bandwidth, available regions, billing unit, and whether capacity is there when the job queue starts.

HostScout hardware data lists NVIDIA A10 24GB with 24 GB VRAM and 600 GB/s memory bandwidth. NVIDIA RTX 4090 24GB also has 24 GB VRAM, with 1008 GB/s memory bandwidth. NVIDIA L40S 48GB doubles the VRAM to 48 GB. NVIDIA A100 80GB has 80 GB VRAM and 2039 GB/s memory bandwidth.

Those numbers are not a universal ranking. A cheaper GPU that keeps the model resident can beat a stronger GPU that spills to CPU memory. A training job also needs checkpoint storage and enough CPU workers to feed the card. For inference, a smaller GPU near users may be better than a distant flagship.

Provider examples for training and inference

HostScout data includes broad cloud providers, specialist GPU hosts, and regional options. Some entries expose a public monthly price. Others expose configuration but not a comparable public price, so treat them as availability examples until the provider page confirms the current bill.

Lambda Labs is useful when the buyer wants GPU shapes such as 1x NVIDIA H100 SXM with 26 CPU cores, 225 GB RAM, and 2816 GB SSD in the HostScout record. Gcore is relevant when region coverage matters; its Everywhere Inference 1x L40S record includes 1 GPU, 16 CPU cores, and 232 GB RAM.

That spread is the point. Training capacity and inference capacity are bought differently. Training can tolerate queueing if the job is batchable and checkpoints are safe. Inference pays for idle capacity because latency spikes are visible to users. The invoice tells the truth here, even when the benchmark chart looks flattering.

When to train on rented GPUs

Rent GPUs for training when the model is large, the training window is occasional, or the team has not measured its final hardware shape. The buyer should care about cancellation, storage persistence, checkpoint recovery, and whether the provider bills by hour or month.

Vultr and Lambda Labs fit this conversation because the records include multi-GPU shapes. Gcore also exposes bare-metal GPU shapes and inference-oriented GPU records. Use those records to shortlist capacity, not to skip the provider-page check.

Avoid a monthly bare-metal commitment while the team is still changing model family, precision, or batch size. The expensive mistake is not a slower epoch. It is paying for a large accelerator while the job fails on storage, memory, or data loading.

When to serve inference near users

Inference is a different server purchase. A live model needs predictable latency, health checks, warm capacity, and a region close to the request path. If most users are in North America or Europe, provider geography can matter more than peak accelerator performance.

DigitalOcean, Scaleway, Hetzner, Hostkey, and Netcup can be reasonable inference candidates when their regions, support boundary, and billing model match the service. Compare them against your measured request rate, not a generic best GPU list.

For inference, VRAM is the hard wall, but latency is the business wall. A model that fits on a smaller GPU and answers near users is often safer than a larger remote host that turns every request into a network problem.

A practical sizing rule

Start with the smallest server that can run a complete representative job, then scale the bottleneck you measured. If the run is CPU-bound, a larger GPU will sit idle. If the run is VRAM-bound, more CPU will not save it. If the run is storage-bound, both can look slow.

For a first shortlist, use CPU-only servers for feature engineering and small models, single-GPU plans for fine-tuning and inference tests, and multi-GPU bare metal only when the training job is parallelised and checkpoints are tested.

HostScout’s public data is useful because it keeps provider plans, hardware records, locations, and prices separate. Affiliate relationships do not change the hardware facts or the order in which you should test bottlenecks. Find the idle line item before buying the impressive one.