New The SREYUN website is live: five in-house products and reference pricing Learn more →

Distributing LLM weights and launching inference on private GPU clusters

AI infrastructure For: AI platform teams self-hosting LLMs Products: dlmodel

Reliably pull ModelScope / Hugging Face weights on network-restricted GPU servers, resume after disconnects, verify and store them, then hand them straight to vLLM / Ollama.

The challenge

  • LLM weights often run to tens of gigabytes, and downloads on private or restricted networks keep breaking and need retries.
  • Several models and versions queue up with no visible progress, and someone has to watch for completions and failures.
  • The download layout does not match the inference service, so files need sorting by hand.

The solution

Deploy dlmodel on the GPU servers:

  • Multi-source fallback: the auto source tries ModelScope first, then an HF mirror, with aria2 multi-connection downloads that step down to resumable HTTP and the native SDK on failure.
  • Resumable: pause, resume and continue from where it stopped; after a restart, tasks are requeued automatically.
  • Verified storage: LFS files are checked against the SHA-256 recorded in the repository and saved in the vLLM layout, sharing the model volume with vLLM / Ollama.
  • Notifications: start, finish, failure and cancel events go to DingTalk, Feishu or WeCom.

The value

  • Reliable access to mainstream open-source weights without a proxy.
  • One path from download to inference, with vLLM start commands generated from the model library.
  • MIT open source, SQLite storage, deployed with one docker compose command.