Monitor and manage remote GPU training jobs
The agent starts or monitors a machine learning training job on a headless GPU rig, checks for CUDA errors, and retrieves loss curves or checkpoints without requiring a display or direct SSH access.
How it works
Changes data in your accountsData and analysis
The agent connects to the remote GPU machine via the MCP server. It checks the status of running training processes using tools like nvidia-smi. If errors are found, it attempts to fix them (e.g., freeing stuck cards). It can also retrieve artifacts like loss curves or model checkpoints. This allows for unattended training runs that survive the chat session.
Prompt
Paste this into an agent that has the servers below connected. Adjust the names and dates to your own.
Prompt for Monitor and manage remote GPU training jobs
Check the status of the current ML training job on the GPU rig. If it's stuck, try to free the card. Otherwise, retrieve the latest loss curve data.MCP servers it needs
Source
The vendor page this playbook is drawn from, and the passage that supports it.
Operate a headless rig from your laptop: watch nvidia-smi , fix a CUDA error, free a stuck card, or grab the loss curve.Remote AI Agent Use Cases: Servers, GPU, Jobs & Files — AI Commander