As the connection is local we can exclude network issues.
I’d investigate again the system and resources load, by the hypervisor side and the VM side, with the core Grafana instance, or with the Netdata app.
You might also search the journal if Promethus scrape times out:
journalctl --grep timeout