Whatever connection they had with men was lost to time. The Rings of Power leaned into that aspect of Hobbit history when Nori and Poppy encountered a Stoor named Merimac in the desert of Rhûn.

However, Nori says something else in episode 2 ... (This is also where most of the dwarves' names from The Hobbit came from.) Name-wise, the inspiration for Gandalf was a dwarf named Gandálfr ...