Should be able to modify the memory settings in
%JDK_HOME%\lib\visualvm\etc\visualvm.confXms and Xmx are in the default_options line.%JDK_HOME%\lib\visualvm\etc\visualvm.confXms and Xmx are in the default_options line.