Removing SVG from dom and memory

Question

I'm doing some music visualisation experiment and it draws data from dancer.js on canvas or svg, depending on chosen visualisation. I have 4 options: clear js on canvas, raphaeljs on both svg, and three.js in the last one.

To keep the sound element and prevent reloading dancer.js i load different scripts in the following manner:

            $('body > a').click(function(){
                $('canvas').remove();
                $('svg').empty();
                $('svg').remove();
                var id = $(this).attr('id');
                audioElement = $('#audio')[0];
                $.ajax({
                    type: "GET",
                    url: "js/"+id+"-ajax.js",
                    dataType: "script"
                });
            });

it should just remove the canvas and svg if there are any and different visualisation are caught by ajax in which there is

  init = function() {
    ...
  }
  init();

they share the same functions like onKick which is changed in the init respectivly to the loaded script. (onKick is for dancer.js to fire it upon a "kick" in the song being played).

it works quite good till the moment you start switching between them. switching between canvas and canvas makes almost no difference, but when i change from canvas to svg, and than back to canvas framarate drops drastically.

May it be because of the svg being held in the memory or some other suggestions?