Is there a performance hit for using extrude() or rotate_extrude()? I did
some simple testing and there doesn't seem to be much of one. I used a
simple 8x8x8 cube done both ways, the same cube with a 6x6 cutout in the
middle, and a sphere(6) with a sphere(3) cut out inside of it. Here's what
I got, giving item, preview time, and render time for a group of 100
objects with the cache flushed between each run:

// cube_p 0:00:00.027 0:00:04.884
// cube_x 0:00:00.018 0:00:04.890
// hole_d 0:00:00.034 0:00:09.503
// hole_x 0:00:00.031 0:00:09.621
// sphere_d 0:00:00.021 0:03:19.630
// sphere_x 0:00:00.032 0:03:08.771

Extrude seems to be competitive. Is that expected?

Code follows if you want to see what I did.

module cube_p() {
cube(8);
};

module cube_x() {
linear_extrude(8) square(8);
};

module hole_d() {
difference() {
cube(8);
translate([1, 1, -1]) cube([6, 6, 10]);
};
};

module hole_x() {
linear_extrude(8) {
difference() {
square(8);
translate([1, 1])
square(6);
};
};
};

module sphere_d() {
$fn = 16;
difference() {
sphere(4);
sphere(3);
}
};

arc1 = [ for (i = [1:2:15]) [4 * sin(i/2 * 360 / 16), 4 * cos(i/2 * 360 /
16)]];
arc2 = [ for (i = [15:-2:1]) [3 * sin(i/2 * 360 / 16), 3 * cos(i/2 * 360 /
16)]];
a1 = concat(concat([[0, arc1[0].y]], arc1), [[0, -arc1[0].y]]);
a2 = concat(concat([[0, arc2[0].y]], arc2), [[0, -arc2[0].y]]);

slice = concat(a1, a2);

module sphere_x() {
$fn = 16;
rotate_extrude() polygon(slice);
};

for (i = [0:10:90], j = [0:10:90]) {
translate([i, j]) sphere_x();
};

Nophead is the expert here, but I think the simple summary is that 2D
operations are much faster than 3D operations, so if you can build your
model by combining 2D shapes and then extruding them, you're better off
than combining 3D operations.

That is:

    linear_extrude(height=10, center=true) {
        difference() {
            square(10, center=true);
            circle(d=9);
        }
    }

is faster than

    difference() {
        cube([10,10,10], center=true);
        cylinder(h=30, d=9, center=true);
    }

The render of multiple copies of sphere_x() and sphere_d() is not a good
way to compare their runtimes. OpenScad will render the sphere once, cache
it and translate this cached to the copy positions. Most of the render time
is spent in the union of the spheres.

To have a fair comparison of the two ways of defining the holed spheres you
should render one sphere at a time with a high value of $fn. As your slice
have a fixed resolution I redefined sphere_x() as:

module sphere_x() {
$fn = 40;
rotate_extrude() {
intersection() {
difference() { circle(4); circle(3); }
translate([0,-5]) square(10);
}
}
};

and compared its render with:

module sphere_d() {
$fn = 40;
difference() {
sphere(4);
sphere(3);
}
};

I found that the render of the rotate_extrude version is 63 times faster
than the other.