I've explained some of my points and given examples to try and help. The code is pretty basic and only serves to be used as an example. Hopefully this will help, I'm willing to explain anything.
There will be cases where there will be issues with the code, but this should help you improve.
Aiming - Target Vector
For aiming, you'll want to start off by aiming at a vector within the target. This could be adjusted to aim at the head height, foot height, etc.
I'd start off by getting the closest vector within them.
One way of doing this would be done by calculating the vector of the current rotations and extending them by the distance to the target.
Java:
Vec3 lookVec = RotationUtil.getVector(yaw, pitch); // yaw, pitch is your last yaw and pitch
double range = mc.thePlayer.getDistanceToEntity(entity);
Vec3 aimVec = mc.thePlayer.getPositionEyes(1F)
.addVector(lookVec.xCoord * range, lookVec.yCoord * range, lookVec.zCoord * range);
but this may produce a vector outside of their bounding box, so we can clamp it to be within.
Java:
AxisAlignedBB bb = entity.getEntityBoundingBox(); // You could shrink this by doing, e.g., .expand(-0.15D, -0.15D, -0.15D)
aimVec = new Vec3( // This clamps aimVec to be within the area of the bound box
Math.max(bb.minX, Math.min(bb.maxX, aimVec.xCoord)),
Math.max(bb.minY, Math.min(bb.maxY, aimVec.yCoord)),
Math.max(bb.minZ, Math.min(bb.maxZ, aimVec.zCoord))
);
You can also apply randomisation before this.
Java:
// Calculate the aim vector
double radius = 0.1D;
aimVec = aimVec.addVector(
RandomUtil.nextDouble(-radius, radius),
RandomUtil.nextDouble(-radius, radius),
RandomUtil.nextDouble(-radius, radius)
);
// Clamp the aim vector to be within the bounding box
This would produce a target location to look at.
Aiming - Smoothing
For aiming, you could limit the maximum amount the rotations will change per update.
A basic implementation would be simply clamping them both by the same value.
You could improve this by calculating the speed relative to the delta yaw and pitch.
Java:
float distance = (float) Math.hypot(deltaYaw, deltaPitch); // Calculate the total distance
float speedYaw = speed * Math.abs(deltaYaw / distance), // Calculate a relative speed for the yaw and pitch
speedPitch = speed * Math.abs(deltaPitch / distance);
Aiming - Randomisation
For aiming randomisation, you can check out the example I gave earlier. applying a random number is pretty basic and you'll want to take further steps.
Aiming - Finding the Rotation
As you mentioned earlier, you tried finding the rotation but it caused severe lag - although this was for randomisation.
When calculating the rotation, it may be behind a wall there the target cannot be hit. You'll want to bruteforce it - but this is costly on the CPU. You can optimise this with parallel-processing/multi-threading but you will have to balance out precision and efficiency.
One method could be creating a list of, e.g., 1000 points within the target's bounding box, going through them, checking if they are visible then adding them to a list, sorting them by closest to the vector and then using that one.
Another method could be to check in a radius starting from the target position and finding the first one closest within the bounding box.
You don't always have to do this, you only have to do it if the target point isn't in view.
Target Sorting
For target sorting, I meant it as in modes such as health, FOV, etc. That would be a nice mode although.