If you host your site on a VPS or dedicated server, then you are able to tweak your server and make your site load significantly faster. The fallowing considers a LAMP platform but some parts might be useful for other platforms as well.

The first thing to consider is a PHP accelerator. I recommended eAccelerator. It will not make your site go significantly faster, but is one of those things that you have to have.

Second thing to do is a little database optimization.
As you might have guessed, the database is almost always the weakest link in the chain. If there's something on the server side taking a long time, in 95% of the cases is a database query.
So, you can start by analyzing your tables and add indexes on columns that you know to be used on WHERE clauses.
Then, you can take a deeper look on those nasty queries. A nice tool is the mysql slow queries log. That is not enabled by default, so you can ask your sys admin to enable it or do it yourself by adding this in my.cnf
log-slow-queries[=file_name]
The log will gather queries that take longer then long_query_time (defaulted to 10 seconds).
After the log has been accumulating enough information, you can begin the slow and painful process of optimizing your database by taking each query and see what's there to do to help it out.
After you make sure the indexes are all in place, and queries don't have necessary joins, think of these couple of tricks:
normalize your data (i.e. store data in a single table, even if you'd normally store it in more then one), thus you avoid joins and don't try to do everything with a singe query. You can use you script to take some of the load. Instead of using some complicated joins, sometimes you can start a loop from a simple query, and at each step you can query for some additional data.
With that, we've barely scratch the surface. Those who want to be proficient should take a look on replication and mysql query cache.

Moving on, we have a cool way to optimize sites. It is becoming more and more used, but it requires a good understanding of cache principles.
Mecached is particular useful for big applications.
With PHP there are 2 extensions for using memcached. An older one named memcached and a newer pecl library having more features named memcache.
Many CMS or frameworks implement caching systems. And if they don't, they should. So, theoretically implementing memcached with their own caching system should not be such a hard job.
Drupal has a nice initiative with the module named, take a guess, memcache but it does not come exactly out of the box and some folks might be scared of that. They shouldn't because it only needs applying a couple of patches for Drupal core and a few modules such as Views and CCK and set a couple of config options. But all you need to do is run a couple of Linux commands.
The wonderful part is that it does not exclude other caching systems. In fact, it works pretty well with boost, which is a another Drupal module that saves a plain html file and serves that directly. If boost does it job excellent for anonymous users (since Drupal does not cache pages for logged users) memcached fills this gap and heps regenerating boost pages quickly but most important helps generating pages quicker for logged users.