Caching and performance improvements

We all know the phenomenon: Your unimportant website was just mentioned on ./ and all of a sudden you have many more visitors. Drupal websites where renowned early on (read: Drupal 3.0) that they were able to withstand such an "attack" quite easily. The reason for this is the page cache which will deliver anonymous visitors the pre-rendered webpage directly from the database. This feature made Drupal quite popular early on and sites such as kerneltrap.org switched to Drupal because of it. Unfortunately, the page cache doesn't work well if you have a very active site that gets new content every minute since each new node or comment will invalidate all the precious cached pages. Also, authenticated users don't get the benefits of the page cache since their pages are usually different for each user. There are many more caches in Drupal: filters, menus, forms (Drupal 6), blocks (Drupal 6). Also, contributed modules also cache various bits and pieces. I want to illustrate the various approaches that the different caches take for storing and removing items, I also want to discuss possible improvements to these caches. Further, Drupal's cache backend is pluggable, e.g. you can write your own cache.inc and do not use the database for caching, but another external application. Memcached is probably the most well known of these. Caches are not the only way to have your Drupal site run fast, but they are the ones that are implemented in Drupal itself. Other options include PHP opcode caches, MySQL query caches, various proxies, running a special webserver for static content, and tuning of your webserver and database. We want to give a short review of these and suggest use cases for small to medium Drupal sites.