One of the things we try hard to do is not reinvent the wheel. That's why we use Joomla! instead of writing our own Content Management System, why we search hard for off-the-shelf components before we write our own. For our servers, we rent from a large, reputable company and use one of the three industry-standard control panels. This is to give our customers the best value for their money by not reinventing wheels and for security, in case we lose a team member as standard skills are much easier to replace. There's one exception to our 'never reinvent the wheel' rule...
and that's of course if the wheel is square. Such was an ongoing struggle we've had: backup. The control panel software we use had automatic scheduled backup facilities: daily or weekly backups, what time, how many to keep, etc. And for a server with very few sites it was probably adequate. One of our servers has about 80 sites and many have e-commerce on them so they have to be backed up nightly. The backup routines started taking longer and longer, with backups overlapping each other and slowing the system down. I wrote a procedure to try to spread them out, but still they piled up and hurt performance. So I finally took a weekend and ripped out the standard backup procedure and replaced it with a more robust one. This new one can do both full backups (everything on the site) and incremental (just what's changed from the previous backup). This is a major win because if you'll think about it, most of the contents of your site don't change from one day to the next. Now instead of keeping three days' worth of full backups with the old software, we can now keep seven days' of backup with the new software because we're only doing one full backup, then a series of daily incrementals.
All in all, you probably don't care about the gory details of backup, but that's just one of the behind-the-scenes things we worry about and try very hard to get right. The bottom line is that instead of slogging down the server for six hours a night, backup now runs smoothly in two hours. As a result, a couple of our sites had perfect 100% uptime for a month. And that's something everyone understands and appreciates.