After months of fighting with our ISP, the fiber was finally dropped into the basement and the fiber patch panel and MUX were installed on top of the router and switch on our 19 inch wallmount rack. A few hours later, we were up and running with a minimal configuration and pinging the ISP. The installation tech left us on our own and we started router configuration for the DMZ, firewall, and NAT. Another half hour and our Macs behind the firewall could see the outside world.
Setting up core services Since the SPARCstations are extremely slow at the command line but work great for months on end at menial tasks, they became the dedicated "core" servers that would take care of tasks such as DNS, outbound Web proxy, and outbound e-mail. We also decided that these two servers would be completely hardened with virtually no services whatsoever. Water would be configured as a bastion nameserver while earth would only handle DNS and outbound Web proxy and e-mail.
The SPARCstation 2 became "earth," the primary server that takes care of all core operations at Sosik-Hamor Networks. This system was the first to be installed because it had an external CD-ROM drive that could be used for FTP installs for other machines. OpenBSD 2.5 boot floppies were downloaded from ftp.openbsd.org and we started an FTP install on earth.
Total installation took around 45 minutes from start to finish, including downloading the approximately 250MB full distribution over the T1. Since the FTP installation method decompresses and installs files on the fly, like Linux, no scratch disk is required. Once finished, an obscure and long root passwd was chosen and the machine was rebooted into single user mode. All services except FTP and Daytime were disabled in inetd.conf and all daemons except named were disabled in rc.conf. Even portmap was disabled because the server would never be used in an open file server environment. The machine was rebooted and brought up on the Net.
With the machine up and running, the first software to be installed was SSH. To get up and running quickly, the pre-built SSH binary package was downloaded from ftp.openbsd.org and installed using pkg_add. The system was now accessible from the outside world, so I started up a few SSH sessions from my Macintosh.
The full i386 and SPARC OpenBSD 2.5 distributions were downloaded from ftp.openbsd.org and put in /home/ftp/pub/openbsd so we could immediately start installing on the other three servers. Anonymous FTP was configured and temporarily enabled for this task, but would be disabled once the other installations were finished. Although anonymous FTP is not a direct security hole, it is one more port to tempt a potential attacker.
Qmail was downloaded from www.qmail.org and installed from source because there hasn't been a direct qmail port for OpenBSD yet. Compiling qmail took a while because of the slow processor, but installation and configuration was painless. TCP wrappers were configured so qmail could only relay e-mail from the DMZ and firewall networks, and then set up as a secondary MX to queue e-mail for any other domains on the network should a primary mail server go down.
Squid was then installed for the outbound Web proxy to help hide the identity of machines in use behind the firewall. Configured with a 100MB disk cache and maximum privacy features enabled, Squid caches often accessed Web pages and hides the USER_AGENT and USER_REFERER strings so make it more difficult for Web servers to track movement from page to page. As with qmail, Squid only allows connections from within Sosik-Hamor Networks.
Forward and reverse nameserver zone files were then created from the default templates Sosik-Hamor Networks' primary domains. Zone transfers were disabled except for other nameservers in the DMZ, which makes it difficult for an attacker to download the forward and reverse maps of the network.
With earth up and running, other machines were brought online over the course of the next two or three days.
Because water, the SPARCstation 1+, would be a bastion nameserver, only minimal packages would be required. Compilers and other niceties were not installed because any patches and upgrades that were needed could be mirrored from earth. The only other package installed was SSH to allow for secure remote logins.
The installation process for water was virtually the same as earth except, instead of wasting bandwidth over the T1, OpenBSD was installed via FTP from earth. After installation was finished and the machine was brought up in single user mode, everything was commented out of inetd.conf and disabled in rc.conf except for named and sshd.
Although water was configured as a secondary nameserver to pull zone transfers from earth, we decided to list it first in the zone files and with InterNIC/NuNIC. That way most DNS requests would come into water first and keep earth free for other duties. Even though DNS isn't a very CPU or network intensive task, a nameserver that handles 100 or 200 domains needs to be carefully configured.
Setting up Web services Most large Web hosting companies run multiprocessor PII and PIII machines to handle their high workload. Since we were just starting out and our old colocated P90 Linux server with 64MB RAM handled 50,000 hits a day without breaking a sweat, we figured that, until we got busy, our two existing AMD K6 systems would be perfect for our personal and client Web servers. Both K6 systems were basically the same, but we opted to use the K6/233 for fire and the more powerful K6/266 for wind.
Fire was built first because we needed a machine to act as our testbed and production server for our corporate and project Web sites. All new configurations and software are tested on fire before going live on wind, and most interactive development packages from the ports tree are installed because fire is also used as the primary staff shell server.
All OpenBSD packages were installed via FTP from earth and all non-essential services were disabled in single user mode before bringing the system up on the Net. Because this was going to be a staff production system, quite a few services were left enabled and installed.
As with all of our systems, SSH was the first software to get installed to allow for secure remote access. Qmail was then installed and configured for virtual domain support with no relaying. With this configuration, qmail only allows inbound email or outbound email originating from localhost. Also, e-mail for each domain is handled by its own individual account and users have full configuration over aliases and forwarding without root intervention.
MySQL was then installed from the ports tree for all Web sites that need database interaction. A simple make; make install in the ports tree took care of everything and the MySQL server was up and running. The only additional change made to the default configuration was to create a mysql user, change the ownership of /var/db/mysql from root to mysql and make safe_mysqld launch as the user mysql user from rc.local.
A database is useless without a bridge to get data to and from the Web server, so PHP3 was installed. This required recompiling Apache from scratch in /usr/src/usr.sbin/httpd, so the Configuration file was modified to add in all the extra modules that we needed, ./Configure was run and then the PHP3 module was installed into the Apache source tree. Other various modules were added as well and then Apache was recompiled and dropped into /usr/sbin. Once compiled, suexec was also compiled and dropped into the Apache sbin directory to allow for secure execution of CGI binaries.
Once all of the servers were set up, niceties such as Emacs and Pine were installed from the ports tree. Deciding which applications get installed is simply a matter of user preference, so take a look through /usr/ports and do a make install for anything that looks interesting. Pre-built binaries are also available from ftp.openbsd.org. Smaller applications are just fine to run from precompiled binary packages, but larger applications that need tuning (Apache, MySQL, etc.) should be compiled from the ports tree on the system they will be running on.
Wind is an exact mirror of fire and acts as the clients server. The only difference between wind and fire is that miscellaneous applications may get installed at each client's request (IRC scripts, etc.).
Akasha is the watchful eye that makes sure all is well out in the DMZ. It sits on a hub between the DMZ switch and the DMZ ethernet port on the router and constantly runs a sniffer and network analyzer to look out for "interesting" traffic. The main purpose for this is to keep track of per-MAC address accounting and to look for denial of service attacks or other nasty packets that may come across the fiber. Depending on the requirements at the moment, akasha may be the Team Internet 486dx2/66 running OpenBSD or a Macintosh IIcx running Mac OS 7.5.5.
Setting up the internal network Now that the DMZ was set up and ready to go, the systems on the LAN inside the firewall needed to be reloaded and configured.
socks.shn.nu Socks was chosen to be the internal file server and Red Hat Linux 6.0 was installed. Of the two disks, the 3.5GB was used as the boot disk with /home for home directories and the 25GB data disk was mounted as one huge 23.5GB partition under /home/warehouse01. As the need arises, more 25GB (or larger) disks will be installed as /home/warehouse02, etc.
Because of the large drives, the entire Red Hat distribution was installed and unused services were disabled for security. Making the jump from Red Hat Linux 5.0 to 6.0 was quite a large step, and I wanted to see everything 6.0 had to offer. So, not only was socks configured as a file server, but also as a user workstation to play with the new window managers that have become available. Even though virtually all Web site development is done using Adobe GoLive under Mac OS, the Linux workstation is used for most systems administration tasks.
To serve our internal Macintosh G3 workstations, socks also runs Netatalk, the UNIX implementation of the AppleTalk protocol. File transfer speeds over the internal 10BaseT LAN are surprisingly fast, but we will be moving to switched 100BaseT eventually to help speed things up. 10BaseT seems fast until you routinely start opening up 20MB Photoshop files for editing. All other services on socks are set up almost exactly like fire with Apache, MySQL, PHP3, etc.
To tie everything together, a Belkin OmniView 6-port KVM switch was used to control akasha, fire, wind and socks from the same keyboard, mouse, and monitor. All machines sit on an 8 foot vented steel rack, and an air conditioner keeps the machine temperature at 72 degrees. The entire NOC can be controlled from the keyboard and monitor hooked up to the OmniView.
DIY: Do It Yourself! Overall, setting up an Internet resource provider isn't that expensive. A mixture of OpenBSD and Linux can make older workstations into perfect servers and keep initial startup costs extremely low. Using existing hardware and 100% open source software, we kept our startup costs under $8,000 and -- even using brand new systems -- easily stayed under $15,000. Also, by dropping fiber into a home business, your home office becomes a business expense with the added benefit of Mb speeds at home!ø
Sean Sosik-Hamor is an Alpha Geek and systems administrator for Lucent Technologies. For open source IT, Sean also hosts the Apache help forum.