A Conversation with Patrick Chu, Part 2By Scott Mitchell
|Patrick Chu is the founder and lead developer for ItsYourTurn.com (IYT), a popular turn-by-turn gaming site that has been in operation since 1998. Developing, testing, and supporting IYT is handled by Patrick, two other full-time software developers, and a part-time support staff. ItsYourTurn.com has over 2,500,000 registered accounts, records around 700,000 game moves per day that result in four million daily page views and a SQL Server 2005 database with over 470 million rows of game moves. Initially, IYT was powered by custom C++ ISAPI extensions, but has since moved over to using ASP.NET 2.0. Clearly there's a lot that can be learned, both about technology and business, from a person with the experience and background of Patrick Chu. I recently emailed Patrick some questions about IYT, both from a business and technology standpoint. He was kind enough to write back with some very detailed and valuable answers that he agreed to let me post here.|
Scott: What does the dev environment look like? What source control do you use? Do you do unit testing? Do you have testing/staging/production servers? What's the model you use for rolling out changes? Do you archive old data offline? Keep everything there? Delete old data? How often do you backup? How much disk space do you use for current games/backups/archived data? What's the database throughput like? Do you use VS.NET to develop? What do you use for front-end design?)
Patrick: The entire database is backed up every day, and we keep separate full backups for each day of the week. So today's backup won't be overwritten until next week. Last August we got burned when our primary backup and our secondary backup were both victims of data corruption on the disk (two different machines), so we are making sure we keep way more backups than we need. In addition, about once a week we save a backup copy to an external USB drive. Soon we will also be transferring the backup files to our office, which is about an hour drive away from our ISP, in case our ISP is destroyed by a tornado, for instance.
As for the application, since we have full copies on both our testing and development machines, we do not back up the web servers on a daily basis.
The database is currently several hundred gigabytes in size, and it keeps growing every day.
We do not do any type of formal unit testing. While we were porting the C++ to C# we had scripts that made sure the output coming from both sides were identical, so we do run large tests when necessary. We don't do any type of load testing at all. Right now the web servers are running at about 10% CPU load (and about 1% network load) and the database is running at about 30%-35%. I estimate we could double our traffic and still be able to handle it with the current hardware configuration. Our load testing strategy consists of having far more hardware than we need -- machines are cheaper than testing staff. We also have hardware that's currently "dark" but ready to be deployed quickly. We've always put "too much" hardware behind the website (it's cheap), and we've been able to put off the performance and load testing for much of our history. About once a week I'll log onto our busiest server during peak hours and make a few moves, and usually the response is instantaneous. I also keep a close eye on the load graphs throughout the day to make sure we're not seeing any strange spikes.
For database tuning, I wait until the database load gets up to 50%-60% CPU, and usually a few days and a few tweaks can get the CPU load down to 30%-40% again. With the web servers doing data caching there's even less need to tune the database, for now.
For performance optimization, I'll run the code through a profiler (we use Ants Profiler from red-gate software) and optimize the slowest parts. Our web server and database server are rarely the performance bottlenecks. The bottleneck is usually a slow connection on the client end, which we can't do much about.
When we make a change, we code it on our desktop/laptop, and make sure it works. Then we push the code up to our machines at Intrex, to our test servers. We'll pull up a few pages on the site (both the changed pages and our most important pages), and if it passes, we'll copy it to our production servers. Command-line scripts handle all of the transfers between computers.
We use Subversion for version control. We often check in code remotely, which Visual Source Safe (VSS) doesn't do well, and after losing code changes in VSS a few times we decided to go to Subversion. We've been on it for over a year and we've been quite happy with it.
Scott: What version of ASP.NET do you use? Any plans for moving to 2.0? Any AJAX plans?
Patrick: We're already using ASP.NET 2.0. Since we used very little of the ASP.NET 1.1 web framework, the port was almost trivial.
Our AJAX code is almost ready for release. It's already running on our test servers, and it rocks (that is to say, AJAX rocks). We'll be converting our game board pages to AJAX, and then we'll be looking at other parts of the site to see where it makes sense.
To describe where our first AJAX implemention is, I have to briefly describe how moves are made on ItsYourTurn.com. From your "game status" page which lists all your games, you click on a game that you're playing, and that game board will be displayed. For a game like chess, you have to first click on the piece to move (one page load), then click on the square you want to move to (another page load), then you enter an optional message to your opponent and submit (one final page load). AJAX is ideal for this type of interaction, because it's all part of a single "transaction": making that one move.
Another annoyance that AJAX fixes is that, if you have a large board, you have to keep scrolling down to see the board every time the page refreshes. With AJAX, since there's no page refresh, you no longer have to scroll down for the intermediate pages-- the board stays where it is.
Obviously, having seen other AJAX implementations, I was aware of how much faster an AJAX implementation is over reloading each page, but to see it in action on ItsYourTurn.com is still jaw-dropping, especially after 8 years using the "old way". Pieces move almost instantaneously. It really is like magic.
The move history in the game will also be AJAX-ed. You will be able to click through your game history without a single page reload. It's FAST! After I finished the initial AJAX implementation I sat there for a long time just clicking through game histories and watching the pieces zip around the board in real-time. It was amazing and unbelievable.
For those of you reading this who play on ItsYourTurn.com, yes, the code is working today, right now, on my laptop, for both the game boards and the move histories. We just have to test it and polish it up, and it goes online. Initially, only checkers will be AJAX-ed, but the other games should follow very quickly. Chess and backgammon are next on the list.
It took me less time to write a short AJAX function than to figure out how to use an existing AJAX library. At some point I'll look at Scriptaculous or RICO and copy some features, but for now the core functionality works for us.
Scott: From the business side: do you do IYT full time?
I've been working full-time on ItsYourTurn.com since February 1998. We went online with a very bare implementation on April 3, 1998, and we were selected as Netscape's top 10 cool sites of the day on April 7, 1998. (remember Netscape, the original portal? Or remember the days when you could actually track which websites went online in a single day, and pick out the 10 best?) That got us some good exposure which allowed us to get to critical mass fairly quickly.
So yes, I was full time even before the site went online. Before that I was a consultant working on very large database design using the Teradata RDBMS. I was working on terabyte databases when there were fewer than ten databases over a terabyte in the whole world, whereas nowadays almost everyone -- myself included -- has a terabyte of disk space on their home system.
We have 2.5 employees including myself: two programmers and a part-time customer support person. I do the programming and system administration. We were up to 5.5 employees in 2000, but they were all jettisoned at various times in 2001, leaving just me to run the site from fall 2001 through summer 2003. I had no choice, the money simply wasn't there for additional staff during the time of the "Big Crash."
Scott: Ah, yes, the "dot bomb" bubble. You clearly started at the beginning of the bubble (1998). What were things like during the dot com boom? What about the impact after the dot com bust? Also, how did you get the idea to start this website back in 1998? Did you forsee the Internet bubble and planned on getting in on the ground floor?
Patrick: In 1994 I first used NCSA Mosaic which a friend of mine described as the "new Gopher". If you don't know what Gopher is -- that is, if you're under the age of 30 -- here's a page about it from 1995, Complete with Windows 3.1 screen shots. I was hooked. I think I spent pretty much the next two days in front of the screen clicking on everything in the Yahoo directory, and surfing out from there. There wasn't a lot to see. The home pages that were online at the time were extremely clever, since they were usually written by tech-savvy academics, and had to be painstakingly constructed by hand without the aid of "HTML for Dummies", which wasn't available at the time.
For someone familiar with FTP and Gopher, HTTP didn't seem like such a radical idea. The Gopher protocol supports primitive menuing and search functionality. You navigate a directory structure, clicking on text files to read them, and image links to view the images. It was directory and file-based. It was a step above FTP because with FTP you usually had to download the file to view it.
So along comes HTTP, and instead of clicking on files in a directory to view them, someone decides which file to display in a given directory, and the images are shown directly on that page, next to the text on the same page, rather than having to click on each image to view them. Nice. You can also browse that same directory to view other HTML files in it (old Gopher habits die hard), or you can even embed the links directly in the page itself to the other files in that directory. But why create the links when everyone can list the directory contents already? And you could create links that pointed to files outside that directory, but really, why would you ever want to? Oh sure, if you had nothing else to do, you could create links to an external dictionary for some of the more difficult words, or you could put in a link to an external research paper if someone wanted more details about a particular subject. But you could already do that before: just put Gopher links in your text files, and the user would simply cut and paste that link into their Gopher client if they wanted to see that file. Easy. That whole HTTP pictures-in-text thing is nice, and you can click on the link instead of cut-and-paste, but it looks like just a supercharged Gopher, which itself is just a supercharged FTP client. I figured it was just another toy for the academics to play with.
While I thought Mosaic/WWW was very cool, I was working on starting a "multimedia" business based on CD-ROMs. Remember the CD-ROM bubble? Everyone said it was going to be big. The Voyager CEO was everywhere, "interactive movies" were being released, magazines were going to go "CD-ROM only"... yes, this was where the real content was going. I mean, were you really going to post all that stuff online in Gopher, or "Gopher plus" a.k.a. HTTP/WWW? How would you charge people for it if you did that?
As you might expect, my multimedia business went nowhere, and I returned to database consulting after several months of trying to launch my multimedia company. During that time, I continued to spend time surfing WWW, and the thought briefly crossed my mind to email Yahoo a resume since I already had several years of database experience under my belt. But from the website (at the time just grey pages with blue links, minimal graphics) it didn't look like anyone was getting paid, so I never sent that email.
Going back further, in 1989 I pitched my co-workers this crazy idea to put all the movies that were playing around the country into a big database, and allow people to access that information via a dial-up text-based BBS system. My co-workers were unenthused.
- How would you get the information into the database and update it? (Hire people to scan the information from newspapers and clean the OCR data).
- How would you make money? (Charge them a fee to access it-- several large BBS were making good money doing just that. Or, sell the listing to a large BBS as an added feature).
- But why would they pay money to access this when they could read movie listings for free in their local paper (well not free, but 25 or 50 cents)? (Because they could search by movie title or type of movie and see all the theaters and start times in one place. And if they travel a lot, they can search everything in one place without having to look for a paper).
ATDT 9,,,18005551212?), and then pay the long distance charges or set up PC Pursuit by writing their own connection and login scripts? Who are you kidding?
So nix the movie listing idea. Something like that would never work. Besides, how many people have a computer with a modem? Not a lot.
Around the same time I was looking at NCSA Mosaic (say, 1994), I found this BBS that had a huge listing (at the time) of music CDs for sale, many for less than the going price in the record stores. I ordered a complete set of Beethoven's String Quartets not knowing if these people would ever send me anything, but sure enough in about week they arrived. I was stunned. This was something brand new. The BBS could list far more CDs than they could print in a catalog, and you could search the database in different ways.
To put this in context: this is back in the day where Tower Records hadn't made it to Orlando yet, and your CD selection in your basic music store was pretty limited. Trips to New York City always included a trip to Tower Records and the Strand bookstore to get CDs and books you couldn't find out in the sticks. An online store/catalog was going to have serious advantages over a printed catalog or a bricks and mortar store where things need to be displayed at eye level (as opposed to being stored on palettes stacked 20 feet high).
But still, it all goes back to the 1989 discussion about movie listings. To make that viable, a lot of people -- regular folks -- would have to buy a computer for their home. And even in 1994, who could imagine that regular folks would spend time online, given that everything was text-based and difficult to navigate? Even Compuserve, which was fairly easy to use, was far too expensive for average folks since they charged a fairly steep per-minute usage fee. And setting up the internal and external modem (which was not included with the computer) was way too complicated.
So I never sent my resume to Yahoo, or any other website. Mosaic was "fun", the "next gopher", but everyone knows the next big thing is CD-ROMs.
OK, let's return to ItsYourTurn.com....
I originally had the idea of turn-based games around 1996, near the date of Netscape's IPO. I was working in Silicon Valley at the time. At a valuation of $4 billion at the close of the first day of trading, many many water cooler and lunch discussions were launched about what to build in our own little corner of the Internet to snag some of that paper wealth.
One of the ideas I pitched was a turn-based board game site. Reception to this idea was lukewarm, with the preference to real-time games over turn-based. However, in this very same crowd, what we were looking for was a turn-based solution: my office mate and I would leave a chess board sitting out, and whenever one of us had spare time, we would make a move on the board. The problem was, we needed the table for meetings, and it looked unprofessional to have a chess board sitting in the office. What we needed was a way to do this online. A search through the 1996 websites yielded nothing. Later, in 1997 at a New Year's Eve party, I was describing my idea to a friend who later became ItsYourTurn.com's second employee (after me), and we decided to try to start the site.
At the time I started, there were two sites doing turn-based games: Richard's Play by Email (still around today), and something called Stan's Chess where you emailed Stan your move on your game, and he would update the database with your move, and a day or so later (or whenever Stan had time) you would see your move made.
Both of these sites had obvious deficiencies: on Stan's site you had to wait for Stan to make your move (and he asked that you not play too many games at once), and Richard's site's interface was email-only, so you had to read this manual of email commands before you could start playing, and you had to know who you were playing with (I see that Richard is developing a web interface for his site, though). We were the first to implement turn-based games on the web, although we've had many imitators since then.
We grew very quickly during the first few years, and also sunk a lot of money into development. The application was small back then, so it was easy to keep adding new features. We were also running on unstable hardware and unstable software (both the OS and the database) so regular multi-hour outages were not uncommon. For the first two years I wore a pager that would go off every time the site went offline (monitored by an in-house monitoring program), and I would inevitably get paged at 3:00 or 4:00 AM for months at a time.
As the Internet Bubble started to gather steam, we started receiving large sums of advertising dollars for sites that did not look like they would be long-term survivors. We also received some large buyout offers which we turned down because the payment was always in stock that wasn't yet liquid (the company hadn't gone public yet). I talked to a few of the companies but eventually turned down all offers because none of them looked like they were viable businesses, and all of them except for Uproar.com all died during the Crash. For Uproar, I spent most of a day in New York City touring their offices and everyone was very nice, but did I really want to go back to the corporate world so soon, and for what I felt wasn't really a lot of money? The decision to pass was pretty easy. (According to Alexa we now have a higher per-million reach number than Uproar.com although I find that hard to believe.)
I will say this: during a bubble, LIFE IS GOOD! If you are able to spend time inside a bubble, appreciate it and savor it for as long as you can. People were paying insane amounts for online ads, and at the beginning the irony was that there weren't many places to put online ads. Of course that soon changed. People were pumping millions of dollars into wonderful games that they were giving away for free, so that was a bit annoying. Money and crazy predictions were everywhere. Attending a web conference was a better high than any hallucinogenic drug. Back then, Forrester/Jupiter/Pundits'r'Us Research was projecting that, in 20 years, 120 billion people were going to have broadband Internet access. You couldn't afford not to start a website. Life was good. Money was everywhere.
While you're in the bubble, do know that it is a bubble, and spend time preparing for the "Day of the Pop." This I learned sitting on the sidelines watching the popping of the CD-ROM mini-bubble. Unfortunately, I had been preparing for the "Day of the Pop" since 1997, when things were just getting started. I undoubtedly missed out on opportunities where I wasn't aggressive enough. People were out there spending up a storm (and some of that ad money was winding up in our hands), but I was holding back in case things went sour. I was cautious too early. Although I generally consider myself an optimist, I had a creeping feeling that this wasn't going to go on forever. As a reminder to myself, from day one I've kept a big ol' rock of fool's gold (iron pyrite) on my desk to remind myself that all that glitters is not necessarily gold. It's still here right in front of me.
Here's one of many countless bubble stories: I went to "Startup Boot Camp" in March 1999 (run by garage.com, they're still around) where the motto was: "Start Up, Kick Butt, Cash Out" (I still have the T-shirt and wear it whenever I need irony). While there, I had a conversation with someone who wanted money for an online accounting application. The conversation went something like this:
Me: "So this website will help people do business accounting online?"It seems hard to believe these days, but these were normal conversations back then. A twenty-something gets $20 million to start furniture.com. (He eventually runs through $100 million before it's shut down, leaving many people without the furniture they ordered.) His experience? His dad owns a third-generation furniture store in North Carolina (close to where I live). This seems more egregious than the Webvan guy -- at least he was a high-end consultant before starting Webvan. And so on.
Budding Entrepreneur (BE): "Yes, we want to make accounting simple. Right now it's too complicated."
Me: "What about Quickbooks? Doesn't that do accounting?"
BE: "Quickbooks and Peachtree and all those other applications are too hard to use. This website will make accounting easy."
Me: "But isn't accounting hard in general?"
BE: "That's only because the existing software makes it hard. Putting it online makes it easier."
Me: "Do you have any accounting experience?"
BE: "I used to use Peachtree and I thought it was too hard to use. Other than that, no."
Me: "How much money are you looking for?"
BE: "I think we'll need $20 million to get started."
My "Day of the Pop" came during Christmas break 2000. While most people put the "Day of the Pop" somewhere in April 2001, I noticed that our ad revenues took a nosedive in the second week of December, just before Christmas. They never recovered. The first day back from Christmas break in 2001 I gave notice to three employees, and then two others were gone in August 2001. Even at the beginning of the company I had another programmer helping me, so for the first time it was just me.
In June 2001 we added subscriptions to the site to survive, and that model has served us well. By the end of 2001 we had removed ads completely from the site, since we were receiving very little money for them. Currently our model is that we're ad-free for paid subscriptions, and ad-supported for the "free" portion of the site.
The Internet Crash was far worse than even me -- Mr. PopMan -- had envisioned. Ad revenues were down over 99% from their peak (matching the corresponding drop in stock prices, for those companies still trading). Ouch! Fortunately we had an application where a subscription model was viable, and it was enough to keep me alive while I kept the site running. I felt that ItsYourTurn.com had good long-term prospects, so keeping it going made sense rather than trying to find a database job in an incredibly hostile tech workplace.
In the fall of 2001 I had no choice but to find ways to automate tasks that were previously being done by hand. I think the crash helped the company by forcing me to find ways to work as efficiently as possible. I think we're stronger now because of it, because we're still enjoying the benefits of programs written during that time. At the time, no new sites were coming online and no new features were being added, so we had time to beef up our back-end software for a few months.