A Conversation with Patrick Chu, Part 3By Scott Mitchell
|Patrick Chu is the founder and lead developer for ItsYourTurn.com (IYT), a popular turn-by-turn gaming site that has been in operation since 1998. Developing, testing, and supporting IYT is handled by Patrick, two other full-time software developers, and a part-time support staff. ItsYourTurn.com has over 2,500,000 registered accounts, records around 700,000 game moves per day that result in four million daily page views and a SQL Server 2005 database with over 470 million rows of game moves. Initially, IYT was powered by custom C++ ISAPI extensions, but has since moved over to using ASP.NET 2.0. Clearly there's a lot that can be learned, both about technology and business, from a person with the experience and background of Patrick Chu. I recently emailed Patrick some questions about IYT, both from a business and technology standpoint. He was kind enough to write back with some very detailed and valuable answers that he agreed to let me post here.|
Scott: Let me throw the canonical job interviewer question at you: What's the greatest weakness you've had with the site/business? (Could be technical or business-related.) What's the biggest success?
Patrick: The biggest weakness of this company, which I think is a weakness shared by many small businesses, is that it's centered too much around me and the skills I have. Over time I've created a development process for myself that's so efficient that it often takes less time for me to code it than it would be for me to write up a description of the problem in a way that I could assign it to the other programmer or to an outsourcing firm. Naturally, this will limit to growth of the site to what one person can do in 10-12 hours a day, which is not ideal. I'm looking at different ways to break through this logjam, and I'm trying to enlist the help of outside firms to take on tasks that are not in my core set of skills (design, for example). Over time, I envision more and more non-core tasks getting sent to outside companies, whether they be local companies (as is the case with our ISP and our design firm) or overseas (contract programming).
On the flip side, I can't think of any single element or event that has accounted for our success. I think the standard rules apply to ItsYourTurn.com as it does to many other small businesses: listen to your customers, keep your eye on your cash flow, be persistent. The rules are easy, but it takes hard work to keep it going day after day.
Scott: Last question - What's the 5-year plan?
Patrick: To prepare for the future, we've spent the last two years restructuring our hardware and development environment to crank out new features as quickly as possible. We're almost done with that process, and we hope the next twelve months will see radical changes in ItsYourTurn.com, for better or worse. New features will start showing up in rapid succession on the site, and we hope to keep up an aggressive pace of change going forward.
I think every site needs to do this to survive, because the early winners in any given niche (currently MySpace, Facebook, del.icio.us, Flickr) need to keep innovating in order to stay on top. Think about the early winners of the past (Excite, Geocities, theglobe.com) that are now gone, having been replaced by the next new thing. Even the early winner in social networking, Friendster, has now been crushed by MySpace. There's nothing that any of these companies are doing that can't be copied by someone else. Yes, they are enjoying a network effect, but even network effects can be overcome -- just look at the downfall of Friendster. Google came late to the game, both to search and to paid search, and managed to become the 800lb gorilla on the Internet because of a superior product. History has shown that the first-mover advantage is small, if it exists at all. The superior product will win out in the long run.
For ItsYourTurn.com, we have seen many ItsYourTurn copycat sites spring into life throughout our tenure, and they will continue to do so, especially if turn-based games become bigger than it is now (which I think it will). Some of the copycats are slimy enough to steal the EXACT game rules for games I've created, and are using the exact game names that we use on ItsYourTurn.com. More than one site is using the EXACT tournament format that created myself. (In this particular case, the tournament format does not qualify for patent protection.) I could go through the expense of patenting and trademarking all our unique concepts and then sue them, but it's expensive and ultimately it's pointless.
The way to win is to provide a superior user experience over our competitors. We need to have the features that people want, provide those features in a user-friendly way, and the site has to be more reliable and faster than everyone else. If one of the copycat sites develops features or games that we think is cool, we'll copy it for our site. If we can copy their popular features faster than they can copy ours, and come up with some unique features of our own, and put heavier marketing muscle behind it, we'll win in the end. We've spent a lot of time re-engineering and re-thinking our development process, and we think we've hit on something that works. The old C++-based process took so much development effort that in the long run it takes less time to rewrite it all and then develop new features in C# using our own in-house web development framework than to work using the "old way". It's painful and tedious to rewrite your code only to have the site look exactly the same as it did before, but if we want to survive for another ten years, really, we have no choice.
Since our August crash last year where we were down for ten days, our backup and hardware configuration has changed radically, for the better, we hope. In August our primary backup got corrupted, and our secondary backup was not usable. We had to send the disks out to a recovery service (Ontrack Data Recovery) to pull the data off the disks, and thank God they were able to do it. Usually they are able to perform the restore remotely which would have put us online much faster, but in our case the corruption as so bad they needed the physical disks shipped to them, which greatly increased our downtime. I spent ten days on a specially-created blog apologizing to our user base, and they were kind enough to forgive the incident and pick up their games where they left off. The disks we use are RAID-ed of course, and have been for the last five years. The RAID hardware was fine even after the crash, so it had to be either the database or the OS that was the source of the corruption.
Since then, we now keep seven different full database backups (one for each day of the week) as well as a full backup copied to a separate unrelated computer, and soon an offsite location as well. Yes, we run full backups every single day -- we don't do any incremental backups. We're slowly moving to a virtual architecture (VMWare Server), and those machines are regularly backed up as well, and will be archived off-site. We have a hot spare available for the database server which can be turned on and active in less than an hour -- it's already sitting in the rack virtually unused. We're also looking at continuous backup options and log shipping options.
Our data center (at Intrex) has a generator that can run for two days without refueling, and we have UPS unit connected to all our servers that will provide power until the generators kick in (it takes about 30 seconds).
The point is, we've spent a great deal of time and expense to make sure the August crash doesn't happen again. Of course, small undetected problems will always creep in (that's the nature of our business), but we are trying to cover everything we can think of. At this point we have far more RAM, CPU power and bandwidth than we need (probably by a factor of 3x or 4x), just to make sure our players get their pages as fast as possible.
Looking beyond ItsYourTurn.com, the next 5-10 years will be a period of great change and growth for the Internet. Our fate will be greatly dependent on what happens to the Internet in general. The plummeting prices of processing power and RAM and hard drives will enable applications that appear very impractical today. I'm not talking about the pundit predictions of social networking or video everywhere or whatever the fad of the day seems to be. I'm talking about applications that seem as ridiculous to us as the 1989 movie listings (see Internet history digression section) seemed ridiculous and impractical.
Some factors that will influence the future direction of the Internet:
- Right now, web programming is still too difficult and repetitive (and yes, that includes the self-styled "easy"
Ruby on Rails web development framework, which I looked at pretty closely as a possible successor to C# and ASP.NET).
All the recent attempts to make programming easy (UML, database objects, Ruby on Rails) have severe limitations and pain points,
and many of them are difficult to learn. Ideally, a designer should be able to design a site in Photoshop, write a
few business rules, and it should just work. That sounds crazy and impossible, but that's where things are headed.
I've looked at third-party code generators (and ASP.NET could be considered something of a code generator), and I think
they're the first step towards this ideal.
For example, as someone who has spent many years typing out database designs, the new diagramming and modeling tools are a godsend. For me, clicking is much faster than typing, especially when creating keys and multi-table joins and table relationships. We need something similar for web development, and that doesn't exist yet, and I don't think ASP.NET (or Ruby or PHP or any other current web framework) is anywhere near that ideal. In my mind I have a very clear architecture of what this would look like, which borrows from the current code generators, ASP.NET, Ruby on Rails, and my own web engine that I've written on top of ASP.NET.
And if I'm already thinking about it, then others are thinking about it as well, and hopefully an open source project will emerge to solve this problem. I say open source because a necessary component will have to be the ability to change the code that's generated, and right now the closed-source code generators won't let you do that.
Code generators don't have to static -- you can also generate code on the fly. We see a little of this in what Ruby on Rails calls their "scaffolding" code where they read the database table definition and create a page on the fly that reflects the correct format of the data being returned from that table. In order to do this, it has to read the table schema every time the page is loaded and start the formatting from scratch, but if we cache that information on the web server, this isn't as inefficient as it first sounds.
- Machines will get faster. We know this. Since development time is far more expensive these days than CPU time,
we need to figure out ways to use the machines inefficiently in order to speed up the development process.
By streamlining the web development process for the human developer, the web development framework of the future will produce
very inefficient code because it will have to do most of the heavy lifting. Web pages will have longer and longer code
paths as human developers push down most of the work to them. Right now web developers spend so much time doing the
same things over and over to set up for the "real" work: implementing the business rules. IIS and Apache are still
unnecessarily difficult to configure. The web framework of the future will come with 90-95% of the code written for the
most common types of pages, it will be very easy to understand and use, and the developer will spend the time filling in
the rest. Once you create the database table, the web page should almost write itself.
The "next web framework" is a question that lots of people are spending many hours thinking about. What we do know is that in five years, ASP.NET 2.0 will seem hopelessly primitive and clunky, just as no one today would be insane enough to write a C++ ISAPI DLL with the HTML embedded into it. Today's tools are orders of magnitude beyond an ISAPI DLL, and in five years we'll see even more amazing new tools hit the market and we'll ever wonder how we endured the grind of the current development process.
- Searching and filtering information is also too hard, and an area that will see great change. It's a sad state
of technology when the spammers are able to out-innovate the current filtering technology, and we still have to rely on
manual blacklisting techniques to filter our spam. However, there are heuristic techniques available today that are going
to change this.
The filter and search technology being used by Amazon and Google (the most visible implementations of intelligent search and match) are still very primitive. The search filters are too narrow and literal. Sometimes you need a certain amount of Google kung-fu to find your answer. What we'll see in the next few years from Google is individualized search. For example, by analyzing not only what I search for but what I click on, Google will adjust its rankings so that the computer language "Ruby" will appear before results about the gemstone, whereas for someone in the jewelry business they only want to see results about ruby the gemstone. Same search yielding different results depending on your past clicking history. This is a trivial example of the types of rankings adjustment that I'm sure they're working on (I don't know for sure), and we're going to see that go online in the next few years.
We're also going to see recent advancements in AI pushed down to us. Right now if I want to read blogs of interest to me, I've got to go somewhere and search for it myself. I have to search about the whole blogging ecosystem in order to find what I'm looking for. On the other hand, it's likely that the kinds of blogs that I want to read are similar to the email I read and the web pages I visit, so why not give me a software agent that runs in the background on my computer that analyzes everything I type and read, then shows me blogs or Yahoo news or Flickr pages prioritized according to my past reading patterns? This is coming. This AI and pattern matching technology exists today, someone just has to program it and productize it.
We're still using stupid and primitive technique to combat spam. Once these new AI filters are in place, spam will no longer be a problem because they will no longer be able to penetrate the filters. As an example, look at SpamBayes, which is the spam filter I use (it's free!). It uses a spam/non-spam classifier based on statistical pattern matching. After the number of message I pump through it (thousands a week, most of it spam), it's extremely accurate, and the keywords that it scans for are customized to me. If it identifies a message as spam, it will search that message for new keywords, and it adjusts itself as the spammers adjust their campaigns. The more you train it, the more accurate it gets. For me, it's by far the most accurate, trouble-free spam filter I've ever used. [Scott: I also use and recommend SpamBayes. See my blog entry, Give SpamBayes a Try.]
To adapt this to a software agent is not that hard. Amazon has what it calls "statistically improbable phrases", which are important keywords that it pulls out of its books, and this background agent could do the same by logging what you type and reading the email you read and the web pages you visit. Then, when it asks you to give you other news articles and blog entries with the same SIPs, it's able to show them all on one page. Instead of a "spam/non-spam" filter we have a "interest/no-interest" classification geared especially to me. The end result is effortless targeted searching where the computer does all the work.
First of all, I think if you're writing programs in the middle tier, you'll be under increasing pressure from automated code generators in the next few years. Tools will be written to automate these middle tier tasks as much as possible. The relentless march to automation using computers armed with ever-more-advanced pattern classification algorithms will start replacing the more mundane programming tasks. Just as factory robots changed the face of manufacturing a few decades ago, IT workers will find that computers will able to do more and more of their tedious work, and that includes a lot of what currently passes for web development work. If you're a developer doing this type of mundane work, your job may be in danger.
Even database designers (my specialty) will be under pressure. Yes, you'll still need database designers for the difficult
projects. But for a web site, really, how hard is it to draw a reasonably relational design with primary/foreign keys joining
the tables? Not hard.
With the graphical modeling tools I've been using, I don't think I can even hand-write a query with the "new"
INNER JOIN syntax (I learned the "old school" syntax) without the help from a tool. Thankfully, I've never
had to -- the tools do it for me. I only need to know the concept -- why do I need to bother with the specifics of the
syntax? I click or drag the relationships between tables, and the SQL is generated for me. I can tweak it if I want,
but the bulk of the work is done for me by the tools, and it does a pretty good job in the overwhelming majority of cases.
The people who will thrive will be the ones at the very top of the food chain, the application designers/architects and the high-end graphic designers. The low- to mid-range graphic designers who are just creating one generic-looking site after another will get pushed out by the template sites like BoxedArt or TemplateMonster where you can buy an attractive site design for like $50. I mean, these templates look great. And most hosting companies also have low-cost designs available. Sure, if you're a Fortune 500 company you don't want to buy a $50 template for your public branding (you may buy one for your Intranet, though), but if you're a furniture store or a repair shop that just wants to post your prices and your hours, one of those is ideal. True, you still might need someone at your hosting company to help you put up the pages and tell you where to edit, but you no longer need to spend thousands of dollars to develop and maintain a web site. The mid-range will disappear, and you'll have cheap templates on the low end and custom graphic design solutions on the high end with very little in between.
If web development becomes easier in the next few years, we're going to see not only a greater number of generic sites on the low end but also the high-end guys pushing the envelope of what's possible with HTTP. With AJAX, things that were impractical before have suddenly become possible, like an online calendar or word processor or spreadsheet, and others. For some tasks it'll be easier and faster to do it online than to store it locally on your machine (example: del.icio.us has replaced my browser "favorites").
So web sites on all levels are going to look much nicer and do more than they do now. How does this affect ItsYourTurn.com (or any other small web site)? How can a tiny shop survive the next few years with pressure from both the low end and the high end?
Two things sum up our strategy for the next five years: focus on the customer and an efficient development process.
Basing a business around these two pillars is a better bet than basing it on the fad word of the day (for example, the current example is "social networking"). The key to business survival, now as it was 150 years ago, is to give the customer a good product at a good price. Make the product reliable and useful. See where they're having problems, and fix it if you can. (Side note: I think outsourcing customer service is a huge mistake, since that can be your cheapest and most reliable form of market research). Fortunately, our customers are not shy about letting us know what they want. Look at sites in your space who are bigger than you, try to figure what they're doing right, and imitate them. And do it QUICKLY, before they can imitate you.
The second part of that equation is, you've got to be able to get your vision online, and do it faster than everyone else. You've got to innovate, you've got to move forward. In an environment where it's so easy to copy your competitor's most popular features, you need to make sure you can copy them faster than they can copy you. If you have a frequently-used development step that's inefficient, the time spent to speed up that step (even if it means stopping development for a short time) will be recovered in a short time, and everything after that is gravy.
I like the prospects for ItsYourTurn.com. The signs look good. We appeal to an older audience, an audience that prefers pawns and dice over laser guns and magic swords. Casual gaming is on the rise, and as baby boomers retire, they will spend more and more of their entertainment time online playing casual games, games that they're familiar with. With the gaming companies still obsessing over the 18-25 male audience, the huge over 40 crowd is still largely ignored. We hope it stays that way for a while, and we hope that the over 40 crowd continues to see ItsYourTurn.com as one of their primary entertainment options. As long as that's the case, we've got a pretty bright future. We've got the resources to keep the site online for as long as people want to play.
We've got a huge list of enhancements that we want to put online, and we also have plans for two new sites. I'm not ready to announce those yet, but if you check back in six months I may be ready to give you the details. Hope you enjoyed the chat, and hopefully we'll chat again. If you'd like to contact me I can be reached at firstname.lastname@example.org.
Scott: Thanks for your time, Patrick!
Patrick: Thanks for listening.