Drastically Reduce Your Development Time (Project Planning and Execution)

We consider ourselves to be a rapid-development, rapid-deployment team. We do most of our projects in half the time than other teams would. Lots of people have asked us how we do it, and we’re finally ready to share our secret with you. As a caveat: this is what works for us– it may not work for you, but we do believe this is the best approach to use when developing any sort of application or product. We also strongly believe that if it weren’t for this approach, developing applications would take twice as long. Try it out for yourself!

The value of an engineering education is not an advanced knowledge of physics or math. Engineering teaches you two crucial skills: problem solving abilities, and the engineering design process. It is the engineering design process that helps us plan out and execute our projects with ass-numbing speed. I’ll first go over a slightly modified engineering design approach with very general examples. Later, I’ll give a detailed example of how we used this approach to develop awesometransfer.

1. Define the Problem
   You must first determine the problem to be solved. This sounds trivial and not really helpful, but I promise you that it is. Determining the problem helps you define your project’s bounds and keeps you on track. “My website doesn’t exist” is not an appropriate problem statement, however “Currently, there is no arena in which college students can network with one another” (could be facebook’s initial problem statement) is. Write your problem statement down, and stare at it for a while. Is that really what you’re trying to solve? Don’t move on until you’re sure your problem statement is solid.

2. Define the Functions Necessary to Solve your Problem
   This step is really important, as it avoids “scope creep,” the tendency for projects to get too large and incorporate too much. It’s also important to leave specific constraints out of your defined functions. For example, if your problem is “to find a way to safely disarm bombs” (think the army’s bomb-exploding robots), one such function would be “a method to control the device.” At this point, it’s important to leave things in general terms. The problem statement does not explicitly say “robot” or “machine”– the wording of the problem allows for the “device” to instead be a trained human working for the bomb squad. The function (“controlling the device”) then does not have to be an RC controller to control a robot, but can be a radio used to communicate to the bomb squad person. Make sure you carefully look at all the steps necessary to solve your problem. Write it all down. As you did in Step 1, stare at it for a while, and make sure everything’s solid before you move on.

3. Research a Whole Lot
   Research possible methods to solve your problem and each of its functions you just outlined. In your research, you might find that it’s maybe not the best idea to have a human defusing bombs. Be sure to research all sorts of possibilities, even zany ones. Don’t lock yourself into a given method yet– you should be looking at all sorts of things. Can you disarm a bomb by throwing another bomb at it? Sure you can! Can you disarm a bomb by setting an EMP off nearby? Some, yes. Others, no. Can you disarm a bomb by using a robot? Learn everything about your problem and related problems during this stage.

4. Explore Technical Hurdles
   This is where you start to realize project constraints. When thinking about disarming a bomb by frying its circuitry with an EMP, you should realize that not all bombs use circuits that can be fried. That’s a significant technical hurdle. So maybe the EMP method isn’t the best way. Write down a list of all the possible problems you might run into while developing your product, and make sure to figure out how serious each hurdle is to your overall project.

5. Brainstorm
   All those technical hurdles you just listed? Write out several ways you can get around each one of them. This is brainstorming, so don’t be afraid to throw completely random things out there. This also works best with a team: sit around and throw ideas out there. Don’t evaluate any of the ideas yet, and don’t shoot anything down. Always use positive feedback when responding to a teammate’s idea, but in general try to avoid evaluating any ideas altogether.
   
   After brainstorming, sit back and then get serious again. Evaluate all the ideas that you and your team wrote down. Don’t be so quick to throw away the crazy ideas– even though an idea might be totally off the wall, it’s possible that one aspect of it has some value. Then select the “best” ideas from your brainstorming session, and formulate a prototype.

6. Build it, then break it
   Spend some time building your prototype. Don’t go crazy over it. Get it so that its core functionality (the things you listed in Step 2) works. Then do everything you can to break it. Throw bad input at it, throw grenades at it, make it listen to bad music, whatever. The point is to utterly destroy it. If it’s not on the ground crying, then you did something wrong. In breaking it, you’ll learn a lot more about what its shortcomings are and how to improve upon its overall design. Ask yourself: why did this break just now?

7. Fix it
   All the stuff you broke in the last step? Fix it! But build it better and stronger. Hopefully you learned a lot about the limitations of your project in the last step as you were breaking it. You should know why a component broke, and now it’s time to fix that at the source. Tackle the underlying problems. You may have to restructure some major part of your design, but it’s for the best. You’ll end up with a robust and powerful product at the end.

8. Physical Design
   Now it’s time for the physical design, or graphical layout of your final product. Always think of the user first. If you’re building a robot that will be exploded every time it defuses a bomb, maybe you don’t need to worry about its aesthetic. If you’re designing a web page, make sure to make it as easy as possible for the user to navigate and use. Don’t let back end considerations get in the way of a user-friendly design. It may turn out that the best design requires ass loads of AJAX and additional hours of development. So what? It’s worth it. You only need to write that code once, but users will (hopefully) use it over and over and over again. Your extra six hours of work will save your 100,000 users a hell of a lot more time than you spent developing it. It’s a necessary sacrifice.

9. Deploy it
   You’re basically done! You should get some of your friends and family to test your product out and give you feedback on it (or, if it’s a bomb defusing robot, let the army test it– not Aunt Fannie). Then deploy and enjoy.

What I just outlined is a slightly modified version of the engineering design process– it’s modified slightly to fit software development a little better. Real engineering design takes into account budget and environmental factors as well, but these are rarely necessary for a software project. If you want, read on, and I’ll describe how we used this to build awesometransfer.

Awesome. How does it work for a website?

Here’s how this works for a web application we developed, awesometransfer.

1. What’s the problem?
   We got frustrated at AIM Transfer not working through some NAT/Firewall situations. Email has file size limits, and is also synchronous (meaning the upload has to complete before the download can even start). Thus our problem statement became: “Users have no way of asynchronously transferring large files through any NAT/Firewall situation.” This took us an hour to fully think about and formulate properly.

2. Functions
   In order to transfer a file asynchronously, what has to happen? First a file must be uploaded, and while it’s uploading (or after) it must be downloaded. Two functions. Easy, right?

3. Research
   Ok, how do you upload files to a web server? There are lots of ways doing it. Since our service was going to be built in Ruby on Rails, we found a simple upload script. How do you download a file from a server? That’s easy too. No problem here.

4. Technical Hurdles
   AHH! It turns out that when you upload a file through lighttpd, it gets temporarily cached while uploading and then gets dumped to disk all at once. Why is that a problem? This means that we’ve broken the “asynchronous” part of our problem description. The sender has to upload the file fully before it can start being downloaded.

5. Brainstorm
   After a little more research and thinking, we decided to give Apache and Perl a try. We tried using CGI.pm to upload a file; same problem there– caching. After a little more thinking and research, we decided to go old-school, before the CGI.pm days, and build a file upload utility in Perl using the form file attachment standard (which we had to look up). We were able to write a script that takes an upload and writes it to disk as it’s coming in. That was the hard part. The rest is a cakewalk.

6. Build it. Break it.
   Once you can get a “streaming upload” working, it shouldn’t be too hard to get a download going, right? Wrong. When you simply request a file from a server, it sends the part of the file that’s there that instant. It has no concept of how long the file should be. So if you were to try downloading a file that was only 50% uploaded, you’d get only 50% of the file, and the download would stop. Damn.

7. Fix it
   Well, we had to write another script to make sure that the download doesn’t stop until the whole file has been sent. This is sort of like a “streaming download of a streaming upload.” It wasn’t that difficult, but we still had to figure out how to make the download “wait” for another chunk of the upload, and also had to figure out how the script would recognize when the upload was finished and the full file available.

8. Design it
   Eric came up with an amazing design for the site. It looked great. It was a bitch to get it to work though. It took me at least two days just to do all the AJAX stuff. Trust me, it was worth it.

9. Deploy it
   After some testing, we launched the site. We made sure users liked it. We barraged it with tests. The whole process took us about two weeks.

And there ends our saga. I hope you got some useful information from this article. We very strongly believe in the engineering design process, and hope it will benefit you some day.