So Many Schemes

Recently, I was advising a friend about startup interviews and came up with this list of technologies that you simply must know about when going into a technical startup interview. You don’t need to be an expert in every one of these, but I think you should be aware of their existence and their high-level overview.

So, in no particular order…

1. Most web applications are database driven, so know about database scaling and performance. Also: Memcached (distributed memory cache), caching proxies like squid, and caching techniques in general.
2. Machine learning and data mining techniques — at least an understanding of their potential. There is way too much to go into here, but play with the open source package called Weka. Also check out the excellent introductory, hands-on book Collective Intelligence by Toby Segaran.
3. Lucene, an open source search engine.
4. If you’re applying (or working at) an interesting startup, you’ll probably have large quantities of data to process, so know about Hadoop, the open source answer to Google’s MapReduce paradigm. Consider in combination with Amazon’s EC2 (see below).
5. The Google File System (GFS) and Google’s BigTable. These projects represent the current cutting-edge in data storage for scalable web applications. But if you want to use them, you’ll have to join Google or use one of these projects that offer some (but not all) of their features: Amazon’s S3 and SimpleDB (see below), MogileFS, Global File System, and Hadoop’s HDFS file system with HBase acting as BigTable, but HBase may not be ready for prime-time just yet. There are, of course, other solutions of varying complexity as well.
6. Amazon web services: S3 (backups, reliable data store; data archive; file serving), EC2 (virtualized, scalable utility computing; file processing; server environments — and you should know something about machine virtualization in general as well), and to a possibly lesser (and unproven) extent SimpleDB (scalable database replacement for some types of applications — you should have experience with MySQL and the SQL language too).
7. Ruby on Rails — even if you’re not working in Ruby or deploying a Rails app, Rails is a powerful environment for rapid prototyping and experimentation, plus a very marketable skill in the current climate.
8. Almost without saying: the obvious frontend interface technologies of HTML, CSS, JavaScript, and Flash/Flex.

All of the above are fairly language agnostic. You should know a couple of programming languages quite well, preferably one scripting language (probably one of Ruby, Perl, Python, or PHP), and one ‘harder’ language such as Java or C++. Be prepared to write code and answer questions in your chosen languages.

When interviewing at a startup, or any place really, make sure to a) explain your thoughts when solving problems (don’t just think to yourself for 5 minutes), b) talk about what excites you (technologically and otherwise) and your awesome side-projects, c) be willing to talk about the flaws as well as the strengths of technologies, d) know something about the technological area of the startup, and e) actually know the subjects you proclaim to understand on your resume.

Everyone is hiring right now (including my employer)! So read up, do some side projects, and good luck!

What do you think about this list? Please suggest technologies and links that I missed!

After writing about online annotation, I received an invite to diigo, a new social web page annotation service. Diigo seems like a pretty cool idea, although the beta, at least, is hampered by some of the same fundamental issues encountered by all online annotation systems that I’ve seen. Diigo allows you to bookmark pages, highlight and annotate portions of pages, and share these content items with your friends, or the whole world, via feeds and tags. Diigo works via a Firefox, Flock (not surprising, as it comes from the same base as Firefox), or IE plugin. Unfortunately, the plugin is not available for Safari yet. With the plugin installed, you can highlight portions of a page’s text and leave annotations. Additionally, you can choose to view comments and annotations left by your friends, or the Internet as a whole.

The hard thing about web page annotation is figuring out how to lock a highlight or annotation to a specific place on a web page. Especially with dynamically generated pages, the content can change often, so how do you know where to put the annotation? I was excited to see how diigo handled this issue, and they seem to have taken the textual route, analyzing the page’s source via JavaScript and looking for text that matches what was originally annotated. This works when the annotated text is unique, but fails if you happen to annotate text the recurs on the page. However, this still works reasonably well. (If you’re interested, you can grab their plugin and unzip the .jar file to see how some of this stuff works. The interesting text selection code is in the core.js file.) I don’t know how to get around the annotation problem, but will keep an eye on diigo as they continue to refine their beta. Perhaps they will think of something that I haven’t.

Solution Watch and TechCrunch already wrote about Diigo.