This past summer Facebook relocated from University Avenue in Palo Alto, CA — where several buildings fan out along the downtown strip — to a new central office in Stanford Research Park. A good friend and two-year veteran of Facebook invited me to check out the new space. When I arrived, a security guard handed me a non-disclosure contract to fill out, a requirement to enter the building. “Just making sure you’re not a Twitter spy,” he said. I can therefore not describe the tour my friend gave, though photos of the new space abound on the Internet. Afterwards, we went out for a drink at the Dutch Goose, a bar popular with techies and Stanford graduate students, where most of this conversation took place. Though forthcoming, my friend was anxious to preserve her anonymity; Facebook employees, after all, know better than most the value of privacy. As she is not permitted to divulge company secrets, and would like to remain employed, her name has been omitted from this interview. It provides an interesting snapshot of the inner workings and culture of Facebook in the summer of 2009.

The Rumpus: On your servers, do you save everything ever entered into Facebook at any time, whether or not it’s been deleted, untagged, and so forth?

Facebook Employee: That is essentially correct at this moment. The only reason we’re changing that is for performance reasons. When you make any sort of interaction on Facebook — upload a photo, click on somebody’s profile, update your status, change your profile information —

Rumpus: When you say “click on somebody’s profile,” you mean you save our viewing history?

Employee: That’s right. How do you think we know who your best friends are? But that’s public knowledge; we’ve explicitly stated that we record that. If you look in your type-ahead search, and you press “A,” or just one letter, a list of your best friends shows up. It’s no longer organized alphabetically, but by the person you interact with most, your “best friends,” or at least those whom we have concluded you are best friends with.

Rumpus: In other words, the person you stalk the most.

Employee: No, it’s more than just that. It’s also messages, file posts, photos you’re tagged in with them, as well as your viewing of their profile and all of that. Essentially, we judge how good of a friend they are to you.

Rumpus: When did Facebook make this change?

Employee: That was actually fairly recently, sometime in the last three months. But other than that, we definitely store snapshots, which is basically a picture of all the data on all of our servers. I want to say we do that every hour, of every day of every week of every month.

Rumpus: So this is every viewable screen?

Employee: It’s way more than that: it’s every viewable screen, with all the data behind every screen. So when we store your photos, we have six versions of your photos. We don’t store the original: we make six different versions on the photo uploader and upload those six versions.

Rumpus: And the difference between them would be sizing, certain areas are zoomed –

Employee: Exactly. Different sizes for the news feed, your profile pic, enlargement.

Rumpus: And these reside on servers in your office?

Employee: No, not in our office. Absolutely not. We have four data centers around the world. There’s one in Santa Clara, one in San Francisco, one in New York and one in London. And in each of those, there are approximately five to eight thousand servers. Each co-location of our servers has essentially the same data on it.

Rumpus: And how many users are you up to now?

Employee: That I can disclose publicly? Two hundred to two hundred twenty million.

Rumpus: And actually?


Employee: That’s just active users. As far as total accounts, including those that are potentially fake, disabled and whatnot, we’re over three hundred million. The two hundred twenty million are users who have logged on and done something with the site in the last thirty days.

Rumpus: You said they’re changing the policy of keeping all information.

Employee: No. They’re never changing that policy. We still keep all information. What I was referring to, is that if anything, we’re going to start deleting more photos for performance reasons. We are the largest photo distributor in the world.

Rumpus: Really? Is that obvious?

Employee: I don’t know the exact figures off the top of my head, but I want to say upwards of a trillion photos, and then think about six copies of each. This is the epitome of a needle in a haystack. When we need to load a webpage in half a second, we need to go and find upwards of a thousand photos — think about your newsfeed — in one get [snaps], and instantaneously. It’s hard to do.

Rumpus: You’ve previously mentioned a master password, which you no longer use.

Employee: I’m not sure when exactly it was deprecated, but we did have a master password at one point where you could type in any user’s user ID, and then the password. I’m not going to give you the exact password, but with upper and lower case, symbols, numbers, all of the above, it spelled out ‘Chuck Norris,’ more or less. It was pretty fantastic.

Rumpus: This was accessible by any Facebook employee?

Employee: Technically, yes. But it was pretty much limited to the original engineers, who were basically the only people who knew about it. It wasn’t as if random people in Human Resources were using this password to log into profiles. It was made and designed for engineering reasons. But it was there, and any employee could find it if they knew where to look.

I should also say that it was only available internally. If I were to log in from a high school or library, I couldn’t use it. You had to be in the Facebook office, using the Facebook ISP.

Rumpus: Do you think Facebook employees ever abused the privilege of having universal access?

Employee: I know it has happened in the past, because at least two people have been fired for it that I know of.

Rumpus: What did they do?

Employee: I know one of them went in and manipulated some other person’s data, changed their religious views or something like that. I don’t remember exactly what it was, but he got reported, got found out, got fired.

Rumpus: Have you ever logged in to anyone’s account?


Employee: I have. For engineering reasons.

Rumpus: Have you ever done it outside of professional reasons?

Employee: I will say, when I first started working there, yes. I used it to view other people’s profiles which I didn’t have permission to visit. I never manipulated their data in any way; however, I did abuse the profile viewing permission at several initial points when I started at Facebook.

Rumpus: How about reading their messages?

Employee: Never individually like that. I would mostly just look at profiles.

Rumpus: Would you suppose that Facebook employees might read people’s messages?

Employee: See, the thing is — and I don’t know how much you know about it — it’s all stored in a database on the backend. Literally everything. Your messages are stored in a database, whether deleted or not. So we can just query the database, and easily look at it without every logging into your account. That’s what most people don’t understand.

Rumpus: So the master password is basically irrelevant.

Employee: Yeah.

Rumpus: It’s just for style.

Employee: Right. But it’s no longer in use. Like I alluded to, we’ve cracked down on this lately, but it has been replaced by a pretty cool tool. If I visited your profile, for example, on our closed network, there’s a ‘switch login’ button. I literally just click it, explain why I’m logging in as you, click ‘OK,’ and I’m you. You can do it as long as you have an explanation, because you’d better be able to back it up. For example, if you’re investigating a compromised account, you have to actually be able to log into that account.

Rumpus: Are your managers really on your ass about it every time you log in as someone else?

Employee: No, but if it comes up, you’d better be able to justify it. Or you will be fired.

Rumpus: I would imagine they take this—

Employee: Pretty seriously. I don’t really fuck around, at all.

Rumpus: They invented a Chief Officer position for it, Chris Kelly, right?

Employee: Chief Privacy Officer Chris Kelly, correct. Running for Attorney General of California.

Rumpus: Is that a standard position at Silicon Valley web companies?

Employee: I think it’s becoming more of a standard officer position, especially with Web 2.0, 3.0, where the model is basically get as much information out there as you can. Obviously, someone needs to step back and make sure there is some information privacy here, or at least as much as we can put in place.

Rumpus: Facebook was probably a big trendsetter in that regard, right?

Employee: In my opinion, we’ve always provided the most nitty-gritty user privacy settings from the beginning. There’s no other site out there that’s this customizable.

Rumpus: Would you like to give your take on the last few rounds of fuck ups, Facebook Beacon, and the recent Terms of Service controversy?

Employee: It’s really hard to judge exactly the way users are going to react. We just didn’t have a good enough beta-testing system in place. When you have a group of twenty engineers working on a project, they think it’s the most beautiful, immaculate thing in the world, and then they build it, and a project manager approves it. Initially, when that was the case, we just pushed it, and if users didn’t like it we pulled it back. That was just our philosophy, one of trial and error. Whereas now we’ve started running psychological analysis, starting to…

Rumpus: Oh really?

Employee: Fuck yeah. Are you kidding me? We do eye-tracking to see where your eyes move while you browse Facebook.

Rumpus: What do you mean by “eye-tracking”?

Employee: For example, when we want to introduce new features, like when we streamlined the browsing of photo albums, you know, where you can click ‘next’ above the photo, and the page stays the same except you get the next photo? We did tests on that, and actually found out it increased the number of page views by 77%, essentially because we were reducing 77% of the page load, and therefore it was loading faster, and thus generating more clicks. We not only reduced our bandwidth, and how much we have to pay for our Internet, but we made the site faster and increased the clicks-per-minute, which is what we’re truly interested in.

Rumpus: So in what other ways do you track behavior, that isn’t necessarily obvious to users?

Employee: We track everything. Every photo you view, every person you’re tagged with, every wall-post you make, and so forth.

Rumpus: So maybe you know about this, maybe you don’t. There’s a paradox with international expansion, because obviously all internet companies aspire to a worldwide market, but as service enters countries without great infrastructure, such as 3rd-world countries, the companies have to provide the infrastructure and the countries don’t actually produce any (or much) ad revenue.

Employee: I don’t know anything about that, actually. The one comment I would make about that, is that we’ve definitely tried to continue expanding to 3rd- world countries. Take Iran — well, Iran is not a 3rd world country — but when the Iranian elections came up, and then the disputes, we found out they were using Facebook as a tool to organize themselves and expose their qualms and discontent with the government. So publicly we translated the entire site into Farsi within 36 hours. It was our second right-to-left language, which was actually really difficult for us. Literally the entire site is flipped in a mirror. The fact that we did it in thirty-six hours — they hired twenty some-odd translators, and engineers worked around the clock to get it rolled out — was pretty fucking phenomenal. We had at least three times as many user registrations per day the first day it was out, and it has been growing. So we’re definitely still serious about foreign outreach. And the thing is, we have such a gigantic market share in the larger sections of Europe, in Australia, in Mexico, in the States and Canada, and that’s where 99.9% of our ad revenue is and probably will be always — or at least will be the next five, ten years. So the fact that we’re breaching into these other markets mostly means just allowing family and friends to connect even more deeply, which is really our ultimate goal.

Rumpus: What’s the creepiest Facebook interaction you have had?

Employee: Well, the weirdest one I’ve ever seen was one I was able to investigate, one of the situations which required me to log into other accounts. This guy had emailed my friend at school a very very odd message, pertaining to the name ‘Caitlin,’ which is her name, and ‘poop.’ It was literally one of the creepiest things I’ve ever seen: a two-page message about the name ‘Caitlin’ and its semantic relation to ‘poop.’ We found out that he had actually sent it to the first two hundred Caitlins he found on Facebook search.

Rumpus: That’s weird.

Employee: Really weird. Out of nowhere, no reasoning. He started sending it twenty times a day, to different Caitlins, for three weeks or so.

Rumpus: What’s the most bizarre?

Employee: I found a fake account created from Berkeley that used the profile picture and information from the brother of one of my very good friends. We looked up the guy who created the original profile, and he had never ever heard of him, never ever met him, obviously had never seen him. But this guy had evidently added him as a friend, and sadly he accepted it, but literally stole all of this guy’s information, created a fake account, and was communicating with himself from the fake account. He was writing on his wall and posting back to the “other person’s” wall. We found out the guy actually had about fifteen fake accounts that he created, stealing other users’ pictures and information to create the accounts, and was actually communicating back and forth with himself. Just to try to make himself appear cool, I guess?

Rumpus: That’s a really sad display of humanity.

Employee: Yeah. That is the most bizarre encounter that comes to mind. Those two are the big instances I’ve seen that made me say, “What the hell is going on?”

Rumpus: So tell me about the engineers.

Employee: They’re weird, and smart as balls. For example, this guy right now is single-handedly rewriting, essentially, the entire site. Our site is coded, I’d say, 90% in PHP. All the front end — everything you see — is generated via a language called PHP. He is creating HPHP, Hyper-PHP, which means he’s literally rewriting the entire language. There’s this distinction in coding between a scripted language and a compiled language. PHP is an example of a scripted language. The computer or browser reads the program like a script, from top to bottom, and executes it in that order: anything you declare at the bottom cannot be referenced at the top. But with a compiled language, the program you write is compiled into an executable file. It doesn’t have to read the program from beginning to end in order to execute commands. It’s much faster that way. So this engineer is converting the site from one that runs on a scripted language to one that runs on a compiled language. However, if you went to go talk to him about basketball, you would probably have the most awkward conversation you’d have with a human being in your entire life. You just can’t talk to these people on a normal level. If you wanted to talk about basketball, talk about graph theory. Then he’d get it. And there’s a lot of people like that. But by golly, they can do their jobs.

Rumpus: So what will be the net effect of running the site on Hyper PHP?

Employee: We’re going to reduce our CPU usage on our servers by 80%, so practically, users will just see this as a faster site. Pages will load in one fifth of the time that they used to.

Rumpus: When’s it coming out?

Employee: When it’s done. Next couple of months, ideally.

Rumpus: So where do these geeks come from?

Employee: I would say at least 70% of Facebook engineers are from Harvard and Stanford.

Rumpus: Wow. I know Zuckerberg went Harvard, what’s the Stanford connection? I mean other than just Palo Alto.

Employee: I don’t think there’s any question that Stanford is the number one CS department in the world.

Rumpus: Stanford engineers invented Silicon Valley.

Employee: They did.

Rumpus: How has the recent move affected the company?

Employee: Facebook just moved offices to Stanford Research Park, which is where the original HP was started. Before it was kind of sprawled out. We had seven or eight offices downtown.

Rumpus: Any changes in atmosphere after the move?

Employee: It was just nice to have everyone in one office. Before, any meetings that happened were inconvenient for most people. I mean, engineering was split up into three offices. It was a pain. Now there’s more unity, more ease of communication. Everything feels more internal. It’s super-friendly. I think the coolest thing about the work environment is the trust. They don’t care what, where, how, when, as long as you get your shit done. If you want to work at a bar, the ball game, a park, the roof, they don’t give a fuck. Just get your shit done. Hence I was able to ditch work, come have two pitchers with you, and I will literally be able to go back and get my work done. And it goes a long way. Because I know I can get these things done. I know I’m going to have to go back. And I may be there until ten or eleven tonight.

Rumpus: I’m sorry we drank all these beers.

Employee: It’s the trust deal. We’re able to do that. We don’t have to worry. We can put our personal lives first, as long as we get our work done.

***

Rumpus original art by Lucas Adams.

http://therumpus.net/2010/01/conversations-about-the-internet-5-ano...