Showing posts with label Flickr. Show all posts
Showing posts with label Flickr. Show all posts

November 05, 2007

Scalable Web Architectures: Common Patterns and Approaches

Ah, I'm having a great time in Berlin! But instead of a personal blog post, this is a "live blogging"-like post. These are my notes on the "Scalable Web Architectures: Common Patterns and Approaches" workshop Cal Henderson (Flickr) gave. As you'll see in this and future posts, this isn't me talking about the workshop but just the notes I took from it...



-- Stuff you have to care about...
  • * What is scalability
  • * Traffic growth
  • * Dataset growth
  • * Mantainability
  • * High Availability
  • * Performance

-- Scalability
  • * Vertical (get bigger)
  • * Sometimes is alright... quicker... and sometimes less expensive, if the software isn't written for H-scalability or when rearchitecting it goes more expensive than to buy just "more RAM" or somthing
  • * Horizontal (get more)

-- Architecture
  • * finding the right balance for good/fast/cheap

-- App servers
  • * state and sessions? please no sessions or at least not local sessions, if you want a scalable app... -> store in a central database, instead of using sticky sessions
  • * no sessions can be implemented:
  • - stash everything in a cookie
  • - sign the cookie
  • - using a $timestamp you can easily see when the cookie expires
  • * "super slim sessions"
  • - no account info in the session (use the previous cookie) but you can fetch database stuff and stuff it in a
  • "super slim session"
  • * the ramsus way - architect your app so that app servers don't know about the other servers,
  • so they can scale horizontaly without problems

# presentation
# markup
| page logic
| business logic
$ database

  • * Availability - everything doubled, if you can re-double it in another datacenter

-- Amazon
  • * S3 - storage
  • * EC2 - compute
  • * SQS - queueing
  • * all scale horizontally, it is cheap when you're small, it's not really cheap with scale

-- Load balancing

-- Parallelizable == easy

-- Databases
  • * usually the hardest part to scale (unless we're doing a lot of file serving)
  • * when starting, vertical scale should be used
  • * usually you need (specially on web) more read power than write power
  • - database replication (writes go to master, reads go to master or slaves)
  • - this sucks when you need to scale writes, 'cause you'll have to add a lot of new slaves to be sleeping...
  • * caching avoids needing to scale
  • * About MySQL clusters: MBD allows a mesh, RSN will be great in an upcoming version

-- Federation (at this moment we're talking about /big/ scaling)
  • * Simple things first: divide your tables and put each slice in each cluster

-- Akamai...

-- Serving Files

-- High Availability
  • * RAID 5 is cheap, RAID 10 has speeeeeed! ;-)
  • * MogileFS
  • * FlickrFS ( http://sourceforge.net/projects/flickrfs/ )
  • * Amazon S3 (cheap!)

-- Field Work

( iamcal.com/talks )

April 10, 2007

Swaptree - review

Swaptree beta logo


Remember last year when I talked about Swaptree? I bet you don't. It was way too long from then till now: in July they were in private beta and they still are, yet they are expanding slowly Swaptree's userbase by inviting more people into the private beta, and last week I was a lucky one. Anyway, you probably don't recall what Swaptree is, so let me quote myself:

So, what's this "swaptree" thing? Swaptree is a free service that allows you to swap things you have and don't want with things you want to have. It just works like this:



You manage a list of books, CD's, DVD's or games you want to trade, and a wishlist with stuff you want to have. To simplify it, you can easily do stuff like import your Amazon wishlist to swaptree. Then, the site does it's own magic by itself and you don't have to worry about anything. When there's something you want available to swap with something you have, you'll be asked if you want to do that trade. If you want to, swaptree manages even the packaging issues so you don't have to bother with nothing:



And when you realize, the swap is done! Easy, isn't it? Well... This is the part when you start wondering about the implications of such a system: it can't be just as easy as that... Well, as you can see through the blogosphere, it really is as simple as that.

So, I've tested the application and I was really surprised about how neat it is. Yet, there were two things that gave me some frustration.

The first thing is that you need to give the ISBN or the UPC of the product you have for swapping. You can't just surf, think "oh, and I also have this and don't want it" amd click on it, as you can do for your "want list". When you click on the "I have this" button in an item, you must provide its UPC or ISBN. That makes things not as easy as they could be, but on the other hand it is understandable if you think that they want some warranty that (a) that product exists (as a tradeable product), (b) you have an original product, and (c) you really have it. But the frustration really comes when you pick a big pile of CD's, books and DVD's you don't want anymore and start inserting their ISBN/UPC's, just to recieve as reply an error message saying "Please enter a valid ISBN number (for books) or UPC code (for CDs, DVDs & games) below.". And, damn, some of those items I'm 100% sure have a valid ISBN/UPC code! First I thought that only "issued in USA" items were on their database, but not even that. Sucks, in a big pile of items I only managed to insert (by now) four.

The other thing is that I added four items and no one had them in their wishlists. OK, less then 24 hours later I could already swap one of those by one of these:

66 | 100 | 3 | 0



but none of those things were in my own wishlist. So, is the model not working? Not really, that's not the issue here. This is a "relational value network", where the service value grows with the number of users and marked items (both as "I have" or "I want"). The growth is exponential, since a trade can be done in several layers: Imagine that I have something you want and you don't have nothing I want: that can be fixed if you have something a 3rd person wants in exchange of something I want. Now add up to 5 layers of complexity. Right! :-) The issue is that, for now, and being this a private beta it's no surprise that this is like that, Swaptree is lacking users.

What leads me to yet another issue, this time about something that could greatly enhance this service "social networking". I'm not a fan of social networks, and I really feel like social networks like Hi5 and Orkut are completely useless, but sometimes they are well-explored: take Flickr as an obvious example. In Swaptree you can see items you want, but you can't see "who has this item" and "what do those people also have". See, if I start browsing over SciFi books, for instance, I might want to see who has some books I liked, and, if they do, what other SciFi stuff are they into.

Finaly, in the previous post I asked this: "is this going to work overseas or only on USA?" Now that I'm in, the answer is... I don't know. Now, you can only choose, on the countries list, United States, but there is a Country list, so I guess they'll soon add more countries. At least I hope so, because until Portugal is in their orchard my swaptree can't give good apples.

January 31, 2007

Shaking Flickr

FlickrWhile Yahoo!'s competitor Google (Flickr belongs to Yahoo!) has created a Flickr alternative (it still isn't public, but it has to be if they want to be the same kind of product as Flickr), they're also suffering some anger from their users (admitedly, including me). They're continuously restraining funcionalities, some limiting even Pro costumers. They're enforcing their single sign-on, pissing off some old-skoolers. And now they're publicizing Windows Vista! WTF?

Flickr is one of those companies that aren't the best in their field, but - because they were the first to get exposure, bennefict from their user base. But, if they keep doing this, will they keep growing?

Flickr growth