6.824 Lecture 16a: Peer-to-peer systems: Content distribution Last lecture was about finding a computer for an object, more in this lecture and next, but focus on how to serve the objects (i.e., the data). Objects can be large, and therefore consume network bandwidth when serving them. Network bandwidth costs money, and many computers have insufficient network bandwidth to serve popular objects, or many objects. Goal is to aggregate network bandwidth from many computers to serve data. For popular objects, make copies on multiple computers. For many, but less popular objects, partition them across the available computers. Once we have multiple copies, it would be good to get objects from a nearby server if there are multiple copies of the data, because that will reduce latency. This pools resources to load balance traffic to peers participating in the scheme. This approach works well for static content (images in Web pages, software distributions, videos, songs, etc.); not so good for dynamic Web pages. For example, a Web page that is generated based on a user query or, worse, involves user modifying data, which introduces consistency issues if there are several copies. A challenge is that we don't know what is popular in advance, and can changes quickly. For example, a certain Web page can become very popular in short period of time, and experience a flash crowd, overloading the source, and in the worst case, make it impossible for anybody to see the Web page. There are a few different instances of the data distribution problem, depending on which link is the limiting factor, what computers you have available to aggregate, etc. The following are instances in that show up in practice: - A home user/company has a popular web page. We want to avoid to be limited by the home user's link. (Coral, Akamai, etc.) - A home user wants a faster Web for all URLs. Cooperative Web caching (CoDeen, etc.). Coral primariy helps for URLs coralized, not for all URLs. This case assumes that the user configures the browser to contact a Web proxy instead, or that all clicks are observed by a proxy close to the browser. - A home user wants to share a popular song or video, but isn't always online and has limited bandwidth. Depending on which machines are being aggregated you get two different cases, again: (1) if the machines are home PCs, then bittorrent is useful; (2) if the machine are powerful, well-connected, then UsenetNews might be good (or Bittorrent). This lecture we look at case 1+2 by studying Coral, and next at 3 by studying Bittorrent. What's the high-level problem that Coral solves? You have cooperating caching proxies scattered over the Internet. Direct browser to nearest cached copy. If not cached nearby, fetch from real server into a nearby cache. Why is this helpful? Might reduce server load. Might reduce delay visible to user. Doesn't Akamai already solve this problem? Only if the website participates and pays. What are the constraints that make it hard? No support from browser (just rely on HTTP/DNS tricks) No support from webserver---no participation needed Popularity of pages changes fast Want latency to coral to be no worse than latency to unloaded source web server. What tools are available? We only get to see DNS and HTTP requests. Assuming "Coralized" names like www.cnn.com.nyucd.edu:8090 Main ideas of coral: - Sloppy DHT - For a client IP, we need to find a proxy for the data. The sloppiness says that as soon as you find some proxy close enough to the client with the data, you return the data, rather than continuing to route to final successor node as in Chord. - URLs are also mapped to proxies. This way, if a proxy doesn't have content, it can retrieve them from other nearby proxies that cached the data. Since we're sloppy, it might not find another proxy, but this just means the data will be retrieved from the webserver (not the end of the world). - DHTs are clustered into layers - keep nodes in DHT layered, with closest layers grouping nodes with small latency, compared to chord, which sees avg. 1/2 cross-world latency * log(n) per lookup. - Coralized urls, since they tack on nyucd.edu to domain, get access to some nameserver implemented by coral, which returns ip of coral proxy near client. Need to make this fast (pings/traceroutes take time), and need to assume that dns resolver that client uses is next to the client, and that the NAT that the dns resolver uses is next to is next to the client as well. What can we achieve with just a bunch of DNS servers for nyucd.edu? Browser probably chooses a nearby server from a list of DNS server. That DNS server can send the browser an A record for one of the proxies? But which one? Idea 1: if DNS server is close (low ping time) to browser, then DNS server can return any proxy close to the DNS server. So we'd want to somehow cause browser to use nearby Coral DNS server. Idea 2: build a database mapping IP net numbers to nearby proxies, each proxy registers its net number, then DNS server looks up browser's IP net number to find proxy. What about browsers not on the same network as a proxy? Might still be nearby proxy. How does Coral cause browser to use a nearby Coral DNS server? L2.L1.L0 trick to have one chance per hierarchy level - address at nyud maps to: L2.L1.L0.www.cnn.com.nyud.net - DNS works in hierarchies separated by '.' - So if coral DNS server at L0.www.cnn.com.nyud.net takes too long, the dns server only returns IPs for L1.L0.www.cnn.com.nyud.net, which probe client more. Those finally return (closer) L2.L1.L0.www.cnn.com.nyud.net, which the resolver will stick to, because it's the most authoritative server for that full subdomain. nodes(level,count,target) to find good "next" DNS server traceroute and hints in DHT to implement nodes() ----------stopped lecture here. How does Coral find a nearby cached copy of a URL? What does Coral store in the DHT? router IP addresses (found w/ traceroute) -> nearby proxy 24-bit IP prefixes -> nearby proxy URL -> proxy If browser is at MIT, and nearest proxy is at BU, will we find it? 5 hops to www.cnn.com takes us to BBN planet. Does Coral handle flash crowds (very popular URLs) well? What might go wrong? Every proxy fetches the URL direct from server. DHT hot-spots. What does Coral do about it? What DHT techniques did they use? Hierarchy for locality. Why don't they just cache along the path? How do they choose clusters? 6.824 Lecture 16b: Peer-to-peer systems: File sharing Today's instance of the content distribution problem. A home user wants to share a popular song or video, but isn't always online and has limited bandwidth. Depending on which machines are being aggregated you get two different cases, again: (1) if the machines are home PCs, then bittorrent is useful; (2) if the machine are powerful, well-connected, then UsenetNews might be good (or Bittorrent). This lecture's case study bittorrent Usages: static bulk content Songs and videos Linux distributions ... Usage model: cooperative user downloads file from someone using simple user interface while downloding, bittorrent serves file also to others bittorrent keeps running for a little while after download completes Goal: get file out to many users quickly Encourage everyone to upload file Challenges: Tracking which peer has what Handling high churn rates Download rate proportional to upload rate Approach: Publisher a .torrent file on a Web server (e.g., suprnova.org) URL of tracker file name, length, secure hash Seed posts the URL for .torrent with tracker Seed must have complete copy of file Every peer that is online and has copy of a file becomes a seed Peer asks tracker for list of peers to download from Tracker returns list with random selection of peers Peers contact peers to learn what parts of the file they have etc. Download from other peers Transport Peers pipeline on top of TCP divide a file part further in 16Kbyte subpieces keep typically 5 requests in flight Which pieces to download strict? rarest first? ensures that every piece is widely available also helps with the seed and bootstrapping rapidly random? avoid overloading seed when starting download parallel download of same pieces? avoid waiting on slowest final algorithm random for first piece, then rarest-first, parallel for last piece Fairness Goal: avoid peers that don't upload Approach: peers maximize download rate, but uploader can choke peers reciprocrate uploading to peers that upload to them Which peers to unchoke? the ones who provide the highest download rate pick best 4 rate is calculated over 20s average best are recalculated every 10s every 30s unchoke unused peer to see if it gives a good download rate What if you are choked by your peers? choke those peers if yon't receive a piece in 1 min What if donwload finished? use upload rate as the metric How well does BitTorrent in practice? Anecdotally it works great---many happy users Difficult to answer precisely, but let's check the measurement paper figures 1, 2, 3, 5, and 6 Efficiency How long does it take for a high-upload capacity find other high-performance peers? If there are a few high-upload peers and many low-upload peers, does Bittorrent perform at its best? (Best = the total download time is at its lowest.) Fairness Do high-upload peers receive download rates proportional to their upload rate? http://www.cs.washington.edu/homes/piatek/papers/BitTyrant.pdf