Project: DartSync

Overview

We all enjoyed the use of Dropbox -- we can have our documents, photos, and videos available anywhere and anytime. All the files we store at Dropbox are automatically synchronized across our personal devices. The advantages of using such cloud-based storage services are many. However, these advantages come with prices. By uploading our data to the centralized, third-party cloud, we are not only imposing a huge traffic burden to the network that connects mobile to the cloud, but also running the risk that our data, sometimes private data, can fall into the wrong hands. These unwanted data leaks can be caused by secretive trading of our data for profits, or accidental mis-management inside the cloud, or malicious outsider attacks.

In this project, we explore a completely different paradigm for synchronizing our personal data across different devices. Rather than relying on the centralized cloud, we enable local file exchange and synchronization across devices. Devices can directly talk to each other to obtain the latest files, and all files are stored locally. The resulting benefits are twofold. First, it naturally protects user's data privacy, since all data are stored locally and controlled by users, and no copies are made on centralized cloud. Second, it constraints traffic within a local network, which not only reduces the network traffic burden to remote cloud servers, but also leads to faster file synchronization by accessing local networks. This design paradigm is towards our vision of a "decentralized cloud".


Specifically, we are implementing a system called DartSync, which is an application that runs on a local always-on node as a tracker, and a set of our personal devices as peers. Users can specify files that they want to synch across devices, and DartSync will synchronize these files across devices upon file update, addition, or removal. Ideally we would like to have you implement the tracker on a Raspberry Pi node, a credit-card size Linux machine supporting Python and C. We choose Pi to implement the tracker because it entails low power to stay always on. We do recognize, however, the possible stability issues of Pi, so we do not require implementing the tracker on Pi. If you can implement your tracker on Pi, that is fantastic. The peer nodes can be our laptops, or the Linux machines in Lab 003, or even smartphones -- it really up to you in terms of how you want to extend your code to different devices. We do not have a strict requirement here, except that you should be able to demo the synchronization across no less than 4 devices.

At the high level, DartSync runs on a tracker (server) and multiple peers (clients). The tracker collects file information from all peers, maintains records of all files, and notifies peers upon any file updates. Note that tracker does not store files, but rather only keeps file information. A peer node monitors a local file directory, communicates with the tracker, and updates files if necessary. To use DartSync, a user specifies a local file directory to synchronize (similar to the Dropbox root directory), and the DartSync system synchronizes all files under this directory across all peer nodes.

For more design details, please refer to this design document. Please keep in mind that we offer you detailed design only for your reference. Feel free to come up with other designs as long as they implement our desired features.

Requirements

Your tasks for this project are 1) building the DartSync system, 2) giving a group presentation and demo, and 3) writing a project report per group, which summarizes your design, possible extensions, and most importantly the lessons you have learned. We will evaluate all these tasks for your final score on this project.

Unlike our previous assignments where we give you well-defined code structure and you mainly fill in the implementation for specific functions, the class project is more open, where you have the freedom to design the code structure and implement functions. That said, your code needs to realize the following basic features:

  • C implementation that can run in any Linux system.
  • TCP connection and data transfer.
  • Local file monitoring.
  • Synchronization of multiple files by comparing file timestamps.
  • File replacement when updating files.
  • Fetching data from multiple peers.
  • Synchronization of multiple file folders.
Here's the list of advanced features we encourage you to try:
  • Implementing tracker on Raspberry Pi.
  • Incremental (delta) file update where only the differences rather than the whole file are transmitted.
  • Resuming from partial download.
  • Resolving conflict.
  • Supporting other OS platforms such as Mac OS and Windows, and other devices such as Android smartphones or iPhones.
  • Any other cool useful features that you can think of...

Meet Our Teams!

Team Name Team Members SVN repo Shepherd
FanTCPants Ross Bower, Andre Chen, Shelley Garg, Joon Cho team1 Xi Xiong
PiratesOfTheC Yining Chen, Tyler Albarran, Oluwadamilola Apoeso, Seok Jun Bing PiratesOfTheC Xi Xiong
Code Crushers Kevin Donahoe, Huimin Zhou, Christopher Frangieh, Binbin Zhou Code-Crushers Yuan Tian
PI-Net Vishal Gaurav, Jing Ge, Di Zhai team5 Yuan Tian
Team C-Port Khizar Hussain, Xiankai Yang, Luyang Li, Shanzae Nadeem Khan Team-C-Port Yuan Tian
byteMe Junmo Kim, Weiqiang Liu, Nisha Kumari, Christopher Kymn byteMe Shawn Xie
C-Men Justin Lee, Charles Li, Joshua Kerber, Tianqi Li C-Men Tom Chang
segfault in our stars Rebecca Liu, Jonathan Kramer, Kun Ma, Nan Hu team2 Ethan Yu
sudi-codians Collin McKinney, Mitchell Revers, Renjie Zhu, Taeho Sung sudi-codians Shawn Xie
Mustang Debanjum Solanky, Dongjun Suh, Amirhossein Sharifzadeh, Lexin Tang team3 Shawn Xie
C-monkeys Victoria Taylor, Naofumi Tomita, Feng Wang, Fanjie Xiao C-monkeys Xi Xiong
Socket Monsters Chimian Wu, Nicholas Fiacco, Kooshul Jhaveri, Jing Geng Socket-Monsters Ethan Yu
Socket Squad Anthony Addo, Justin Gualtieri, Shivang Sethi, Xiaoting Zou team4 Tom Chang
SevenDeadlySYNs Jacob Weiss, Adam Grounds, David Harmon, Matthew McFarland SevenDeadlySYNs Ethan Yu