ReCode /

Version Control

Call me arrogant, but when I'm not happy with how something works, besides looking for a better alternative, I start thinking: "now how'd I do this if I were to design it". Usually those thoughts are a total waste of time since I don't have the expertise or the capacity to get to anything tangible, but occasionally I get motivated and start working on problems that are way too ambitious for me to take on.

One of the projects that I've started years ago and never finished is a version control system. The main problem I have with Subversion, Perforce, Git and the like is that they're way too slow for us game developers. In games, besides code, we need to store very many and very large binary blobs. So many that even over a gigabit local network syncing the entire project tree to the latest revision is a major pain.

The net effect is that developers become hesitant to syncing: your local revision of the project is in good working order, why would you bring in changes that might break it? The answer, of course, is that frequent syncing makes merging easier and more reliable. Yes, you might sync to something broken, but why is this a problem? You have a version control system right? You should be able to undo a situation like that in no time!

But you can't. It takes forever to sync! So you keep working, until you're ready to commit. At this point you *should* first sync, but you don't. It takes forever to sync and forever more to undo a bad sync!

Is it possible for syncing to work faster? After all, the version control system already uses binary deltas and you can't easily improve the speed of your network. So it boils down to how many and how big your files are, nothing can be done about that right? Not quite.

Regardless of how large the files are, they are all being created and modified by people; and regardless of how productive people are, in practice all changes they make are highly localized. People work on individual files, and compared to the size of the entire project tree even Photoshop files are tiny on their own.

So, there is no reason whatsoever for the version control system to wait for me to request a sync before it begins to transfer the data. In fact, there is no reason whatsoever to wait for someone to commit a change. As soon as a file is modified locally, that change can be mirrored to all other users who are likely to need it in the future. That way whenever they're ready to sync, the operation would involve moving a bunch of local files. And that *is* fast.

Does such a version control system exist? Is it possible to emulate this behavior with existing software? Is there any reason why such a design is undesirable?

Andrew Armstrong — 11 January 2011, 09:05

That sounds awesome, especially if you used multicast somehow (because it's background it'd not matter if it lost some packets here and there as long as it does file checks), which would cut down on network congestion too, even for large files.

As long as hard drives were large, there would hardly be any reason not to do that, as long as there are 2 local syncs - the active one you're using and the background one (containing changed files only presumably!).

Would be fun to know if it exists anywhere already for sure.

Geoff — 15 June 2011, 11:13

This sounds like a suggestion to implement shared network drives as version control.

You would also have to solve the problem of people modifying one (potentially) large file multiple times in quick succession, possibly causing network congestion and beyond that; read/write errors or lock contention due to attempting to update the file while the file is still being broadcast to other computers over the network. Your best bet would probably instead be to sync files in an automated manner at scheduled times throughout the day (varying frequency by file size if desired).

Add comment:
Sign as author:
Add 1 to 263 and enter it here:

Formatting hint: when posting comments, surround code blocks in [@ and @].