Main Content

Magnolia Community Forums: Announcements: Git migration starting April 4th !


  • gjoseph
    gjoseph
    Full name: Grégory Joseph
    Posts: 1,031
    Last post: Sep 15, 2015 9:56:30 AM
    Git migration starting April 4th !
    #1 by gjoseph on Mar 28, 2012 12:55:42 PM

    Hi all,

    This is a post about the upcoming migration of Magnolia's codebase from Subversion to Git. No panic is to be had, read on for details.

    Since about a year, we decided to take the plunge and move to Git. Last December, I started looking into tools to help us migrate to Git. [1]

    Unfortunately, since then, we've been fairly quiet about it, and I've been so deep into setting up the necessary servers, trying and verifying various migration procedures and planning all sorts of details, that I've even forgotten to talk about it here.

    So this is it. We're finally moving to Git. Some have been waiting for this for years, and some are probably still a little afraid of this. It's going to be alright.

    What does it mean to you ? Well, depending on what you do with Magnolia's source, it will mean nothing at all, or perhaps just not much. First off, the existing Subversion (SVN) will remain accessible. Each project which gets migrated to Git will be switched to "read-only", but there are no plans on shutting it down. If you regularly pull sources from our SVN repository, this will continue working, but trunks and branches will not be updated: you will have to clone the new Git repositories instead.

    Once migrated, the projects will be hosted on the new Git server; it currently has a Gitweb interface, visible at http://git.magnolia-cms.com. Details on how to access and use the new repositories are at http://wiki.magnolia-cms.com/display/DEV/Git+usage.

    Oh, and another great news: we will also start mirroring our open source repositories to GitHub at https://github.com/Magnolia. We love the GitHub interface, and we suppose you do too.

    Starting next week (Wednesday April 4th, i.e right after 4.5.2 is released), we will start migrating our codebase to Git. We're currently planning to migrate projects in "chunks", in the following order:
    * main project (i.e currently at http://svn.magnolia-cms.com/svn/community/magnolia)
    * ce and ee bundles
    * all community and enterprise modules
    * build tools

    Forge projects will be migrated "on demand": if you have a project on the Magnolia Forge and would like to use our Git repository, let me know ! We are also taking this opportunity to "fix" a little mistake we did recently - it wasn't clear what a Forge module should be, and we started adding some of our own modules to the Forge; this means the CMIS and Device Detection modules will be migrated to the modules/directory of the Git server. Details and reasoning on the wiki. [2]

    In the meantime, please let me know if you have any concerns or questions, of course. I'll send a pair of other emails before and after migration "chunks".

    Looking forward for your pull requests !

    Cheers,

    -greg

    [1] http://dev.magnolia-cms.com/~gjoseph/git-at-magnolia
    [2] http://wiki.magnolia-cms.com/display/FORGE/Home
    http://wiki.magnolia-cms.com/display/DEV/Git+usage

  • gjoseph
    gjoseph
    Full name: Grégory Joseph
    Posts: 1,031
    Last post: Sep 15, 2015 9:56:30 AM
    Re: Git migration starting April 4th !
    #2 by gjoseph on Apr 4, 2012 12:29:27 PM

    We are about to start migrating now. As a reminder, Subversion will remain accessible, but the parts being migrated will be in read-only mode.
    We're stating with the main project. Keep an eye on the Git repositories structure wiki page for up-to-date information.

    Cheers,

    -greg

    [1] http://wiki.magnolia-cms.com/display/DEV/Git+repositories+structure

  • gjoseph
    gjoseph
    Full name: Grégory Joseph
    Posts: 1,031
    Last post: Sep 15, 2015 9:56:30 AM
    Re: Git migration starting April 4th !
    #3 by gjoseph on Apr 5, 2012 11:56:15 AM

    Folks, the git-svn-clone command takes *way* longer than expected. There are seems to be some voodoo problem with the VM this server is running, so we're fixing that now, and see if it helps.

    So far, we're only migrating the main project [1]. Modules and bundles should be *much* faster. In the meantime, svn for all projects is still available. Only [1] is read-only. Keep your patches warm :)

    We probably won't start migrating modules until Tuesday.

    Sorry for any inconvenience this might cause. As always, if anything is unclear, if you have concerns or questions, please send them my way !

    -greg

    [1] http://svn.magnolia-cms.com/svn/community/magnolia

  • gjoseph
    gjoseph
    Full name: Grégory Joseph
    Posts: 1,031
    Last post: Sep 15, 2015 9:56:30 AM
    Re: Git migration starting April 4th !
    #4 by gjoseph on Apr 10, 2012 12:42:18 PM

    Well, migration for the main project is still running. Since it is taking much longer than expected, we might switch back to writeable mode for this part of SVN. While this is running, we will start migrating modules now. Our first two targets: imaging and dms. If all goes well, we will migrate more modules right away, hopefully today. We will keep the status updated on the wiki: http://wiki.magnolia-cms.com/display/DEV/Git+repositories+structure

    Cheers,

    -greg

  • gjoseph
    gjoseph
    Full name: Grégory Joseph
    Posts: 1,031
    Last post: Sep 15, 2015 9:56:30 AM
    Re: Git migration starting April 4th !
    #5 by gjoseph on Apr 13, 2012 7:54:44 PM

    Ok folks.. here's some sad news. We've had tons of quirks with the migration that my tests hadn't revealed earlier. So here's what we're gonna have to do for now:

    * modules are fast to migrate. There's been a lot of little issues with empty directories and moved projects, so we'll rework on our migration script/procedure until it's all super smooth. In the meantime, we will "reopen" the corresponding svn location for writing.

    * the migration of the main project takes forever, and we're unsure it'll be good enough, if and when it ever finishes (because the same issues as with the modules are likely to occur) - so we'll leave it running, inspect it, and perhaps start it again once we're more confident. I read today that the Hudson project's migration took 3 weeks, so that, in a sense, is reassuring, we're probably not doing this completely wrong. In the meantime, we'll also reopen that svn.

    We'll notify you of further progress, of course. As far as modules are concerned, it could be that we get through next week… but I won't promise anything anymore.

    Sorry for any inconvenience these shenanigans might have caused. Cheers,

    -greg

  • gjoseph
    gjoseph
    Full name: Grégory Joseph
    Posts: 1,031
    Last post: Sep 15, 2015 9:56:30 AM
    Re: Git migration starting April 4th !
    #6 by gjoseph on Apr 30, 2012 10:50:24 AM

    Hi there,

    After much testing and messing around with Git, figuring out a few itchy corners of the migration, we're now ready to do it "for real".

    As soon as 4.4.7 is out of the way, we'll re-start a background process that will migrate the main project (/community/magnolia) from a mirror of our Subversion repository. This might take as long as 3 weeks to complete (yes, yikes); we're hoping for this round to be faster, because we've figured out a few quirks, but it WILL take a relatively long time.

    In the mean time, we'll migrate modules as described prior. If we start on Wednesday, which is the current plan, I expect that a good bunch of modules will be migrated by the end of this week, and all of them by the end of the next, barring any other unexpected issues.

    Once the long running process of migrating the main project is done, we'll finally put it in read-only mode on the master Subversion, and migrate the "delta" that will have happened since then, thus finalizing this overly long process.

    For details on usage and status, see the following two pages:

    * http://wiki.magnolia-cms.com/display/DEV/Git+usage
    * http://wiki.magnolia-cms.com/display/DEV/Git+repositories+structure

    Hopefully, I'll find the time to write down some of the quirks we found out in a blog post at some point in the near future.

    Cheers,

  • gjoseph
    gjoseph
    Full name: Grégory Joseph
    Posts: 1,031
    Last post: Sep 15, 2015 9:56:30 AM
    Re: Git migration starting April 4th !
    #7 by gjoseph on May 9, 2012 3:17:38 PM

    Good news, everyone !

    Most community modules are now migrated and available for general consumption. More specifically, as I write this, the following modules are now available over Git: blog, categorization, commenting, data, device-detection, dms, docutools, excelrenderer, form, google-analytics, groovy, imaging, inplace-templating, public-user-registration, resources, rssaggregator, standard-templating-kit and standard-templating-kit-prototype.

    They should be visible at http://git.magnolia-cms.com/gitweb/, and cloneable anonymously with git clone http://git.magnolia-cms.com/modules/foobar.git

    For developer access (i.e. to be able to push), use https instead, or ssh with git clone git@git.magnolia-cms.com:modules/foobar. To use the former, you need an account at Magnolia. To use SSH access, I'll need to register your SSH public key on our server; send it my way if you'd like that.

    For details on usage and status, the following two pages should still be your go-to place for information:
    * http://wiki.magnolia-cms.com/display/DEV/Git+usage
    * http://wiki.magnolia-cms.com/display/DEV/Git+repositories+structure

    If anything doesn't work as expected, if anything is unclear, or if you have any question, leave it here, or leave a comment on the wiki, and I'll try to address it asap.

    We will now proceed with "step 2" of the migration of the main project: we'll put SVN for community/magnolia in read-only mode and migrate it (we've migrated 95% of it in silence over the week-end) - if all goes well, it should be entirely available on Git later tonight or tomorrow.

    Cheers,

    -greg

  • dlipp
    dlipp
    Full name: Daniel Lipp
    Posts: 50
    Last post: Nov 17, 2014 3:56:47 PM
    Registered on: Sep 16, 2010
    Re: Git migration starting April 4th !
    #8 by dlipp on May 10, 2012 2:55:55 PM

    Hi everyone,

    another bunch of community modules has been successfully migrated to git:

    blossom (including blossom-extension-data & samples)
    magnolia-4-5-migration
    magnolia-module-google-sitemap
    magnolia-module-observation
    magnolia-module-openid
    magnolia-module-poll
    magnolia-module-scheduler
    magnolia-module-snippet
    magnolia-module-todolist
    magnolia-module-webdav
    modeshape-support

    You'll find the details on wiki and gitweb as before - more modules to follow soon...

    Regards,

    Dan

  • gjoseph
    gjoseph
    Full name: Grégory Joseph
    Posts: 1,031
    Last post: Sep 15, 2015 9:56:30 AM
    Re: Git migration starting April 4th !
    #9 by gjoseph on May 16, 2012 12:32:42 PM

    Hey, enterprise modules have also been migrated. Details on the same usual pages :)

You don't have the permission to post on this thread

Sign in

To login on this forum, you can use your Magnolia Forge, Support or Partner account, or, below, your Google, Yahoo! or OpenID account. If you have trouble logging in, or any other sort of issue, please let us know in the Meta forum, on the user-list, or simply by email at forum-admin at magnolia-cms dot com.

* Required

... or sign in with:

  • icon http://{your-openid-url}
  • icon
  • icon https://me.yahoo.com/