syncing variations on a repository git

I am trying to figure out how (if possible) to set up multiple git repositories to behave in a specific way. Basically, we have multiple repositories for multiple clients, however they are meant to all share a system directory. When a change is made to the system directory for one client, it should be made to the system directory for all clients. Now for the complication, we have multiple environments that we use for a CI development system. We currently have a production environment, client rev environment, qa environment, and development environment. These translate nicely into different branches because it means that when a workflow is approved for the next environment, we can just merge that workflow branch into the appropriate environment branch and get just the changes from that workflow added into that environment.

The problem is our code structure is set up in a way that we have system directory that is our base code, a custom directory that has client specific code (including files that get used in replacement of files in the system code) and a client specific configuration directory. Note we are not on an object oriented system which makes development even more complicated because there's no inheritance or extensions to make customizations easier.

Our goal is to convert this development system into GIT. The problem we're having is maintaining that shared system directory. We've tried submodules, but submodules require a lot of maintenance to prevent overwriting or mispointing them and don't work quite as well with permanent environment branches because of the excessive tedious amount of work to make sure that the client qa branch is always pointing to the lead commit of the qa branch in the system submodule and not to a lead commit in a WF branch.

Our current IDE is eclipse which doesn't have subtree integration (though what I've read of subtrees is that they can be just as complicated as submodules). What we really need is parallel branches across multiple repositories that get propogated. So for example if I make a workflow branch in a client repo, a WF branch gets made in the system repo and in all other client repos. And when I make a change and merge that workflow into qa, the system repo's wf branch also gets merged into the qa branch, but only with the changes to the system folder, and not changes that were made to the custom and config directories.

I was wondering if there is a good structure to make something like this work. Everything I've read on other sites and sources has said the solution is just don't do it that way, however that decision is not made by me. The higher ups are determined to keep our current development system.

One thought I had is to see if there's a way to do something like a passthrough where we create a repository that has just the system module. Then for each client we clone that repo and add the custom and config directories. Then when we do development we clone that client's repo. I was wondering if by this method, if we made a change and committed and pushed to upstream, if this would push changes all the way to the top level repo. Also when we pull down changes, if we could pull all the way from the top level repo. The other side of this would be having to set up .gitignore in the client repos in such a way that when pushing up stream it ignores the custom and config directories but when pulling downstream it includes those directories.

Any thoughts on this will be appreciated. Also our code host is a private gitlab server.

Sorry this is so long, but I feel we have a very unique (not good) system that requires a bit of explanation.

Future Git User

This seems more like an organization and architecture problem than git issue but I'll give it a go.

Rather than creating a repo, have you considered creating a branch for each client and then do merge/pull as you need and then push.

Deploying via git is not recommended, git is not a great way to deploy. Its design to be open, record changes and make sharing of code easy so if someone accidentally commits credentials, they are recorded forever unless you do some crazy things (and these crazy things will break the code for everyone else). Most sane users use git to commit and share their changes, build a specific branch into a module/pkg/image/container/something-stable-and-constant and then deploy that module elsewhere to their clients.

If you still wish to use it for deployment, checkout git-hooks. You can setup some hooks (e.g update or pre-receive) that automatically pull the code when the branch is updated.

Another alternative would be more all of the shared directory into a cloud storage service or document-based NoSQL server. It helps you maintain a single source that everyone can fetch from. This won't be perfect and eventual consistency will cause few troubles here and there, but will be worth it. I'd only advise against it for cases where files being on the client's disk right-away is important.