Git DevOps

Last Updated 3/9/2021, 11:49:41 AM UTC
About 8 min read

git is the most commonly used version control system and forms the backbone for reliable devops workflows.
It allows you to maintain change control for all of your configuration data, scripts, programs, docs, etc so that you can:

Know who changed what and when. For example when, who and why changed a firewall rule, an Oracle spfile or tnsnames.ora file.
You should not have people asking questions like:
- 'where's that config file or script?'
- 'there's a bunch of versions for this scripts all over the place. Which one should I use?'
Faster onboarding of new team members. Everything they need to know about is in git
Reliable handover from one team member to another. Everything they need to know about is in git
Partition your assets:
- One or more git repos for all of your core assets
- One or more git repos for all of your customer specific assets
Implement robust CICD (Continuous Integration/Continuous Delivery) workflows:
- automation opportunities
- consistent and reproducible deployments
Automatically distribute your assets from a single location to multiple sites
Take advantage of native git workflows to implement reliable develop-test-deploy cycles.
Security
- centralized access control
- out of the box audit trail of who changed what and when
Trust:
- only accept signed (PGP) commits and releases
- only run code and/or configurations from trusted signatures
Allow customers to vet and authorize which assets can be deployed where and when.

# Myrmex Git repo structure

Create a directory with a name of your choosing inside a git repo and set it as the catalog.git_repo/root in your myrmex-sd.conf. For example, if you create directory myrmex/assets in git, then configure myrmex like so:

[catalog]
  # how often to check for catalog updates
  refresh_interval = "2m"

  # list of git repos to sync catalog from
  [[catalog.git_repo]]
    url      = "https://myserver/git/my-repo.git"
    branch   = "master"
    root     = "myrmex/assets/"
    user     = "a user"
    password = "a password"

Myrmex will periodically scan this git directory for updates and fetch them locally. Content inside this directory is expected to be organized as follows:

myrmex/assets/
├── actions
│   └── tasks
│       └── conf     # action plugin tasks configuration files
├── bin              # plugin and extension binaries by type and architecture
├── collectors
│   └── tasks
│       └── conf     # collector plugin tasks configuration files
├── dashboards       # dashboard json formatted definitions organized under arbitrary directory structures  
├── env              # agent environment files organized under arbitrary directory structures  
├── events           # event metadata organized under arbitrary directory structures  
├── logmon
│   ├── parsers      # log monitoring parser definitions
│   └── patterns     # log monitoring parser pattern definitions
├── metrics          # metric metadata organized under arbitrary directory structures  
├── notifications
│   └── templates    # notification template definitions
├── sql              # SQL query files organized under arbitrary directory structures  
├── sql-reports      # SQL report definitions  under arbitrary directory structures  
├── templates
│   └── task         # metric collection task template definitions
└── visualizations   # visualization definitions organized under arbitrary directory structures

# Git Servers

The system can integrate with any type of git server; either hosted (github, bitbucket, gitlab, etc) or your own private git server.

# Git lingo

clone, is a copy of a repository on your local machine. You can work on it offline and only fetch or push changes to the remote repo when required.
branch, a parallel version of the repository which does not affect the primary (master) branch. By convention, the master branch always contains assets which are stable and can be delivered to a production environment. So, always create a branch to make and test changes; and when ready merge to the master branch.
commit, is an individual change to one or more files on your local repository clone.
diff, the difference in changes between two commits.
fetch, refresh with the latest changes from the remote repo, without merging them. You can compare fetched changes with your local branch.
merge, takes the changes from one branch and applied them to another.
pull, a fetch followed by a merge.
push, send your committed changes to the remote repo.
tag, a reference to a specific point (commit) in the repo history.
release, synonymous to tag

# Working with git

Install git CLI or use your favorite IDE (vscode, github desktop, etc) with git intergration.

Clone a repo

over ssh:

mkdir my_devops_project
cd my_devops_project
git clone some_user@my-git-host:/data/git/repos/my-repo.git
cd my-repo
git config --global user.email "some_user@arisant.com"
git config --global user.name "some_user"

over http:

mkdir my_devops_project
cd my_devops_project
git clone http://localhost:80/git/my-repo.git
cd my-repo
git config user.email "some_user@arisant.com"
git config user.name "some_user"
git config credential.helper cache

Add or modify files and push back to git

git fetch
git add .
git commit -m "a sensible comment"
git push # if in bare repo then git push origin master

View history of changes

$ git log --abbrev-commit
commit b3091ed
Author: some_user <some_user@arisant.com>
Date:   Fri Mar 1 06:21:30 2019 -0500

     test httpd

commit 92a667b
Author: someuser <some.user@arisant.com>
Date:   Fri Mar 1 11:05:49 2019 +0200

     initial commit
$ git diff b3091ed 92a667b
index 405f653..e6d69d1 100644
--- a/README.md
+++ b/README.md
@@ -1,3 +1 @@
# Global Connect myrmex devops repo
-
-## from httpd

# Workflow for continuous delivery

master is always production ready
modifications are performed on branches
merge from branch to master when ready to deploy

# Workflow

Pull the latest from master
```
git pull
```
Create branch (do not use branches that contain things like your name. they should be task/topic/feature specific such as alert_log_parser)
```
git checkout -b my_branch_with_a_sensible_name
```
Perform work
Incorporate changes from other users in the branch via git fetch/merge or git pull
Push the feature branch and create pull request for peer review (optional)
```
git push -u origin my_branch_with_a_sensible_name
```

After peer review merge into master

git checkout master
git pull origin master
git merge --no-ff my_branch_with_a_sensible_name
git push origin master

Tag your release

git tag -a 1.1.2 -m "version 1.1.2 release"
git push 1.1.2

Remove unwanted branch

git branch -d my_branch_with_a_sensible_name
git push origin :my_branch_with_a_sensible_name

# Integrate myrmex server to git

You can configure one or more repositories to pull assets from. The configuration order is significant, if the same asset (path) exists in multiple repos the first configured repo will be preferred even though newer versions of the same asset might exist in lower priority repos.

# catalog configuration. you need to configure at least one git repo to sync configurations from 
[catalog]
  # how often to check for catalog updates
  refresh_interval = "2m"

  # list of git repos to sync catalog from
  [[catalog.git_repo]]
    url    = "http://localgit:8008/git/my-repo.git"
    branch = "master"
    root = "assets/"
    user   = "your user name"
    password   = "your password"

  [[catalog.git_repo]]
    url    = "https://github.com/arisant/myrepo-git.git"
    branch = "master"
    root = "myrmex-assets/"
    user  = ""
    password   = "your one time access token"

# Mirror the Arisant distribution Repo

You may choose to mirror the Arisant distribution repo if you are inside a network that cannot access https://github.com/arisant/myrmex-dist.git or you do not want the system to automatically fetch updates from this repo.

# Setup your mirrored repo

Initialize a bare repo (as per Setting up a local git server) on a node inside your network so that the myrmex-sd server can access it. Make sure that the repo filesystem path you choose is under the GIT_PROJECT_ROOT path of your apache configuration.

mkdir /data/git/repos/myrmex-dist-mirror.git
cd /data/git/repos/myrmex-dist-mirror.git
git init --bare
git config core.sharedRepository group

# Clone the Arisant distribution repo

On a network node that has access to the internet and your internal network, clone https://github.com/arisant/myrmex-dist.git

git clone --mirror https://github.com/arisant/myrmex-dist.git

To mirror to your internal repo:

cd myrmex-dist.git
git push --mirror http://[apacheserver]:[port]/git/myrmex-dist-mirror.git

To update your mirrored git with changes from the Arisant repo:

cd myrmex-dist.git
git fetch -p origin
git push --mirror http://[apacheserver]:[port]/git/myrmex-dist-mirror.git

If you want to vet the content from the Arisant repo before pushing to your internal repo:

cd myrmex-dist.git
git fetch -p origin
cd ..
git clone myrmex-dist.git
### updated content now in directory myrmex-dist
### once it passes your vetting process do:
cd ../myrmex-dist.git
git push --mirror http://[apacheserver]:[port]/git/myrmex-dist-mirror.git

Once you cloned myrmex-dist.git you can pull subsequent changes for vetting like so:

cd myrmex-dist.git
git fetch -p origin
cd ../myrmex-dist
git pull
## vet your changes in this directory and then:
cd ../myrmex-dist.git
git push --mirror http://[apacheserver]:[port]/git/myrmex-dist-mirror.git

← Monitoring Automation Alert Notifications →