My cheat on ci/cd pipeline
1 CI/CD Definition
Continuous integration is a coding philosophy
and set of practices
that drive
development teams to implement small changes
and check in code to version
control repositories
frequently. Because most modern applications require
developing code in different platforms and tools, the team needs a mechanism
to integrate and validate its changes.
It automates your software delivery process
, or automating the stages of app
development
.
2 Overview
Continuous integration
(CI) and continuous delivery
(CD) embody a culture, set
of operating principles, and collection of practices that enable application
development teams to deliver code changes more frequently and reliably. The
implementation is also known as the CI/CD pipeline.
The technical goal of CI is to establish a consistent
and automated
way to
build, package, and test applications
. With consistency in the integration
process in place, teams are more likely to commit code changes more
frequently, which leads to better collaboration and software quality.
Continuous delivery
picks up where continuous integration ends
. CD automates
the delivery of applications
to selected infrastructure environments. Most
teams work with multiple environments other than the production, such as
development and testing environments, and CD ensures there is an automated way
to push code changes to them.
Continuous integration and continuous delivery require continuous testing
because the objective is to deliver quality applications and code to
users. Continuous testing is often implemented as a set of automated
regression, performance, and other tests that are executed in the CI/CD
pipeline.
2.1 CI/CD Piplelines
Pipelines are the top-level component of continuous integration, delivery, and deployment. Pipelines (in Gitlab? or in general?) comprise:
Jobs
, which define what to do. For example, jobs that compile or test code.Stages
, which define when to run the jobs. For example, stages that run tests after stages that compile the code.
Jobs
are executed by runners.
Multiple jobs
in the same stage
are executed in parallel
, if there are enough
concurrent runners.
- If all jobs in a stage
succeed
, the pipeline moves on to thenext stage
. - If any job in a stage
fails
, the next stage is not (usually) executed and the pipelineends early.
- In general, pipelines are
executed automatically
requiring no intervention once created. However, there are also times whenyou can manually interact
with a pipeline. - A typical pipeline might consist of four stages, executed in the following
order:
- A
build stage
, with a job called compile. (cisco calls it "change stage
" - A
test stage
, with two jobs called test1 and test2. - A
staging stage
, with a job called deploy-to-stage. - A
production stage
, with a job called deploy-to-prod. - That's it.
- A
From cisco learning on CI-CD pipelines This is a common-pipeline:
You commit a change for example.yml to a code repository. The repository executes syntax and sanity checks, code review rules, and: - Accepts your commit and notifies the CI/CD server to run tests. - Rejects your commit based on failed checks. CI/CD prepares an environment and runs predefined tests for any Ansible playbook: - pip install ansible==2.8.1 - ansible-version - ansible-playbook example.yml --syntax-check - ansible-playbook -i staging_inventory.cfg -check - ansible-playbook -i staging_inventory.cfg -vvvv You are notified that you can propagate your changes to the production environment.
2.2 Cisco's CI/CD Pipeline
Consists of:
- source code repository
- build stage (source code, dependencies, and static code analytics)
- test stage (unit and integration tests. ([end-to-end tests])
- deploy stage (packaging, staging, and production)
The pipeline can be triggered automatically with a commit, or manually by a DevOps engineer.
A runnable instance is built during the build stage from the source code and the needed dependencies
Followed by:
2.3 Configuring CICD
From:
Pipelines
and their component jobs
and stages
are defined in the CI/CD
pipeline configuration file for each project.
- Jobs are the basic configuration component.
- Stages are defined by using the stages keyword.
For a list of configuration options in the CI pipeline file, see the GitLab CI/CD Pipeline Configuration Reference.
You can also configure specific aspects of your pipelines through the GitLab UI. For example:
- Pipeline settings for each project.
- Pipeline schedules.
- Custom CI/CD variables.
2.4 Tools integration
There are a lot of tools that can help you implement a CI/CD pipeline. Tools
based on Git (GitLab, GitHub
, and Bitbucket
) all provide you with a code
repository
as well as with the CI/CD pipeline
, while tools like Jenkins or
CircleCI do not provide software repositories. Tools such as Jenkins,
CircleCI, rely on code repositories, hosted somewhere else, to
trigger
the pipeline with the help of /APIs
.
2.4.1 Jenkins
One of the two most common CI/CD pipeline tools. The other is Gitlab.
From cisco training:
"Jenkins uses a file called Jenkinsfile for the configuration. Jenkinsfiles
are written in a programming language called Apache Groovy. Groovy takes the
syntax of Java
and combines the features of Python
(and other scripting
languages), such as defining a variable without specifying a type.
The main difference of Jenkins from GitLab is that the extensions of native
functionalities are done with the help of plug-ins, which usually translates
to expensive maintenance. GitLab
on the other hand is open-core
, which means
that any changes done to the GitLab codebase are automatically tested and
maintained."
2.4.2 Gitlab
GitLab also provides a pipeline as code, which is an upgrade from the GUI type of pipeline. This approach gives DevOps teams more freedom in the pipeline process. It also makes rollbacks much easier, and it provides a built-in lint tool that makes sure the YAML file is valid, as well as version control and audit trails.
- A CI/CD pipeline might consist of four stages, executed in the following
order:
- A
build stage
, with a job called compile. (cisco calls it "change stage
") - A
test stage
, with two jobs called test1 and test2. - A
staging stage
, with a job called deploy-to-stage. - A
production stage
, with a job called deploy-to-prod.
- A
A GitLab pipeline consists of:
Commit
: A change in the codeJob
: Runner instructionsPipeline
: A group of jobs divided into different stagesRunner
: A server or agent that implements each job separately, which can spin up or down if neededStages
: Parts of a job (for example, build or tests). Multiple jobs inside the same stage are executed in parallel.
3 CICD Phases
- Build phase
- application is compiled
- Test phase
- code is tested
- Release phase
- app delivered to the repository
- Depoly phase
- code deployed to production
This is similar, but does NOT match Cisco's CICD Phases:
- Commit
- Build ( with source code plus dependencies )
- Test
- Deploy
3.1 CICD Phases (8 phase version)
- Plan Scrum + Agile for frequent microincremental changes
- Code Using and IDE
- Build rapidly and incrementally mergee code commits
- Test automated verification of enhancements, often with test-driven deployment practices. Often with pytest, unittest or other test frameworks
- Release repository commits and documrentation
- Deploy Update the code base
- Operate in production, with monitoring and orchestartion
- Monitor & Optimize data collection, analysis, feedback with machine learning that closes the loop back to plan.
3.2 Cisco DevNet CICD view (old course)
3.2.1 Coding phase
Developers write code
3.2.2 Software deliver pipeline (Before CICD)
- Integration: developers submit all their code and this is integrated Compile / build followed by execute tests.
- Delivery Phase On tested and working code, create final artifacts, then make available for usage.
- Deployment Deployment engineers install the code and configure it for use.
3.2.3 Software delivery pipeline with CICD
4 NetDevOps Configuration Pipeline
"Network as Code"`
- Network config stored in git (or other vcs, source control)
- changes are proposed in code branches
- CICD build servers deplooy and test proposed configurations (with pyATS plus others)
- Successfull configs automatically pushed to "productions"
4.1 Tools Needed.
5 NetDevOps CICD in practice
Some common tools that help with CI/CD are
Jenkins
handles both CI and CD sideTekton Pipeline
CI/CD Framework for Kubernetes that provides a standard cloud-native CI/CD experience with containersSpinna ker
For CD in multi-cloudGoCD
A CI/CD server focused on modeling and visualizationConcourse
open-sourced continuous "thing-doer"Screwdriver
handles CD