My cheat on ci/cd pipeline

Home

1 CI/CD Definition

Continuous integration is a coding philosophy and set of practices that drive development teams to implement small changes and check in code to version control repositories frequently. Because most modern applications require developing code in different platforms and tools, the team needs a mechanism to integrate and validate its changes.

It automates your software delivery process, or automating the stages of app development.

ci-cd.png

Figure 1: CI/CD

2 Overview

Continuous integration (CI) and continuous delivery (CD) embody a culture, set of operating principles, and collection of practices that enable application development teams to deliver code changes more frequently and reliably. The implementation is also known as the CI/CD pipeline.

The technical goal of CI is to establish a consistent and automated way to build, package, and test applications. With consistency in the integration process in place, teams are more likely to commit code changes more frequently, which leads to better collaboration and software quality.

Continuous delivery picks up where continuous integration ends. CD automates the delivery of applications to selected infrastructure environments. Most teams work with multiple environments other than the production, such as development and testing environments, and CD ensures there is an automated way to push code changes to them.

Continuous integration and continuous delivery require continuous testing because the objective is to deliver quality applications and code to users. Continuous testing is often implemented as a set of automated regression, performance, and other tests that are executed in the CI/CD pipeline.

2.1 CI/CD Piplelines

Pipelines are the top-level component of continuous integration, delivery, and deployment. Pipelines (in Gitlab? or in general?) comprise:

  • Jobs, which define what to do. For example, jobs that compile or test code.
  • Stages, which define when to run the jobs. For example, stages that run tests after stages that compile the code.

Jobs are executed by runners.

Multiple jobs in the same stage are executed in parallel, if there are enough concurrent runners.

  • If all jobs in a stage succeed, the pipeline moves on to the next stage.
  • If any job in a stage fails, the next stage is not (usually) executed and the pipeline ends early.
  • In general, pipelines are executed automatically requiring no intervention once created. However, there are also times when you can manually interact with a pipeline.
  • A typical pipeline might consist of four stages, executed in the following order:
    • A build stage, with a job called compile. (cisco calls it "change stage"
    • A test stage, with two jobs called test1 and test2.
    • A staging stage, with a job called deploy-to-stage.
    • A production stage, with a job called deploy-to-prod.
    • That's it.

cicd-process.png

Figure 2: CI/CD Process

From cisco learning on CI-CD pipelines This is a common-pipeline:

You commit a change for example.yml to a code repository.
The repository executes syntax and sanity checks, code review rules, and:

- Accepts your commit and notifies the CI/CD server to run tests.
- Rejects your commit based on failed checks.

CI/CD prepares an environment and runs predefined tests for any Ansible playbook:

- pip install ansible==2.8.1
- ansible-version
- ansible-playbook example.yml --syntax-check
- ansible-playbook -i staging_inventory.cfg -check
- ansible-playbook -i staging_inventory.cfg -vvvv

You are notified that you can propagate your changes to the production environment.

2.2 Cisco's CI/CD Pipeline

Consists of:

  • source code repository
  • build stage (source code, dependencies, and static code analytics)
  • test stage (unit and integration tests. ([end-to-end tests])
  • deploy stage (packaging, staging, and production)

The pipeline can be triggered automatically with a commit, or manually by a DevOps engineer.

cicd-example.png

Figure 3: CI/CD Example using Anisbl3e

A runnable instance is built during the build stage from the source code and the needed dependencies

cisco-cicd-phases.png

Figure 4: Cisco's CICD Phases 1 & 2

Followed by:

cisco-cicd-phases3.png

Figure 5: Cisco's CICD Phases 3 & 4

2.3 Configuring CICD

From:

Pipelines and their component jobs and stages are defined in the CI/CD pipeline configuration file for each project.

  • Jobs are the basic configuration component.
  • Stages are defined by using the stages keyword.

For a list of configuration options in the CI pipeline file, see the GitLab CI/CD Pipeline Configuration Reference.

You can also configure specific aspects of your pipelines through the GitLab UI. For example:

  • Pipeline settings for each project.
  • Pipeline schedules.
  • Custom CI/CD variables.

2.4 Tools integration

There are a lot of tools that can help you implement a CI/CD pipeline. Tools based on Git (GitLab, GitHub, and Bitbucket) all provide you with a code repository as well as with the CI/CD pipeline, while tools like Jenkins or CircleCI do not provide software repositories. Tools such as Jenkins, CircleCI, rely on code repositories, hosted somewhere else, to trigger the pipeline with the help of /APIs.

2.4.1 Jenkins

One of the two most common CI/CD pipeline tools. The other is Gitlab.

jenkins-pipeline.png

Figure 6: Jenkins Pipeline with Python Unit Test done in the Test stage

From cisco training: "Jenkins uses a file called Jenkinsfile for the configuration. Jenkinsfiles are written in a programming language called Apache Groovy. Groovy takes the syntax of Java and combines the features of Python (and other scripting languages), such as defining a variable without specifying a type.

The main difference of Jenkins from GitLab is that the extensions of native functionalities are done with the help of plug-ins, which usually translates to expensive maintenance. GitLab on the other hand is open-core, which means that any changes done to the GitLab codebase are automatically tested and maintained."

2.4.2 Gitlab

GitLab also provides a pipeline as code, which is an upgrade from the GUI type of pipeline. This approach gives DevOps teams more freedom in the pipeline process. It also makes rollbacks much easier, and it provides a built-in lint tool that makes sure the YAML file is valid, as well as version control and audit trails.

  • A CI/CD pipeline might consist of four stages, executed in the following order:
    • A build stage, with a job called compile. (cisco calls it "change stage")
    • A test stage, with two jobs called test1 and test2.
    • A staging stage, with a job called deploy-to-stage.
    • A production stage, with a job called deploy-to-prod.

A GitLab pipeline consists of:

  • Commit: A change in the code
  • Job: Runner instructions
  • Pipeline: A group of jobs divided into different stages
  • Runner: A server or agent that implements each job separately, which can spin up or down if needed
  • Stages: Parts of a job (for example, build or tests). Multiple jobs inside the same stage are executed in parallel.

3 CICD Phases

  1. Build phase
    • application is compiled
  2. Test phase
    • code is tested
  3. Release phase
    • app delivered to the repository
  4. Depoly phase
    • code deployed to production

This is similar, but does NOT match Cisco's CICD Phases:

  1. Commit
  2. Build ( with source code plus dependencies )
  3. Test
  4. Deploy

cisco-cicd-phases.png

Figure 7: Cisco's CICD Phases

3.1 CICD Phases (8 phase version)

  • Plan Scrum + Agile for frequent microincremental changes
  • Code Using and IDE
  • Build rapidly and incrementally mergee code commits
  • Test automated verification of enhancements, often with test-driven deployment practices. Often with pytest, unittest or other test frameworks
  • Release repository commits and documrentation
  • Deploy Update the code base
  • Operate in production, with monitoring and orchestartion
  • Monitor & Optimize data collection, analysis, feedback with machine learning that closes the loop back to plan.

3.2 Cisco DevNet CICD view (old course)

3.2.1 Coding phase

Developers write code

3.2.2 Software deliver pipeline (Before CICD)

  1. Integration: developers submit all their code and this is integrated Compile / build followed by execute tests.

devnet-cicd.png

Figure 8: Devnet CICD Process Integration
  1. Delivery Phase On tested and working code, create final artifacts, then make available for usage.

devnet-cicd-delivery.png

Figure 9: Devnet CICD Process Delivery
  1. Deployment Deployment engineers install the code and configure it for use.

devnet-cicd-deployment.png

Figure 10: Devnet CICD Process Deployment

3.2.3 Software delivery pipeline with CICD

devnet-cicd-flow.png

Figure 11: Devnet CICD Flow

4 NetDevOps Configuration Pipeline

"Network as Code"`

  • Network config stored in git (or other vcs, source control)
  • changes are proposed in code branches
  • CICD build servers deplooy and test proposed configurations (with pyATS plus others)
  • Successfull configs automatically pushed to "productions"

4.1 Tools Needed.

5 NetDevOps CICD in practice

Some common tools that help with CI/CD are

  • Jenkins handles both CI and CD side
  • Tekton Pipeline CI/CD Framework for Kubernetes that provides a standard cloud-native CI/CD experience with containers
  • Spinna ker For CD in multi-cloud
  • GoCD A CI/CD server focused on modeling and visualization
  • Concourse open-sourced continuous "thing-doer"
  • Screwdriver handles CD

5.1 Home