https://www.linkedin.com/pulse/essential-devops-skills-thomas-theakanath
An experienced DevOps engineers would be able to describe most of the technologies that I would describe in the following sections. This is a comprehensive list of DevOps skills for comparing one’s expertize and a reference template for acquiring new skills.
Knowledge of Infrastructure
A DevOps engineers should have good understanding of both classic (datacenter based) and cloud infrastructure components, even if the team would have dedicated infrastructure engineers.
Classic infrastructure
- How real hardware - servers and storage devices - are racked and networked, and, accessed from both corporate network and Internet.
- The provisioning of shared storage to be used across multiple servers and the methods available for that.
- Infra and methods for load balancing.
Virtualization Basics
- Hypervisors
- Virtual machines
- Object storage
- Running virtual machines on PC and Mac - Vagrant, VMWare etc
Cloud infrastructure
- Core cloud computing and storage components as they are implemented in one of the popular virtualization technologies, VMWare or OpenStack.
- The idea of elastic infrastructure and options available to implement it.
Networking Basics
- Network layers
- Routers, domain controllers etc
- Networks and subnets
- IP address
- VPN
- DNS
- Firewall
- IP tables
- Network access between applications; ACL
- Networking in cloud, Amazon AWS as an example.
Load Balancing
- Load balancing infra and methods
- Geographical load balancing.
- Understanding of CDN.
- Load balancing in cloud
DevOps Toolchain
An important category of skills a DevOps engineer should have is the experience of using specialized tools for implementing various DevOps processes. While Jenkins, Ansible etc are known to most everyone, other tools might be obscure, or not very obvious such as the importance of knowing one major monitoring tool in-and-out. Some tools like source code control system are shared with development teams.
The list here has only examples of basic tools. An experience DevOps engineer would have used some application or tool from all or most of these categories.
Source-Code Management (SCM) System
- Expert level knowledge of an SCM system such as Git or Subversion
- Code branching best practices, such as Git-Flow
- Importance of checking in Ops code to SCM system
- Using GitHub
Bug Management System
- The engineer should have used a major bug management system such as Bugzilla or Jira
- Workflow related to the bug filing and resolution process
- Experience integrating SCM system with bug resolution process, using triggers or REST APIs.
Collaborative Documentation System
- Wiki basics
- Experience using MediaWiki, Confluence etc
- Why DevOps projects have to be documented?
- How documents were organized on a wiki based system for the past projects?
Build and CI
- Build on Jenkins
- Jenkins as Continuous Integration (CI) platform
- CI Platform features: Integration with SCM system, Secret Management, ssh based Access Management, Scheduling and chaining of build jobs, Source-code change based triggers, Worker/Slave nodes, REST API support and Notification management.
Artifacts Management
- What are artifacts and why they have to be managed?
- Experience using a standard artifacts management system such as Artifactory.
- Caching third-party tools and dependencies in-house.
Configuration Management
- Configuration Management explained
- Experience using any Configuration Management Database (CMDB) system.
- Experience using open-source tools such as Cobbler for inventory management
- Agent-less and agent-driven enforcement of configuration
- Ansible, Puppet, Chef, Cobbler etc as examples
Orchestration and Deployment
- The workflow of released code getting into production.
- Pushing code to production, use of ssh based tools such as Ansible.
- On-demand or Continuous Delivery (CD) of code from Jenkins.
- Agent-driven code pull to update production environment.
- Deployment strategies: With or without impact on software service.
- Code deployment in cloud: Using auto-scaling groups, machine images etc.
Monitoring
- Monitoring categories: system, platform, application, business, last-mile, log management and meta-monitoring.
- Status based monitoring, as in Nagios
- Data-driven monitoring, as in Zabbix
- Last-mile monitoring, as done by Pingdom or Catchpoint
- Log management, as done by ELK
- Monitoring SaaS solutions, Datadog and Loggly as examples.
System Tools and Methods
To get an automation project up and running, a DevOps engineer would build new things such as configuration objects in an application and code snippets or full-blown programs. However, a major part of the work would be to glue many things together at the system level on the given infrastructure. Such efforts are not different from traditional system integration work, and, in my opinion, the ingenuity of an engineer at this level determines his/her real value on the team. It is easy to find cookbooks, recipes and best practices for vendor-supported tools, but, it would take experience working on diverse projects to gain the necessary skills-set to implement robust integrations that have to work reliably in production.
The important system level tools and techniques are listed here.
Access Management
- Users and groups on Linux
- Use of service accounts for automation
- Sudo command, /etc/sudoers file, Password-less access
- Using LDAP and AD for access management
- Remote access using ssh
- SSH keys and related topics
- scp, sftp and related tools
- SSH key formats
- Managing access using configuration management tools
Password Management
- Use of gpg for password encryption
- Tools for password management such as KeePass
- md5 for for encryption
- Remote access with authentication from automation scripts
- Managing API keys
- Jenkins plugins for password management
Build
- Basics of compilers such as gcc and javac
- make and Makefile, Ant, Maven, Gradle etc
- Code libraries in C++, Java, PHP, Perl, Ruby, Python etc.
- Build artifacts such as .so files, jar and war
- Running builds from Jenkins
Packaging
- Packaging files: zip, tar, gzip etc
- Packaging for deployment: rpm, debian, dnf, zypper etc.
- Packaging for cloud: AWS AMI,VMWare template. Use of Packer.
- Docker - Container for microservices
Artifacts Management
- Use of artifacts repository: distribution and release of builds; meeting build and deployment dependencies
- Serving artifacts from shared storage volume
- Mounting locations from cloud storage services such AWS S3
- Artifactory as artifacts server
File Transfer
- scp, rsync, ftp and their SSL counterparts
- Via shared storage
- File transfer with cloud storage services such as AWS S3
Deployment
- Code push using system level file transfer tools
- Scripting using ssh libraries such as paramiko
- Orchestrate code push using configuration management tools
Job Management
- Use of crontab
- Running jobs in the background; use of nohup
- Use of screen to launch long running jobs
- Jenkins as process manager
Files and Storage
- Typical uses of find, df, du etc
Linux Distributions
- A comparison of popular distributions
- Checking OS release and system info
- Package management differences
Text Processing
- Typical uses of sed, awk, grep, tr
- Scripting using Perl
- Regular expressions
- Support of regular expressions in Perl and Python
Troubleshooting Toolkit
Sample usages and steps to install these tools:
- nc
- netstat
- traceroute
- vmstat
- lsof
- top
- nslookup
- ping
- tcpdump
- dig
- sar
- uptime
- ifconfig
- route
Programming Primer for DevOps
One of the attributes that helps to differentiate a DevOps engineer from other members in the operations team, like sysadmins, DBAs, and operations support staff, is his/her ability to write code. The coding/scripting skill is just one of the tools in the DevOps toolbox, a powerful one however, that a DevOps engineer would maintain as part of practising his/her trade.
Coding is the last resort when things cannot be integrated by configuring and tweaking the applications and tools that are used in an automation project.
Scripting
Bash scripting essentials: Many times a few lines of bash script could be the best glue code integrating two components in the whole software system. DevOps engineer should have basic shell scripting skills and bash is the most popular right now.
Python: If a script would have to deal with external systems and components, or, it would be more than just a few lines of command-lines and dealing with fairly complex logic, it might be better to write that script in an advanced scripting language like Python, Perl or Ruby.
Knowledge of Python would make your life easier when dealing with DevOps applications such as Ansible, which uses Python syntax to define data structures and implement conditionals for defining configurations.
Web Programming
One of the categories of projects a DevOps engineer would end up doing is building dashboards. Though dashboarding features are found with most of the DevOps tools, those are specific to the application, and, there will be a time when you may require to have a general purpose dashboard with more dynamic content content than just static links and text.
Another requirement is to build web UI for provisioning tools to present those as self-service tools to user groups.
In both these cases, deep web programming skills are not required. Knowledge of a web programming friendly language such as PHP and a Javascript/CSS/HTML library like Composer would be enough to get things started. It is also important for the DevOps engineer to know the full-stack, in this case LAMP, for building and running the web apps.
Configuration Languages
Almost every application and tool that is used for building, deploying and maintaining software systems uses configuration files. While manual reading of these files might not require any expertize, a DevOps engineer should know how config files in such formats are created and parsed programmatically.
A DevOps engineer should have good understanding of these formats:
- INI
- XML
- JSON
- YAML
The engineer should also know how these formats are parsed in his/her favorite scripting language.
REST API
The wide acceptance of REST API as a standard to expose features that other applications can use for system integration, made it a feature requirement for any application for it to be taken seriously. The knowledge of using REST API has become an important skill for DevOps engineer.
- HTTP/HTTPS: REST APIs are based on HTTP/HTTPS protocol and a solid understanding of its working is required. Knowledge of HTTP headers, status codes, and main verbs GET, POST and PUT.
- REST API basics: Normal layout of APIs defined for an application.
- curl and wget: Command line tools to access REST API and HTTP URLs. Some knowledge of the support available for HTTP protocol in scripting languages will be useful and that would be an indication of working with REST APIs.
- Authentication methods: Cookie based and OAuth authentication; API keys; Use of If-Match and If-None-Match set of HTTP headers for updates.
- API management tools: If the application you support provides an API for the users, most probably, its usage will be managed by some API Gateway tool. Though not an essential skill, experience in this area would be good if one works on the API provider side.
Programming with Data Repositories
There was a time when mere knowledge of programming with RDBMS was enough for an application developer and system integrator to manage application data. Now with the wide adoption of Big Data platform like Hadoop and NOSQL systems, to process and store data, a DevOps engineer would need varied requirements, from one project to another. Core skills are the following:
- RDBMS: MySQL, Oracle, SQL Server, Postgres etc are still used and knowledge of one or more is important.
- Setting up & configuring MySQL: As an open-source database used with many other tools in the DevOps toolchain, I consider this a basic requirement for a DevOps engineer. If one hasn’t done this, he/she might not have done enough yet.
- ANSI SQL: Basic DML and aggregation functions.
- ODBC: Tools depend on this to access a variety of data repositories and therefore configuration of ODBC on Linux is needed sometimes.
- Running queries from bash script: How to run a database query via a database client from bash script and use the output. The MySQL client, mysql, is a good example.
- Database access from Perl/PHP/Python: All the major scripting languages provide modules to access databases and that can be used to write robust automation scripts. Examples are Perl DBI and Python’s MySQLdb module.
- NOSQL: There is a plethora of NOSQL systems, but knowledge of popular applicatios among those, Cassandra and MongoDB, would be better.The skills should be similar to those discussed above for RDBMS.
- Hadoop: The default platform for Big Data processing and storage. Basic knowledge of Hadoop suite of tools - HDFS, HBase, Hive, Pig etc - would be useful.
Programming for Cloud
Those who had built cloud infrastructure with focus on automation and versioning would know some of these (or similar) tools:
- cloud-init: cloud-init can be used to configure a virtual machine when it is spun up. This is very useful when a node is spun up from a machine image with baseline or even application software already baked in.
- AWS CLI: If the application runs on Amazon AWS, knowledge of AWS CLI is needed which would be handy to put-together simple automation scripts.
- Terraform: HashiCorp’s Terraform is an important tool if the focus would be to provision infrastructure as code (IaaS). Using this, infrastructure can be configured independent of the target cloud or virtualization platform.
- Packer: It can be used to build machine images for a variety of virtualization technologies and cloud platforms like AWS, and it is useful if the infrastructure is provisioned in a mixed or hybrid cloud environment.
Error Handling
In a rush to get things rolled out, one of the things left half-done is adding enough error handling in scripts. Automation scripts that are not robust can cause major production issues which could impact the credibility of DevOps efforts itself. A DevOps engineer should be well aware of the best practices in error handling and logging.
- Importance of error handling in automated scripts
- Error handling in bash
- Error handling in Python
- Logging error in application and system logs
Jumping the High Bar
Looking at these requirements for a DevOps engineer, it is pretty clear that he/she should have variety of skills to succeed at work. And that throws up a hiring problem also, in finding skilled engineers. In think, a template like this should be used only for assessing the current experience of a prospective hire. The needed skills can be picked up on the jobs that demand deep knowledge in certain areas. Therefore, the focus should be to hire smart engineers who have a track record of picking up new skills, rolling out innovative projects at work, and, contributing to reputed open-source projects