Databases for Network Engineers

Understanding Databases is a must for network engineers nowadays.
We should understand databases, not only relational databases such as Mysql but also nonrelational databases (aka NOSQL), which are commonly used with Big Data.

For example we are living now in the era of Streaming Telemetry and Model Driven Telemetry (MDT).
Streaming telemetry is a new approach for network monitoring in which data is streamed from devices continuously with efficient, incremental updates.

Your Network devices such as a switch are the publisher, Your machine (where the application reside) that getting the data is the subscriber.
One of the major benefits of model-driven telemetry is that you can define the frequency and amount of data that the network device will stream back to the collector or application.

Telemetry subscription is a subscription that is used to define the set of data that is requested as part of the telemetry data.

The telemetry subscription allows you to choose the subset of the data for which you want to receive information.
There are two types of subscriptions that are used in telemetry on Cisco IOS XE Software systems:
• Dynamic (also known as dial-in) using YANG and Netconf
• Configured subscriptions (also known as dial-out) using YANG and gRPC

Switches, Routers support Streaming telemetry and can be configured to send data for instance, when CDP or BGP neighbors changed (on-change Telemetry publications); we can also get data about our switch CPU status in the last few minutes and keep getting these data every few minutes (periodic Telemetry publications)

All these data will be sent to your machine and stored in nonrelational databases from the type (Time-series database) such as InfluxDB

Another example, Cisco SD-WAN (Viptela) , all Statistics saved in your NMS (vManage) in nonrelational databases from the type (Document-based database) such as Elasticserach.
While vManage Configuration database saved in ( Graph-based database) such as Neo4j

You should know how to query these databases and how to visualize data in it using visualization tools such as Grafana or Kibana

Links:
https://developer.cisco.com/docs/ios-xe/#!streaming-telemetry-quick-start-guide
https://blogs.cisco.com/developer/getting-started-with-model-driven-telemetry
https://blogs.cisco.com/developer/model-driven-telemetry-sandbox

To download any of these databases in VM for free and practice with:
https://bitnami.com/stacks/virtual-machine

Common used nonrelational databases (aka Unstructured-Data DB)

• Key-value database: A travel blog on a website, which uses a key-value database. Each value (in this case, a blog post) is stored under a different key, and the keys represent URIs on which the blog is available.

• Document-based database: Suitable for a successful startup company that uses an application with fast-changing specifications. Data is saved as semistructured documents, facilitating change management.

• Column-based database: For business analysis, where huge amounts of data need to be processed, a column-based database is perfect because the data is stored in columns, instead of rows. Most of the operations are performed on only one column, so performance is much greater than with row-based databases.

• Time-series database: With the rise of smart cars and the Internet of Things (IoT), gigabytes of telemetry data are being generated each day. The data is sent back to a vendor for analysis and stored in a simple time-series database.

• Graph-based database: Social media has become very popular in the past few years. Due to a huge number of complex relations between entities, the social media application uses a graph-based database, where data is defined and traversed via nodes (entities) and edges (relations).

Soft Code Your Secrets

Hard coding your secrets in python or any programming language is something you should avoid.

What is your secrets ?

User credentials , API Keys , API Tocken or Cookies etc

You should always Soft coding it and there are many ways to do that , let me show you a few of them in the following script I created.

In this script you will see three options for soft coding your secrets.

Option 1 With help of using “Input” built-in function and  “getpass” python library we can allow the user to interactively type his credential.

“getpass” python library reads the input from the user as Password and not showing while not showing what characters he is typing.

Code:

import getpass

username=str(input(‘Type Your Username:\n’))

password=getpass.getpass()

print(“User Credential is” + ” ” + “Username is” + ” ” + username + ” ” +”Password is” +” ” + password)

Option 2 you can save your username and password as Environment variables in windows then we call it using “os” python library.

Code:

import os

input(“Press Enter to continue…”)

print(“*” * 20  + ” “+  “Method 2″ +” ” +”*” * 20)

username2 = str(os.environ.get(‘USER’))

password2 = str(os.environ.get(‘Password’))

print(“User Credential is” + ” ” + “Username is” + ” ” + username2 + ” ” +”Password is” +” ” + password2)

Option 3 you can save your username and password in a file then  call the  credential  using ” Python-dotenv ” python third party library.

Python-dotenv reads key-value pairs from a .env file and can set them as environment variables. It helps in the development of applications following the 12-factor principles.

https://pypi.org/project/python-dotenv/

You will need to install first using pip install Python-dotenv command

Code :

from dotenv import dotenv_values

input(“Press Enter to continue…”)

print(“*” * 20  + ” “+  “Method 3″ +” ” +”*” * 20)

user_Credential_from_envfile = dotenv_values(“.env”)

print(user_Credential_from_envfile)

Script code avaiable here in my github account:

https://github.com/YasserAuda/Hard-Code

output of this script:

What is Cisco SD-Branch?

What is SD-Branch?

Simply means you can create and build a branch with few steps from a centralized location, where you can build your branch with routers, switches, WLCs , Firewall, Win/Linux Servers, and WAN Edge routers.

How?

Actually, all these branch gears and servers I mentioned above will be Virtual machines (VMs) we call it in this solution Virtual Network Functions (VNFs) using Cisco Enterprise Network Functions Virtualization (NFV)  

Which Hypervisor will be used?

Cisco had its own answer called Enterprise NFV Infrastructure Software (NFVIS)

https://www.cisco.com/c/en_in/products/routers/enterprise-nfv-infrastructure-software/index.html

But what if I would like to have Polo Alto Firewall as VM, can I?

Yes for sure, Cisco support many Thrid party VNFs , check below list that going to grow over time

https://www.cisco.com/c/en/us/solutions/collateral/enterprise-networks/enterprise-network-functions-virtualization-nfv/nfv-open-ecosystem-qualified-vnf-vendors.html

Where this NFVIS software and VNFs will exist, it will need hardware for sure ?

yes we have multiple solutions to host it but the common way is hosting in an appliance called Enterprise Network Compute System (ENCS)

Add alt textNo alt text provided for this image

Ok NFVIS is software in ENCS hardware that helps me to create VMs (aka NFV) to create my Branch devices, but how I will access it to do that?

You can access it using CLI/GUI/REST API or You can use Controllers and Orchestrators such as

Cisco NSO

Cisco vManage

Cisco DNA-C

Cisco MSX

Imagine how pretty it is to use vManage to create WAN Edge VM and other required VMs remotely for your Branch

In May 2021 there is a new 2 days course for that, it is

Implementing Cisco Enterprise SD-Branch (ENSDBI)

Cisco SD-WAN Automation

Someone was asking me about Cisco SD-WAN Automation.

Mainly you use REST API to communicate with vManage.

To learn more about the Viptela API Library and Documentation, consult the product documentation:

https://sdwan-docs.cisco.com/Product_Documentation/Command_Reference/Command_Reference/vManage_REST_APIs

They are also available by accessing the documentation through https://{{vmanage}}:{{port}}/apidocs.

You can even try it for free:

using https://devasc-sdwan-1.cisco.com/apidocs/

Log in using username devnetuser and password RE!_Yw519_27

More in networkacademy.io blog: https://www.networkacademy.io/ccie-enterprise/sdwan/cisco-sd-wan-rest-apis

The Viptela REST API calls expose the functionality of Viptela software and hardware features and of the normal operations you perform to maintain Viptela devices and the overlay network itself.

In REST API terminology, each of these features or operations is called a resource.

A resource is an object with a type, associated data, relationships to other resources, and a set of methods that operate on it.

Resources are grouped into collections.

Each collection contains a single type of resource, and so is homogeneous.

In the Viptela REST API, the collection of resources is present at the top level of the API.

The Viptela REST API resources are grouped into the following collections:

Monitoring: This collection views status, statistics, and other information about operational devices in the overlay network. Viptela devices collect monitoring information about themselves every 10 minutes. After collecting these statistics, each Viptela device places them in a zip file. The vManage server retrieves these zip files every 10 minutes or, if the vManage server cannot log in to the device, it retrieves them whenever it is next able to log in.

Real-Time Monitoring: This collection retrieves, views, and manages real-time statistics and traffic information. Real-time monitoring information is gathered in real time, approximately once per second.

Configuration: This collection creates feature and device configuration templates, retrieves the configurations in existing templates, and creates and configures vManage clusters.

Administration: This collection manages users and user groups, views audit logs, and manages the local vManage server.

Device Inventory: This collection collects device inventory information including serial numbers and system status.

Certificate Management: This collection manages certificates and security keys.

Troubleshooting Tools: This collection provides tools to help troubleshoot devices, determine the effect of policy, update software, and retrieve software version information.


Here is example I created
I used DevNet always on sandbox and I used this script which is part of it in Cisco CLN ENAUI materials.
The script will ask vManage for devices that exist in your Viptela org.
Try it by yourself. I am open to answer any question about the content of this script, But you need to ask it in the below CLN link.
https://learningnetwork.cisco.com/s/question/0D53i000010uW2YCAU/cisco-sdwan-python-script-example

Script:


Script Output:

Script can be downloaded from my github :
https://github.com/YasserAuda/SD-WAN

SD-WAN Automation can be done using REST API tools such as python requests module or Ansible uri module

There is another python module can be used called Sastre 

Sastre python module tutorial and how to install:

https://github.com/CiscoDevNet/sastre

Sastre is available in two flavors:

Sastre: Public open-source under MIT license available on Cisco DevNet repository. Supports a limited set of tasks.

Sastre-Pro: Cisco licensed version, supporting the full feature-set. Sastre-Pro is available for customers with a CX BCS subscription and Cisco internal at Cisco eStore.

Build Your Lab for DevNet DEVCOR Study

I am using kali as my Linux Distro , Also I recommend to use Ubuntu or Debian if you do not like Kali.
https://www.offensive-security.com/kali-linux-vm-vmware-virtualbox-image-download/

My Hypervisor-type 2 is VMware® Workstation 15 Pro.

How to install Install Visual Studio Code on Kali Linux 2020

sudo apt update
sudo apt install curl gpg software-properties-common apt-transport-https
curl -sSL https://packages.microsoft.com/keys/microsoft.asc | sudo apt-key add
echo “deb [arch=amd64] https://packages.microsoft.com/repos/vscode stable main” | sudo tee /etc/apt/sources.list.d/vscode.list
sudo apt update
sudo apt install code

How to Install docker
sudo apt update
sudo apt install -y docker.io
sudo systemctl enable docker –now

How to Install ansible
sudo apt install ansible

How to Install puppet
sudo apt-get install puppet

How to Install git
sudo apt install git

Python 3 already installed in kali 2020.x to run it type python3
How to Install pip for python3

sudo apt-get install python3-pip

How to Install geany (Sometimes I used instead of VSC)
sudo apt-get install geany

VMS you might need like gitlab and MongoDB ,Neo4j ,PostgreSQL ,MySQL ,ELK ,NGINX Open Source:
https://bitnami.com/stacks/virtual-machine

You will need also GNS3 to practice with ansible

https://www.gns3.com/software/download

Python libraries you will work with during the course:
flask
socket
MySQLdb
os
sys
argparse
subprocess
json
unittest
requests
urllib3
netmiko
meraki
pprint
time
datetime
yaml
logging
fdm
ucsmsdk

Devnet Sandboxes to practice with webex teams , Intent API , Meraki , etc:
https://developer.cisco.com/site/sandbox/

Play with docker online:
https://labs.play-with-docker.com/

UCS Platform Emulator UCSPE 4.1
You will need UCS Manager so you can communicate with it using APIs

https://community.cisco.com/t5/unified-computing-system/ucs-platform-emulator-downloads-ucspe-4-1-2cpe1-ucspe-4-0-4epe1/ta-p/3648177

Terraform basic files structure is required , you can use the following links:

https://collabnix.github.io/terraform/

https://mikhail.io/lab/terraform/

Webex Teams and Meraki and how to ChatOps between them

You can use DevNet Sandboxs but first create account in webex Teams

Kindly note that

kubernetes and kubectl command are also required from you

For practice you can use https://www.katacoda.com/

Graylog from https://www.graylog.org/

HAProxy from http://www.haproxy.org/

Good Luck

Yasser Ramzy Auda