HTML Technical Cheat Sheet

• encoded in ASCII
• in HTTP/1.1 messages were openly sent across a tcp connection.
• in HTTP/2 the once human readable messages are broken up into HTTP frames

Composed of three elements:

• GET zintis.net HTTP/1.1

Optional a case-insensitive string followed by a colon (':') and a value whose structure depends upon the header. The whole header, including the value, consist of one single line, which can be quite long.

For example:

• content-type: application/json
• Authorization: Basic BDz6bmV0uXqlcjpDaXkjTzEQMyE=
• Accept: application/json # this is the MIME type you will accept
• Accept-Language: en-us, en-GB;q-0.9, lv-eu-0.9, fr-eu-0.2
• User-Agent: Mozilla/4.0
• Connection: Keep-Alive

The q=0.9 is the quality factor rating. between 0 and 1, 1 being most preffered.

Always seperates the headers (one line) and the body (one or many lines)

Final part is the body. Usually get, post, head, delete requests do NOT have a body, because they are just fetching resources. But if a body is there it will have either Content-type or Content-length matching file. or a multi part resource body such as HTML Forms, where each part contains different information.

The status line of an HTTP response, called the status line, contains the following three elements:

• see section above.

Always seperates the headers (one line) and the body (one or many lines)

• same as in HTTP requests, but typically most of what the user is asking for is returned in the body of an HTTP response.

Typically of three categories

1. Single-resource bodies consisting of a single file of known length, defined by the two headers: Content-Type and Content-Length
2. Single-resource bodies consisting of a single file of unknown length, encoded by chunks with Transfer-Encoding set to chunked
3. Multiple-resource bodies consisting of a multipart body, each containing a different section of information, but these are rare.

identifies the resrouce ./org-graphics/url.png~

tells you where to locate or find it zintis.net

uses a URN scheme: urn:people:names:alice.

curl https://deckofcardsapi.com/api/deck/dec40fca2d5/draw/?count=3 | python -m json.tool The uri is the usual scheme://host/path/?query The full uri is:

           userinfo          host     port
       ┌───────┴───────┐ ┌────┴─────┐ ┌┴┐
http://john.doe:password@www.acme.com:123/forum/questions/?tag=networking&order=newest#top
└─┬─┘ └───────────┬─────────────────────┘└─┬─────────────┘└────────┬──────────────────┘└┬─┘
scheme         authority                      path                  query             fragment

curl uses:

• -X followed by a request verb, GET, PUT, POST, PATCH, or DELETE or
• -H followed by a header such as a token to send with the requests.

  Device:~# curl -i -k -X "POST" "https://10.85.116.30:443/restconf/data/Cisco-IOS-XE-native:native/logging/monitor" \
  >      -H 'Content-Type: application/yang-data+json' \
  >      -H 'Accept: application/yang-data+json' \
  >      -u 'admin:admin' \
  >      -d $'{
  >   "severity": "alerts"
  > }'
  
  

In negotiating between acceptable presentation formats, the clients can give a preference for one format over another. This is done through the q-factor ranging from 0 (unacceptable) to 1 (most preferred). 1 is default.

The q-factor is included (optionally) in the Accept- headers with a

• semicolon
• q=0.5

For example Accept-Language: en-US;q=0.8, en-GB;q=1, en-IN;q=0.8, *;q=0.7

From Cisco Devnet: A media type represents a general category and the subtype, which identifies the exact kind of data. A general type can be either discrete (representing a single resource) or multipart, where a resource is broken into pieces, often using several different media types (for example, multipart/form-data).

These are the common Mime-Types you may see in REST APIs:

• Application -
• Audio
• Image
• Text
• Video

• UTF-8 or ISO 8859-1 Sets the preferred character sets, such as UTF-8 or ISO 8859-1. It is important when displaying resources in languages that include special characters.

Requests a previous version of the resource, denoted by the point in time with datetime. The value must always be older than the current datetime.

Sets the preferred encoding type for the content.

The preferred natural language. Useful for various localizations.

All these headers support quality-factor weighting.

It allows the user or user agent to indicate the relative degree of preference for that media range, using the q-value scale from 0 to 1. The default value is q=1.

A request header that prefers U.S. English over British English but still prefers British English over Indian English would look like this, with 1 being the default, so if it is omitted, it is 1: These are identical

• Accept-Language: en-US, en-GB;q=0.9, en-IN;q=0.8, *;q=0.7
• Accept-Language: en-US;=1.0, en-GB;q=0.9, en-IN;q=0.8, *;q=0.7

From Cisco Devnet: These two requests and their corresponding response are for the same data but use different content types.

Request Header:

• GET http://www.example.com/api/menu Accept: application/json Accept-Language: en-GB

Response:

• { "menu": { "name": "crisps" } }

Request Header:

• GET http://www.example.com/api/menu Accept: application/xml Accept-Language: en-US

Response:

• <?xml version="1.0" encoding="UTF-8"?> <menu> <name>chips</name> </menu>

(i.e. the server makes the decision)

1. client submits a request to a server.
2. The client informs the server which media types it understands
   • ("Accept" header and quality-factor weighting).
3. The server then supplies the version of the resource that best fits the request. Often, redirection is used to point the client to the correct resource.

(scales much better than server-driven negotiation) i.e. the server does NOT decide, but merely dumps a list of multiple choices of supported types. the client decides based on this list.

1. an user agent (or any other client) submits a request to a server. The agent resides on the client side, i.e. browser
2. The server responds and provides the agent with available representations on the server and their locations, usually with a "300 Multiple Choices" response that depends on the application implementation.
3. The user agent then makes another request to the desired URL for the actual resource.

Simple Object Access Protocol (on WSDL) SOAP has been around since ?? and is often the underlying layer to some of the more complex web services. (based on WSDL, Web Services Description Lanaguage) WSDL is an interface for describing functionalities offered by web services, which in turn make discover and integration with a remote web service very straightforward.

SOAP is extensible. Features can be added without major updates to the implementation. Security (WS-Security or WSS) and WS-Addressing can be added to SOAP.

SOAP is neutral. SOAP is not protocol specific and is able to operate over several different transport protocols, such as HTTP, TCP, Simple Mail Transfer Protocol (SMTP), and so on.

SOAP is independent It supports any programming model (object-oriented, functional, imperative, and so on), platform, and language.

The SOAP spec defines the messaging framework in four parts

1. Envelope - Ids the XML document as a SOAP message. Is required
2. Header - contains the SOAP header information.
3. Body - contains the remote call, parameters, and response information.
4. Fault - Provides information about any errors that occurred..

Example of a SOAP request:

 POST /Quotation HTTP/1.0
Host: www.xyz.org
Content-Type: text/xml; charset = utf-8
Content-Length: nnn

<?xml version = "1.0"?>
<SOAP-ENV:Envelope
   xmlns:SOAP-ENV = "http://www.w3.org/2001/12/soap-envelope"
   SOAP-ENV:encodingStyle = "http://www.w3.org/2001/12/soap-encoding">

   <SOAP-ENV:Body xmlns:m = "http://www.xyz.org/quotations">
      <m:GetQuotation>
         <m:QuotationsName>MiscroSoft</m:QuotationsName>
      </m:GetQuotation>
   </SOAP-ENV:Body>
</SOAP-ENV:Envelope>

Example of a SOAP request:

POST /Quotation HTTP/1.0
Host: www.xyz.org
Content-Type: text/xml; charset = utf-8
Content-Length: nnn

<?xml version = "1.0"?>
<SOAP-ENV:Envelope
   xmlns:SOAP-ENV = "http://www.w3.org/2001/12/soap-envelope"
   SOAP-ENV:encodingStyle = "http://www.w3.org/2001/12/soap-encoding">

   <SOAP-ENV:Body xmlns:m = "http://www.xyz.org/quotations">
      <m:GetQuotation>
         <m:QuotationsName>MiscroSoft</m:QuotationsName>
      </m:GetQuotation>
   </SOAP-ENV:Body>
</SOAP-ENV:Envelope>

Simple, light-weight RPC protocol, encoded in JSON with just these few data types:

• string
• integer
• boolean
• null

And just these commands:

• params
• method
• id

It supports notifications like sending data to the server that does not require a response. (useful for asynchronous updates and batch requests that have multiple requests inside one request body)

A standard protocol for procedure call support (lightweight) can run over HTTP, but is NOT a REST API. defines the REQUEST and RESONSE format only for sending communication from one device to another. Can send CLI commands.

Watch the transaction ids (to repond to the correct request)

Request:

[
  {
    "jsonrpc": "2.0",
    "method": "cli",
    "params": {
      "cmd": "show ip int brief vrf management",
      "version": 1
      },
    "id": 1
   }
 ]

This example had "method": "cli" which returns nice structured data, jsonrpc data. This format is the easiest way for programs to consume it. The other option is "cli-ascii" which would return a more traditional, human readable cli output, as one raw string. Some older "screen scraping" scripts may want this type of output.

Response

{
  "jsonrpc": "2.0",
  "result": {
   "body": {
    "TABLE_intf": {
     "ROW_intf": {
      "intf-name": "mgmt0",
      "prefix": "10.10.20.58",
      "ip-disabled": "FALSE",
      "iod": 2,
      "proto-state": "up",
      "link-state": "up",
      "admin-state": "up"
      }
     }
    }
   }
   "id": 1
  }

See also JSON-RPC :

Similar to SOAP, however less structured, with fewer constraints. It supports the basic types plus more complex types like Base64, array, datetime, struct. It also support basic http authentication

2019 XML-RPC "reboot" http://reboot.xmlrpc.com is a re-issue, this time supporting JSON.

See the dedicated org file on NETCONF

Network Configuration Protocol (NETCONF) is a network device configuration management protocol that provides mechanisms to install, manipulate, and delete configurations on network devices. It also provides a mechanism for notification subscriptions and asynchronous message delivery.

Has four layers:

ever increasing in popularity, it is open source, built with performance in mind. Performance comes from using HTTP/2, so supports multiplexing requests.

gRPC uses a special format for data serialziation, called Protocol Buffers. (which is a compressed binary format, before sending it)

REST natively uses HTTP Methods.

But REST also gets benefits from standard HTTP cacheing, compreshion, encryption as well.

CRUD –> Create Read Update Delete

Compare that with NETCONF actions:

The ? because there RESTCONF does not have these functions???

Bare bones:

import requests
import json

url = 'https://{{amp4e_client_id}}:{{amp4e_api_key}}@{{amp4e_host}}/v1/computers'

response = requests.get(url=url, params=None)
status_code = response.status_code
content = json.loads(response.content)

print(f' Status code: {status_code} ')
print(f' Response content: {content} ')

Now with headers:

import requests
url = 'https://{{amp4e_client_id}}:{{amp4e_api_key}}@{{amp4e_host}}/v1/computers'
url = "https://85f5553ffa9425b99189: kWg1gxce-c14a-4shb-ss3a-a49hs2362yhdnapi.amp.cisco.com/v1/computers"

payload={}
headers = {
  'Authorization': 'Basic ODVmNTUlas4mYTk0Mj23ggadfhr5uzdfgDQxZGUshdfhje34uyh34rtgfdbJdfgsfhhshhkerzQ1'
}
response = requests.request("GET", url, headers=headers, data=payload, timeout=400)

print(response.text)

Notice there are more than one way to use requests. I should investigate if some are deprecated or they do different things.

Initial values get shared when you share your collection, so DON'T put your keys here.

The same collection can be played against a production or a test environ, so postman environements lets you set different variables when in one or another.

Know how to get code.

1. global
2. collection (initial) (broadest)
3. environment (initial) (narrowest)
4. ? I think there is also an override a specific request, i.e. adding a header manually. I think that that overrides my environment settings, and collection, adn global. >>>> I need to test this. <<<<<

Use URL parameters to set the page size limit, and page number as the offset. Called offset pagination. Easy and safe. Mostly involves safely passing those parameters to SQL as shown. Because all the data needed is contained in the request, offset-pagination is stateless.

Sometimes the API takes care of pagination by itself. You get links to relevant pages in a response. These links contain the reference itself, and links to the previous page, and the next page. If next page link is empty or missing, you can assume the current page is the last page.

Native API pagination saves time, and makes code easier on the client, and increases efficiency, especially with large datasets.

RFC 5988 specifies how pagination is to work with this Cisco standard.

This also utilizes the HTTP standards, this time using URL parameters.

For example:

Even the best designed API can get bogged down in high load conditions. Fortunately there is rate limiting built into either the server side, or the client side.

Client-side rate limiting is usually implemented in the client application and limits the client from performing a large number of tasks that are costly for the API itself (with a timer, loading bar, or something similar). Here you are limiting the rate of API requests sent to the web server.

• more effective
• can be per user, say 1000 requests API calls per day, or if you pay a premium account 500 API calls per hour.
• limiting requests/second would mitigate DOS attacks
• web servers can limit rates based on region too, i.e. Canada can limit requests/second from Russia or China.

Is server-side limiting, that puts a limit on the size of the accepted payload, in bytes. Think video upload to a cloud storage site.

By reducing the size of the body within a HTTP request sent by the client, you reduce the chance of that request being corrupted while transporting, increase the speed it can be processed, and limit damage someone malicious could inflict by sending a 1GB size picture for example.

If payload is exceeded in size, the server will return 413 Request Entity Too Large

API calls can be throttled using:

The response code 429 often includes a Retry-After header that tells the client how long to wait before trying again.

It makes sense to expect a certain amount of problems when using remote APIs. Therefore, your code should handle those errors, and even, where available, check on the health of the service, before sending a big request. Therefore, writing good code means checking for status of APIs and waiting to retry rather than just crashing with a Traceback. For example a client might use this to check API health:

Example of a simple API Health Check

def checkStatus(url):
    try:
    # Check API Status
    status = requests.get(url + "/api/status")
    if status.status_code != 200:
        raise Exception("Unexpected status code: " + status.status_code)
   except Exception as e:
       print(f" API endpoint encountered an error: {e} ")

More detailed client might implement something that retries reading users 10 times with 30 second intervals in-between. Each time, the client checks for HTTP code 200 OK because that means the request did not time out.

Also, you requested "users" but the response might contain different data, so it is best practice to trust but verify, i.e. make sure you are getting what you expected to get.

import requests

def getUsers(url):
    max_retries = 10
    timeout = 30

    for retry_count in range(max_retries):
    try:
        api_users = requests.get(url + "/api/users")
    except Exception as e:
        print("Error encountered while reading users.. Retrying")

    if api_users.status_code == 200:
        return api_users
    else:
        time.sleep(timeout)

    raise Exception("Retry limit reached!")   

You can use these time-out schemes:

• linear for example (5, 10, 15, 20)
• exponential for example (1, 2, 4, 8, 16, 32)

Timeouts produce 408 Request Timeout (unless server is acting as a proxy in which case it would be 504 Gateway Timeout ) Unless the server is a simplistic one, in which case youtmight only see 404 Not Found

uses the native/built-in http authentication

• Con: every API request must have authentication details (same for API key). (compare with custom token where no authentication details are sent. Instead, a token is sent)

uses pre-generated API keys that client adds to a header (usually X-API-Key)

• Con: every API request must have authentication details (same for basic auth). (compare with custom token where no authentication details are sent. Instead, a token is sent)
• client manually adds the pre-generated key to the request
• server generates the key in an earlier basic authenticated web session
• the key is Base64 encoded, so it can be transported via HTTP
• key can use HTTP headers, either Authentication: or Authorization:
• key can use URL parameter
• key can use cookies, (i.e. as a header)

  The server receives the request, parses it, decodes the key, compares to a table containing valid keys, then either allows or sends 401 Unauthorized error in the response.

In python, using the requests built-in module it would look like:

headers = {'X-Api-Key': 'gahwo135n2nf'}
payload = {'user': {'name': name, 'age': age}}
response = requests.post(url | '/users', headers=headers, data=payload)

Invalid keys lead to 401 Unauthorized response error codes.

If key is good, but you are not authorized to do the job, you will gets 403 Forbidden.

• Weak, as the same key is sent in every single request, can be compromised
• Custom tokens solves this problem, but they are for users, not apps/scripts

Under Meraki's dashboard, under account profile, I can generate a new meraki key, or revoke a compromised meraki key:

Using these keys as a URL parameter

• GET /users? api_key=28fnhu781gab44

Using these keys as an HTTP cookies

• GET /users Cookie: api_key=28fnhu781gab44

A dynamcially generated token is used. Also known as access token.

Here's the process:

1. a client tries to access the web server
2. client is not authenticated, so client is redirected to an authentication server. where client provides username:password (i.e. basic auth). This is done with a POST request for example to: htps://sandboxdnac2.cisco.com/dna/system/api/v1/auth/token
   • username:password in base64 encoding (lets say that is stored in {{myusepass₆₄}} in the headers of the request.
   • Authorization: {{myusepass_64}} would be the header used by Postman.
3. Authentication server validates the credentials given, and generates a custom, time-limited, signed authentication token and returns it in the response
4. client requests now contain this proper, valid token in headers, or URL parameters or cookies?, and that is passed to the API
5. The API services validates the token and servers up the response.

In theory this is what is happening:

The authentication server is often an external service (for example, OpenID) that is used in combination with an authorization mechanism (for example, OAuth 2.0). The tokens themselves often contain some authorization data as well. Authentication servers are commonly used to issue tokens that can be used with many different services from different vendors (for example, using Google authentication to log in to an unrelated site that has implemented Google sign-in).

The term that is commonly used for this type of authentication (and authorization) is single sign-on (SSO). SSO is gaining in popularity, because it is very useful for services that require third-party authorization (for example, sharing an article from a news site on a social media account) or for businesses that offer several different services but want users to use the same identity for all of them (for example, Google Apps, your company enterprise environment, and so on).

The pros of custom token authentication include: Pros:

A modern standard for authorizing network connections between disparate services.

• allows an app to post to your twitter feed.
• allows a login to github using Google account
• allows IOT device, like a temperature sensor and its cloud based back end, post daily temperature readings in a google sheets, or google docs
• allows an API or CLI on your laptop access remote public cloud service without storing your username/password in that API, or CLI.
• Allows SSO, a single corporate user login access to a bunch of websites, services, and applications.

As you can see OAuth2.0 is really just an implementation of Customer Token

• API keys need to be store, and they are static, so they are not as secure.
• Basic auth encodes username:password in Base64, but that is not encryption just an encoding, that is easy to decode, so less secure.
• hashed credentials are more secure, because credentials are never sent in the clear.
• Basic http auth parameters

To handle HTTP authentication through a proxy, one can use special proxy headers.

There are several:

• Anonymous (no auth)
• Basic (Base64-encoded credentials as username:password)
• Bearer (HTTP implementation of custom token authentication)
• Digest (MD5-hashed credentials)
• Mutual (2-way auhentication)
• Others (uncommon) such as HMAC, HOBA, AWS, OAuth, Negotiate…

These ALL should be used with TLS, i.e. https as a minimum.

Example:

import requests
import base64

def authenticate(url, username, password):

    auth_type = 'Basic'
    creds = '{}:{}'.format(username, password)
    creds_b64 = base64.b64encode(creds)
    headers = {'Authorization': '{}{ {}'.format(auth_type,creds_b64)}

    try:
    response = requests.get(url, headers=headers)

    if response.status_code != 200:
        raise Exception('Authentication error: {}'.format(response))
    except Exception as e:
    print('Authentication unsuccessful!  Error returned was: {}'.format(response))

    print('Authentication successful!')

This example, taken from Cisco Devnet shows how one can create a new user, using API key authentication.

def createUser(url, name, age):
    headers = {'X-API-Key': '28fnhu781gab44'}
    payload = {'user': {'name': name, 'age': age}}

    try:
    response = requests.post(url + '/users', headers=headers, data=payload)

    if response.status_code == 401:
        raise Exception('API key invalid')
    elif response.status_code == 403:
        raise Exception('API key not authorized!')
    elif response.status_code != 200:
        raise Exception('API error: {}'.format(response))

    except Exception as e:
    print(f'User creation failed!  Error:{e}')
    print('User creation failed!  Error:{}'.format(e))

    print(f' User {name} created successfully!')

Usually REST API documentaiton is written to open standards, such as OpenAPI(Swagger) or RAML. Cisco DevNet site publishes them under these menus. Start at developer.cisco.com

Right out of help(api) where api = WebexTeamsAPI() in python.

#+BEGIN_EXAMPLE An access token must be used when interacting with the Webex Teams API.

• provide identity, called "public certificates"
• signed by a trusted third party, called "certificate authorities".
• public keys are then retrieved from this third party, to make it harder for both pairs of keys between two parties being intercepted, and replaced, in a "man in the middle" attack.

Only a few global certificate authorities exist. They are GoDaddy.com, IdenTrust, Comodo, Let's Encrypt.

• use a trust chain, of signed certificates that are themselves signed by other trusted certificates.
• Info in a certificate is defined in the X.509 standard.

Initiated by the client.

This depicts using RSA encryption algorithm. Other algorithms may or may not exchange secrets the same way, i.e. keyless SSL, Diffe-Hellman

Risks are at the TLS handshake phase, and HTTPS itself, though low, or a compromised CA. All the attacks attempt to be a MIM, man in the middle.

Risks are:

• HTTPS spoofing fake certificates are sent to the client, thus making the client trust your connecion as a real one
• SSL hijacking fake authentiation keys are copied to the client and server, thus getting full access (unencrypted in the middle) to the connection.
• SSL stripping converts the https connection to a http connection by interrupting the TLS handshake and exploiting HTTP redirects
• Downgrade attacks abuses systems that try to support backward compatibility by insisting the connection use an older, less secure or less protected version of s/w in order to exploit known vulnerabilities in the old s/w.
• Software bugs for unpatched systems, known bugs are disastrous, for example, heartbleed was such an exploit.
• Phishing attacks trick the user to accept a fraudulent certificate. 'nuff said.

Keep your certs, your API keys, your tokens secure!! They are the weakest link in your security chain. The solution is proper Credential Management.

Principles to live by:

1. Avoid hard-coding credentials in scripts/programs
   • lead to loss of portability
   • loss of flexibility
   • poor management. think about upgrading credentials in all your various scripts and files. this would take lots of time and be error prone
   • is insecure if for instance you store your code in a revision mgt system such as git. There for the taking/stealing!!
2. Use softcoding of your credentials. That means retrieving your credentials from an external source for instance:
   • environmental variables meraki_user = os.environ["MERAKI_USER"]
   • external env modules
   • configuration files # you could import configparser
   • use cookies

     cookies = {'cookies': f'api_key={}'}
     

3. mind your logging First of all, take care when you write messages to a log file, that those messages never contain credentils. Secondly, a good idea to periodically search your logs for credentials, and fix them before someone else finds them and hacks you.
   • Use a true-fals switch for logging sensitive info. i.e. log lots in dev, but very little in prod environments.
   • Regularly "mask" credentials in logs, if they ever do appear. Similar to the concept of egress filtering on firewalls that strip credit card info, or credentials on traffic exiting a firewall.
   • implement credential partitioning for users, with different credentials needed for different tasks.
4. Mind your http headers if you send credenti~als as headers, make sure it is https and not http.
5. Encrypt data at rest In transit it should be encrypted by TLS, but at rest use symetric key encryption to encrypt/decrypt credentials when you need them. Will need effective key management to keep track of the different keys for the different functions of your project. Encrypting data at rest can be:
   • full disk encryption
   • file system encryption
   • database encryption