XML, JSON, YAML Key:Value Notation

Data serialization languages have the following characteristics:

• format syntax
• concept of an object
• key-value notation
• array or list
• importance of whitespace
• case sensitivity

The all have this key value pair concept. For XML the "key" is actually the tag that has to have a closing tag. <tag> value </tag> or <key>value</key>

For JSON it is "Key":Value, whitespace not significant, but curly braces are as are commas.

For YAML it is Key:Value no commas, but whitespace indentation defines the structure. Your IDE will handle that for you, including emacs. You can also have your IDE highlight those indentations as an option.

• whitespaces are ignored
• whitespaces are ignored
• whitespaces however are significant in certain cases.
• key value pairs can be nested
• .xml file ending
• NO predefined tags. i.e. they are all user defined
• opening tags <mytag>
• closing tags </mytag> # must match opening tags.
• comments are <!-- this is a comment and it ends with a -->
• start with XML prologue wihch starts and ends with <? and ?> respectively. For example <?xml version="1.0" encoding="UTF-8"?>
• special encoding (i.e. if you want to send a < as part of the data) are used with the character's numeric encoding
  • < is < > is >

XML also provides a good method of storing data that is universal becuase it is computer language neutral.

• more complicated but also more powerful than json, because not predefined. Any customized tags can be defined. (compare with HTML tags <h1>, <h2> etc. but less secure due to its complexity
• XLT XML Language Tags can be predefined that specify a

• Every XML document has a root element containing one or more child elements
• Path is a way of addressing a particular element inside a tree
• namespaces provide you with name isolation for potentially duplicate names for an element. i.e. there could be many places where "address" will appear. namespaces lets you deal with this ambiguity/overlap, even when an XML document may be built from several YANG models, as they often are.

<?xml version = "1.0" encoding = "UTF-8" ? >
<!--  hey dude, this comment was left by Zintis -->
<!--  just to show what comments look like in xml -->
<!--  ########################################### -->

  <Interface xmlns = "ietf-interfaces">
    <name> Gigabitethernet2</name>
    <description>
      WAN firewall
    </description>
    <enabled>   </enabled>
    <ipv1>
      <address>
	<ip>172.17.18.19</ip>
	<netmask>255.255.255.0</netmask>
      </address>
      <address><ip>172.16.16.1</ip><netmask>255.255.255.0</netmask>
      </address>
    </ipv4>

<interfaces xmlns="urn:ietf:params:xml:ns:yang:ietf-interfaces">
<!--                   comment left by zintis               -->
   <interface>
       <name>GigabitEthernet1</name>
       <type xmlns:ianaift="urn:ietf:params:xml:ns:yang:iana-if-type">ianaift:ethernetCsmacd</type>
       <enabled>true</enabled>
       <ipv4 xmlns="urn:ietf:params:xml:ns:yang:ietf-ip">
	   <address>
	       <ip>198.18.133.212</ip>
	       <netmask>255.255.192.0</netmask>
	   </address>
       </ipv4>
       <ipv6 xmlns="urn:ietf:params:xml:ns:yang:ietf-ip"/>
   </interface>
   ... 
</interfaces>

https://www.w3.org/XML/

The xml module is built-in to python It has two submodules,

1. minidom and
2. Elementree

Also be aware that The xml.etree.ElementTree module is not secure against maliciously constructed data. See xml vulnerabilities link for *.

For example, this code, known as the 'billion laughs bomb' will expand out to 10⁹ "lol"s

<?xml version="1.0"?>
<!DOCTYPE lolz [
 <!ENTITY lol "lol">
 <!ELEMENT lolz (#PCDATA)>
 <!ENTITY lol1 "&lol;&lol;&lol;&lol;&lol;&lol;&lol;&lol;&lol;&lol;">
 <!ENTITY lol2 "&lol1;&lol1;&lol1;&lol1;&lol1;&lol1;&lol1;&lol1;&lol1;&lol1;">
 <!ENTITY lol3 "&lol2;&lol2;&lol2;&lol2;&lol2;&lol2;&lol2;&lol2;&lol2;&lol2;">
 <!ENTITY lol4 "&lol3;                                               ;&lol3;">
 <!ENTITY lol5 "&lol4;                                               ;&lol4;">
 <!ENTITY lol6 "&lol5;   you get the idea.  Don't do this yourself   ;&lol5;">
 <!ENTITY lol7 "&lol6;                                               ;&lol6;">
 <!ENTITY lol8 "&lol7;                                               ;&lol7;">
 <!ENTITY lol9 "&lol8;&lol8;&lol8;&lol8;&lol8;&lol8;&lol8;&lol8;&lol8;&lol8;">
]>
<lolz>&lol9;</lolz>   # this triggers the "bomb" and is simply 10 lol8s, which
                      # in turn is 10 lol7s ... to the last "lol" 

Any code that allows for references in itself is vulnerable, and so external sources of data should be scurbbed clean to prevent heavily nested entities.

Here is a YAML version: a: &a ["lol","lol","lol","lol","lol","lol","lol","lol","lol"] b: &b [*a,*a,*a,*a,*a,*a,*a,*a,*a] c: &c [*b,*b,*b,*b,*b,*b,*b,*b,*b] d: &d [*c,*c,*c,*c,*c,*c,*c,*c,*c] e: &e [*d,*d,*d,*d,*d,*d,*d,*d,*d] f: &f [*e,*e,*e,*e,*e,*e,*e,*e,*e] g: &g [*f,*f,*f,*f,*f,*f,*f,*f,*f] h: &h [*g,*g,*g,*g,*g,*g,*g,*g,*g] i: &i [*h,*h,*h,*h,*h,*h,*h,*h,*h]

This module is an additional model that needs to be pip installed. Here is the https://pypi.org/project/xmltodict/ link, which describes it as : "`xmltodict` is a Python module that makes working with XML feel like you are working with [JSON](http://docs.python.org/library/json.html), as in this ["spec"] (http://www.xml.com/pub/a/2006/05/31/converting-between-xml-and-json.html):"

This is a 3rd party package from codespeak. It uses ElementTree API. The lxml package has XPath and XSLT support

Here is some sample python code using lxml, from pythonlibrary.org

from lxml import etree

def parseXML(xmlFile):
    """
    Parse the xml
    """
    with open(xmlFile) as fobj:
	xml = fobj.read()

    root = etree.fromstring(xml)

    for appt in root.getchildren():
	for elem in appt.getchildren():
	    if not elem.text:
		text = "None"
	    else:
		text = elem.text
	    print(elem.tag + " => " + text)

if __name__ == "__main__":
    parseXML("example.xml")

mapping key: value The mapping key does NOT have to be enclosed in double quotation marks.

Mark McGwire: {hr: 65, avg: 0.278}
Sammy Sosa: {
    hr: 63, 
    avg: 0.288
  }

Sex Pistols:

• Paul Cook
• Steve Jones
• Johnny Rotten
• Sid Vicious

The Pogues:

• Shane MacGowan
• Jem Finer
• Andrew Ranken
• Phillip Chevron
• David Coulter
• Cait O'Riordan

The whitespaces are important in indentation. Indentation has meaning.

A great source to see YAML detail is the YAML documentation in ansible.com

layout file starts with three dashes. These dashes indicate the start of a new YAML document. YAML supports multiple documents, and compliant parsers will recognize each set of dashes as the beginning of a new one.

Next example, we see the construct that makes up most typical YAML documents: a key-value pair. company is a key that points to a string value:

• company: "acm explosvies ltd."

YAML supports more than just string values. The file starts with five key-value pairs. They have four different data types, each of which is shown in the following example:

---
company: "acme explosives ltd."
pi: 3.1415926535
happy: true
iterations: 6
users:
  - superuser
  - operator
  - admin
  - enduser
ietf-interfaces:
  - interface1:
      name: GigabitEthernet2
      description: WAN firewall
      enabled: true
      ietf-ip:
      ipv4:
        address:
          ip: 172.16.17.18
          netmask: 255.255.255.0
  - interface2:
      name: GigabitEthernet6
      description: load balancer
      enabled: true
      ietf-ip:ipv4:
        address:
          ip: 172.16.21.1
          netmask: 255.255.255.0
  - interface3:
      name: GigabitEthernet48
      description: core routers
      enabled: true
      ietf-ip:ipv4:
        address:
         ip: 10.10.1.254
         netmask: 255.255.255.0
  - interface4:
      addressess:
      - ip: 172.16.17.18
        netmask: 255.255.255.0
      - ip: 172.16.16.1
        netmask: 255.255.255.0
      - ip: 192.168.17.1
        netmask: 255.255.255.12

• company is a string. "acme explosives ltd."
• pi is a floating-point number. 3.1415926535
• happy is a boolean. true
• iterations an integer. 6
• users is a list (array). users has four elements inside it,
  • each denoted by an opening dash.
• ietf-interfaces is a dictionary

You can enclose strings in single or double-quotes or no quotes at all. YAML recognizes unquoted numerals as integers or floating point.

user elements I have indented with two spaces. Indentation is how YAML denotes nesting. The number of spaces can vary from file to file, but they must be consistent within a file. Tabs are NOT allowed.

ietf-interfacers dictionary has four more elements inside it, each of them indented.

We can view ietf-interfaces as a dictionary that contains four other embedded dictionaries. YAML supports nesting of key-values, and mixing types.

There are only two types of structures used in YAML, Lists and Maps.

YAML supports a "folding" syntax. i.e. linebreaks are presumed to be replaced by sapces

several options exist:

• import dicttoxml
• xmlstring = dicttoxml(mypythdictionary)

import untangle # xml parser library myresponse_python - untangle.parse(myresponse)

Here is a small example of parsing data to json, and yaml

import json
import yaml
from pprint import pprint

with open('myfile.json','r') as json_file:
    ourjson = json.load(json_file)

pprint(ourjson)

print(ourjson['expires_in'])

print("The access token from JSON is: %s" % ourjson['access_token'])

print("\n\n---")    # to add the required three --- to start a yaml file.
print(yaml.dump(ourjson))   # yaml.dump takes json and outputs yaml

Where the file myfile.json looks like this:

{
 "access_token":"ZDI3MGEyYzQtNmFlNS00NDNhLWFlNzAtZGVjNjE0MGU1OGZmZWNmZDEwN2ItYTU3",
 "expires_in":1209600,
 "refresh_token":"MDEyMzQ1Njc4OTAxMjM0NTY3ODkwMTIzNDU2Nzg5MDEyMzQ1Njc4OTEyMzQ1Njc4",
 "refreshtokenexpires_in":7776000
}

So basically we used:

• x = json.load(jsonfile)
• x = yaml.dump(ourjson)
• x = yaml.safe_load(yamlfile)

Note: the difference between yaml.load and yaml.safe_load:

• yaml.safe_load(sys.stdin) is the same as
• yaml.load(sys.stdin, Loader=yaml.SafeLoader)

And

• yaml.full_load(sys.stdin) is the same as
• yaml.load(sys.stdin, Loader=yaml.FullLoader)

import yaml

with open('config.yaml') as f:
    try:
    dict = yaml.load(f, Loader=yaml.FullLoader)
    print(dict)
    except yaml.YAMLError as e:
    print(e)

is the same as:

import yaml

with open('config.yaml') as f:
    try:
    dict = yaml.full_load(f)
    print(dict)
    except yaml.YAMLError as e:
    print(e)

PyYAML offers the SafeLoader. From the docs: "SafeLoader(stream) supports only standard YAML tags and thus it does not construct class instances and probably safe to use with documents received from an untrusted source. The functions safe_load and safe_load_all use SafeLoader to parse a stream."