dictparse

A simple, slim and useful, zero-dependency utility for parsing dictionaries or dictionary-like objects.

It's particularly useful for parsing incoming request data in REST APIs & web applications, for example in the case of Flask, parsing form data from request.form, query string arguments fromrequest.args or JSON data from request.json.

The dictparse design takes inspiration from Python's own argparse library, similar to the ArgumentParser class , taking input as a dictionary or dictionary-like object, enforcing rules, types, applying functions, default values and returning a NameSpace, with values mapped to attributes.

Installation

pip install dictparse

Example

The following code is a Python program that takes takes some data in the form of a dictionary and parses it:

>>> from dictparse import DictionaryParser
>>> parser = DictionaryParser()
>>> parser.add_param("name", str, required=True)
>>> params = parser.parse_dict({"name": "FooBar"})
>>> params.name
'FooBar'

Creating a parser

The first step is to create the DictionaryParser object

>>> from dictparse import DictionaryParser
>>> parser = DictionaryParser(description="Create a new user")

Adding parameters

Adding parameters to the parser is done by making calls to the add_param method. These calls tell the DictionaryParser how to handle the values passed in and turn them into the desired output, enforcing rules , changing types and transforming values based on the arguments passed to the add_param method.

>>> parser = DictionaryParser()
>>> parser.add_param("name", str, required=True)
>>> parser.add_param("language", str, choices=["python", "javascript", "rust"])
>>> parser.add_param("tags", str, action=lambda x: x.split(","))
>>> params = parser.parse_dict({"name": "FooBar", "language": "python", "tags": "foo,bar,baz"})
>>> params.name
'FooBar'
>>> params.language
'python'
>>> params.tags
['foo', 'bar', 'baz']
>>> params.to_dict()
{'name': 'FooBar', 'language': 'python', 'tags': ['foo', 'bar', 'baz']}

If the parser does not find a value matching the name, the default value is None

Arguments available for add_param

DictionaryParser.add_param(
    name: str,
    type_: Optional[Union[Type[str], Type[int], Type[float], Type[bool], Type[list], Type[dict], Type[set], Type[tuple]]] = None,
    dest: Optional[str] = None,
    required: Optional[bool] = False,
    choices: Optional[Union[list, set, tuple]] = None,
    action: Optional[Callable] = None,
    description: Optional[str] = None,
    default: Optional[Any] = None,
    regex: Optional[str] = None
) -> None

• name: The parameter name (required - See note below)
• type_: The common parameter type (The parser will attempt to convert the parameter value to the given type)
• dest: The destination name of the parameter (See note below)
• required: If True, enforce a value for the parameter must exists
• choices: A list, set, or tuple of possible choices
• action: A function to apply to the value (Applied after any type conversion)
• description: A description of the parameter
• default: A default value for the parameter if not found
• regex: A regular expression to match against (Sets the parameter to None if the match is negative)

Note - The name and dest parameters must comply with standard Python variable naming conventions (only start with a letter or underscore & only contain alpha-numeric characters), not be a Python keyword and not start and end with a double underscore (dunder)

Parsing the data

After creating the parser and adding parameters to it, data can be parsed by calling the parse_dict method, passing in the data to be parsed. This returns a NameSpace object.

DictionaryParser.parse_dict(
    data: Dict[str, Any], 
    strict: Optional[bool] = False, 
    action: Optional[Callable] = None
) -> NameSpace:

• data: A dictionary or dictionary-like object
• strict: If True, raises an exception if any parameters not added to the parser are received
• action: A function to apply to all parameters (after any type conversion and after action passed to add_param)

The NameSpace object

A NameSpace object is returned when calling parse_dict and contains the parsed data after applying your rules defined when adding arguments.

Parameters can be accessed as attributes of the NameSpace using dot notation:

>>> parser = DictionaryParser()
>>> parser.add_param("age", int, required=True)
>>> params = parser.parse_dict({"age": 30})
>>> params.age
30

A standard AttributeError will be raised if an attribute is accessed without being added to the parser:

>>> params.foo
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'NameSpace' object has no attribute 'foo'

if the dest parameter is supplied when adding a parameter in add_param, the value can only be accessed by using the dest value:

>>> parser = DictionaryParser()
>>> parser.add_param("bar", str, dest="foo")
>>> params = parser.parse_dict({"bar": "bar"})
>>> params.foo
'bar'

The NameSpace object has the following available methods:

get

NameSpace.get(
    name: str, 
    default: Optional[Any] = None
) -> Union[None, Any]:

Calling the get method on the NameSpace and passing in a key works in the same way as calling get on a dictionary , returning either the value for the parameter requested or None if the NameSpace does not have that attribute.

An optional default value can be supplied using the default parameter to be returned if the attribute does not exist.

>>> parser = DictionaryParser()
>>> parser.add_param("age", int, required=True)
>>> parser.add_param("weight", int)
>>> params = parser.parse_dict({"age": 30, "height": 1.9})
>>> params.weight
None
>>> params.get("age")
30
>>> params.get("foo", 42)
42

to_dict

NameSpace.to_dict() -> dict

Returns a dictionary of the parsed parameters.

>>> parser = DictionaryParser()
>>> parser.add_param("one", str)
>>> parser.add_param("two", int)
>>> parser.add_param("three", list)
>>> params = parser.parse_dict({"one": "one", "two": 2, "three": [1, 2, 3]})
>>> params.to_dict()
{'one': 'one', 'two': 2, 'three': [1, 2, 3]}

to_dict() accepts an optional parameter exclude, a list of keys to exclude from the returned dictionary

>>> from dictparse import DictionaryParser
>>> parser = DictionaryParser()
>>> parser.add_param("csrf_token", str, required=True)
>>> parser.add_param("name", str)
>>> parser.add_param("email", str)
>>> params = parser.parse_dict({"csrf_token": "xxyyzz112233", "name": "foo", "email": "[email protected]"})
>>> params.to_dict(exclude=["csrf_token"])
{'name': 'foo', 'email': '[email protected]'}

get_param

Returns a Param object

>>> from dictparse import DictionaryParser
>>> parser = DictionaryParser()
>>> parser.add_param("names", list, default=[])
>>> params = parser.parse_dict({"names": ["foo", "bar"]})
>>> names = params.get_param("names")
>>> names.name
'names'
>>> names.value
['foo', 'bar']
>>> names.default
[]

Param objects are hold all data associated with the parameter, as can be seen below in the Param.__init__ method:

class Param(object):

    def __init__(
            self,
            name: str,
            type_: Optional[Union[Type[str], Type[int], Type[float], Type[bool], Type[list], Type[dict], Type[set], Type[tuple]]] = None,
            dest: Optional[str] = None,
            required: Optional[bool] = False,
            choices: Optional[Union[list, set, tuple]] = None,
            action: Optional[Callable] = None,
            description: Optional[str] = None,
            default: Optional[Any] = None,
            regex: Optional[str] = None,
            value: Optional[Any] = None
    ):

Note - The NameSpace will be assigned the value for dest if supplied in add_param

>>> from dictparse import DictionaryParser
>>> parser = DictionaryParser()
>>> parser.add_param("foo", str, dest="bar")
>>> params = parser.parse_dict({"foo": 42})
>>> param = params.get_param("bar")
>>> param.name
'foo'
>>> param.dest
'bar'
>>> param.value
'42'

Flask example

An example of parsing JSON data sent in a POST request to a Flask route:

from app.users import create_user

from flask import Flask, request
from respond import JSONResponse
from dictparse import DictionaryParser


def create_app():

    app = Flask(__name__)

    @app.route("/", methods=["POST"])
    def post():

        parser = DictionaryParser(description="Create a new user")

        parser.add_param("name", str, required=True)
        parser.add_param("age", int)
        parser.add_param("password", str, required=True, action=lambda x: x.encode("utf-8"))
        parser.add_param("interests", list, action=lambda x: [i.strip() for i in x])
        parser.add_param("level", float, default=1.5)
        parser.add_param("stage", str, choices=["alpha", "beta"])

        try:
            params = parser.parse_dict(request.get_json())
        except Exception as e:
            return JSONResponse.bad_request(str(e))

        user = create_user(
            name=params.name,
            age=params.age,
            password=params.password,
            interests=params.interests,
            level=params.level,
            stage=params.stage
        )

        return JSONResponse.created(user.to_dict())

    return app


if __name__ == "__main__":
    app = create_app()
    app.run()

Exception handling

Exceptions will be raised in the following scenarios:

ParserTypeError

Raised when a parameter cannot be parsed to the type declared in add_param

from dictparse import DictionaryParser
from dictparse.exceptions import ParserTypeError

parser = DictionaryParser()
parser.add_param("age", int)

try:
    params = parser.parse_dict({"age": "thirty"})
except ParserTypeError as e:
    print(e)  # Invalid value 'thirty' for parameter 'age', expected 'int' not 'str'

ParserTypeError contains the following attributes:

• param: The parameter name (str)
• value: The parameter value (Any)
• expected: The expected type (type)

ParserRequiredParameterError

Raised when parse_dict is called and a parameter is required, but not found

from dictparse import DictionaryParser
from dictparse.exceptions import ParserRequiredParameterError

parser = DictionaryParser()
parser.add_param("name", str)
parser.add_param("email", str, required=True)

try:
    params = parser.parse_dict({"name": "John Doe"})
except ParserRequiredParameterError as e:
    print(e)  # Missing required parameter 'email'

• ParserRequiredParameterError has a single attribute param, the name of the parameter (str)

ParserInvalidChoiceError

Raised when parse_dict is called and parses a value not defined in the choices parameter of add_param

from dictparse import DictionaryParser
from dictparse.exceptions import ParserInvalidChoiceError

parser = DictionaryParser()
parser.add_param("name", str)
parser.add_param("language", str, choices=["python", "bash"])

try:
    params = parser.parse_dict({"name": "John Doe", "language": "javascript"})
except ParserInvalidChoiceError as e:
    print(e)  # Parameter 'language' must be one of '['python', 'bash']', not 'javascript'

ParserInvalidChoiceError has the following 3 attributes:

• param: The parameter name (str)
• value: The parameter value (Any)
• choices: The available choices added via add_param (list|set|tuple)

ParserInvalidParameterError

Raised calling parse_dict with strict set to True

The strict parameter enforces the parser to only accept parameters that have been added to the parser

from dictparse import DictionaryParser
from dictparse.exceptions import ParserInvalidParameterError

parser = DictionaryParser()
parser.add_param("name", str)
parser.add_param("language", str, choices=["python", "bash"])

try:
    params = parser.parse_dict({"name": "John Doe", "language": "python", "email": "[email protected]"}, strict=True)
except ParserInvalidParameterError as e:
    print(e)  # Invalid parameter 'email'

ParserInvalidParameterError has a single attribute param, the name of the parameter (str)

Other runtime considerations for parse_dict

If an invalid data type for data is passed to parse_dict (such as a list or string), it raises a ParserInvalidDataTypeError

from dictparse import DictionaryParser
from dictparse.exceptions import ParserInvalidDataTypeError

parser = DictionaryParser()
parser.add_param("name", str)

try:
    params = parser.parse_dict([{"name", "John Doe"}])
except ParserInvalidDataTypeError as e:
    print(e)  # Invalid type for 'data', must be a dict or dict-like object, not 'list'

try:
    params = parser.parse_dict("foo")
except ParserInvalidDataTypeError as e:
    print(e)  # Invalid type for 'data', must be a dict or dict-like object, not 'str'

Tests & coverage

A test suite is available in the tests directory with 100% coverage (15/Sep/2020)

Name                      Stmts   Miss  Cover
---------------------------------------------
dictparse/__init__.py         1      0   100%
dictparse/exceptions.py      37      0   100%
dictparse/parser.py         106      0   100%
tests/__init__.py             0      0   100%
tests/test_parser.py        310      0   100%
---------------------------------------------
TOTAL                       454      0   100%