Stringify Lambdas and Derived Types

[pauldhein]

Overview

We need to convert the lambda functions and derived types to be saved as string objects in a generated AIR.json file. The reason for this change is to support the portability of GrFN objects (a lambdas Python file is path-dependent during evaluation/execution). Completing this change will remove the necessity for outputting a _lambdas.py file.

@skdebray and @cl4yton, I am adding you both to this GH issue so you can track the progress.

Task assignments

• @pratikbhd: convert the lambda function generation to the new stringification style (see examples below)
• @hlim1: convert the derived type generation to the new stringification style (see the example derived-type below)
• @pauldhein: Create a secure sandbox that allows us to use eval() and exec() to load lambda functions and derived types without compromising security for future users of GrFN JSON

If anyone has any questions about their individual tasks please add a comment in this GH issue and mention me in the comment.

Lambda function examples

Examples

# Example 1 ==========================================
# Old style
def PETPT__petpt__IF_0__assign__albedo__1(msalb: Real, xhlai: Real):
    return 0.23 - ((0.23 - msalb) * np.exp(-((0.75 * xhlai))))
# New style
"lambda msalb,xhlai:(0.23-((0.23-msalb)*np.exp(-((0.75*xhlai)))))"
# ====================================================

# Example 2 ==========================================
# Old style
def PETPNO__petpno__assign__sbzcon__0():
    return 4.903e-09
# New style
"lambda :4.903e-09"
# ====================================================

# Example 3 ==========================================
# Old style
def PETPT__petpt__IF_0__decision__albedo__2(
    COND_0, albedo_0: Real, albedo_1: Real
):
    return albedo_0 if COND_0 else albedo_1
# New style
"lambda COND_0,albedo_0,albedo_1:albedo_0 if COND_0 else albedo_1"
# ====================================================

# Example 4 ==========================================
# Old style
def some_fake_lambda(x, y, z):
    return x+y, y+z, x+z
# New style
"lambda x,y,z:(x+y,y+z,x+z)"
# ====================================================

Important notes

• Use minimal spacing when creating these strings as I have shown above
• Type identifiers are going to be dropped in this representation (extra : are illegal in lambda expressions)
• Parentheses must be used if we ever end up returning more than a single object from a lambda expression

Derived-type example

In this section I am providing an example of a derived type from Fortran. @hlim1 I am going to show what the current derived-type class code should look like in the current form, and then I will show how to stringify that.

Original Fortran

type soiltype
  integer nlayr
  character (len=10) slno
  character(len=12), dimension(nl) :: texture
  character*50 sldesc, taxon
  logical, dimension(nl) :: coarse
  real ales, dmod
  real, dimension(nl) :: alphavg
end type soiltype

Current python output

@dataclass
class soiltype:
    nlayr: int
    slno: str
    texture: List[str]
    sldesc: str
    taxon: str
    coarse: List[bool]
    ales: float
    dmod: float
    alphavg: List[float]

Stringified form

"@dataclass\nclass soiltype:\n\tnlayr:int\n\tslno:str\n\ttexture:List[str]\n\tsldesc:str\n\ttaxon:str\n\tcoarse:List[bool]\n\tales:float\n\tdmod:float\n\talphavg:List[float]"

Important notes

• The List annotation is imported from the python types module and it needs to be capitalized
• No spaces are necessary between the colon and the type annotation
• The following sequence is needed before each new data member: \n\t
• A newline character is added after @dataclass and a space is necessary after the class keyword

[pauldhein]

A solution for the eval()/exec() security concern

(@cl4yton and @skdebray can you both please verify that I am not missing any obvious security concerns beyond those discussed in this comment?)

Using eval()/exec() to load our lambda functions and derived-types into the Python global namespace means that we are also opening the door to executing any other runnable Python code that may have been in the strings where we expected to find a lambda function or derived-type. This could lead to many bad outcomes since Python gives execution access to some pretty hefty modules (e.g. os/sys/subprocess/urllib/ssl/email/etc.) that could do plenty of damage to a users system as well as steal and use sensitive information.

Sandboxing calls to eval()/exec()

However, these modules cannot be used unless they are already imported in the Python namespace where eval()/exec() are being used. So the first step of my security solution is to isolate the places where we plan on loading code with eval()/exec() in a sandbox environment where the only modules imported are the ones we need for loading lambdas and derived-types.

Scanning for malicious imports

The second part to my solution is ensuring that no new modules are imported inside the stringified code that we interpret with eval()/exec().

For calls to eval(), the only way to import a module is to access one of the Python builtins or runtime variables that can import a module. Below, I have identified an exhaustive list of all such items that could be used to import a module. To ensure the safety of eval() commands, I search for any of these strings and stop execution if they are found.

• __import__(<module-name>): This runtime variable can be used like the import keyword to import a module via its name as a string
• globals()/locals(): Both of these builtins have access to a __load_module__ function that can be used in the same was as __import__ above

To secure a call to exec() we need to do the same process that was done above for eval() but we also need to search for a regular import statement of the form:

[from <some-place>] import <module-name> [as <some-alias>]

I have added that to the checklist for an exec() and will stop execution if any import occurrence is found in the same manner as I do for eval().

Wrapping up

I believe that the steps outlined above are sufficient for securing any usage of eval()/exec() for evaluating lambda functions and derived-types. If anyone can think of a security flaw that I haven't addressed please let me know.

@pratikbhd and @hlim1 to support this process, please collect a list of all potential items that need to be imported in the sandbox environment so that we can evaluate the lambda functions and execute the derived-types.

[cl4yton]

Two additional modules to look out for:
imp (now deprecated in since 3.4): https://docs.python.org/2/library/imp.html
importlib (now the favored replacement to imp): https://docs.python.org/3/library/importlib.html
Both provide some additional module loading and definition magic.
Other than that, @pauldhein, this looks good to me

[pauldhein]

@cl4yton thanks for the suggestions. I’ll add these to the list of “illegal modules” for or sandbox.

I also found another runtime variable named __loader__ that we will need to check for because it can also be used to import modules.

[hlim1]

@pauldhein I' just pushed the code for stringfying the derived type and add it into the _AIR.json file. However, there are 2 things that I would like to confirm. (1) Where this string's to be added? (2) What should we do with the existing lambdas file?

(1) We currently have a GrFN spec for a derived type like below:

"types": [
    {
      "name": "",
      "type": "type",
      "attributes": [],
    },
]

For example,

"types": [
    {
      "name": "@type::derived-types-03::@global::mytype_1",
      "type": "type",
      "attributes": [
        {
          "name": "a",
          "type": "integer"
        },
        {
          "name": "b",
          "type": "real"
        }
      ],
    }
]

I've added another attribute "code" right under "attributes" and added the stringfied code there like "code": "__stringfied_code__". I just want to confirm this is where you want it to be placed.

"code": "@dataclass\nclass soiltype:\n\tnlayr:\n\tslno:str\n\ttexture:List[str]\n\tsldesc:str\n\ttaxon:str\n\tcoarse:List[bool]\n\tales:float\n\tdmod:float\n\talphavg:List[float]"

This is the output of the above example that you gave.

(2) I have not removed the code for producing _lambdas.py file. Do you want me to remove the code so we don't produce the lambdas in the lambdas file or should I just leave it as it is for now?

[pauldhein]

@hlim1 great point, I forgot to address this (@pratikbhd you'll need to know this information too).

New argument locations

• Derived-types: Create a new argument named code in the type object at the same level as name, type, and attributes. (@hlim1 it sounds like you've already accomplished this).
• Lambda functions: Create a new argument named code in the function object of the corresponding lambda statement. It should be at the same level as name and type. @pratikbhd see the example below.

Lambda function example

{
  "function": {
    "name": "PETPT__petpt__assign__td__0",
    "type": "lambda",
    "code": "lambda tmax,tmin:(0.6*tmax)+(0.4*tmin)"
  },
  "input": [
    "@variable::tmax::-1",
    "@variable::tmin::-1"
  ],
  "output": [
    "@variable::td::0"
  ],
  "updated": []
}

Derived-type example

{
  "name": "@type::derived-types-03::@global::mytype_1",
  "type": "type",
  "code": "@dataclass\nclass soiltype:\n\tnlayr:\n\tslno:str\n\ttexture:List[str]\n\tsldesc:str\n\ttaxon:str\n\tcoarse:List[bool]\n\tales:float\n\tdmod:float\n\talphavg:List[float]",
  "attributes": [
    {
      "name": "a",
      "type": "integer"
    },
    {
      "name": "b",
      "type": "real"
    }
  ],
}

[pratikbhd]

@pauldhein This has been implemented as per the recent push to the dssat_pet branch. Could you verify this from your end?