Extend valid arguments for `derivative`

[gwhitney]

Currently (from the demo):

h(x)=x^2+3x+7
  h(x)

derivative(h,"x")
  TypeError: Unexpected type of argument in function derivative
     (expected: string or Node or number or boolean,
      actual: function, index: 0)

It would seem reasonable to extend derivative to apply to a "function" since it is not doing anything on such an argument now. Moreover, it would seem good when the argument is "function" for the return type to be "function" as well, so that one could do hypothetically:

h(x) = x^2+3x+7
  h(x)
dh = derivative(h, "x")
  dh(x)
dh(1)
  5 // 2x+3 evaluated at 1

Additionally, do you see any problem with making derivative take rawArgs so that e.g. derivative(x^2+3x+2,x) [Note: no quotes] would work as well? Are there cases when the user would want there to be evaluation in either argument that derivative couldn't figure out that it should do, or where the meaning would be ambiguous depending on whether derivative decided to do evaluation ?

Finally, is there any reason why the second argument, giving the variable, shouldn't be optional, and default to the lexicographically first free variable that appears in the first argument? With all of these changes, we could write derivative(x^2+3x+2) and get back 2*x+3 which would seem very reasonable.

[josdejong]

I think it makes sense to keep the algebraic derivative separate from a numeric one. When you have an arbitrary function as input (like h in your example), you can only calculate a numeric derivative. I guess we could do a trick and keep the parsed node tree attached to functions defined in the expression parser, though I'm afraid that will only be confusing in the end. (Or is that actually an interesting idea to think through?)

It would be great to be able to do enter derivative(x^2+3x+2,x) directly. I think we discussed that long ago in some issue, but I can't find it again.

Finally, is there any reason why the second argument, giving the variable, shouldn't be optional, and default to the lexicographically first free variable that appears in the first argument?

That would be a nice improvement. There is no special reason that it's not there :)

[gwhitney]

. I guess we could do a trick and keep the parsed node tree attached to functions defined in the expression parser, though I'm afraid that will only be confusing in the end. (Or is that actually an interesting idea to think through?)

Well, that's exactly what I was proposing. In fact, I assumed that was already happening. I guess by analogy with javascript itself, which keeps the source code for functions even once they've been byte-compiled. I don't see why in mathjs if the parse tree is kept on an appropriately named property of the function it will cause much if any trouble.

We must still reckon with the case that the user passes a non-mathjs-parsed function in through an external scope. For now I think derivative would punt unless/until we want to make an automatic differentiation package a dependency of mathjs. (One probably already exists, it is totally feasible.)

[josdejong]

🤔 yeah, it's an interesting idea.

To put a bit broader perspective on this idea: in general, it would be nice to be able to write expressions with symbolic computations as follows:

x = symbol()
h1 = x^2 + 3x
h2 = 3x + 7
h3 = h1 + h2
g = derivative(h3)

So, not writing with parse and providing a string, but really have the symbols and expressions as first class citizen, being able manipulate and work with them just like regular variables. If that would be possible, it's probably not needed to have derivative accept a function, since you'll probably start working in a different way. What do you think?

[gwhitney]

Well, with just very tiny tweaks (see the demonstration PR #2470), all of the ingredients already exist in mathjs for a mode of evaluation in which all unspecified variables are interpreted as symbols, so that "evaluation" of an expression will result in either a concrete value (a number or BigNumber or Fraction or Matrix or Complex or etc.) or in an expression (Node) containing (only) the unspecified, free variables of the input expression (besides constants, operators, and functions). So with very little work, we could offer an alternative function symbolicEvaluate which would operate as in your last example (but just assume that all unknown variables are symbols), or even replace the current behavior of evaluate with this alternate behavior (i.e., rather than throwing an error in the case of an undefined variable, return a Node expressing how the result of evaluation would depend on any remaining undefined variables).

The value of sometimes designating some variables as symbols and others as not is unclear to me... In other words, why should one have to do anything special to designate 'x' as a symbol when we just naturally write expressions with all kinds of symbols in them all the time, and mathjs naturally handles them symbolically? Or in still other words, it seems to me it just comes down to what happens when an undefined variable is encountered. Right now, we always throw an error. But we could (always, or just in a certain mode, indicated by a different function or a flag) just leave such a variable as a SymbolNode and continue on our merry way. (Of course, if there is truly a use case in marking just some variables as 'symbols' so that they will be allowed in the expression mode of evaluation results while those not designated will throw errors if encountered, that could be done, too -- it's just somewhat more work.)

So from that perspective, the point of #2470 is to show that it's unnecessary to add mathematical operations on Node objects (in addition to the numbers, BigNumbers, Complex, etc. that they are already defined on) to accomplish this mode of evaluation -- mathjs has symbolic manipulation embedded in its core already. (The basic idea is that resolve takes care of "evaluating" SymbolNodes to the extent that the scope dictates that they should be, and then all of the rest of the possible evaluation in light of what (if anything) remains undefined is handled by simplifyConstant.) If you look at the diffs in that PR, you will see they are really minimal. I mean, to flesh this out to something truly suitable for merging would require a bit more work (in particular, it would be pleasant to deal with the - and / not combining nicely with + and * unless you temporarily rewrite them), but it's not a major project.

For your convenience, I'll post the output of the demonstration example file examples/symbolic_evaluation.js as the next comment, so you don't have to pull the branch and run it if you don't want to.

To sum up: let me know if you'd like mathjs to go in any directions suggested by #2470, and/or if there's anything else in particular you'd like me to try in terms of making derivative (and similar symbolically-oriented expressions, like maybe rationalize) more comfortably integrated with mathjs and/or facilitate "working with symbols and expressions as first class citizens" in other ways.

[gwhitney]

> node examples/symbolic_evaluation.js
1. By just simplifying constants as fully as possible, using
the scope as necessary, we create a sort of "symbolic" evaluation:
Evaluating: 'x*y + 3x - y + 2' in scope {y: 7}
  --> Expression x * 7 + 3 x - 5

2. If all of the free variables have values, this evaluates
all the way to the numeric value:
Evaluating: 'x*y + 3x - y + 2' in scope {x: 1, y: 7}
  --> 5

3. It works with matrices as well, for example
Evaluating: '[x^2 + 3x + x*y, y, 12]' in scope {x: 2}
  --> Expression [10 + 2 * y, y, 12]
Evaluating: '[x^2 + 3x + x*y, y, 12]' in scope {x: 2, y: 7}
  --> [24/1, 7/1, 12/1]
(Note all the fractions because currently simplifyConstant prefers
them. That preference could be tweaked for this purpose.)

4. This lets you more easily perform operations like symbolic differentiation:
Evaluating: 'derivative(sin(x) + exp(x) + x^3, x)'
  --> Expression cos(x) + exp(x) + 3 * x ^ 2
(Note no quotes in the argument to 'derivative' -- it is directly
operating on the expression, without any string values involved.)

5. You can also build up expressions incrementally:
Evaluating: 'derivative(h3,x)' in scope {h3: h1 + h2, h1: x ^ 2 + 3 x, h2: 3 x + 7}
  --> Expression 2 * x + 6

6. Some kinks still remain at the moment. The scope is not affected
by assignment expressions, and scope values for the variable of differentiation
disrupt the results:
Evaluating: 'derivative(x^3 + x^2, x)'
  --> Expression 3 * x ^ 2 + 2 * x
Evaluating: 'derivative(x^3 + x^2, x)' in scope {x: 1}
  --> Expression derivative(2, 1)
(We'd like the latter evaluation to return the result of the
first differentiation, evaluated at 1, or namely 5. However, there is not (yet)
a concept in mathjs (specifically in 'resolve') that  'derivative' creates a
variable-binding environment, blocking off the 'x' from being substituted via
the outside scope within its first argument.)

But such features can be implemented.

[josdejong]

ahhh, that is really cool 😎 . So the "trick" you've implemented here is to introduce a flag unwrapConstants which makes it possible to resolve expressions like h3 = h1 + h2, where h1 and h2 are expressions too, right? It is a really smart idea, especially since this is such a simple approach.

I was thinking in a bit of a different direction: implementing first-class support for Nodes in all mathjs functions. That would be much more work to implement: we will have to extend each of the functions of mathjs with a new signature which allows handling Node instances. I.e.:

// pseudo code...

export const createAdd(...) {
  return typed('add', {
    // ...
    'Node, Node': (a, b) => {
      return new OperatorNode('+', 'add', [a, b])
    }
  }) 
}

export const createSqrt(...) {
  return typed('sqrt', {
    // ...
    'Node': (x) => {
      return new FunctionNode('sqrt', [x])
    }
  })
}

// ...

I have the feeling that from a practical point of view, both approaches would allow a user to do the same thing, it's only implemented in a different way. At this point I'm still a bit in search on what would be the use case. Having a clear use case could help to think though if there is added value in one or the other.

I finally found the earlier discussions on this, see: #38, #218 (different I think), #1732, #1747

[gwhitney]

Well, resolving and simplifying is the "on the cheap" approach, avoiding having to deal with arithmetic on expressions, since it didn't seem like it could possibly be worth it to modify every function's source file to add Node signatures to the typed-function. Plus the "proof of concept" you linked to having build problems suggesting an issue with circularity kind of scared me off. But this "cheap" approach is also not quite as powerful -- it doesn't allow any functions not built for handling symbolic expressions suddenly to do so. In particular, even if the source code for det only uses math.XXX functions like multiply and add internally, the approach in the demo PR won't magically let you compute a symbolic determinant as envisioned in #1732. Only your suggested approach, making Node another data type that the primitives can deal with, and so functions which are defined only in terms of the primitives just get them "along for the ride", would do that. But then it seems to me there should be a more generic way to do it, since all of the primitives simply operate on Nodes by tree building; they don't actually do any computation. The same is true of det() on top of those primitive tree-building operations: it won't give the answer @m93a wanted by itself, it will give huge tree if blindly executed in terms of tree-building primitives. All of the real computation in a symbolic determinant comes in simplify() anyway, which is in the end how I ended up interested in derivative(), because of my interest in simplify, and derivative being an inherently symbolic operation. So anyhow, I figured I would see how far I could get with just the simplify() part, so to speak, and the answer is fairly far.

I guess what I am coming to is that the two approaches are not necessarily in opposition, but are rather complementary to each other.

I don't currently have a goal in my other projects for which this style of symbolic computation is a must, so I am afraid I don't have a use case. I was offering this in the spirit of extending mathjs sphere of utility, and then it may attract use cases that we can employ as a guide to further development.

So anyhow, let me know if there's a direction you want me to pursue in this arena.

[gwhitney]

Actually my last comment got me to thinking that the "symbolic" implementation of all of the standard functions is identical (except for the function name) and doesn't need any knowledge of the operation -- it just builds more tree structure. Thus, it can be implemented generically if typed-function allows a handler when no signature matches, without defining any new cases for any existing operators. I've followed up that idea with an alternate demo implementation of symbolic computation, #2475, for comparison purposes. For convenience, I will post the output of its version of the symbolic_computation example, so you can see the pros and cons.

Personally, I think one approach or the other would make a natural extension to mathjs that would help it move further toward a bona fide CAS. On the other hand, I don't want to push a noodle here: I don't have an immediate practical need for for either approach, just a strong feeling that if we have symbolic operations like derivative, they ought to be less clumsy to take advantage of than they currently are. So anyhow, let me know if you'd like mathjs to pursue either approach or some variant thereof to the point of actually being suitable for merging. Otherwise, I will leave this topic be until there is a drive for it.

[gwhitney]

> node examples/symbolic_evaluation.mjs
1. By just evaluating all unknown symbols to themselves, and
providing a typed-function handler that builds expression trees when there is
no matching signature, we implement full-fledged symbolic evaluation:
Evaluating: 'x*y + 3x - y + 2' in scope {y: 7}
  --> Expression x * 7 + 3 * x - 7 + 2

2. If all of the free variables have values, this evaluates
all the way to the numeric value:
Evaluating: 'x*y + 3x - y + 2' in scope {x: 1, y: 7}
  --> 5

3. It works with matrices as well, for example.
Evaluating: '[x^2 + 3x + x*y, y, 12]' in scope {x: 2}
  --> [10 + 2 * y, y, 12]
Evaluating: '[x^2 + 3x + x*y, y, 12]' in scope {x: 2, y: 7}
  --> [24, 7, 12]
(Note there are no fractions as in the simplifyConstant
version, since we are using ordinary 'math.evaluate()' in this approach.)

4. However, to break a chain of automatic conversions that disrupts
this style of evaluation, it's necessary to remove the former conversion
from 'number' to 'string':
Evaluating: 'count(57)'
  --> TypeError: Typed function type mismatch for count called with '57'
(In develop, this returns 2, the length of the string representation
of 57. However, it turns out that with only very slight tweaks to "Unit," 
all tests pass without the automatic 'number' -> 'string' conversion,
suggesting it isn't really being used, or at least very little.

5. This lets you more easily perform operations like symbolic differentiation:
Evaluating: 'derivative(sin(x) + exp(x) + x^3, x)'
  --> Expression cos(x) + exp(x) + 3 * x ^ 2
(Note no quotes in the argument to 'derivative' -- it is directly
operating on the expression, without any string values involved.)

6. Doing it this way respects assignment, since ordinary evaluate does:
Evaluating: 'f = x^2+2x*y; derivative(f,x)'
  --> [2 * (x + y)]

7. You can also build up expressions incrementally and use the scope:
Evaluating: 'h1 = x^2+5x; h3 = h1 + h2; derivative(h3,x)' in scope {h2: 3 * x + 7}
  --> [2 * x + 8]

8. Some kinks still remain at the moment. Scope values for the
variable of differentiation disrupt the results:
Evaluating: 'derivative(x^3 + x^2, x)'
  --> Expression 3 * x ^ 2 + 2 * x
Evaluating: 'derivative(x^3 + x^2, x)' in scope {x: 1}
  --> TypeError: Typed function type mismatch for derivative called with '2, 1'
(We'd like the latter evaluation to return the result of the
first differentiation, evaluated at 1, or namely 5. However, there is not (yet)
a concept in math.evaluate that  'derivative' creates a variable-binding
environment, blocking off the 'x' from being substituted via the outside
scope within its first argument. Implementing this may be slightly trickier
in this approach since ordinary 'evaluate' (in the absence of 'rawArgs'
markings) is an essentially "bottom-up" operation whereas 'math.resolve' is
more naturally a "top-down" operation. The point is you need to know you're
inside a 'derivative' or other binding environment at the time that you do
substitution.)

Also, unlike the simplifyConstant approach, derivative doesn't know to
'check' whether a contained variable actually depends on 'x', so the order
of assignments makes a big difference:
Evaluating: 'h3 = h1+h2; h1 = x^2+5x; derivative(h3,x)' in scope {h2: 3 * x + 7}
  --> [3]
(Here, 'h1' in the first assignment evaluates to a
SymbolNode('h1'), which ends up being part of the argument to the eventual
derivative call, and there's never anything to fill in the later definition
of 'h1', and as it's a different symbol, its derivative with respect to 'x'
is assumed to be 0.)

Nevertheless, such features could be implemented.