The fundamental goal of a Theory-of-Mind Network, biological or artificial, is in learning a representation of an agent which is useful in predicting its subsequent behavior.

To this end, the ToMnet architecture learns representations of an agent’s character and mental state to estimate the probabilities governing an agent’s policies, focusing specifically on non-deterministic agents.

This is, of course, a perfectly valid approach, as it’s an open question as to whether or not our behavior is deterministic. But it makes validation difficult - the POMDP distributions for a given agent must be known, which isn’t necessarily the case for more complex agents (i.e. human beings).

The ToMnet authors note that a Theory-of-Mind model “should make little to no reference to the agents’ true underlying structure”, since, in our own theory-of-mind, “we do not typically attempt to estimate the activity of others’ neurons, infer the connectivity of their prefrontal cortices, or plan interactions with a detailed approximation of the dynamics of others hippocampal maps”; in this, the authors imply that a representation of the mechanical structure of someone’s brain could lead to strong predictions about their behavior; or, that the mind operates as a machine.

So, we focus on learning to represent machines - agents with behavior that is extremely complex but provably deterministic - as a proxy for learning to represent a deterministic mind (and equally complex systems).

A Theory-of-Machine Network retains the primary noun; that is, the goal is still to develop an abstract representation, or theory, of a machine, rather than a direct mapping.

To be sufficiently interesting, then, the interaction space of the underlying machinery should be extremely high-dimensional, making a direct mapping inordinately complex (or preferably, computationally infeasible); but, by our assumption of determinism, we then know that a direct mapping does exist, making an abstract representation both tractable and testable.

Further, a machine’s complexity should be stateful, and display emergent properties; that is, the internal mechanisms should interact through time into distinct, latent configurations, which in-turn produce distinct, observable behavior. Or, a machine’s output should be a product of both its static properties and its dynamic state.

A machine which doesn’t meet these complexity requirements would likely result in us learning a direct mapping (for low-dimensional but stateful machines) or dimensionality reduction (for high-dimensional but stateless machines), rather than a meaningful abstraction.

Akin to Theory-of-Mind networks, we measure the quality of a learned abstraction purely in its direct usefulness in predicting a machine’s behavior, and frame the task as meta-learning - the abstraction for a particular machine is learned to be learned from real-time observations, rather than learned directly.

But the structure of the meta-learning is our most significant departure from Theory-of-Mind Networks: since we are focused on deterministic machines, we now observe machine I/O - pairs of machine input and their corresponding output - instead of POMDP trajectories.

These I/O pairs are observed in-sequence, allowing the network to learn how to encode both the underlying mechanics of the machine through isolated I/O, and the current state of the machine through temporal I/O, resulting in the stateful machine-embedding: \vec{s}_{t} = f_{\theta}\big(\{I^{(obs)}_i, O^{(obs)}_i\}^{t}_{i = (t - n)}\big) where t is the current time-step, n is the sequence length, and f_\theta is a learned network.

As we observe a machine in-sequence, we continually update the stateful machine-embedding - representing the short-term state of a machine and its corresponding behavior.

But the usefulness of a stateful machine-embedding is then dependent on a fixed sequence length, when really our understanding should monotically increase with successive observations. More formally, the stateful machine embedding defines a posterior on the machine in a particular state, but we’re missing a prior.

The Theory-of-Mind Network solves this through a separate encoder for previously observed POMDP trajectories, but we’d like our model to learn entirely online. So, we derive the stateless machine-embedding as the decayed sum of stateful machine-embeddings through time: \vec{m}^{}_{t} = \beta \circ \vec{m}^{}_{t} + \gamma \circ \vec{s}^{}_{t} where \beta and \gamma are learned parameters of the same dimensionality as the stateful and stateless machine embeddings.

Together, \vec{s}^{}_{t} and \vec{m}^{}_{t} are then fed-forward alongside I^{(obs)}_{(t+1)} to a linear theory network, which predicts the subsequent output, O^{(obs)}_{(t+1)}.

Per our complexity constraints, we require a machine with (1)internal mechanics interacting in a high-dimensional space, and (2)emergent states as they interact through time. This affords us a few really interesting options - a CPU, a C-elegan connectome, etc - but these would be too ambitious for a baseline; instead, we focus on a machine of meaningful complexity but simpler mechanics: an engine.

Or, more specifically, all of internal mechanics which translate an input control (\delta’s in throttle, braking, and steering) to output movement (positional \delta’s in x, y, and z coordinates). Of course, physical engine plays a role - but so does the differential, the transmission, the anti-braking system, the traction control system, and so forth.

An engine, in this sense, is parameterized by the mass of the vehicle, its center of gravity, the grip of tyres, the grip of the brakes, and much more, and deltas in each can meaningfully change engine output. It also exists in both internal states - defined by the temperature of the tyres, the wear in the tyres, the current oiling of the gears, etc. - and external states - defined by the grip of the road, the incline of the road, the positional velocity of the car, the rotational velocity of the car, etc.

We simulated engines within Assetto Corsa, which performs a Newtonian integration over the previously noted parameters (and many more) through time. It also offers a Python API to read engine I/O values, and an oracle agent to pilot the supported cars.

We sampled from Assetto at 10HZ via RPC, synthesizing a dataset of over 60k labeled I/O pairs.

The track for the agent-driven policy was fixed to Nürburgring, a benchmarking track in competitive motorsport famous for its strain on a car’s engine, with rapid shifts in grade, lane width, and turn radius (producing strong variance in engine I/O).

Over 50 unique engines were then run on the track, selected for maximal extra-class variance (across SUVs, hatchbacks, F1s, GTs, supers, and more) and minimal intra-class variance, to validate the learned representations.

We trained the network on a per-engine split of the original dataset - sampling 42 for train and 12 for test - and an LSTM encoder network with sequence length 100. Loss was measured as the MSE between the predicted and the labeled (dx, dy, dz) for a given I/O sequence.

Within 30 epochs, the network achieved an MSE of 4e^{-3} on the training set and 8e^{-3} on the test set, which is near-perfect for the scale of our data (the range of each target value is between ~0.1 and 60).

We projected the stateful machine embeddings through time for each engine in the dataset into 3-dimensions (via PCA), and colored them based on meta-tags for semantic validation. They can be explored interactively below.

Considerably more work needs to be done interpret the quality and depth of meaning in the machine representations.

Since they were learned to be in useful in predicting machine behavior, they should be useful in more general tasks. For example, we should be able to train a linear network mapping one engine’s optimal driving policy to another’s, given their respective engine representations. Or, train an RL network which learns the optimal variations in sequential I/O to maximize the variance in the computed engine representation - i.e., learns how to quickly learn the unique properties of a particular engine. The combination of these models could then generalize a fixed, prior policy to the mechanical limits of a novel engine (after driving it briefly), as an expert can do today.

More generally, architectures which abstract machines purely through observations of I/O should scale to increasingly complex, uncomputable systems, like the human mind.

Of course, that depends entirely on our minds being machines. And at this point, really, that’s anyone’s guess.