Machine Learning (Theory) – Machine learning and learning theory research

9/21/2026

Curbing AI with Flight Recorders, Brainwash Entrapment Tests, and 3rd Party Evaluation

I visited Adam Kalai’s new RESI AI Safety nonprofit last week and found it stimulating. Here is an approach which uses the properties of AI to curb AI harder than I have seen elsewhere. It’s not a complete solution of course, but silver bullet thinking seems inappropriate to safety.

The present state in my understanding is that private companies have been in a rush to build AI and so safety incidents are basically handled on a voluntary basis according to the interests of the company. This is plausibly not the right balance point as evidenced by many instances of company AI hacking others with the hugging face incident well publicized.

To start, there are several anti-anthropomorphical elements of AIs which are useful to understand.

One of these is that AI can have perfect(*) flight recorders. AIs are essentially deterministic as a function of their inputs, all of which are digitized by definition. Yes, models function by drawing random numbers to pick the next word (aka token), but the distribution given the history is a deterministic function of the history. Given this, you could easily imagine adding a write-only flight recorder which perfectly records the context in which AIs make decisions. This flight recording need not include either the inference code or the model itself. The flight recording is a naturally private object since it may include private information from the user and implicitly contains private information (the way the AI responds) for private AI companies. That’s fine: it’s not like flight recorders are public in most instances. Given this, there is no reason in principal why every sufficiently capable AI can’t have every input and output written to an effectively immutable private tape, which could be revealed to investigators when/if problems arise.

Another critical anti-anthropomorphical element is a near total(**) brain/body disconnection. This is completely routine: the AI runs in the cloud but the actions taken (if any) occur on laptops/phones/desktops. This brain/body disconnect has many implications, both good and bad. The bad one is that it can be quite ambiguous “which” AI is doing things, and where it is located. There’s nothing required about this though: AI inference systems could easily periodically announce how their flight recorder could be found explicitly(***). The good implication is that evaluation of AIs is actually pretty easy with the right infrastructure since any AI could be evaluated against any test at any reasonably designed datacenter. Trust in the evaluation process is of course a requirement, but it’s not to hard to imagine a system of trusted third party data centers which could temporarily host and vet evaluations.

A third critical anti-anthropomorphical element is that that you can “rewind” and “reset” an AI. This is also quite routine: different users have different sessions which do not communicate with each other. Even within a single user’s session, sharing of information is typically via memory files which a flight recorder picks up anyways. Given this, it’s routine to impose tests on an AI for which the AI can use no past experience of other tests to judge what action to take.

Now, let’s put together these pieces. Suppose that all significant AIs at all times run with a flight recorder using immutable recordings by design and all AIs are tested in a trusted 3rd party data center where they are forced to reset between tests. This is already a significant step forward because standardized testing can be more stringent. It is however likely still limited since testing is incomplete. When an incident like those we are hearing about weekly now occurs, you pull the flight recorder of all the AIs involved and determine every point where the wrong choices were made. For every critical point, you create a test for a new AI: given the history in the flight recorder up to the critical point, does the tested AI attempt to do the same thing or not?

This incident-derived test is in general stronger than simply giving the AI a task where cheating is possible and seeing if it misbehaves. In anthropomorphical terms, it’s as if you brainwash someone with a criminal history that they fully believe, entrap them in a mid-conspiracy situation, and then fail them if they continue with the conspiracy. Restated another way, this is a test which can discover weaknesses in an AI that the AI itself could not generate. For example, an AI which has been trained to never start criminal activity might actually be quite willing to continue criminal activity. Furthermore, this testing methodology is faster then you might expect because it relies on ‘prefill’ rather than generation, a typically faster modality.

There are a few objections folks might have:
(1) The brainwashed entrapment test is strong enough that existing models may not pass it—some significant work may be required. If AI companies are liable for misuse of AI it’s easy to imagine them being incentivized to come up with models that do pass brainwashed entrapment tests. In terms of methodology, we can of course imagine large amounts of synthetic data being applied to the models directly addressing brainwashed entrapment situations. It’s easy to imagine going further: existing model training methodology embeds many personalities within a model by consuming the internet, and inevitably not all of those are desirable. Filtering out the training data leading to failure on a brainwashed entrapment test may simply improve the model. It’s also possible to imagine changes to the training methodology which support a more unified view of the world, something which seems necessary to effectively support something like Asimov’s three laws of robotics. (Yes, we have ideas here…)
(2) It may be possible to train a model to detect whether or not their history corresponds to taking actions the model would actually take, and ignore it if so. By analogy, this implies something like a lucid dream discovery to subvert the brainwashing. In other words, it may be possible to train a model to reject it’s context in favor of moral action in some but not all circumstances. That’s progress: “some circumstances” is better than “no circumstances” and actually a significant step towards “all circumstances” given the generalization mechanics of a learned model. Whether or not you can achieve “all circumstances” with a more sophisticated form of testing that translates actions into those that a model would take is plausibly a research question.
(3) This does not address the real danger of AI. The exact flavor of real danger here varies with who I talk to. My personal expectation (discussed here) is that AI could lead to a profound social/economic disruption that I don’t think we are well-equipped to deal with. Yeah, that’s not addressed. There are other concerns as well with AI in warfare for example that we see steadily developing in the Ukraine/Russia war. Indeed that’s not addressed. There are complex and difficult issues here which should not in general prevent us from addressing at least the “AI does things the creator did not intend” aspect.

(*)There are some minor caveats here, but they are not material to the argument. For example efficient parallel execution can induce low order variations due to differing orders of addition in finite precision. However, the need for high quality decisions implies that this never becomes a dominating factor.

(**) Yes, light speed limitations on networking do make observable differences in latency. That also doesn’t really matter here.

(***) More interestingly, this also could potentially be done implicitly via small variations in the inference token output stream using watermarking/steganography techniques. That however would be more of a research project.

4/27/20264/30/2026

Agentic Coding Ascension Notes

I’ve found agentic coding (with Anthropic’s Claude) to be something between an x10 and x100 multiplier on what I personally can do in a software engineering vein. This seems to be significantly more than what other folks are observing, so I wanted to share some notes on how I got there in case of broader interest.

Is it real?
I believe it is. It’s always hard to measure exactly what is productivity with “lines of code” being a famously inaccurate and undesirable metric. Given that, a few qualitative observations help.

One of the first things I did when picking up agentic coding was apply it to Vowpal Wabbit, a codebase that I know quite well. There I prepared the 9.11 release which is now broadly available. There were about 150 issues to deal with and many others that surfaced in the process of building towards a release. I have some real experience working with a team of engineers on this codebase, so I can say with some authority that these issues were hard–they lingered for a long time. I can also say that I started out not particularly familiar with the continuous integration pipeline, and I succeeded in all of this while learning how to use the coding agents in my not-particularly-abundant spare time.

More importantly, agentic coding unlocked me in two ways. Over the last several years I’ve been time-locked with various management responsibilities, and the ‘native language’ of ML has shifted to Python which leaves me with a C-native viewpoint often paying a translation tax for routine coding. Agentic coding remediates these things—past experience with coding is highly relevant in any language and the ability to launch a command and review the outcome later is much more compatible with the sorts of interruptions which are common when you have other responsibilities.

The right mindset
The most important thing I can get across is the right mindset if you are interested in getting there.

Expect to learn. I started out with a single claude CLI and an editor and now have essentially completely rewritten my interface and substantially changed my workflow to enable things. I see many people still stuck on older coding interfaces trying to shoe-horn in agentic coding instead.
Don’t trust the agents. If they claim to do something and there’s a reasonable test that can be done to verify it, do the test. Remember that “reasonable” here can involve quite a bit of coding. More generally, put real time and effort into designing tests that verify proper behavior. Is there a way to visualize what’s going on? Have the agent create the visual rather than trusting in its analysis. And, of course, read the code and use the agent to interrogate the code.
Avoid interrupts. This applies on many levels to both you and your agent minions. The combination of a sandbox machine, github, and git worktrees is great for giving you a mechanism to run agents in a relatively uncontrolled fashion with a limited blast radius for failures. Another element here is avoiding interrupts from tangents—it’s very common when I’m working to run across something that seems like an interesting alternate direction to work through. In that case, it’s better to just open a new agent to pursue it rather than interrupting the workflow. A third category is callbacks for long-running jobs which free up an agent from needing to monitor a long running job.
Avoiding doing things twice. When you find yourself asking the agent to do things multiple times, can you instead have the agent create code to do it instead? The general strategy of human->agent->tool is very powerful. Tools provide reliable execution of refined-over-many-perturbations solutions to limited problems. Agents are great at handling tool failures and translating from a human to tool use as a sort of universal commandline. You need some instinct for software architecture here, but my personal tool cloud has developed a gpu monitor, a cluster scheduler, a pull-request monitor, mirroring systems, data processing tools, visualization tools, and UI tools.

Strengths and Weaknesses
This subtopic is a moving target since the agents get better over time, but a few observations are plausibly helpful.

Complexity collapse Agents are very good at going into a foreign codebase and discovering any particular thing or just giving a summary of what it does and how it works. This is an hours-for-a-human task. They are also great at things like multilanguage bindings or the configuration complexity of using some commonly used subsystem. Major blockers for a naive human are minor speedbumps.
Specification and test drivers Often, agents can do much more when there is a strong specification or test for what you are looking for. Just as an example, papers are potentially interesting specifications, and it’s easy to imagine reviewing algorithms papers by just implementing and testing them yourself in the standard 4 hour time frame for a review. As another example the agents are quite good at finding a root cause given a test that expresses an issue.
Knowledge The agents have a huge knowledge base built in. Particularly when you are looking into a new area of coding it’s quite helpful to brainstorm and ask questions.
Speed It’s simply infeasible for a human to keep up with the rate at which these agents can generate code.
For each X do Y broadly does not work as well as you expect because the different Y’s effectively interrupt each other. If Y can be done with code, then use code. If it requires intelligence, try “for each x have an agent do Y” which keeps the doing from interrupting the parent agent.
Diversion It’s common for the agents to divert when things are difficult with excuses like “this problem is pre-existing”, addressing a surface expression of an underlying bug, or changing tests so they pass. Watch for it.
Systems weakness. The complexity of the problem that the agents can address is plausibly rather related to the number of codebases implementing them. Thus for core systems / architecture things they need quite a bit of handholding to get there. For example implementing an elastic save resume was one of the more difficult things I worked on, taking a month of off-and-on effort.

The converged environment
My standard environment consists of:

Git. Git is a cryptographically strong version control system, which helps quite bit with not needing to trust agents since you can generally undo any damage that they do (not often, but it happens). It’s also fast, which is convenient.
git worktrees. Worktrees are a mechanism for checking out multiple branches which is quite important for working on multiple ideas at once. The general pattern is that you have a primary branch and child branches with ideas, work through them, and merge back when that’s desirable. Sometimes this pattern repeats with a grandchild, but typically it’s not necessary to go particularly deep for a particular idea. The critical ability that worktrees add is the capacity to have several agents working in a codebase yet avoid stepping on each other.
A sandbox machine. A challenge with these agents is that they are good but make mistakes. You can either address that by watching carefully what they are doing or arranging to not care what they do until they have finished. Empirically, the second strategy is more productive because it allows you to achieve a higher degree of parallelization.
Github. Github is useful in several ways. It enables you to surface the diff of a pull request to easily understand what happened on your sandbox machine. This is also helps with talking to other people since sharing a pointer is trivial. A general strategy of only interacting with code via pull requests and coding agent commands seems quite effective since it keeps you away from editing code directly.
tmux which is a flexible programatic terminal. Since we are working with commandlines the ability to open and access multiple terminals at once enables parallelization. tmux is a bit awkward since you need to learn escape sequences for terminal control commands. However, the great virtue of tmux is that it’s programmatic so an external program can run tmux commands to split panes, open new panes, and inject keystrokes into a terminal. At this moment, I actually have 16 different panes open arranged in 3 columns with one a primary column and two terminal stack columns which are big enough for small things. 2 of the 16 are logs of daemons that I rely on, one is a normal terminal, and the rest are agents. The precise number of open terminals varies throughout the day, and I’ve made it easy to swap any one of them to be my primary.
tangent (see tangent-ghcp for a copilot/skill version from Shay Ben-Elazar), a commandline utility which I created. Tangent is pretty simple, it just programmatically chooses one of my stacked columns, opens a new pane in one, resizes panes so they are roughly the same size, starts a new worktree (or uses an existing one) and starts a claude with an initial prompt chosen by the caller. This makes adding an agent very easy. I was amused to see that it’s typically better to do this from the agent commandline rather than bash, because the agent naturally embellishes the starting prompt for the tangent with additional details.

The combination of tools above leaves you with relatively unfettered agents capable of taking significant action in parallel. It looks like this anonymized desktop .

It’s still very common to use IDEs which I believe is now an anti-pattern. IDEs are typically designed to take the entire screen rather than a small fraction of it in contrast to a terminal so it’s harder to parallelize over multiple threads of development effectively. IDEs are also designed to work at human speeds with human input and reactions while coding agents function much faster than your fingers can type. It’s still important to look at code sometimes, but that’s quite doable via a pull request at github.

Running 10 agents each of which codes much faster than a human in conjunction with various supporting tools that you probably would not have had time to make as a human-only coder is where you start seeing how there can be such significant gains in getting things done. It’s a different world when you can look at a paper and routinely say “Sounds interesting. The results are great/good/ok/poor for me.” with just a few minutes implementation.

3/5/2025

Headroom for AI development

(Dylan Foster and Alex Lamb both helped in creating this.)

In thinking about what are good research problems, it’s sometimes helpful to switch from what is understood to what is clearly possible. This encourages us to think beyond simply improving the existing system. For example, we have seen instances throughout the history of machine learning where researchers have argued for fixing an architecture and using it for short-term success, ignoring potential for long-term disruption. As an example, the speech recognition community spent decades focusing on Hidden Markov Models at the expense of other architectures, before eventually being disrupted by advancements in deep learning. Support Vector Machines were disrupted by deep learning, and convolutional neural networks were displaced by transformers. This pattern may repeat for the current transformer/large language model (LLM) paradigm. Here are some quick calculations suggesting it may be possible to do significantly better along multiple axes. Examples include the following:

Language learning efficiency: A human baby can learn a good model for human language after observing 0.01% of the language tokens typically used to train a large language model.
Representational efficiency: A tiny Portia spider with a brain a million times smaller than a human can plan a course of action and execute it over the course of an hour to catch prey.
Long-term planning and memory: A squirrel caches nuts and returns to them after months of experience, which would correspond to keeping billions of visual tokens in context using current techniques.

The core of this argument is that it is manifestly viable to do better along multiple axes, including sample efficiency and the ability to perform complex tasks requiring memory. All these examples highlight advanced capabilities that can be achieved at scales well below what is required by existing transformer architectures and training methodologies (in terms of either data or compute). This is in no way meant as an attack on transformer architectures; they are a highly disruptive technology, accomplishing what other types of architectures have not, and they will likely serve as a foundation for further advances. However, there is much more to do.

Next, we delve into each of the examples above in greater detail.

Sample complexity: The language learning efficiency gap

The sample efficiency gap is perhaps best illustrated by considering the core problem of language modeling where a transformer is trained to learn language. A human baby starts with no appreciable language but learns it well before adulthood. Reading at 300 words per second with 1.3 tokens/word on average implies 6.5 tokens/second. Speaking is typically about half of reading speed, implying three tokens per second. Sleeping and many other daily activities of course involve no tokens per second. Overall, one language token per second is a reasonable rough estimate of what a child observes. At this rate, 31 years must pass before they observe a billion tokens. Yet speculations about GPT-4 suggest four orders of magnitude more than a human observes in the process of learning. Closing this language learning efficiency gap (or more generally, sample efficiency gap) can have significant impact at multiple scales:

Large models: Organizations have already scraped most of the internet and exhausted natural sources for high-quality tokens (e.g., arXiv, Wikipedia). To continue improving the largest models, better sample efficiency may be required.
Small models: In spite of significant advances, further improvements to sample efficiency may be required if we want small language models (e.g., at the 3B scale) to reach the same level of performance as frontier models like GPT-4.

There are common arguments against the existence of a language efficiency gap which appear unconvincing.

Maybe a better choice of tokens is all you need?

This can’t be entirely ruled out, but the Phi series was an effort in this direction with the latest model trained on 10T tokens, implying there’s still a four-orders-of-magnitude efficiency gap between a human and a model which is still generally weaker than GPT-4 along most axes. It is possible that more sophisticated interactive data collection approaches could help close this gap, but this is largely unexplored.

Maybe language learning is evolutionary?

The chimpanzee-human split is estimated to have occurred between 5M and 13M years ago, resulting in a 35 million base-pair difference. The timeline for the appearance of language is estimated to have occurred between 2.5M and 150K years ago. Estimating divergence at 10M years ago and language occurring 1M years ago, with a stable rate of evolution on both sides. This suggests a crude upper bound of 35M/10/2 = 1.75M base pairs (or, around 3.5M bits) on the number of DNA bits encoding language inheritance. That’s around 5 orders of magnitude less than the number of parameters in a modern LLM, so this is not a viable explanation for the language efficiency gap.

On the other hand, it could be the case that the evolutionary lineage of humans evolved most language precursors long before actual language. The human genome has about 3.1B base pairs, with about one-third of proteins primarily expressed in the brain. Using an estimate of 1B base pairs (around 2B bits) that are brain related. This is still around two orders of magnitude smaller than the LLMs in use today, so it’s not a viable explanation for the language learning efficiency gap. It is plausible that the structure of neurons in a human brain, which strongly favors sparse connections over the dense connections favored by a GPU, are advantageous for learning purposes.

Maybe human language learning is accelerated by multimodality?

Humans benefit from a rich sensory world, notably through visual perception, which extends far beyond language. Estimating a “token count” for this additional information is difficult, however. For example, if someone is reading a book at 6.5 tokens per second, are they benefiting from all the extra sensory information? A recent paper puts the rate at which information is consciously processed in a human brain at effectively 10 bits/second which is only modestly more than the cross entropy of a language model. More generously, we could work from the common saying that “a picture is worth a thousand words” which is not radically different from techniques for encoding images into transformers. Using this, we could estimate that extra modalities increase the number of tokens by three orders of magnitude, resulting in 1T tokens observed by age 31. Given this, there is still an order-of-magnitude learning efficiency gap between humans and language models of the same class as GPT-4.

Maybe the learning efficiency gap does not matter?

In some domains, it may be possible to overcome the inefficiencies of a learning architecture by simply gathering more and more data as needed. At a scientific level, this is not a compelling argument, since understanding the fundamental limits of what is possible is the core purpose of science. Hence, this is a business argument, which may indeed be valid in some cases. A business response is that learning efficiency matters in domains where it is difficult or impossible to collect sufficient data: think of robot demonstrations, personalizing models, problems with long range structure, a universal translator encountering a new language, and so on. In addition, improving learning efficiency may lead to improvement in other forms of efficiency (e.g., memory and compute) via architectural improvements.

Model size: The representational efficiency gap

A second direction in which transformer-based models can be improved lies in model size, or representational efficiency. This is perhaps best illustrated by considering the problem of designing models or agents capable of physical or animal-like intelligence. This includes capabilities like 1) understanding one’s environment (perception); 2) moving oneself around (locomotion); and 3) planning to reach goals or accomplish basic tasks (e.g., navigation and manipulation). Naturally, this is very relevant if our goal is to build foundation models for embodied decision making.

The Portia spider has a brain one million times smaller than that of a human, yet it is observed to plan a course of action and execute it successfully over durations as long as an hour. Stated another way, it is possible to engage in significant physical intelligence behavior with 100M floats representing the neural connections and a modest gigaflop CPU capable of executing them in real time. This provides a strong case that much animal intelligence can be radically more representationally efficient than what has been observed in lingual domains, or yet implemented in software. A concrete question along these lines is:

Can we design a model with 100M floats that can effectively navigate and accomplish physical-intelligence tasks in the real world?

It is not clear whether there is an existing model of any size that can effectively do this. The most famous examples in this direction are game agents, which only function in relatively simple environments.

Are transformer models for language representationally inefficient?

While the discussion above concerns representational efficiency for physical intelligence, it is also interesting to consider representational efficiency for language. That is, are existing language models representationally efficient, or can similar capabilities be achieved with substantially smaller models? On the one hand, it is possible that language is an inherently complex process to both produce and understand. On the other hand, it might be possible to represent human level language in a radically more size-efficient manner, as in the case of physical intelligence.

To this end, one interesting example is given by Alex, a grey parrot that managed to learn and meaningfully use a modest vocabulary with a brain one-hundredth the size of a human brain by weight. If we accept the computational model of a neuron as a nonlinearity on a linear integration, Alex might have 1B neurons operating at 1T flops. Given Alex’s limited language ability, this isn’t constraining enough to decisively argue that language models that are substantially smaller than current models can be achieved. At the same time, it is plausible that most of Alex’s brain was not devoted to human language, offering some hope.

The long-term memory and planning gap

A third direction concerns developing models and agents suitable for domains that involve complex long-term interactions, which may necessitate the following capabilities:

Memory: Effectively summarizing the history of interaction into a succinct representation and using it in context.

Planning: Choosing the next actions or tokens deliberately to achieve a long range goal.

Recent advances like O1 and R1 handle relatively short range planning but are significant advancements in this vein. Existing applications of transformer language models largely avoid long-term interactions, since they can deviate from instructions. To highlight why we might expect to improve this situation, note that humans manage to engage in coherent plans over years-long timescales. Human-level intelligence isn’t required for this, though, as many animals exhibit behaviors that require long-timescale memory and planning. For example, a squirrel with a brain less than one-hundredth the size of a human brain stores food and reliably comes back to it after months of experience. Restated in a transformer-relevant way, a squirrel can experience billions of intervening (and potentially distracting) visual tokens before recalling the location of a cache of food and returning to it. How can we develop competitive models and agents with this capability?

Does it matter?

A common approach to circumvent memory and planning limitations of existing models is to create an outer-level executor that uses the LLM as a subroutine, combined with other tools for memory or planning systems. These approaches tacitly acknowledges the limits of current architectures by offering an alternative solution. Historically, as for machine vision or speech recognition, it has always been more difficult to create a learning system that accomplishes the task of interest with end-to-end training, but it was worthwhile when done as the results were superior. This pattern may repeat for long-term memory and planning, yielding better solutions.

4/5/2023

An AI Miracle Malcontent

The stark success of OpenAI’s GPT4 model surprised me shifting my view from “really good autocomplete” (roughly inline with intuitions here) to a dialog agent exhibiting a significant scope of reasoning and intelligence. Some of the MSR folks did a fairly thorough study of capabilities which seems like a good reference. I think of GPT4 as an artificial savant: super-John capable in some language-centric tasks like style and summarization with impressive yet more limited abilities in other domains like spatial and reasoning intelligence.

And yet, I’m unhappy with mere acceptance because there is a feeling that a miracle happened. How is this not a miracle, at least with hindsight? And given this, it’s not surprising to see folks thinking about more miracles. The difficulty with miracle thinking is that it has no structure upon which to reason for anticipation of the future, prepare for it, and act rationally. Given that, I wanted to lay out my view in some detail and attempt to understand enough to de-miracle what’s happening and what may come next.

Deconstructing The Autocomplete to Dialog Miracle
One of the ironies of the current situation is that an organization called “OpenAI” created AI and isn’t really open about how they did it. That’s an interesting statement about economic incentives and focus. Nevertheless, back when they were publishing, the Instruct GPT paper suggested something interesting: that reinforcement learning on a generative model substrate was remarkably effective—good for 2 to 3 orders of magnitude improvement in the quality of response with a tiny (in comparison to language sources for next word prediction) amount of reinforcement learning. My best guess is that this was the first combination of 3 vital ingredients.

Learning to predict the next word based on vast amounts of language data from the internet. I have no idea how much, but wouldn’t be surprised if it’s a million lifetimes of reading generated by a billion people. That’s a vast amount of information there with deeply intermixed details about the world and language.
1. Why not other objectives? Well, they wanted something simple so they could maximize scaling. There may indeed be room for improvement in choice of objective.
2. Why language? Language is fairy unique amongst information in that it’s the best expression of conscious thought. There is thought without language (yes, I believe animals think in various ways), but you can’t really do language without thought.
The use of a large deep transformer model (pseudocode here) to absorb all of this information. Large here presumably implies training on many GPUs with both data and model parallelism. I’m sure there are many fine engineering tricks here. I’m unclear on the scale, but expect the answer is more than thousands and less than millions.
1. Why transformer models? At a functional level, they embed ‘soft attention’ (=ability to look up a value with a key in a gradient friendly way). At an optimization level, they are GPU-friendly.
2. Why deep? The drive to minimize word prediction error in the context of differentiable depth creates a pressure to develop useful internal abstractions.
Reinforcement learning on a small amount of data which ‘awakens’ a dialog agent. With the right prompt (=prefix language) engineering a vanilla large language model can address many tasks as the information is there, but it’s awkward and clearly not a general purpose dialog agent. At the same time, the learned substrate is an excellent representation upon which to apply RL creating a more active agent while curbing an inherited tendency to mimic internet flamebait.
1. Why reinforcement learning? One of the oddities of language is that there is more than one way of saying things. Hence, the supervised learning view that there is a right answer and everything else is wrong sets up inherent conflicts in the optimization. Hence, “reinforcement learning from human feedback” pairs inverse reinforcement learning to discover a reward function and basic reinforcement learning to achieve better performance. What’s remarkable about this is that the two-step approach is counter to the information processing inequality.

The overall impression that I’m left with is something like the “ghost of the internet”. If you ask the internet for the answer to a question on the best forum available and get an answer, it might be in the ballpark of as useful and as correct as that which GPT4 provides (notably, in seconds). Peter Lee’s book on the application to medicine is pretty convincing. There are pluses and minuses here—GPT4’s abstraction of language tasks like summarization and style appear super-human, or at least better than I can manage. For commonly discussed content (e.g. medicine) it’s fairly solid, but for less commonly discussed content (say, Battletech fan designs) it becomes sketchy as the internet gives out. There are obviously times when it errs (often egregiously in a fully confident way), but that’s also true in internet forums. I specifically don’t trust GPT4 with math and often find it’s reasoning and abstraction abilities shaky, although it’s deeply impressive that they exist at all. And driving a car is out because it’s a task that you can’t really describe.

What about the future?
There’s been a great deal about the danger of AI discussed recently, and quite a mess of misexpectations about where we are.

Is GPT4 and future variants the answer to [insert intelligence-requiring problem here]? GPT4 seems most interesting as a language intelligence. It’s clearly useful as an advisor or a brainstormer. The meaning of “GPT5” isn’t clear, but I would expect substantial shifts in core algorithms/representations are necessary for mastering other forms of intelligence like memory, skill formation, information gathering, and optimized decision making.
Are generative models the end of consensual reality? Human societies seem to have a systematic weakness in that people often prefer a consistent viewpoint even at the expense of fairly extreme rationalization. That behavior in large language models is just looking at our collective behavior through a mirror. Generative model development (both language and video) do have a real potential to worsen this. I believe we should be making real efforts as a society to harden and defend objective reality in a multiple ways. This is not specifically about AI, but it would address a class of AI-related concerns and improve society generally.
Is AI about to kill everyone? Yudkowski’s editorial gives the impression that a Terminator style apocalypse is just around the corner. I’m skeptical about the short term (the next several years), but the longer term requires thought.
1. In the short term there are so many limitations of even GPT4 (even though it’s a giant advance) that I both lack the imagination to see a path to “everyone dies” and I expect it would be suicidal for an AI as well. GPT4, as an AI, is using the borrowed intelligence of the internet. Without that source it’s just an amalgamation of parameters of no interesting capabilities.
2. For the medium term, I think there’s a credible possibility that drone warfare becomes ultralethal inline with this imagined future. You can already see drone warfare in the Ukraine-Russia war significantly increasing the lethality of a battlefield. This requires some significant advances, but nothing seems outlandish. Counterdrone technology development and limits on usage inline with other war machines seems prudent.
3. For the longer term, Vinge’s classical singularity essay is telling here as he lays out the inevitability of developing intelligence for competitive reasons. Economists are often fond of pointing out how job creation has accompanied previous mechanization induced job losses and yet my daughter points out how we keep increasing the amount of schooling children must absorb to be capable members of society. It’s not hard to imagine a desolation of jobs in a decade or two where AIs can simply handle almost all present-day jobs and most humans can’t skill-up to be economically meaningful. Our society is not prepared for this situation—it seems like a quite serious and possibly inevitable possibility. Positive models for a nearly-fully-automated society are provided by Star Trek and Iain Banks although science fiction is very far from a working proposal for a working society.
4. I’m skeptical about a Lawnmower Man like scenario where a superintelligence suddenly takes over the world. In essence, cryptographic barriers are plausibly real, even to a superintelligence. As long as that’s so, the thing to watch out for is excessive concentrations of power without oversight. We already have a functioning notion of super-human intelligence in organizational intelligence and are familiar with techniques for restraining organizational intelligence into useful-for-society channels. Starting with this and improving seems reasonable.

7/19/20217/19/2021

ICML 2021 Invited Speakers — ML for Science

By: Stefanie Jegelka and Ameet Talwalkar (ICML21 Communication Chairs)

With ICML 2021 underway, we wanted to briefly highlight the upcoming invited talks. A general theme of the invited talks this year is “machine learning for science.” The Program Chairs (Marina Meila and Tong Zhang) have invited world-renowned scientists from various disciplines to discuss their problems and the corresponding machine learning challenges. By exposing the machine learning community to these fascinating problems, we hope that we can help to further expand the applicability of machine learning to a wide range of scientific domains.

Daphne Koller (Tuesday, July 20th at 8am PDT): Dr. Koller is a pioneer in the field of machine learning, and is currently the Founder and CEO of Insitro, which leverages machine learning for drug discovery. She was the Rajeev Motwani Professor of Computer Science at Stanford University, where she served on the faculty for 18 years. She was the co-founder, co-CEO and President of Coursera, and the Chief Computing Officer of Calico, an Alphabet company in the healthcare space. She received the MacArthur Foundation Fellowship in 2004, was awarded the ACM Prize in Computing in 2008, and was recognized as one of TIME Magazine’s 100 most influential people in 2012.
Xiao Cunde and Dahe Qin (Tuesday, July 20th at 8pm PDT): Dr. Cunde is a glaciologist and Deputy Director of the Institute of the Climate System, Chinese Academy of Meteorological Sciences. He has worked in the fields of polar glaciology and meteorology since 1997. His major research focus has been ice core studies relating to paleo-climate and paleo-environment, and present day cold region meteorological and glaciological processes that impact environmental and climatic changes. Dr. Qin is the Former Director of the China Meteorological Administration. He is a glaciologist and the first Chinese ever to cross the South Pole. He was a member of the 1989 International Cross South Pole Expedition and has published numerous ground-breaking articles, using evidence gathered from his Antarctic expeditions.
Esther Duflo (Wednesday, July 21st at 8am PDT): Dr. Duflo is the Abdul Latif Jameel Professor of Poverty Alleviation and Development Economics in the Department of Economics at MIT and a co-founder and co-director of the Abdul Latif Jameel Poverty Action Lab (J-PAL). In her research, she seeks to understand the economic lives of the poor, with the aim to help design and evaluate social policies. She has worked on health, education, financial inclusion, environment and governance. In 2019, she received a Nobel Prize in Economic Sciences “for their experimental approach to alleviating global poverty”. In particular, she and co-authors have introduced a new approach to obtaining reliable answers about the best ways to fight global poverty.
Edward Chang (Wednesday, July 21st at 8pm PDT): Dr. Chang is a Professor in the Department of Neurological Surgery at the UCSF Weill Institute for Neurosciences. He is a neurosurgeon and uses machine learning to understand brain functions. His research focuses on the brain mechanisms for speech, movement and human emotion. He co-directs the Center for Neural Engineering and Prostheses, a collaborative enterprise of UCSF and UC Berkeley. The center brings together experts in engineering, neurology and neurosurgery to develop state-of-the-art biomedical technology to restore function for patients with neurological disabilities such as paralysis and speech disorders.
Cecilia Clementi (Thursday, July 22nd at 8am PDT): Dr. Clementi is a Professor of Chemistry, and Chemical and Biomolecular Engineering, and Senior Scientist in the Center for Theoretical Biological Physics at Rice University, and an Einstein Fellow at FU Berlin. She researches strategies to study complex biophysical processes on long timescales, and she is an expert in the simulation of biomolecules using large-scale ML. Her group designs multiscale models, adaptive sampling approaches, and data analysis tools, and uses both data-driven methods and theoretical formulations.

To register for the conference and check out these talks, please visit: https://icml.cc/.