AI Concepts 101

Software engineering has split into three eras, and leaders need vocabulary to steer teams through the current one.

Software engineering has moved through three eras, and this module gives you the vocabulary for the current one. Software 1.0 is hand-written logic. Software 2.0 moved logic from code into curated datasets and neural networks. Software 3.0 turns natural language into the programming interface, which is what makes agents and conversational coding possible. Understanding this shift is the starting point for steering any team that builds with AI.

The module maps the new mental model piece by piece. A large language model works as the central processing unit of an application. Context is its finite working memory, and position matters: tokens near the start and end of the window receive the most attention, while material in the middle can be ignored. Embeddings translate any content into vectors, where proximity equals similarity of meaning, and vector databases act as the persistent store. Agent loops supply control flow through the Thought, Action, Observation, Repeat pattern known as ReAct. Tool calling converts model text into structured function calls the surrounding system can execute.

After this module you will be able to talk about latency in concrete terms, separating time to first token, which drives perceived responsiveness, from tokens per second, which drives completion speed. You will understand why evaluations replace traditional tests for non-deterministic behavior: a curated dataset of inputs and correct outputs, scored by an LLM acting as judge against criteria such as faithfulness and relevance. You will be able to insist on evals as part of the definition of done and to budget for context management, not just model tokens.

The common failures follow from missing this vocabulary. Teams confuse unit tests with evals and ship untested non-determinism. They expect the model to know facts that must come from retrieval. They let a single agent accumulate responsibilities until its context collapses. Reviewing agent systems at the level of the loop, not the individual prompt, prevents most of this.

In one paragraph

Software 1.0 is hand-written logic. Software 2.0 shifted logic from code to curated datasets and neural networks. Software 3.0 turns natural language into the programming interface, making agents and vibe coding possible. In this new paradigm, a large language model becomes the central processing unit of an application, with context as its working memory, embeddings and vector databases as its persistent store, agent loops as its control flow, and tool calling as its hands. Evaluations replace traditional tests for non-deterministic behavior.

Key ideas

  • An LLM can be modeled as a non-deterministic database. The prompt is the query, and the parameters are the stored patterns.
  • Context is finite working memory. Tokens near the start and end receive the most attention; material in the middle can be ignored.
  • Embeddings translate any content type into vectors. Proximity in that space equals similarity of meaning.
  • Time to first token drives perceived responsiveness. Tokens per second drives completion speed. Both matter.
  • Evals use a curated dataset of inputs and correct outputs, with an LLM-as-judge scoring new outputs against criteria such as faithfulness and relevance.
  • Agent loops follow a Thought, Action, Observation, Repeat pattern. This is the ReAct pattern and is the default agent architecture today.
  • Tool calling converts a model's text output into structured function calls the surrounding system can execute.

What this means for leaders

  • Insist on evals as part of the definition of done for any AI feature.
  • Budget for context management, not just model tokens.
  • Treat agent loops as the unit of measurement when reviewing agent systems, not individual prompts.

Failure modes to watch for

  • Confusing traditional unit tests with evals and shipping untested non-determinism.
  • Expecting the model to know facts that must come from retrieval.
  • Letting a single agent accumulate responsibilities until its context collapses.

This module is part of the Foundations track in the free AI Learning Hub. Source material: AI Concepts 101.