How Large Language Models Actually Work, A Security Professional's Primer

Large language models can feel like magic, and magic is impossible to secure. This is a working security professional's mental model of how LLMs really work: enough detail to reason about how they break, no math required. It's also the exact foundation CompTIA SecAI+ Domain 1 expects you to have.

First, where do LLMs sit?

People use "AI," "machine learning," and "LLM" interchangeably, but they're nested ideas. Getting the hierarchy straight is the first thing the exam checks, and it makes every later attack easier to place.

An LLM is a specific kind of deep-learning model built on an architecture called the transformer. That one architecture is behind nearly every model you've heard of, which is why understanding it generalizes so well.

Everything is tokens

An LLM never sees words the way you do. The first thing it does is tokenize: chop text into small pieces (a token is roughly ¾ of a word) and convert each into a number. Those numbers become embeddings: long lists of values that place each token in a "meaning space," where related concepts sit near each other.

From there, the model does exactly one thing, over and over: predict the next token. Given everything so far, what token is most likely to come next? Append it, then repeat. A paragraph of fluent text is just that loop run hundreds of times.

Attention: how the model decides what matters

The transformer's key trick is self-attention. As it processes each token, the model weighs how much every other token should influence it. In "the trophy didn't fit in the suitcase because it was too big," attention is what lets the model connect "it" to "trophy" rather than "suitcase."

You don't need the linear algebra for the exam. You need the intuition: attention is the mechanism that lets a model use context, and context is attacker-reachable. If an attacker controls part of the context (a document you asked the model to summarize), they're influencing the same machinery your legitimate instructions use.

Training vs. inference: two very different phases

This is the distinction that organizes almost all of AI security. A model has two lives, and they're attacked in completely different ways.

Training is where the model learns. It ingests an enormous corpus and slowly adjusts billions of internal weights until it predicts text well. This is rare, costly, and (crucially) it's where the model's behavior is baked in. Tamper with the training data and you tamper with the model itself. That's the home of data poisoning and backdoor attacks.

Inference is every time you actually use the model. The weights are frozen; the model just runs the prediction loop on your input. Nothing it sees here changes what it "knows." This is the home of prompt injection, jailbreaks, and sensitive-information disclosure: attacks that manipulate the input rather than the model.

Putting it together: the security view

Once the mental model clicks, the whole SecAI+ threat landscape lines up neatly:

The single most important takeaway is the one that trips up newcomers: the model cannot reliably separate instructions from data. Everything is tokens, attention blends them together, and a frozen model has no privilege boundary inside its context window. Defenses don't come from asking the model nicely. They come from controlling what reaches it and constraining what it can do.

This is the ground floor of CompTIA SecAI+. Our SecAI+ Domain 1 study guide walks all 18 topics: tokenization, transformers, the training lifecycle, and the attack surfaces above, with exam-style questions and a downloadable PDF.

#SecAIplus #AISecurity #LLM #MachineLearning #Transformers #PromptInjection #CyberSecurity #CompTIA #AIThreats #InfoSec