Deepfake technology runs on three families of generative AI models: autoencoders, generative adversarial networks (GANs), and diffusion models. Each learns the appearance of a face or scene from training data, then produces new images or video that match those patterns. Understanding how the three differ is no longer an academic exercise. The architecture behind a piece of synthetic media determines the artifacts it leaves, the detection methods that catch it, and how quickly your defenses go stale.
This guide explains how each architecture works, what makes it good at faking reality, and what that means for anyone trying to catch the result. It is written for security architects, IDV product managers, ML engineers, and compliance officers who want a working technical understanding without reading a 50-page survey paper. No prior machine learning background is assumed, though a little helps.
- Deepfakes are built on three generative architectures: autoencoders, GANs, and diffusion models. Many real-world pipelines combine more than one.
- Autoencoders use a shared encoder and two decoders to swap one face onto another, and remain the workhorse for targeted video impersonation.
- GANs, introduced by Ian Goodfellow in 2014, pit a generator against a discriminator and made photorealistic synthetic faces possible.
- Diffusion models, dominant since 2023, build images by reversing a noising process and now lead on high-end image and video quality.
- Each architecture leaves a different signature, so a detector tuned for one can be blind to another. Ensemble detection and continuous retraining are essential.
What Powers a Deepfake?
A deepfake is synthetic media in which a person's face, voice, or likeness is generated or swapped using deep learning. The common thread across every method is the same: a neural network is shown thousands of examples of a target, learns the statistical structure of how that target looks, and then generates new outputs that fit the same distribution.
The differences come down to how the network learns and how it generates. An autoencoder learns to compress a face and rebuild it. A GAN learns by competing against a built-in critic. A diffusion model learns to reverse a process of adding noise. All three produce convincing fakes, but they take very different routes to get there, and those routes leave very different fingerprints behind.
It helps to separate two things people often lump together. Face swapping replaces one person's face with another while keeping the original expression, pose, and lighting. Full synthesis creates a person or scene that never existed. Autoencoders dominate the first. GANs and diffusion models excel at the second, and increasingly at both. For a broader tour of the tools and pipelines that assemble these models into finished fakes, see our guide on how deepfakes are made.
Autoencoders: The Original Face-Swap Engine
The earliest viral deepfakes, the ones that launched apps like FakeApp and later DeepFaceLab, were built on autoencoders. The architecture is the oldest of the three and still the workhorse for video face swaps.
How an autoencoder works
An autoencoder is a neural network with a deliberate bottleneck in the middle. It has two halves. The encoder takes an input image and compresses it into a small set of numbers called a latent representation, or latent vector. The decoder takes that compressed representation and tries to rebuild the original image. Because the bottleneck forces the network to throw away detail, it cannot simply memorize and copy pixels. It has to learn the meaningful structure of a face: where the eyes sit, how the mouth shapes a smile, the angle of the head. Training minimizes reconstruction loss, the difference between the input and the rebuilt output. The result is lossy, so a reconstructed face never has quite the detail of the original.
The face-swap trick
A single autoencoder just rebuilds what it sees. The clever part of a face swap is the configuration. You train one shared encoder together with two separate decoders, one for person A and one for person B. The encoder sees faces of both people and is forced to learn a common representation of "a face" that works for either identity. Decoder A only ever learns to rebuild person A. Decoder B only ever learns to rebuild person B.
Once training is done, you perform the swap by deliberately mismatching the parts. You feed an image of person A through the shared encoder to get the latent representation of their expression and pose, then you route that representation through decoder B instead of decoder A. Decoder B treats the incoming code as if it were a distorted version of person B and rebuilds it as person B's face, but carrying person A's original expression and head position. The output is person B wearing person A's performance. To make the encoder generalize rather than memorize, the input faces are usually warped during training, nudging the network to learn flexible features instead of exact copies.
Strengths and weaknesses
Autoencoder swaps are stable to train, work well on video frame by frame, and only need footage of two specific people, which is why they remain popular for targeted impersonation. The weaknesses are equally characteristic. Because the two identities have to share an encoder, results are best when the faces are similar in shape and angle. The bottleneck and the blending step leave seams at the hairline, jaw, and skin edges, and lighting or skin tone often fails to match the target scene. In video, those imperfections produce a subtle flicker between frames. Variational autoencoders (VAEs) improve on the basic version by shaping the latent space to follow a smooth Gaussian distribution using a technique called the reparameterization trick, which reduces some artifacts and deformations.
Generative Adversarial Networks: The Realism Arms Race
GANs, introduced by Ian Goodfellow and colleagues in 2014, were the architecture that made deepfakes genuinely alarming. They pushed synthetic faces from obviously fake to photorealistic, and they are the reason a website can serve you an endless stream of people who do not exist.
How a GAN works
A GAN pits two neural networks against each other. The generator starts with a vector of random noise and transforms it into a synthetic image. The discriminator is a binary classifier that looks at an image and judges whether it is real, drawn from the training data, or fake, produced by the generator. The two are trained together in what is formally a minimax game, a contest where one network's gain is the other's loss.
The standard analogy is a counterfeiter and the police. The generator is a forger trying to print convincing fake currency. The discriminator is the inspector trying to spot the fakes. Every time the inspector catches a flaw, the forger learns from it and refines the next batch. Every time a fake slips through, the inspector tightens its standards. Over millions of rounds, the generator is pushed toward outputs the discriminator can no longer distinguish from real. At that point you have a generator capable of producing highly realistic synthetic faces from nothing but random noise.
What GANs are used for
GANs power several distinct deepfake tasks. Full face synthesis creates entirely fictional people, with StyleGAN being the best-known architecture for this, learning to separate high-level attributes like pose and identity from fine details like freckles and stray hairs. Other GAN variants handle attribute editing, changing age, expression, hair color, or even swapping identity across many target domains at once, as StarGAN does. GANs are also frequently bolted onto autoencoder face-swap pipelines as a finishing stage, sharpening the output and pushing realism higher than the autoencoder alone could reach.
Strengths and weaknesses
GANs generate images in a single fast pass once trained, and they produce sharp, high-fidelity results. Their famous weaknesses are training instability and mode collapse, a failure where the generator gets lazy and produces only a narrow range of outputs because it found a few that reliably fool the discriminator. Getting a GAN to train cleanly is notoriously finicky. From a detection standpoint, the most useful fact is that GANs leave consistent statistical fingerprints in the frequency domain, regular patterns invisible to the eye but detectable by a trained model. Those fingerprints are how many detectors caught GAN content reliably, right up until diffusion models changed the game.
Diffusion Models: The Current State of the Art
Since 2023, diffusion models have become the dominant approach to high-quality image and video generation. They are the architecture behind Stable Diffusion, DALL-E, Midjourney, and the new wave of text-to-video systems. For deepfakes, they represent a clear step up in quality and a real problem for detectors built for the previous generation.
How a diffusion model works
Diffusion takes the opposite intuition from a GAN. Instead of generating an image in one shot, it learns to build one gradually out of noise. There are two processes to understand.
The forward process is fixed and simple. Take a real image and add a small amount of Gaussian noise. Repeat over many steps until the image is destroyed and nothing remains but pure static. This trajectory is known and requires no learning. The reverse process is where the model is trained. A neural network, usually a denoising U-Net or a transformer, learns to undo a single step of noising: given a slightly noisy image, predict what the slightly cleaner version looked like.
Generation then runs the reverse process from scratch. You start with a frame of pure random noise and apply the trained denoiser over and over, typically 20 to 50 steps with modern samplers, each step removing a little more noise. What emerges is a clean image drawn from the same distribution as the training data. To control the output, the model is conditioned on extra information: a text prompt, a reference photo, or an identity embedding that tells the denoiser whose face to converge toward. Many production systems, including Stable Diffusion, run this process in a compressed latent space rather than on raw pixels, which is what makes them fast enough to be practical.
Why diffusion produces better fakes
Diffusion models optimize a likelihood-based objective rather than fighting an adversary, which makes them far more stable to train and largely immune to the mode collapse that plagues GANs. That stability translates into better diversity and cleaner, more controllable conditioning on text and reference images. Because the iterative denoising process averages over many possible trajectories, the output tends to carry fewer of the harsh high-frequency artifacts that GANs produce. The trade-off is speed and compute. Running 20 to 50 denoising steps is slower than a GAN's single pass, though improved samplers continue to narrow that gap.
The Three Architectures Compared
The fastest way to hold all three in your head is side by side. Each was the state of the art in its moment, and each is still in active use, often combined within a single pipeline.
What This Means for Detection
The reason the architecture matters so much to defenders is that each one leaves a different signature, and a detector tuned for one can be blind to another. This is not a minor tuning issue. It is the central technical reason detection accuracy degraded sharply in 2023 and 2024, as attackers shifted from GANs to diffusion and existing detectors found themselves looking for the wrong fingerprints.
Autoencoder face swaps betray themselves at the boundaries. Because a swapped face is blended onto an existing head and scene, the telltale signs are seams at the hairline and jaw, mismatched skin tone or lighting, resolution differences between the face and its surroundings, and temporal flicker across video frames. GANs leave regular, almost periodic patterns in the frequency domain, statistical fingerprints that a model can learn even though no human would ever see them. Diffusion models shift those fingerprints to different parts of the frequency spectrum entirely, produce fewer obvious high-frequency artifacts, and tend to fail instead on fine semantic detail such as hands, teeth, jewelry, or background text.
The practical conclusion is that no single detection signal covers the field. A frequency-domain detector built for GANs misses diffusion content. A boundary-artifact detector built for face swaps misses fully synthesized faces. This is why modern systems rely on ensembles that cross-correlate multiple signals and, critically, are retrained continuously against the newest generators. For a deeper look at what still works and what no longer does, see our guide on how to spot a deepfake, and for where this capability belongs in a verification pipeline, where deepfake detection fits in an IDV stack.
Common Misconceptions
A few persistent myths get in the way of clear thinking about deepfake technology.
"Deepfakes are made by one technology." They are not. Three distinct architecture families are in active use, and serious pipelines frequently combine them, for example an autoencoder swap polished by a GAN finishing stage.
"GANs are the current standard." They were the standard, and they are still widely used, but diffusion models now lead on quality for high-end image and video synthesis. Treating GAN detection as sufficient leaves a large blind spot.
"If you can detect one kind, you can detect them all." This is the most expensive mistake. Artifacts are architecture-specific. A detector trained only on GAN output can perform near random on diffusion content.
"Higher visual quality means undetectable." Realism to the human eye and detectability to a model are different things. Diffusion output can look flawless while still carrying statistical signatures a trained detector can flag.
"Autoencoder swaps are obsolete." They are not. For targeted video impersonation of a specific person, autoencoder pipelines like DeepFaceLab remain among the most common methods in the wild.
Practical Recommendations
If you are responsible for defending against synthetic media, a few principles follow directly from how the technology works.
Assume all three architectures, plus voice synthesis, are in play. Your detection coverage should not be tied to a single model family. Favor ensemble detection that combines frequency analysis, boundary and blending analysis, and semantic consistency checks, because each catches what the others miss.
Treat detection as a continuously maintained capability, not a one-time purchase. Generators evolve on a timescale of months, and a detector validated against last year's models is not validated against this year's. Ask any vendor how often their models are retrained and against which current generators.
Insist on explainability. In regulated settings, a black-box score is not defensible. A verdict that points to the specific artifact, the seam, the frequency anomaly, the inconsistent texture, is one your compliance team, auditors, and courts can actually act on. This is the design principle behind DuckDuckGoose's DeepDetector, which surfaces visual evidence for every flag rather than returning a bare confidence number.
Finally, pair detection with process. Even the strongest detector should sit alongside out-of-band verification for high-value actions, so that a single missed fake does not become a single point of failure.
Frequently Asked Questions
What is the difference between a GAN and a diffusion model?
A GAN generates an image in one pass by having a generator compete against a discriminator that tries to spot fakes. A diffusion model generates an image gradually, starting from pure noise and removing it step by step using a network trained to reverse a noising process. Diffusion is more stable to train and currently produces higher-quality results, while GANs are faster at generation time.
Which AI model is used to make deepfakes?
There is no single model. The three main families are autoencoders, which dominate video face swaps, GANs, which excel at synthesizing realistic faces, and diffusion models, which now lead on overall image and video quality. Many real-world pipelines combine more than one.
How do autoencoders create face swaps?
By training one shared encoder with two separate decoders, one per identity. The encoder learns a common representation of a face. To swap, you feed person A's image through the encoder and then through person B's decoder, producing person B's face carrying person A's expression and pose.
Why are diffusion models better at deepfakes than GANs?
Diffusion models train against a stable likelihood-based objective instead of an adversary, so they avoid the training instability and mode collapse that affect GANs. The result is more diverse, more controllable output with fewer harsh artifacts. The trade-off is slower generation.
Do deepfakes leave artifacts that can be detected?
Yes, though the artifacts depend on the architecture and are often invisible to people. Autoencoder swaps leave boundary seams and flicker, GANs leave frequency-domain fingerprints, and diffusion models leave different spectral signatures plus occasional errors in fine detail like hands or text. Trained detectors find these even when humans cannot.
Are GANs still used for deepfakes in 2026?
Yes. GANs remain in wide use for face synthesis and as finishing stages in face-swap pipelines, even though diffusion models have overtaken them for top-end image and video quality.
Why does a detector that works on one deepfake fail on another?
Because each architecture leaves architecture-specific artifacts. A model trained to recognize GAN frequency patterns has never learned diffusion signatures, so it can perform little better than chance on diffusion content. This is why ensemble detection and continuous retraining are necessary.
.png)
About the author
