Auxiliary classifier generative adversarial networks (ac-GAN)

The p-GAN model workflow employs auxiliary classifier generative adversarial networks (ac-GAN) [18] trained to generate synthetic VUs and associated VU regions. ac-GAN is a variation of generative adversarial networks [14], which utilizes a generator and discriminator to create realistic images. Fig 2 shows the general structure of the ac-GAN model.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. The ac-GAN structure.

The generator (G) is a deep neural network that transforms a feature vector of the class label (c) and a random noise (z) to an image, and the discriminator (D) distinguishes between the real images and fake images and predicts the class labels for each image.

https://doi.org/10.1371/journal.pone.0280316.g002

The training was done by iteratively improving the generator and discriminator. The loss consists of two components, the discriminative loss L_D on distinguishing between real and fake images and an α weighted classification loss L_C in the accuracy of predicting class labels, i.e. where,

• Pre-processing: The same 9 slices of VUs were kept and rescaled from 112 x 224 to 64 x 64. For this specific dataset of VUs, although being 3D images, they were treated as 2-dimensional images with 9 channels and learnt with a 2D ac-GAN model, as the 3-dimensional structure in VUs were not very strong due to large slice thicknesses.
• Feature vector: Apart from the additional 3 dimensions for class labels (region encoded as a length 3 one-hot vector), a 100-dimensional standard Gaussian random noise was used as the input to the generator.
• Generator: The generator consisted of five 2-dimensional deconvolutions (transposed convolution) layers with 512, 256, 128, 64 and 9 output channels, respectively. A kernel of size 4 was used in all layers. The first layer had a stride of 1 and 0 padding to up sample the image size from 1 x 1 to 4 x 4, then the rest layers were set with stride size 2 and padding 1 to double the image size. Rectified Linear Unit (ReLU) activations were used in the first 4 layers and the hyperbolic tangent (tanh) activation was used in the last layer to restrict the output pixels within the range of [–1, 1].
• Discriminator: The discriminator consisted of five 2-dimensional convolution layers with 64, 128, 256, 512, 64 output channels, respectively, and each layer was set with kernel size 4. A stride size 2 and padding 1 was used in all layers but the last one, and the last layer was set with stride size 1 and padding 0. Leaky ReLU activations were used in all layers with a negative slope of 0.2. After passing through the convolution layers that extract information from images, VUs were then sent through two separate, fully connected layers with sigmoid and softmax activation, respectively, to produce the decision of real vs fake as well as the predicted classification label.
• Loss: The classification loss was weighted by α = 1 during training.
• Optimization: Training was done with Adam optimizer18 with learning rate 1e-4 for both the generator and the discriminator.
• Model Selection: When performing hyperparameter search for training the model, visual inspection, adversarial and classification loss during training were monitored, as well as classification on the A2209 test set.
• Synthetic Data: To generate the synthetic dataset, VU location was first simulated from the empirical distribution of training data. The sampled VU location was then utilized to condition the generator and generate the entire image.
• Reproducibility: Trial data used for the GAN training is restricted, but synthetic VU data used for the diversity results are available. Additionally, all Python codes are fully available.

Real vs. synthetic ResNet-18 performance

Prior to the training of F_synthetic, a synthetic dataset was created using the trained generator. The synthetic dataset size and class distribution was designed to approximate the scans from F2305 by generating 10,000 synthetic images, whereas the class assignment was sampled from a distribution p_C, p_T, p_L = 0.25, 0.55, 0.2) where the probability was denoted for cervical, thoracic and lumbar, respectively.

Two ResNet-18 [19] based classifiers F were trained for the VU region. For the first classifier F_real F2305 was used as the training set, while the second classifier F_synthetic was trained solely on synthetic data. In both cases, A2209 was used as an independent test set. Both models used the same hyperparameters and an identical training procedure.

• Optimization: Stochastic gradient descent with a learning rate of 10⁻⁴ and momentum of 0.9 was used, and networks were trained for 20 epochs with a batch of 32.
• Loss: Cross-entropy loss was applied.

A single global ROC curve was computed for each classifier by averaging the class specific ROC curves using equal weights (macro-average). The 95 percentile confidence intervals for the AUCs were computed using bias-corrected and accelerated bootstrapping [20] with 2000 resamples.

Image morphing

To generate the synthetic image, a tensor was inputted that concatenates a random noise z∈R¹⁰⁰ and a condition c∈R³. The tensor c was a hot-encoding vector representing the vertebra location:

Thus, for all three locations:

To morph an image from a location to a new one, the tensor c can be modified to impact the generator, thus leaving untouched z. Linear interpolation was utilized between source condition c_source and the target one c_target. The equation below describes condition for step i (out of n) to morph from k to k′:

With,

Using the definition above, then to morph from cervical to lumbar the computation becomes:

Privacy

To evaluate privacy, a list of candidate samples was created that was composed of 1000 images: 1/3 from training, 1/3 from validation and 1/3 from test. For pairwise attacks, the comparison between synthetic and candidate was done in pixel space and embedding space. For the pixel space, similarity was computed as the minimum Euclidean distance between one candidate and all the synthetic samples. The embedding distance was obtained by training a Uniform Manifold Approximation and Projection (UMAP) trained on 5000 images (3000 from the trainset, 1000 from the validation set, and 1000 from the test set). The UMAP compressed the samples down to 64 features, and the feature distance was obtained as the minimum Euclidean distance on features. For the distribution attacks, the threshold for 2 points to be considered as neighbours was that their distance was part of the 1^st percentile. A privacy leakage is defined as finding traces of the training dataset within the synthetic dataset.

Fidelity

The first assessment was to evaluate whether the synthetic VUs were visually indistinguishable from real VUs for a given location. Fig 3A shows a collection of real and synthetic data. Visually, synthetic samples look like real ones and expose different structures depending on their conditioned region location (i.e., cervical, thoracic and lumbar). Inspired from recent work on face morphing with GAN [21] Fig 3B shows that generator behaviour can be controlled by continuously changing the condition aspect from the input noise tensor [21]. This was achieved by using a linear regression between the source and target region wanted. This result demonstrated that the generator was correctly conditioned and could create specific VU locations on demand.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Fidelity.

A) Examples of real and synthetic vertebrae and their corresponding vertebral unit (VU) locations. Each row shows 4 consecutive vertical slices for a real and fake VU (each slice is of size 64 x 64). The first two rows are cervical VUs, followed respectively by thoracic and lumbar. B) Morphing an image from one location to another was done by interpolating the condition c (keeping z untouched as the latent variable). One step for each morphing step was utilized, thus leaving intermediate states as 50% / 50% morph between two locations.

https://doi.org/10.1371/journal.pone.0280316.g003

Diversity

The real and synthetic datasets were evaluated to determine if they have similar global distribution while having distinct individual VU samples as shown in Fig 4A. To obtain a meaningful low dimensional representation, a UMAP transform was applied to the dataset of VUs. UMAP [22] is a powerful non-linear dimensionality reduction technique. It is similar under many aspects to t-SNE [23] but differs by the fact that the transformation from high dimension to low dimension is learnable. This was leveraged by training the UMAP only on real data and then applying the transform to unseen real data as well as synthetic data. The first finding was that VU location distribution was separable and similar between real and synthetic samples, showing that both datasets have similar global distribution. The second finding was that the synthetic VUs covered most of the distribution in the low dimensional representation space without overlapping with real VUs, implying that the synthetic samples were indeed distinct. Those findings confirmed that the GAN could synthetize data that behaves like real VUs and that its diversity (phenotypic differences) matched the real distribution as the mode was not observed to collapse. The quality of the synthetic dataset was further examined in terms of its capability to serve as a functional equivalent to a real dataset for downstream training tasks by training two ResNet-18 [24] based classifiers F_real and F_synthetic with real and synthetic data respectively as seen in Fig 4B. Fig 4C shows the ROC curves for F_real and F_synthetic. It was observed that, while F_synthetic slightly underperforms compared to F_real, the synthetic dataset could approximate the distribution in the original data when tested for VU region. The slight discrepancy in performance was to be expected as the conservation of privacy could be expected to come at a cost in data quality. While, in principle, the performance of F_synthetic could asymptotically reach the one of F_real, in practice there is a trade-off between overfitting the training set and replicating the classification statistics of a network trained solely on the training set.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. Diversity.

A) Low dimensional UMAP representation of real and synthetic vertebral units with locations cervical, thoracic and lumbar labelled as blue, green and red respectively. B) Confusion matrices of F_real (left) and F_synthetic (right) when tested on A2209. The predicted class for a sample image x_i corresponds to max(F_real(x_i)) and max(F_synthetic(x_i)) respectively. The class distribution of the test set corresponds to p_C, p_T, p_L = (0.23, 0.55, 0.23). C) ROC and AUC for F_real (left) and F_synthetic (right) when tested on A2209. Every class specific ROC curve was computed by treating the other two classes as negative instances, effectively reformulating the classification for each class as a binary problem.

https://doi.org/10.1371/journal.pone.0280316.g004

Privacy

In this section, the privacy leakage assumption was restricted to only sharing a synthetic and finite dataset rather than the trained generative model itself. This restriction made privacy attacks much more difficult because they could not be conducted by reverse engineering the network’s output and there was no direct access to the generative model’s latent space. This meant only attacks that rely directly on comparing candidate images with the synthetic dataset could jeopardize privacy. An obvious case of privacy threatening behaviour in GAN is overfitting. A major overfitting corresponds to a generator outputting synthetic images that are exact copies from training images. Other convergence issues could be problematic, such as cases where the generator only learns to reproduce a few samples from the training set (mode collapse).

Two main potential vulnerabilities were identified for a synthetic dataset:

• Pairwise attacks: where finding a synthetic image which is similar to a given sample would prove that this sample was used during training.
• Distribution attacks: where a high density of synthetic images cluster around one or a few real images.

To assess the robustness to these attacks, the synthetic images were simulated by using a “candidate dataset” (S1 Fig) and evaluated as to which ones could be traced back to the training set.

Robustness to pairwise attacks

For robustness to pairwise attacks, the similarity between a candidate sample and all images were computed from the synthetic dataset. A relatively small L2 distance may indicate that the candidate was used for the generator training. The distribution of minimum distances between candidate and synthetic images are represented in Fig 5A. On this graph, the lowest distances are expected to be train-synthetic pairs and the largest ones to be test-synthetic due to some overfitting behaviour from the GANs. In the privacy-threatening scenario, images from the training set are easy to identify by a simple anomaly detection task. An anomaly detection with reference distances is available on S2 Fig. On the observed synthetic dataset however, it is impossible to reliably identify candidates originating from the training dataset as compared to the examples coming from validation set. This is also confirmed using an embedding space obtained by applying a UMAP. The synthetic dataset hence shows robustness to pairwise attacks, in both pixel and embedding space.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 5. Simulation of pairwise attacks and distribution attacks.

A) Simulation of pairwise attacks. A small distance indicates that the candidate is likely from the training set. Graphs 1–2 are toy examples of privacy threatening scenario with easily identifiable candidates from training. Graphs 3–4 are the observed behaviour of our synthetic dataset. B) Simulation of distribution attacks. A large cluster indicates that the candidate is likely coming from the training set. Graphs 1–2 are privacy threatening and graphs 3–4 are observed behaviour on the synthetic dataset. C) 2D projection of the synthetic and training dataset. Shades of yellow-red are density levels of synthetic images and blue dots are individual training examples.

https://doi.org/10.1371/journal.pone.0280316.g005

Robustness to distribution attacks

To evaluate the robustness to distribution attacks, clusters of synthetic images were identified around a given candidate. The clusters were defined by counting the number of synthetic images in the candidate neighbourhood. These results are shown in Fig 5B. A large cluster was expected to form around candidates from the training set, with the other candidates to have a small number of neighbours. The privacy-threatening scenario (left figures) reflects this hypothesis. The observed behaviour of the synthetic data is shown on the two most right figures. Candidates from the test set rarely belong to large clusters unlike candidates from training or validation. In fact, the largest clusters overwhelmingly form around training samples. Even if this method can help identify a handful of training examples, it is ineffective for most of them. Indeed, the size of the training set and the synthetic dataset are around the same (around 10’000 samples). This means that on average, one training sample can be only traced back to one synthetic sample. This means that by construction, the generated synthetic dataset is safe from distribution attacks.

It is also worth noting that the embedding space was not uniformly populated. As seen on Fig 5C, a high density of synthetic images is in general linked to a high density of real images (blue dots on the figure) and not due to the GAN generation collapsing and only generating one type of sample. A detailed detection of training samples is available in the S3, S4 Figs and S1 Table.

Limitations

Ensuring privacy is a challenging aspect of this project, as defining a robust method to assess similarity between images is complex [38]. Due to sample size limitation, VUs were investigated instead of full spines images. This approach increased the effective sample size by pooling all time points and all VU locations, thus transforming the 108 patients into 7832 VUs. The GANs were able to be trained with sufficient data, at the cost of limiting the generation of synthetic images to the smaller VU piece. Conditioning the pGAN on a clinical metric was also investigated (VU inflammation score) but did not produce the desired results as the classifier task is more challenging than the prediction of a VU location. Several reasons can explain this: imbalance between the number of positive and negative samples, as well as the intra-reader discrepancies, led to similarly noisy results that proved too hard to analyse. Finally, in the present approach, the synthetic dataset has a fixed privacy tolerance that is a factor of the real dataset and the pGAN model convergence but cannot be modulated based on specific needs. Privacy was computed ad hoc rather than optimized, which may limit the control on the level of privacy needed for some applications.

Future steps

Images generated by the pGAN model were realistic, informative and private. Additional analyses could be performed to further confirm those attributes. Privacy was computed as the distance of the synthetic images from the original datasets (in a form of attacks). The chance of generating a synthetic image with pGAN that resembles a real image is low, as the high dimensionality of the sampling space minimises the probability of synthesising an image with parameters identical to an original image. Further efficiencies in the method can be considered when confirming that the network did not reproduce a real sample, as compared to after each sample generation. A direction for further research to enhance the sample privacy could be to determine where the real images lie in the latent space of the generator and then create a careful sampling strategy to avoid sampling around these points. A future focus would be to generate complete synthetic spines instead of independent VUs. This would require balancing the trade-off between VU consistency and memory, since generating 21 VUs at once is extremely memory intensive and only a weak correlation would be required.