Ajinkya Tejankar

I am a 3rd year PhD student at University of California, Davis. My advisor is Dr. Hamed Pirsiavash.

Research: I primarily work on self-supervised learning for images. My focus thus far has been improving the quality of features learnt by self-supervised models. This has involved using knowledge distillation for improving smaller models and improving the loss functions for better semantic level grouping. Recently, I have become interested in analyzing the failure modes of these methods. Specifically, how changing the input dataset affects the representations of the final model? I am also interesting in vision-language pre-training.

Previously: I obtained my MS in Computer Science from UMBC in 2020. I interned at Meta AI in summer 2021, 2022 and at Matroid in summer 2020. Even before that I was a Software Engineer at Tavisca from 2014 to 2017.

Publications

(*) denotes shared first authorship

CVPR

Defending Against Patch-based Backdoor Attacks on Self-Supervised Learning

Ajinkya Tejankar, Maziar Sanjabi , Qifan Wang , Sinong Wang , Hamed Firooz , Hamed Pirsiavash , and Liang Tan

In Conference on Computer Vision and Pattern Recognition (CVPR) Jun 2023

Abs PDF Code

Recently, self-supervised learning (SSL) was shown to be vulnerable to patch-based data poisoning backdoor attacks. It was shown that an adversary can poison a small part of the unlabeled data so that when a victim trains an SSL model on it, the final model will have a backdoor that the adversary can exploit. This work aims to defend self-supervised learning against such attacks. We use a three-step defense pipeline, where we first train a model on the poisoned data. In the second step, our proposed defense algorithm (PatchSearch) uses the trained model to search the training data for poisoned samples and removes them from the training set. In the third step, a final model is trained on the cleaned-up training set. Our results show that PatchSearch is an effective defense. As an example, it improves a model’s accuracy on images containing the trigger from 38.2% to 63.7% which is very close to the clean model’s accuracy, 64.6%. Moreover, we show that PatchSearch outperforms baselines and state-of-the-art defense approaches including those using additional clean, trusted data.
ECCV

Constrained Mean Shift Using Distant Yet Related Neighbors for Representation Learning

KL Navaneet* , Soroush Abbasi Koohpayegani* , Ajinkya Tejankar*, Kossar Pourahmadi , Akshayvarun Subramanya , and Hamed Pirsiavash

In European Conference on Computer Vision (ECCV) Jun 2022

Abs PDF Code

We are interested in representation learning in self-supervised, supervised, and semi-supervised settings. Some recent self-supervised learning methods like mean-shift (MSF) cluster images by pulling the embedding of a query image to be closer to its nearest neighbors (NNs). Since most NNs are close to the query by design, the averaging may not affect the embedding of the query much. On the other hand, far away NNs may not be semantically related to the query. We generalize the mean-shift idea by constraining the search space of NNs using another source of knowledge so that NNs are far from the query while still being semantically related. We show that our method (1) outperforms MSF in SSL setting when the constraint utilizes a different augmentation of an image from the previous epoch, and (2) outperforms PAWS in semi-supervised setting with less training resources when the constraint ensures that the NNs have the same pseudo-label as the query.
CVPROral

Backdoor Attacks on Self-supervised Learning

Aniruddha Saha , Ajinkya Tejankar, Soroush Abbasi Koohpayegani , and Hamed Pirsiavash

In Conference on Computer Vision and Pattern Recognition (CVPR) Jun 2022

Abs PDF Code

Large-scale unlabeled data has spurred recent progress in self-supervised learning methods that learn rich visual representations. State-of-the-art self-supervised methods for learning representations from images (e.g., MoCo, BYOL, MSF) use an inductive bias that random augmentations (e.g., random crops) of an image should produce similar embeddings. We show that such methods are vulnerable to backdoor attacks – where an attacker poisons a small part of the unlabeled data by adding a trigger (image patch chosen by the attacker) to the images. The model performance is good on clean test images, but the attacker can manipulate the decision of the model by showing the trigger at test time. Backdoor attacks have been studied extensively in supervised learning and to the best of our knowledge, we are the first to study them for self-supervised learning. Backdoor attacks are more practical in self-supervised learning, since the use of large unlabeled data makes data inspection to remove poisons prohibitive. We show that in our targeted attack, the attacker can produce many false positives for the target category by using the trigger at test time. We also propose a defense method based on knowledge distillation that succeeds in neutralizing the attack.
NeurIPS W.

Can we train vision and language zero-shot classification models without syntax?

Ajinkya Tejankar, Bichen Wu , Saining Xie , Madian Khabsa , Hamed Pirsiavash , and Hamed Firooz

In NeurIPS SSL Theory and Practice Workshop Dec 2022

Abs PDF

Natural language supervision in the form of image captions was recently shown to be an effective way of training zero-shot image classification models. In this work, we focus on teasing out what parts of the language supervision are essential for training zero-shot models. Through extensive and careful experiments, we show that replacing intact captions with Bag-of-Words (BoW) does not significantly degrade the zero-shot performance. Surprisingly, we can even slightly improve the performance on some datasets by balancing the frequency of words in BoW.
BMVC

SimReg: Regression as a Simple Yet Effective Tool for Self-supervised Knowledge Distillation

KL Navaneet , Soroush Abbasi Koohpayegani , Ajinkya Tejankar, and Hamed Pirsiavash

In British Machine Vision Conference (BMVC) Nov 2021

Abs PDF Code

We present a simple framework to improve performance of regression based knowledge distillation from self-supervised teacher networks. The teacher is trained using a standard self-supervised learning (SSL) technique. The student network is then trained to directly regress the teacher features (using MSE loss on normalized features). Importantly, the student architecture contains an additional multi-layer perceptron (MLP) head atop the CNN backbone during the distillation (training) stage. A deeper architecture provides the student higher capacity to predict the teacher representations. This additional MLP head can be removed during inference without hurting downstream performance. This is especially surprising since only the output of the MLP is trained to mimic the teacher and the backbone CNN features have a high MSE loss with the teacher features. This observation allows us to obtain better student models by using deeper models during distillation without altering the inference architecture. The train and test stage architectures are shown in the figure below.
ICCVOral

Mean Shift for Self-Supervised Learning

Soroush Abbasi Koohpayegani* , Ajinkya Tejankar*, and Hamed Pirsiavash

In International Conference on Computer Vision (ICCV) Oct 2021

Abs PDF Code Website

Most recent self-supervised learning (SSL) algorithms learn features by contrasting between instances of images or by clustering the images and then contrasting between the image clusters. We introduce a simple mean-shift al- gorithm that learns representations by grouping images to- gether without contrasting between them or adopting much of prior on the structure or number of the clusters. We simply “shift” the embedding of each image to be close to the “mean” of the neighbors of its augmentation. Since the closest neighbor is always another augmentation of the same image, our model will be identical to BYOL when using only one nearest neighbor instead of 5 used in our experiments. Our model achieves 72.4% on ImageNet lin- ear evaluation with ResNet50 at 200 epochs outperforming BYOL. Also, our method outperforms the SOTA by a large margin when using weak augmentations only, facilitating adoption of SSL for other modalities.
ICCV

ISD: Self-Supervised Learning by Iterative Similarity Distillation

Ajinkya Tejankar*, Soroush Abbasi Koohpayegani* , Vipin Pillai , Paolo Favaro , and Hamed Pirsiavash

In International Conference on Computer Vision (ICCV) Oct 2021

Abs PDF Code Website

Recently, contrastive learning has achieved great results in self-supervised learning, where the main idea is to pull two augmentations of an image (positive pairs) closer com- pared to other random images (negative pairs). We argue that not all negative images are equally negative. Hence, we introduce a self-supervised learning algorithm where we use a soft similarity for the negative images rather than a binary distinction between positive and negative pairs. We iteratively distill a slowly evolving teacher model to the stu- dent model by capturing the similarity of a query image to some random images and transferring that knowledge to the student. Specifically, our method should handle unbal- anced and unlabeled data better than existing contrastive learning methods, because the randomly chosen negative set might include many samples that are semantically simi- lar to the query image. In this case, our method labels them as highly similar while standard contrastive methods label them as negatives. Our method achieves comparable results to the state-of-the-art models.
NeurIPS

CompRess: Self-Supervised Learning by Compressing Representations

Soroush Abbasi Koohpayegani* , Ajinkya Tejankar*, and Hamed Pirsiavash

In Advances in Neural Information Processing Systems (NeurIPS) Dec 2020

Abs PDF Code Website

Self-supervised learning aims to learn good representations with unlabeled data. Recent works have shown that larger models benefit more from self-supervised learning than smaller models. As a result, the gap between supervised and self- supervised learning has been greatly reduced for larger models. In this work, instead of designing a new pseudo task for self-supervised learning, we develop a model compression method to compress an already learned, deep self-supervised model (teacher) to a smaller one (student). We train the student model so that it mimics the relative similarity between the datapoints in the teacher’s embed- ding space. For AlexNet, our method outperforms all previous methods including the fully supervised model on ImageNet linear evaluation (59.0% compared to 56.5%) and on nearest neighbor evaluation (50.7% compared to 41.4%). To the best of our knowledge, this is the first time a self-supervised AlexNet has outper- formed supervised one on ImageNet classification.