Poorvi Hebbar

I am a Masters in Computer Vision (MSCV) student in the Robotics Institute at Carnegie Mellon University, advised by Prof. Shubham Tulsiani. I completed my undergraduate education with a Major (with Honors) in Computer Science and Minor in Physics from Indian Institute of Technology, Bombay in 2021. My primary interests lie in Computer Vision, Artificial Intelligence, Machine Learning, and Optimization.

I am fortunate to have worked with Prof. Ganesh Ramakrishnan, Prof. Nutan Limaye, Prof. Amit Sethi, and Prof. Soumyajit Mukherjee at IIT Bombay, Ramanan Sekar and Sundar Subramanian at Qualcomm Research, Zhitao Lu, Akshaya Sivakumar, and Gaurav Priyadarshi at Goldman Sachs, and, Nityendra Singh at Google.

E-mail-1 | E-mail-2 | Resume | GitHub | LinkedIn

That's me

MS in Computer Vision
Aug 2022 - Dec 2023


Autonomous Driving Intern
May 2023 - Aug 2023


Goldman Sachs
Quantitative Risk Analyst
Jun 2021 - Jul 2022


Google Research Intern
Jul 2021 - Sep 2021


IIT Bombay
B.Tech. Computer Science
Jul 2017 - May 2021


Our brains can interpret what we see effortlessly. I envision a world where computers emulate this ability to decipher visual information and perform complex modeling tasks that currently depend on human cognition or are not yet in our range of perception. I aspire to provide this reliable visual intelligence to artificial agents, with the ultimate goal of making effective autonomous solutions that can assist mankind. Specifically, I am interested in building robust perception systems that can infer the structure of their environment, percieve interactions, and then actively interact in the real world.

My work focuses on 3D vision, mainly on human-object interactions and pose estimation tasks. I am currently working on my capstone project "Reconstructing Human Object Interactions (HOIs) from Internet Videos" along with Yufei Ye, advised by Prof. Shubham Tulsiani. Prior to joining CMU, I have worked mainly on detection of skew robust HOIs, pose-estimation and future pose prediction tasks as an undergraduate student at CSE, IIT Bombay. I also have experience working on anomaly detection in proctoring videos as a part of my Bachelors thesis with Prof. Ganesh Ramakrishnan.

Key Projects

Diffusion Guided Reconstruction of Hand Object Interactions from Internet Clips
Yufei Ye, Poorvi Hebbar, Abhinav Gupta, Shubham Tulsiani
ICCV 2023 (Oral)

Given a monocular video of a hand interacting with a rigid object, we aim to reconstruct the underlying hand-object interactions, i.e., the 3D shape of the object, its pose in every frame, along with per-frame hand meshes and camera poses. Our key insight is to incorporate both view consistency across multiple frames and a data-driven prior of the geometry of hand-object interactions as the object of interest is often partially observed from everyday clips

paper | code (Coming soon)

3D Pose Estimation and Future Pose Prediction
Poorvi Hebbar, Srijon Sarkar, Rishabh Dabral, Ganesh Ramakrishnan
Bachelor's Thesis II

We implement PoseBERT with relative positional embedding to learn pose‑representations from monocular videos. Training the model via a contrastive task (distinguishing true latent from sampled pose‑based negative distractors, we obtain retrieval scores of 277 (259) and estimat pose with 148 (144) MPJPE with(out) a quantizer module. We further use an auto‑regressive OpenAI GPT2 model to predict human motion and achieve competitive mean absolute errors on large-interval (>400ms) actions of the Human 3.6M dataset.

Thesis | slides | code

Anomaly Detection in Proctoring Videos
Poorvi Hebbar*, Rachit Bansal*, Rishabh Dabral, Ganesh Ramakrishnan (collaboration with CodeTantra)
Bachelor's Thesis I

We developed an LSTM‑autoencoder model based on human pose features to estimate the reconstruction error & degree of anomaly in proctoring scenarios. We validated the formulation on NTU‑RGB & MPI‑Inf and personally-collected video datasets action‑recognition datasets to detect aberrant video segments. The code was later adopted by Codetantra, a national online exam conducting platform in India. We aim to optimize the framework in a synchronous setting using online sub‑modular maximization & video summarization.

Thesis | slides | code

Skew-Robust Detection of Human Object Interactions in Videos
Apoorva Gupta, Poorvi Hebbar, Rishabh Dabral, Ganesh Ramakrishnan
Under review

We extract visual features of humans & objects from the input video segments using ResNext and RoI trajectories and implement a novel framework to discern HOIs in a multi‑label setting by classifying the aggregated RoI features. We have outperformed state‑of‑the‑art methods by a significant margin (8%) on the large scale in‑the‑wild VidHOI dataset.

slides | code

Robust Classification of Histology Images Exploiting Adversarial Autoencoders
Nikhil Cherian Kurian, Gurparkash Singh, Poorvi Hebbar, Shreekanya Kodate, Swapnil Raney, Amit Sethi

We propose a novel weighing scheme of training instances based on likelihood of the encoded features in latent space. A consistent improvement in the AUC scores is observed, implying a robust feature generation with optimized priors. Classification accuracy of 80.9% was obtained on noisy multi‑labeled histology datasets.

paper | report | slides | code

Detection of Brittle Shear Zones in Mesoscale Photographs
Poorvi Hebbar, Soumyajit Mukherjee, Narayan Bose
Journal of Indian Geophysical Union (JIGU) 2022

We process images from brittle shear zones to discern fracture planes & their orientation, obscured to naked eye. Using Edge detection & quantization methods, we tried to accurately classify and label fractures without human supervision.

paper | report | code

Predicting the Remaining Useful Lives of Jet Engines
Poorvi Hebbar, Nutan Limaye (collaboration with Tvarit AI)

With sensor measurements of jet engines, we designed an optimized pipeline for predicting their Health Indices (HI). We encoded the intermittent data into a series of images with 21 features and filled the gaps using generative images. A Transfer Learning setup was successfully employed to predict the HI of jet engines with as much as 30% input holes.

report | code

Work Experience

Qualcomm AI Research
Autonomous Driving Intern (May 2023 - Aug 2023)
Managers: Sundar Subramanian, Ramanan Sekar

- Optimized the attention in Cross View Transformers to improve latency from 3.3 to 113 fps for BEV Perception
- Conducted Neural Architectural Search to find backbones that are ~32% faster, while sustaining baseline accuracy
- Surveyed state-of-the-art research in efficient networks & designed a novel search space with proper efficacy analysis
- Accelerated data loading by 2x for cloud based distributed training & integrated with Hardware-In-Loop on Vertex AI

Goldman Sachs
Quantitative Risk Analyst (Jun 2021 - Jul 2022)
Managers: Zhitao Lu, Gaurav Priyadarshi

- Backfilled reliable time series for swap rates and Euro future prices of various currencies to aid the LIBOR Transition
- Developed benchmark, volatility & data quality tests to bless‑unbless trained pricing models based on performance
- Drafted an automated ML framework to identify anomalies in risk metrics(VaR, Topsheet) & expect mitigation values
- Assisted the validation, reconciliation and sign off of a vast amount of risk metrics in the firm’s portfolio

Google Research
Software Engineering Intern (Jul 2020 ‑ Sep 2020)
Manager: Narendra Singh

- Designed a digital content recommendation app for viewing media based on the past left‑right swaps of posts
- Trained a matrix factorization model with collaborative filtering and embedding of the memes vocabulary
- Customized newsfeed based on preferences & general popularity to balance the viewer’s taste & awareness of trend

report | slides | demo | code

Goldman Sachs
Data Analyst Intern (May 2020 ‑ Jul 2020)
Managers: Akshaya Sivakumar, Gaurav Priyadarshi

- Presented a best fitting model to predict PnL & explain the expected changes in Portfolio Risk due to security shocks
- Reframed situations in Gaussian copulas & used Lasso‑based regression optimization on hypothesised parameters
- Achieved 97.3% validation accuracy & reduced the PnL estimation time by 47% for 5K pandemic market‑scenarios

Teaching Assistantships

Awards and Scholarships

Course projects

Improving Latent Diffusion with perceptual mask-aware loss
Abhinav Gupta*, Poorvi Hebbar*, Vibhakar Mohta*
Course: Visual Learning and Recognition (Prof. Deepak Pathak), Spring 2023, CMU

AI-based image generation has advanced significantly with diffusion models, which combine context using text prompts for realistic and diverse images. However, these models struggle with complex objects like human faces and bodies. This work aims to address this limitation by optimizing diffusion latents using a mask-aware loss on human faces and body. By focusing on important aspects like human faces and poses, this can serve as a foundation for finetuning pre-trained diffusion models on more sophisticated loss functions.

slides | report | code

iNERF for 6D pose estimation
Poorvi Hebbar*, Abhishek Tandon*, Siva Ramakrishnan*
Course: Mathematical Fundamentals of Robotics (Prof. Michael Erdmann), Fall 2022, CMU

We implemented the iNeRF paper leveraging Inverse NeRFs for direct Pose Estimation instead of using 3d-2d correspondences. iNeRF takes 3 inputs: an observed image, an initial estimate of the pose (identity matrix), and a NeRF model representing a 3D scene or an object in the image. Different from NeRF, we start from an estimated initial camera pose and iteratively refine the pose (through gradient descent) following the NeRF rendering procedure to compute the loss with the observed image. iNeRF propagates the gradients to the estimated pose and is hence able to recover the correct camera poses by aligning the rendered and observed images.

slides | report

Facial Emotion Recognition
Poorvi Hebbar*, Nama Hari Krishna*, Dileep Kumar*, Ashruth Doddela*
Course: Artificial Intelligence and Machine Learning (Prof. Ganesh Ramakrishnan), Fall 2020, IIT Bombay

A face emotion recognition system comprises of two step process i.e. face detection (bounded face) in image followed by emotion detection on the detected bounded face. We use feature-based cascade classifiers (real time and fast) to detect frontal face in an image. We then train a classification CNN model architecture which takes bounded face (48 x 48 pixels) as input and predicts probabilities of 7 emotions (happy, neutral, angry, disgust, scared, sad and surprise) in the output layer.

report | code

Period Verification and Distance Estimation for UZ Com
Poorvi Hebbar*, Neilabh Banzal*, Pranav Deo*, Prathamesh More*
Course: Astrophysics (Prof. Varun Bhalerao), Spring 2022, IIT Bombay

We estimated the time period of UZ Com, an RR Lyrae variable star using 35 images from the GROWTH India Telescope. Employing an automated approach to calculate zero point using the Photoutils library & Aperture Photometry Tool, we verified the period using the peak value of Lomb‑Scargle periodogram. We further used the Period‑Luminosity relation & metallicity to estimate the star’s absolute magnitude, distance & inclusion in an open globular cluster: Latham I

slides | proposal | code

Mutual support system for elderly and youth
Poorvi Hebbar*, Saral Mittal*, Kshitiz Singhal*
Course: Innovation by Design (Prof. B K Chakravarthy), Fall 2020, IIT Bombay

We built an experience sharing platform, MilJhul, to promote cross‑generational interactions and knowledge exchange. A wireframe model was designed as a prototype for the android app in Figma based on insights of the elderly people.

blog | prototype

Image Quilting for Texture Analysis and Transfer
Poorvi Hebbar*, Sagar Kalsaria*, Aneesh Shetty*
Course: Digital Image Processing (Prof. Suyash Awate), Spring 2020, IIT Bombay

We implemented the original paper to synthesise novel visual appearance of images by coalescing small patches of existing ones. We then extended the algorithm to successfully re‑render an object with a texture transferred from a different 2D image.

report | code

Scene Recognition
Poorvi Hebbar
Course: Computer Vision (Prof. Min Hyok Kim), Spring 2019, KAIST

We obtain the vocabulary of visual words with required features through K‑means clustering and PCA on the scenes. Visualization of images as histogram of words was done using the Bag of words and Spacial Pyramid Representation. We then built a Multi‑class Support Vector Machine with 15 binary one‑vs‑all SVMs to classify with 89% validation accuracy.

report | code

Camera Calibration and Stereo Depth Estimation
Poorvi Hebbar
Course: Computer Vision (Prof. Min Hyok Kim), Spring 2019, KAIST

We estimated the camera’s intrinsic and extrinsic parameters by using Zhang’s method on different views of an image. The disparity and the depth map of given gray image pairs was obtained by the plane‑sweeping algorithm.

report | code

Secure Personal Cloud
Poorvi Hebbar*, Nikhil Reddy Ramolla*, Rajdeep Yadav*
Course: Software Systems Lab (Prof. Soumen Chakraborty), Fall 2018, IIT Bombay

We designed an encrypted cloud‑based file system with database management & accessed it securely via a server. We also mplemented backup service using socket programming along with file‑sharing across multiple linux clients, automated synchronization and auto rendering of common media types on the web client ‑ in NodeJS + Django.

report | code

Conway's Game of Life
Poorvi Hebbar*, Nikhil Reddy Ramolla*
Course: Abstractions and Paradigms for Programming (Prof. Amitabh Singhal), Spring 2018, IIT Bombay

Conway-racket is a Conway's game of life implementation on racket (no, not 🎾). We made a simulator with a grid which can be set for multiple sizes, grids and topolozies and which uses many preloadeable scripts to run it. In other words, we programmed a cellular automaton and used the simulator on files encoded with Run-Length-Encoding.

report | code
Activities and Extracurriculars
  • Sports Roll of Honor by Athletics team, IIT Bombay (2021)
  • Sports Person of the Year by Athletics team, Hostel10, IIT Bombay (2020)
  • Department Academic Mentor - organized help sessions, assisted students in academics, general concerns and helped them cope with the curriculum (2020-21)
  • Initiated a computer literacy program for cafeteria workers under the National Service Scheme at IITB (2020)
  • Internship Coordinator - built and forstered relations with firms and universities to secure and streamline internships and placements for 2000+ students of IIT Bombay (2019-20)
  • Choreographed a broadway performance at the Annual Insync Dance Show; won People’s Choice Award (2020)
  • Represented IIT Bombay in the Long jump and 800m long distance race at the Inter‑IIT Sports, Kharagpur (2020)
  • Won the Gold Medal in Kho‑Kho & Bronze Medal in Women‑800m race at Aavhan, IITB’s Sports festival (2019)
  • Institute Student Satellite Team - modeled the mechanical structure of IITB’s second satellite, Advitiy (2018-20)


Last updated July 2023. Thanks Jon Barron for the template!