Tailoring Skip Connections for More Efficient ResNet Hardware

Skip connections are valuable for training Residual Neural Networks (ResNets), helping them converge to a better solution faster. But skip connections add overhead to the hardware implementation; they require additional on-chip memory and other resources and larger memory bandwidths. Skip connections are like training wheels on a bicycle – they are helpful for learning but get in the way once the learning process is completed.

Tailor is a hardware-software codesign technique that modifies skip connections to be more hardware efficient while maintaining overall accuracy. Tailor gradually transforms a ResNet to remove or shorten the skip connections while iteratively retraining the network. Tailor can remove skip connections on smaller networks. As the network gets larger, removing the skip connections reduces accuracy. In these cases, Tailor makes the skip connections shorter. Perhaps non-intuitively, this reduces resources as the hardware synthesis tool (hls4ml in this case) will implement these shortened skip connections on local memories, which incurs very little additional overhead.

An original ResNet with skip connections in place. The same ResNet model with removed and shortened skip connections.

Tailor started in CSE 237C class in Fall 2020. That was during the height of the pandemic, so the class was totally online. Co-authors Olivia “Liv” Weng, Gabriel Marcano, and Nojan Sheybani were students in this class. Co-author Alireza “Ali” Khodamoradi and Ryan Kastner were course instructors. Ali came up with the idea and pitched it as a potential class project. The project went through several submissions and many rejections, but each time Liv and other co-authors took the reviewer’s comments, added more experiments, and polished the writing. The paper improved and picked up a few other co-authors along the way. Rejections are frustrating but a common (and rarely talked about) aspect of publishing in top venues. It was a journey that resulted in a very strong set of research results.

Links: The Tailor paper in the ACM Transactions on Reconfigurable Technology and Systems, “Davis” summarizes the work nicely, Edge Impulse highlights the project

Qualcomm Innovation Fellowship for Hardware Security Verification

Since 2009, the annual Qualcomm Innovation Fellowship has enabled Qualcomm to help foster the research and development of new technologies in key areas of interest to the semiconductor industry through investments in Ph.D. students and their forward-thinking ideas. Members of winning teams are awarded a fellowship and partner with a Qualcomm mentor to help facilitate their proposed research plan.   

For this year’s fellowship, Qualcomm listed “Secure System Design” as one of their eight areas of interest. Upon seeing this, Andy Meza (UCSD) was inspired to join forces with current research collaborator and former UCSD labmate Colin Drewes (Stanford) in order to submit a proposal, given their shared interest and background in hardware security research. From December 2022 to July 2023, Andy and Colin’s research proposal, “Facilitating Security Verification via Hyperflow Analysis” made its way through the Abstract phase (182 teams), the Full Proposal phase (90 teams), and the Finalist Presentation and Live Q&A phase (43 teams) in order to end up among the winning proposals (18 teams).

Andy Meza and Colin Drewes, 2023 Qualcomm Innovation Fellows.

The funding and mentorship from the Qualcomm Innovation Fellowship will enable Andy and Colin to continue their research into automated methodologies for detecting, exploiting, and, ultimately, mitigating security vulnerabilities in hardware designs. It will also support their sushi-eating and coffee-drinking endeavors, which, as any serious researcher knows, is a critical part of the research process.

Congrats Andy and Colin!

Junkyard Computing

With over 1.2 billion smartphones shipped in 2021, it’s clear that they are a ubiquitous part of our lives. Smartphones have a relatively short lifespan; the average upgrade cycle is two years. People replace them for different reasons — they want a better camera, newer software features, the screen becomes too broken to be usable, or the battery life doesn’t retain a charge. However, even if the screen is cracked or the battery life has diminished, the discarded devices can be repurposed to perform valuable computations.

The relative carbon costs for smartphones. Manufacturing or embodied carbon dominates, far surpassing the carbon associated with the energy required to power the device over its lifetime.

The rapid consumption of smartphones comes with significant environmental costs. Manufacturing smartphone computer chips is incredibly carbon-intensive. This embodied carbon outweighs the carbon costs of powering the smartphone over its entire lifetime. Repurposing smartphones to extend their life is critical to reducing their carbon footprint. 

The Junkyard Computing Project aims to repurpose these unwanted smartphones for useful computation. This computational stockpile represents a huge wasted potential; these smartphones have a high-performance and energy-efficient processor, extensive networking capabilities, and a reliable built-in power supply. This project gives a second life to older, discarded, idle smartphones.  The project’s research addresses critical technical questions of transforming a user-optimized, interactive device into a robust, reliable device capable of long-term, unattended operation.  It develops new metrics to capture manufacturing and operational carbon costs, couples these with economic models, and establishes a roadmap for the best opportunities for old electronic devices.  Finally, the project tests these ideas at scale to empirically establish how to use phones-as-compute and phones-as-sensors. We have shown how to repurpose smartphones as a web server for the project, as cloudlets for microservices for social media websites, and as wildlife monitoring sensors.

Jen Switzer‘s ASPLOS Lightning Talk

The project has gained a lot of attention. In the popular press, it had articles on Hacker News and Hackaday. Our ASPLOS paper has over 50,000 downloads making it the most downloaded paper in ASPLOS history (28 years). It was given a Distinguished Paper Award at ASPLOS. We were awarded a National Science Foundation grant as part of the Design for Environmental Sustainability in Computing program. A Google gift allows us to work closely with Dave Patterson, Herman Schmit, and other Googlers to develop racks of repurposed smartphones.

Finally, it should be noted that “Junkyard Computing” is the unofficial project name. The National Science Foundation politely asked us to remove “Junkyard Computing” from the title of that grant. Of course, we obliged. We wouldn’t want an overzealous senator to wrongly accuse us of wasting taxpayer money! We promise that we are spending US taxpayer dollars responsibly. Dave Patterson coined the name “Redundant Array of Inexpensive Smartphones (RAIS)” for cloud repurposing efforts. He didn’t think that “junkyard” was a good way to sell the project to Google’s upper management. Who are we to argue with a Turing Award winner and legendary acronym maker? Regardless of the name, our research will find new homes for powerful but unwanted smartphones and lead to a more holistic, carbon-centric view of computation


We had a busy, but enjoyable time at the International Symposium on FPGAs. It is always great to visit the beautiful Monterey Peninsula.

The festivities started on Sunday, when Dustin Richmond, Ryan Kastner, Jeff Goeders, and Mirjana Stojilović organized the Third Workshop on Security for Custom Computing Machines (SCCM) on Sunday. This hybrid event had 30+ people in person and another 30+ online. The presentations have all been posted on the SCCM website, which also has videos from last year. As part of this, Colin Drewes gave the first public presentation of our Pentimenti project, which describes a novel temporal analog side channel in FPGAs.

Point Lobos State National Reserve

On Monday, Colin presented his UCSD MS research developing a time-to-digital converter (TDC) for FPGA security applications. TDCs are common sensors for measuring the on-chip voltage and inferring behaviors of other system tenants. Our Tunable Dual-Polarity TDC (repo, paper) can hone in on important times in the computation, e.g., around clock transitions, and extract more information about co-tenant computations. The project was one of the three best paper nominations. Unfortunately, we did not get the best paper, but we are honored to be in the top three. The research was a strong, multi-university collaboration between UC Santa Cruz, UC San Diego, and Georgia Tech Research Institute. Congrats to all the authors!

On Tuesday, Olivia Weng presented her research on co-design optimizations for implementing neural networks on FPGAs. This research is part of the Tailor project, which performs network architecture modifications that remove or shorten skip connections to optimize on-chip memory architectures.

NSF CSGrad4US Fellowship

Andres (Andy) Meza was awarded a 2022 National Science Foundation Computer and Information Science and Engineering (CISE) Graduate Fellowship (CSGrad4US). CSGrad4US “aims to increase the number and diversity of domestic graduate students pursuing research and innovation careers.”

Andy is a Research and Development Staff in our group working on hardware security and ML hardware acceleration. He plays a critical role in much of our group’s research. These include hardware security verification projects with industry (Intel, Cycuity, OpenTitan, Leidos, the Semiconductor Research Corporation) and academia (Cynthia Sturton’s group and Calvin Deutschbein). As if this isn’t enough, he also works on the hls4ml project on topics related MLPerf Tiny benchmarks and fault tolerance. To date, he has seven(!) publications at top venues with several more in submission.

Andy started with our group as an undergraduate. After graduation, he continued working on research projects and moved into his R&D staff role. Andy is applying for Ph.D. programs and will undoubtedly continue his research career as a Ph.D. student starting next academic year. We hope he stays at UCSD!

Congrats Andy on this well-deserved honor!

Prof. Dustin Richmond Starts at UC Santa Cruz

Kastner Research Group alum Dustin Richmond started as an assistant professor in the Department of Computer Science and Engineering at UC Santa Cruz this Fall. Dustin got his Ph.D. in 2018 and spent a few years as a post-doc at the University of Washington.

For those of you that are keeping count, that means Dustin has transformed from a Husky (undergraduate) to a Triton (Ph.D.), back to a Husky (post-doc), and now a Banana Slug?! Clearly, he likes the West Coast.

While he has been gone for awhile, he is certainly not forgotten. We have continued to work with Dustin (and now his new research group) on projects related to FPGA security.

Dustin thinking about how one day he will be a banana slug.

Dustin in the much easier, carefree days of graduate school.

Colin Drewes: An Award-Winning Master Researcher

Congratulations to Colin Drewes for being awarded the 2022 UCSD Department of Computer Science and Engineering MS Research Award. Colin worked in our research group for almost three years – first as a CS undergraduate and during the past year as an CSE MS student. He was the de factor leader on one of our major research projects – a large multi-institution project between UCSD, University of Washington, and Georgia Tech Research Institute. The results of this project are an impressive string of publications (a couple still pending). Colin continues his graduate career as PhD student at Stanford University working with Caroline Trippel starting this Fall. Colin – you will be missed. Best of luck at Stanford!

Colin appears around the 15 second mark

RAGE Best Presentation Award

Congratulations to Andy Meza for giving the best presentation at the Real-Time and Intelligent Edge Computing Workshop co-located with the Design Automation Conference. Andy presented our research on “Safety Verification of Third-Party Hardware Modules via Information Flow Tracking” a collaborative project with Francesco Restuccia and Jason Oberg from Cycuity. The work describes how to use information flow tracking to verify the safety of bus interactions among on-chip hardware resources.

Dr. Peter Tueller Rocks His PhD

A long overdue congratulations to Dr. Peter Tueller for successfully defending his Ph.D.

Peter joined our group in Fall 2016 despite me doing one of the worst recruiting jobs ever. He was doing outstanding undergraduate research to build a fleet of underwater vehicles. Around the same time, we were developing localization algorithms for swarms of underwater vehicles. It seemed like a perfect fit! I saw his UCSD graduate application many times — several colleagues forwarded it to me saying that I should make sure not to miss it. At the time I was on a sabbatical and not looking to take on more students. After the third or fourth time of someone telling me I should check out his application, I finally got around to it. I made him a very late offer, which was right before visit day. I asked if he could come and stick around another day so that I could meet him as I was traveling during the official visit days (again, I wasn’t planning on taking students that year).

Our first meeting still sticks in my mind. We met up at a taco restaurant in La Jolla Shores. I figured I didn’t have much of a shot at convincing him to come to UCSD. I was apologizing for making a late offer, not being a visit day, etc., but also trying my best to convince him that I was still really interested in working with him. I was doing a terrible job of recruiting him. But he rather nonchalantly stated that he was planning to come to UCSD. Lucky me!

Peter had a Ph.D. career that I will forever be jealous of. He spent a couple of months at the University of Haifa in Israel working with Roee Diamant on an underwater sonar project. He made expeditions to Guatemala to perform 3D scanning at the archaeological sites of El Zotz and Tikal. He worked most summers at NIWC Pacific developing different underwater technologies. He was the leader and driving force behind getting the FishSense project up and running. He got a SMART scholarship that allowed him to pursue Ph.D. research largely of his choosing related to 3D vision systems. This allowed him to move directly into a job at NIWC after graduation, which was what he wanted to do all along.

And all of this was done as a second career to being a rock star.

Despite being a rock star, Peter was a cool and calming influence on our group. I always looked forward to our weekly meetings — great research discussion and lots of laughs. He was a tremendous mentor to undergraduates and the other new members of our research group.

Congrats Dr. Tueller! You will be missed. Keep on rocking!

Engineering a Smarter Surfboard

The Smartfin project was highlighted in a ThisWeek@UCSD article. The Smartfin holds a microcontroller, temperature sensor, inertial measurement unit, and wireless radio — all embedded into a surfboard fin. This allows surfers to opportunistically gather oceanographic data in the near-shore environment, which is otherwise challenging for more traditionally sensors on buoys and moorings. Engineers for Exploration students are working this summer as part of the NSF-funded REU Site to solidify the data collection process, and develop in-house ability to manufacture Smartfins in a low-cost and open-source manner.