Julia Chou

Experiences From a 10 Day Silent Meditation Retreat




The latter half of 2018 saw me turn my attention towards focusing on personal development and self-improvement, and as part of these efforts, I began to explore ways to strengthen my emotional intelligence and self-awareness. I’ve paid a lot of attention to improving my physical fitness and nutrition for a while now, but I realized that one of my biggest shortcomings was failing to pay corresponding attention to my mental and emotional health. I wanted to be able to communicate better, understand others, and understand how to make more mindful, balanced, and deliberate decisions.

I picked my way through some literature surrounding improving your social and emotional intelligence, and I noticed that a common pattern among many successful people is their practice of meditation, to which they give credit for drastically improving their quality of life and emotional stability.

I have never particularly identified myself as either being very spiritual or religious, and I had always thought of meditation as something one practices for those reasons. Perhaps I had retained a Western unfamiliarity with the practice that made the concept seem foreign to me, though it is something people have been practicing for thousands of years. However, reading through people’s personal testimonials about how meditation can improve one’s self-awareness and focus, help create emotional balance, reduce anxiety, and remove distractions opened my eyes to the fact that there are very pragmatic reasons outside of spirituality and religion to explore meditation.

As I was reading through a book called Never Eat Alone, the author spoke of his time at a 10 day Vipassana meditation retreat taught by a man named S.N. Goenka as a truly transformative and transcendent experience that changed his whole worldview. He now retreats for 10 days of silent meditation on a yearly basis.

Intrigued, I found the website for that Vipassana course and learned they offered a session at a location about an hour and a half outside of San Francisco (where I’m currently based), so I decided to go ahead and register. The course took place from December 23rd to January 3rd, so I would be spending the bulk of my holiday time off there. The organization is run on a donation basis, providing courses for free to new students and only accepting contributions from those who have completed the full 10 days. They even helped coordinate a rideshare board, so I didn’t have any financial or logistical concerns.

I’m a fairly introspective and contemplative person already, so spending 10 days sitting around, not talking to anyone, and discovering new ways to improve myself sounded like something that would be completely in my wheelhouse. There was much to potentially gain and little to lose!

As of this writing, I’ve had some time to calm down and process my thoughts from my experiences at this retreat. I’m still in a little bit of disbelief that I so nonchalantly signed up to meditate in silence all day for 10 straight days and actually made it through. Below are some observations I’ve taken away from my time there.



DAY 0

I arrived at the retreat on the afternoon of December 23rd, the day before the first official day of meditation began. I discovered that the site was actually a Catholic youth summer camp that the Vipassana center rented out in the winter to host this class. The organizers did their best to create a neutral space by taping over signs and covering up any indication that this was anything but a place intended for quiet contemplation. 

I went through the women’s registration line where I signed a liability waiver and was assigned a top bunk in a 12-person cabin. I was also encouraged to turn in my cell phone for safekeeping during the course, but I opted to hold onto it and just keep it turned off in the bottom of my bag throughout the course. Reading and writing materials were prohibited.

Students had the opportunity to mingle and get to know one another a little bit before Noble Silence officially began. The term is a little grandiose, but effectively it means that students must submit to the silence of body, speech, and mind, and no communication whatsoever, whether through speech, hand gestures, or eye contact, is permitted. Students were only allowed to converse with teachers or assistants if they had any questions or concerns about the practice or the technique.

A large portion of the student population seemed to be from the Bay Area, with several outliers from states like Florida and New York. Paradoxically, it seemed to me, some students came with their partners or families to this silent meditation retreat to labor in isolation together. 

There were numerous returning students, but many, like myself, had little to no experience with meditation whatsoever and were just there to challenge themselves, try new things, and learn what meditation was all about.

Overall, there were about 240 students in total.

During orientation, the deep, croaking voice of S.N. Goenka (who passed away in 2013 but lives on through audio and video recordings in these courses) played through the speakers and instructed us to fully surrender yourself to the practice of Dhamma and Buddhism and to adopt several precepts throughout the next 10 days. Five of the precepts forbade you from killing, stealing, lying, engaging in sexual misconduct, or taking any intoxicants. If you were a returning student, you had to adopt three additional precepts instructing you not to eat after noon, wear any sensory decorations such as jewelry, or sleep in high or luxurious beds.

The vows seemed to just be a formality, as we were all in an environment that made it extremely difficult to disobey most of the precepts. I wondered idly, though, whether or not the top bunk in my 12-person cabin could be considered “high” or “luxurious.”

At 8:00 pm, Noble Silence was officially underway.

DAYS 1–2

Each day at 4:00 am, volunteers went around to each cabin clanging a gong to awaken its residents. They would perform this duty before each meditation session, the last of which ended at approximately 9:30 pm. Students had only a few hours set aside for eating and resting. Each evening, we would sit and watch a discourse video in which Goenka would explain meditation techniques, answer some frequently asked questions, and share little Zen koans. Then, he would introduce a new progression in our meditation technique for us to practice the following day.

All students sat in the same meditation hall, and men and women were seated in separate halves of the room. Each student had his or her own designated blue meditation mat and cushion, though extra cushions, blankets, and meditation benches were available if so desired. I amused myself watching some students construct veritable pillow fortresses on top of their meditation mats. We were to spend hours upon these mats, eyes closed, face forward, and back straight, for the next 10 days.

We began by learning anapana meditation, which is the practice of simply observing your respiration. As you observe your respiration, you train your mind and sharpen your focus in order to concentrate fully on the way your breath feels as it rushes through your nostrils and across your upper lip. As you did this, you were supposed to keep any distracting thoughts at bay.

I duly sat, focusing on my breath, trying not to think and trying not to think about not thinking.

I quickly discovered that I’m not nearly as patient a person as I thought I was. I became frustrated within the first few days by being physically confined to a small campground and instructed to not partake in any physical activity beyond simple stretching. Course boundaries were clearly demarcated by yellow caution tape and signs, and students were not to cross these lines. 

I’m accustomed to living alone in a studio, so it was quite an adjustment for me to share a cabin with 11 other people. It was alarmingly easy to dehumanize people and think of them as nothing but mindless, thoughtless automatons when you can’t look at them and you don’t even know their names or the sound of their voices. I gave people nicknames based on the names of colleges or tech companies emblazoned on their sweats. When I couldn’t empathize with them much at all, it became all the easier for their loud footsteps, snores, coughing, and sneezing to drive me insane.

During the meditation sessions, part of me thought it was a little morbid that we were all listening to recordings of a man who died years ago. But in a way, it’s a beautiful thing that his legacy lives on through these courses and still impacts so many people.

DAY 3

As I adjusted to the schedule and became accustomed to the daily routine, my frustration and irritation grew. I began to berate myself for running off into the woods to sit around and think about myself as if I had no real problems to deal with or no better way to spend my time. I could have been lifting weights, taking online courses, or working my way through my reading list. Instead, I was trapped within the tiny confines of this children’s summer camp without a book, a journal, or a phone. At the same time, however, it was nice to be able to give free rein to my thoughts and my thoughts alone.

I must not have been alone in my frustrations, as one person in my cabin gave up and left early. Then there were eleven.

It vaguely crossed my mind how odd it was that I was spending Christmas and New Year’s with a large group of strangers in total silence. I wondered what was going on in the outside world, what kind of celebrations people were getting up to, what new political scandals had occurred, and how financial markets were performing. (As it turned out, the government was shut down and financial markets were experiencing a serious case of the doldrums).

Throughout the day, I often found myself practicing the art of sleeping while sitting up rather than that of meditation.

I thought I caught the occasional whiff of weed around the cabins and the dining hall, which was somehow both surprising and unsurprising.

There were moments of peace and serenity. The scenery was calm, the air was clear, the sun was bright, and the days were beautiful. Once or twice, I caught the sight of several deer bounding through the camp. At night, the stars were just gorgeous, shining with such clarity and brilliance that you would never witness in the city. 

Outside of these moments of gratitude, I quickly started to look forward to getting back to my regular life. I started a nightly ritual of mentally incrementing how much progress I had made through the course.

10% done… 20% done… 30% done…

DAY 4

On the fourth day, we were introduced to the practice of vipassana meditation. The purpose behind spending the first few days concentrating on your breath was to sharpen your ability to concentrate on the sensations in a small area of the body, and in vipassana, you then extend that focus to other areas of the body, starting from the top of the head and traveling down to the toes. 

In the process of doing so, you become aware of the sensations in your body, whether that be itching, aching, numbness, coldness, or the feeling of your heart pulsing. Then, you observe them as they are, divesting any sort of positive or negative interpretation of them. As you focus on them, you notice that these sensations are constantly shifting, illustrating one of the principles of the whole meditation experience: everything (whether that be emotions, sensations, life situations) is fluid, impermanent, and in constant flux. If you can recognize that, you can allow yourself to be fully aware of the present moment but also prevent yourself from becoming too fixated on it because you know it’s not going to last.

With this practice of self-observation, you come to realize that nothing is inherently good or bad. Your own biases and perceptions are the only things telling you that they are.

As Goenka says…

One must learn to be aware of all the different sensations without reacting to them, accepting their changing, impersonal nature. By doing so, one comes out of the habit of blind reaction, one liberates oneself from misery.

DAY 5

Some days, students were called up in groups to meet with their teachers, where the teachers would provide pointers and make sure you were practicing the technique properly.

At these times, I answered questions like “Do you feel your equanimity growing stronger?”

Why, yes, I suppose so.

To the best of my understanding, equanimity means having the ability to sit on a mat for hours on end, back aching, butt-sore, legs numb, and think “Eh. This will pass.”

It seems that forcing yourself to endure discomfort is a universal way to train your mind and strengthen your resilience to adversity.

I do wonder, though, whether maintaining the same posture for hours and cutting off circulation ever results in any health issues for long-time meditation practitioners.

The passage of time was excruciating and each day seemed interminable. My only source of entertainment was the constant stream of my own internal dialogue, and the inability to record any of my thoughts was the most painful part of the whole process.

It would have been much easier if I were allowed to acknowledge the existence of the hundreds of other people here. Though I don’t miss conversing with people when I’m actually alone, it was just absurd that we were a group of about 240 people walking around in contrived little bubbles of isolation bumping into each other in the meditation hall, waiting together in line at the dining hall, and going to sleep and waking up together. It would have been such an eerie scene for an unknowing person to stumble upon, this silent community in the woods. 

Once, however, one of my cabin-mates asked me in a whisper whether or not I had a pair of shower shoes, and my heart nearly burst with happiness at this moment of human connection.

DAY 6

Little moments of joy prevailed. Even though the organizers and teachers did their best to maintain an air of solemnity in the meditation hall, and I had difficulty keeping my composure on many occasions. 

People were nodding off all over the place. Once, the girl next to me fell asleep, slipped off her meditation bench, and nearly face-planted into the girl in front of her.

As it turned out, no matter how diligently Noble Silence was observed, the meditation hall was anything but silent. In fact, it was difficult not to become irritated by all the distracting sounds. One guy kept burping every 10 minutes or so and managed to keep that up for several days. Multiple people caught colds, and their coughing and sneezing became a fixture during meditation sessions. I had a most uncompassionate fear of being infected by someone around me.

The managers overseeing the women were strict about showing proper respect in the hall by not lying down, stretching excessively, or pointing your feet towards the teachers and would come to chastise you if they saw you doing any of the above. Meanwhile, I would occasionally glance over to the men’s half of the hall to see some of them sprawled out and actually snoring on their meditation mats.

Once, I saw a man blowing a kiss towards the direction of the women’s side of the meditation hall. He was wearing a wedding ring, so presumably, that was for his wife. 

“How sweet,” I thought. “But also, how dare he break the rules!”

On another occasion, one of the men rushed out of the hall to the bathroom in order to throw up violently, and the whole hall of 240 people pretended nothing was amiss and just carried on with their meditations.

We were warned that the practice of meditation may bring forth a lot of deep-rooted negative emotions, and this may ostensibly have been a physical manifestation of the eradication of the complexes nestled in the deepest levels of his consciousness. Alternatively, it is also possible that he just ate a bad piece of fruit.

DAY 7

I felt I had reached a tipping point. I was about three-quarters of the way through the course, and I tapped an inner reserve of patience and discipline. The cadence of each day was familiar to me now, and I felt I could calmly pass through the rest of my time here.

Watching Goenka speak in his discourse videos was like watching a kindly old grandpa reading you little parables and teaching you how to build character. At this point, I was really buying into his messages of compassion, peace, patience, tolerance, and forgiveness. I mean, who wouldn’t? There is so much beauty in the human spirit, so much love in the soul! It was completely worth coming to this retreat in order to experience these uplifting moments of clarity and connection.

DAY 8

…Only a few more days left of this stupid shit.

DAY 9

The more fixated I became on my liberation, the slower the time flowed. I looked forward to Goenka’s discourses, as they were the only source of any stimulation throughout the entire stay.

I was sitting deep in thought after one particular discourse one evening, pondering a parable Goenka had just told. In this story, a man receives a simple silver ring with a few words etched onto it: This will also change. The man wears the ring and goes throughout life, growing crops through high seasons and low seasons, the ring serving as a constant reminder that both bountiful harvests and droughts come and go.  I sat pondering Nature’s inherent impermanence and thinking of all the times in my life when I became overly attached to a set of circumstances only to become devastated when they were inevitably altered when my musings were abruptly interrupted by the horrible fart of a girl sitting near me. 

I guess this also illustrates change.

Despite the managers' attempts to maintain order and respect, the atmosphere in the meditation hall was unfortunately punctuated by frequent Noble Farts.

I have a suspicious nature, and the vegetarian menus they planned frequently featured beans quite prominently, so I think they definitely knew what they were doing.

It was around this time that I also realized that I had gained the ability to tell how much time had passed during meditation hours based on the degree of numbness in my butt.

DAY 10

10:00 am  -  Noble Silence ended. 

Noble Speech was now permitted.

They arranged for Noble Silence to end a day before students left the retreat in order to provide a buffer to ease their transition back into regular society. Talking in the meditation hall and physical contact were still prohibited.

A huge amount of relief washed over me when I was finally able to witness that these people I had been living in silence with for the past week and a half are actually thinking, feeling, and sensing human beings who are really funny and very kind.

This speaks more to my own preconceived notions and biases, but I was wondering how many people were proponents of the healing powers of spiritual energy and mystical auras. In fact, I did overhear some mention of Reiki and crystals, but most of the people I talked to were not quite that free-thinking. 

We discussed the practicalities of adopting the precepts into our ordinary lives (what constitutes sexual misconduct, anyway? Is it improper to have sexual desires?). We learned each other’s occupations as accountants, engineers, contractors, and project managers. We shared life stories, traded Instagram handles, speculated about the lives of our teachers outside of these meditation retreats, and joked about the little absurdities we had noticed during our time of silence but couldn’t share.

As it turned out, the four teachers who taught this session were a pair of married couples. The two women were a school teacher and principal in the education system in the Bay Area. I didn’t get the chance to find out what the men’s occupations were.

I was only slightly disappointed to learn that they weren’t some celestial beings who annually descended from their higher plane of existence in order to teach this course.

DAY 11

Goenka had one last lesson yet to impart before the course officially ended, and that was to practice sending metta, or a sense of loving-kindness, to others after each meditation session. After spending so much time in introspection and building your self-awareness and establishing your own sense of emotional balance, you then reflect that back upon those around you.

After the final meditation session of the course, the organizers set up information booths in the dining hall so students could pick up literature on how to maintain this practice now that they’d finished this course. The way they set up these displays, complete with plastic tables and corrugated cardboard tri-fold boards, reminded me of science fairs at my old elementary school.

I picked up a booklet about how to incorporate meditation into your life as a daily habit and a brochure titled The Art of Living: Vipassana Meditation.

In their closing notes, we were all encouraged to join the Vipassana Facebook group and volunteer to serve in future iterations of the course before finally being released to go our own ways.



Overall Takeaways

This was a positive experience and I’m glad I did this; however, I wouldn’t opt to repeat this particular course. I think dedicating 10+ hours a day to meditation is excessive and an inefficient use of my time. 

That’s not to say I don’t believe the technique is incredibly beneficial, however. I do believe that making a daily habit of simply being calm for a little bit will do wonders for reducing stress and anxiety. 

I found that reintroducing myself to all the sights, smells, and sounds in San Francisco was a little bit jarring. It wasn’t until then that I truly realized how many distractions there are in my day-to-day life and how they were constantly fragmenting my attention. As a result of being able to identify these distractions, I am better positioned to eliminate or ignore them and redirect my attention toward tasks that I want to complete. I used my phone’s “Digital Wellbeing” feature to limit the time I spend on my phone each day and blocked off two hours a day for more productive tasks like reading or writing. I hope that this increased focus is something I can maintain through the practice of meditation.

Goenka recommended that students meditate two hours each day: one hour in the morning and one hour at night; however, I think I’ll aim for thirty minutes a day for now.

Everybody gets a little something different out of going through even the same experiences, so I’m not sure to whom I would recommend this course. You might emerge feeling like you’ve just undergone some spiritual metamorphosis, as many have done, or you might come out thinking it’s all utter bullshit. However, if any of this sounds intriguing to you in any way, you can check out their website for yourself and explore further.

As for me, I found my time here immersive and valuable, and I plan to continue reading up on Buddhism and other belief systems on my own. I’m looking forward to seeing where this path leads me.

On Finding Solace From an Existential Crisis


The mass of men lead lives of quiet desperation

I wished to live deliberately, to front only the essential facts of life, and see if I could not learn what it had to teach, and not, when I came to die, discover that I had not lived

-- Henry David Thoreau


These two of Thoreau’s most famous quotes from Walden were the words that knocked me free from the clutches of an arrogant self-assuredness I hadn’t known possessed me in my sophomore year of college. In its place, I felt the creeping numbness of a slow existential dread. For most people, college is a time of self-exploration and self-discovery, a chance to test the limits of your own capacity and establish the foundations of your career. As for myself, I took to heart that very last goal to the exclusion of all others. I was going to graduate with a degree in Computer Science, snag a lucrative job in Silicon Valley, work for a few decades, save up money, and finally retire to an idyllic life squirreled away in a cabin in the woods where I would write the rest of my days away.

I was struggling at the time, pouring a lot of my energy into balancing three jobs alongside the study of a subject that did not come intuitively to me, to say the least. During my sophomore year, I decided to take a literature course with an amazing professor who introduced me to the works of Thoreau, resulting in a dramatic shift in my perspective. Suddenly, everything I was doing seemed like an endless and meaningless toil. I realized that this future I was so single-mindedly pursuing was only a vague specter and I had no idea if it was going to be what I wanted when I got it. What were the things that were going to be important to me when I woke up some 40 years down the road? When it came time for me to die, was I going to discover that I had never truly lived?

It wasn’t so much that choosing a career path based primarily on the money-making potential is inherently wrong, it was the realization that I had made that commitment without much deliberation or forethought whatsoever as to what else in my life I wanted to achieve and what I would be sacrificing in order to pursue this. I knew that my natural affinity gravitated towards literature and the art of language and story more so than hard science and engineering, but I was afraid that the opportunity cost of investing any of my time and energy towards nurturing those interests would be my chance to break into the realm of software engineering and a life free from financial destitution.

This crisis of identity fed into a larger, more general crisis of existentialism itself. After all, how could I decide what meaning I wanted to add to the world if I didn’t even have a proper understanding of myself and my own values and beliefs?

At the time, I sought to find some answers in Albert Camus' work exploring a concept he referred to as “absurdity,” that is, the act of trying to seek meaning in a meaningless world. Camus' spin on the subject is that any attempt to shape the chaos of the world into some form of sense is futile, and he presents the Greek mythological character Sisyphus as the ideal characterization of the struggle with absurdity. Sisyphus is a man who is doomed in the afterlife to an eternal cycle of pushing a boulder to the top of a mountain only to have it roll back down again, and Camus uses his labor to exemplify the toils of mankind at large. It is only on Sisyphus' journey back down the mountain that he has full agency. He accepts the inevitable futility of his efforts and decides to embrace it regardless, and in that moment, he has complete mastery of his fate.

Reading through these concepts helped put my mind at ease just a little. I imagined that all I had to do was accept that anything I did in life was eventually meaningless, so I might as well drive my initial agenda forward. I pushed aside my thoughts in order to focus on making my engineering career a priority. The next few years witnessed me graduating with my Computer Science degree, moving to San Francisco, and finding work as a software engineer at a tech startup. On the surface, it appeared that I was well on track to achieve my initial dreams and aspirations, but a vague, uncomfortable feeling of malaise persisted through my everyday life.

As it goes, a recent loss of a very important connection to me, along with its associated hopes, dreams, and expectations, unearthed all these underlying insecurities and sent me reeling deeper into an identity crisis than I have ever been. It’s inevitable that one experiences several major losses throughout their lifetime, but the worst ones are the ones that completely blindside you and leave you with a lingering sense of incompletion, of missed opportunity. You become helplessly mired in thoughts of how you could have done things differently, done more, or just been better.

Everything around me suddenly seemed to take on a new sense of gravity, and I found myself questioning and reevaluating every single aspect of my life. Does every relationship, every material possession, every belief, every value, every motivation, and every goal I have truly embody who I am, what matters to me, and what kind of life I really want to lead?

I was completely swept away in a maelstrom of despair, self-inadequacy, and doubt, and almost overnight, my emotional well-being completed a beautiful and transcendent metamorphosis into a flaming pile of garbage. There has always been a slight disconnect between how I feel and what I think, such as when I feel irritated by having to sit in heavy traffic even though I know there’s nothing I can do about it. Never before have I felt such a gaping chasm between the two, however, and the small, hyper-rational part of me could only shrink away in awe and incredulity to observe the mad stampede of emotions rampaging through me. I’ve usually been the sort of person for whom rationality and logic ought to prevail in all decision-making, but I suppose for matters of the heart, reason can only follow. I could only use this as an opportunity for some serious self-examination, to try to embrace my emotions and assess and harness them to drive my own behavior.

It just so happened that a former college roommate of mine scheduled a visit during this time, providing me some refuge in the old comforts of a familiar friendship. She’s an amazing, well-grounded, thoughtful, and hilarious person, and seeing her again reminded me of a lot of wonderful qualities inherent in her that I respect and aspire to attain myself. We wandered all over the city, wove in and out of tattoo parlors and dive bars, and took a road trip up to Tahoe, all the while having fun and being ridiculous at a time when having fun and being ridiculous have been the absolute last things on my mind.

She’s the sort of person with whom I can talk about anything and everything, and every conversation with her is a chance to learn something new. Our conversations freewheeled from immigration reform and border-control policies, having children (always a topic of discussion among women, it seems), the morality of suicide, and what it means to be happy to what we think our own deepest flaws are. The ideas that we discuss and bounce off one another never hit a wall where one person simply refuses to change their point of view. Rather, the only barrier we hit is when one of us realizes that she knows too little about the subject for the conversation to be productive any longer.

As much as this has been a time for me to reflect on how I need to work on myself and the next steps I need to take in order to grow and improve, this recent visit my friend paid me also opened my eyes to just how amazing the interconnection of human relationships can be. As I reconsider all of my initial anxieties in college about finding purpose, I realize that the works of Thoreau and Camus that affected me so profoundly emphasized only a solitary search for meaning, making no mention of what reassurance an individual might find in the associations with those around her. I’m now starting to properly appreciate how many wonderful, kind, and generous people there are around me. I treasure the friendships that I have very highly, and I’m making it my mission to live a life where those I call my friends can say that I had a positive impact on them.

This has been a textbook exercise in psychosocial development for me, in which I have been trying to reconcile my own sense of self within the context of my surrounding social environment. Through it all, I think I’ve finally managed to consolidate what carries meaning for me from all of the experiences that I’ve had so far. What’s important to me is continual self-improvement, measured in part by how I foster and cultivate my relationships with those around me. I want to always be exploring more, learning more, and becoming a better person, and I want to find the people in my life that make me a priority and focus on them as well. To that end, I’m setting tentative, longer-term goals for myself and taking the smaller, more immediate steps necessary in order to reach them. For instance, I imagine that somewhere along the path of my software engineering career, I’ll be confronted with the choice of becoming either a manager or a more senior individual contributor. Regardless, I’ll need to work on my leadership skills, so for now, I’m setting immediate goals to improve my speaking, writing, empathy, self-awareness, and overall communications skills.

I am still being constantly surprised by the seemingly limitless bounds of my own arrogance and naiveté. I also know that I’ll never reach a state where I am completely secure about all the decisions I’ve made and the life I lead. However, instead of being dragged down by the weight of my own doubts and insecurities, I’m learning to find peace in the acknowledgement of the perpetuity of my imperfection. Clearly pinpointing all of my flaws will allow me to confront them and work through them, laying out the path I need to follow and allowing me to incorporate an element of deliberation in the everyday choices I make.

I know there is still a great deal of uncertainty in my future. The steps I’m taking today will likely lead me to places that look wildly different from where I expected to be in 10, 20, and 30 years. However, I believe true importance lies not in what the larger narrative of my life ultimately looks like but in being mindful and finding the joy in the instances of human connection in each of the small moments that comprises it. If I die tomorrow, I’ll be at peace believing I made something of the time that I had here.

Though I rather suspect that some thread of existential agony will weave permanently throughout my state of being, this most recent struggle has helped me to uncover which ideals truly matter to me at this moment, and I have found much solace in that.

Sidekiq-ifying Emails at Reflektive

At Reflektive, we offer the option of sending reminder emails after kicking off performance review cycles in order to notify participants of that cycle that they should fill out their reviews. We used Delayed::Job, an asynchronous priority queue system pioneered by the Shopify engineering team, to handle the delivery of those emails.

However, we noticed that there was a huge bottleneck when it came to delivering emails to cycles with large numbers of participants. For instance, one company that had approximately 30,000 employees would clog up our worker queues for hours, preventing other companies from sending out their own emails while this large job was still being processed.

One solution might have been to just throw more hardware at it, to spin up more DJ workers to hammer through the job queues. However, we decided to explore the option of using Sidekiq as an alternative asynchronous job processing system.

Some of the factors leading to choosing Sidekiq included the fact that Sidekiq has higher concurrency than Delayed::Job because it leverages threads as opposed to single processes.

In addition, Delayed::Job stores its queued jobs in a database table (in our case, Postgres) while they’re waiting for a DJ worker to process it. We opted to pair Sidekiq with Redis, an in-memory datastore. This results in a much lower I/O cost because the threads processing Sidekiq jobs don’t have to make database queries every time they fetch a new job.

Below are some of Sidekiq’s performance statistics (obtained from their Github page)


Converting a Delayed::Job to Sidekiq

The process for converting a Delayed::Job worker to Sidekiq was fairly straightforward. The general structure of our DJ email worker code looked something like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
class DelayedJobEmail
  attr_reader :employee

  def initialize(employee)
    @employee = employee
  end

  def perform
    schedule_email(employee)
  end
end

# Enqueueing this email job:
job = DelayedJobEmail.new(Employee.find(123))
Delayed::Job.enqueue(job)

The equivalent Sidekiq implementation of that email worker would look like this:

1
2
3
4
5
6
7
8
9
10
11
12
class SidekiqEmail
  include Sidekiq::Worker
  sidekiq_options queue: :process_email

  def perform(employee_id) # Note that Sidekiq does not accept Employee objects
    employee = Employee.find(employee_id)
    schedule_email(employee)
  end
end

# Enqueueing this email job:
SidekiqEmail.perform_async(123)

It’s worth noting that Sidekiq will perform a JSON dump and load of arguments passed into its workers, so it only accepts pure JSON data types. This increases the consistency of data for edge cases where data has changed from underneath in the time that a job has spent enqueued and waiting for a worker to process it, but it will increase the total number of database calls made.


Testing Sidekiq Jobs

Of course, for every piece of code, we have to write corresponding tests. Below are various methods of testing Sidekiq jobs in Minitest. They are all different ways of testing the same thing.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
require 'sidekiq/testing'

class SidekiqEmailTest
  def setup
    Sidekiq::Testing.fake! # Should be fake by default
  end

  def teardown
    Sidekiq::Worker.clear_all # Ensure jobs don't linger between tests
  end

  test '#perform delivers an email (Sidekiq fake mode)' do
    SidekiqEmail.perform_async(123)
    assert_equal(1, SidekiqEmail.jobs.size)
    SidekiqEmailWorker.drain
    # Assert that email was sent properly
  end

  test '#perform delivers an email (Sidekiq inline mode)' do
    Sidekiq::Testing.inline! do
      SidekiqEmail.perform_async(123)
      # Assert that email was sent properly
    end
  end

  test '#perform delivers an email (direct testing)' do
    SidekiqEmail.new.perform(123)
    # Assert that email was sent properly
  end
end


Configuring Sidekiq Workers

Now that we have set up enqueueing Sidekiq jobs, it is a simple matter to spin up a process to start working on those jobs. The Sidekiq rake command allows you to configure the concurrency, weight, and queues for each process as follows:

1
bundle exec sidekiq -c 20 -q process_email,2 -q deliver_email,1

The -c flag controls the concurrency of the worker, in this case, 20. The -q commands specify which queues this worker will be pulling jobs from, and the ,2 and ,1 after the queue names indicate the weighted priority given to these queues. In this case, jobs from the process_email queue will be processed twice as frequently as those from the deliver_email queue.


Monitoring Queues

We use NewRelic to monitor our application’s performance, and we wanted to be able to leverage NewRelic to provide further transparency and statistics for our new Sidekiq queues as well. To this end, we set up a Heroku dyno to continually publish our Sidekiq queue sizes as custom events to our NewRelic instance. This way, we can see whether or not a queue is backed up, investigate the causes of any bottlenecks that we observe, and determine whether or not we should allocate more workers to process jobs.

The code we used to publish custom NewRelic events looked something like as follows:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
module QueueSizeMonitor
  def self.publish_queue_information
    Sidekiq::Queue.all.each do |queue|
      publish(queue.name, queue.size)
    end

    publish('scheduled', ::Sidekiq::ScheduledSet.new.size)
    publish('retries', ::Sidekiq::RetrySet.new.size)
    publish('dead', ::Sidekiq::DeadSet.new.size)
  end

  def self.publish(queue_name, queue_size)
    ::NewRelic::Agent.record_custom_event('QueueSizeMonitor', name: queue_name, size: queue_size)
  end
end

And afterwards, we were able to query those custom events in NewRelic and compile a chart to visualize how our Sidekiq queue sizes vary over time.


Results and Next Steps

After some monitoring, we discovered that switching from Delayed::Job to Sidekiq resulted in about a 4.5x improvement in throughput! We were able to complete a 30,000 employee email campaign in under an hour. We did hit some snags where we discovered that the concurrency we had set for our Sidekiq workers was a little too high and was resulting in a lot of open database connections at once, and we ended up scaling that down.

Moving forward, we are pushing to convert more Delayed::Job work over to Sidekiq.

Rot13 in Ruby

Just for fun, I decided to try my hand at implementing a Rot13 program, which is a simple problem which takes a string as its input and encodes it by moving each letter 13 characters forward or backward. For example, since there are 26 letters in the alphabet, we can assign indices to each letter starting at 0 for 'a' and ending at 25 for 'z'. The Rot13 program would switch (or rotate) the character 'a' with 'n' since it has the index 13.

Giving this program the string input 'apple' would (hopefully) return 'nccyr'.

To that end, the following is the first pass I made at implementing the program:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
class Translator
  ALPHABET = 'abcdefghijklmnopqrstuvwxyz'

  def translate(string)
    result = ''
    string.split(//).each do |char|
      if char == ' ' #preserve spaces if string has multiple words
        result << char
        next
      end
      result << translate_char(char)
    end
    result
  end

  private

  def translate_char(char)
    lowercase = char.downcase
    is_uppercase = char != lowercase
    index = ALPHABET.index lowercase
    translated_index = index - 13
    if translated_index < 0
      translated_index += 26
    end

    #preserve case in original string
    if is_uppercase
      ALPHABET[translated_index].upcase
    else
      ALPHABET[translated_index]
    end
  end
end

In this solution, I created a class Translator that defines constant ALPHABET string representing the alphabet and an instance method #translate that takes in a string, splits it into an array of its characters, translates each character using a private method #translate_char, and then returns a string composed of those translated characters.

The private #translate_char method checks the capitalization of the character, checks the index of the character against the ALPHABET constant, and then rotates that character by returning the appropriate character in ALPHABET at that given index minus 13.

All in all, this got the job done.

However, I later discovered Ruby’s String#tr method, which allows you to replace characters in a string. It is similar to the #gsub method, but while #gsub can match complex patterns with complex results, #tr can only replace fixed characters.

In any case, you can call 'hello'.tr('el', 'ip') to get the result hippo since it will replace every 'e' with 'i' and every 'l' with 'p'.

It turns out that the above implementation can be greatly simplified by the following:

1
2
3
4
5
class Translator
  def translate(string)
    string.tr('a-zA-Z', 'n-za-mN-ZA-M')
  end
end

Spaces and punctuation are preserved since it only checks the string for those characters and replaces them accordingly, ignoring all else. I also realized at this point that my original solution would not have accounted for punctuation or special characters and would have stripped them out in the result.

Using the String#tr method, I was able to simplify my ~30 line class down to 5 lines in a more idiomatic (and more effective!) Ruby solution.

Pretty cool.