Julia Chou

On Finding Solace From an Existential Crisis


The mass of men lead lives of quiet desperation

I wished to live deliberately, to front only the essential facts of life, and see if I could not learn what it had to teach, and not, when I came to die, discover that I had not lived

-- Henry David Thoreau


These two of Thoreau’s most famous quotes from Walden were the words that knocked me free from the clutches of an arrogant self-assuredness I hadn’t known possessed me in my sophomore year of college. In its place, I felt the creeping numbness of a slow existential dread. For most people, college is a time of self-exploration and self-discovery, a chance to test the limits of your own capacity and establish the foundations of your career. As for myself, I took to heart that very last goal to the exclusion of all others. I was going to graduate with a degree in Computer Science, snag a lucrative job in Silicon Valley, work for a few decades, save up money, and finally retire to an idyllic life squirreled away in a cabin in the woods where I would write the rest of my days away.

I was struggling at the time, pouring a lot of my energy into balancing three jobs alongside the study of a subject that did not come intuitively to me, to say the least. During my sophomore year, I decided to take a literature course with an amazing professor who introduced me to the works of Thoreau, resulting in a dramatic shift in my perspective. Suddenly, everything I was doing seemed like an endless and meaningless toil. I realized that this future I was so single-mindedly pursuing was only a vague specter and I had no idea if it was going to be what I wanted when I got it. What were the things that were going to be important to me when I woke up some 40 years down the road? When it came time for me to die, was I going to discover that I had never truly lived?

It wasn’t so much that choosing a career path based primarily on the money-making potential is inherently wrong, it was the realization that I had made that commitment without much deliberation or forethought whatsoever as to what else in my life I wanted to achieve and what I would be sacrificing in order to pursue this. I knew that my natural affinity gravitated towards literature and the art of language and story more so than hard science and engineering, but I was afraid that the opportunity cost of investing any of my time and energy towards nurturing those interests would be my chance to break into the realm of software engineering and a life free from financial destitution.

This crisis of identity fed into a larger, more general crisis of existentialism itself. After all, how could I decide what meaning I wanted to add to the world if I didn’t even have a proper understanding of myself and my own values and beliefs?

At the time, I sought to find some answers in Albert Camus' work exploring a concept he referred to as “absurdity,” that is, the act of trying to seek meaning in a meaningless world. Camus' spin on the subject is that any attempt to shape the chaos of the world into some form of sense is futile, and he presents the Greek mythological character Sisyphus as the ideal characterization of the struggle with absurdity. Sisyphus is a man who is doomed in the afterlife to an eternal cycle of pushing a boulder to the top of a mountain only to have it roll back down again, and Camus uses his labor to exemplify the toils of mankind at large. It is only on Sisyphus' journey back down the mountain that he has full agency. He accepts the inevitable futility of his efforts and decides to embrace it regardless, and in that moment, he has complete mastery of his fate.

Reading through these concepts helped put my mind at ease just a little. I imagined that all I had to do was accept that anything I did in life was eventually meaningless, so I might as well drive my initial agenda forward. I pushed aside my thoughts in order to focus on making my engineering career a priority. The next few years witnessed me graduating with my Computer Science degree, moving to San Francisco, and finding work as a software engineer at a tech startup. On the surface, it appeared that I was well on track to achieve my initial dreams and aspirations, but a vague, uncomfortable feeling of malaise persisted through my everyday life.

As it goes, a recent loss of a very important connection to me, along with its associated hopes, dreams, and expectations, unearthed all these underlying insecurities and sent me reeling deeper into an identity crisis than I have ever been. It’s inevitable that one experiences several major losses throughout their lifetime, but the worst ones are the ones that completely blindside you and leave you with a lingering sense of incompletion, of missed opportunity. You become helplessly mired in thoughts of how you could have done things differently, done more, or just been better.

Everything around me suddenly seemed to take on a new sense of gravity, and I found myself questioning and reevaluating every single aspect of my life. Does every relationship, every material possession, every belief, every value, every motivation, and every goal I have truly embody who I am, what matters to me, and what kind of life I really want to lead?

I was completely swept away in a maelstrom of despair, self-inadequacy, and doubt, and almost overnight, my emotional well-being completed a beautiful and transcendent metamorphosis into a flaming pile of garbage. There has always been a slight disconnect between how I feel and what I think, such as when I feel irritated by having to sit in heavy traffic even though I know there’s nothing I can do about it. Never before have I felt such a gaping chasm between the two, however, and the small, hyper-rational part of me could only shrink away in awe and incredulity to observe the mad stampede of emotions rampaging through me. I’ve usually been the sort of person for whom rationality and logic ought to prevail in all decision-making, but I suppose for matters of the heart, reason can only follow. I could only use this as an opportunity for some serious self-examination, to try to embrace my emotions and assess and harness them to drive my own behavior.

It just so happened that a former college roommate of mine scheduled a visit during this time, providing me some refuge in the old comforts of a familiar friendship. She’s an amazing, well-grounded, thoughtful, and hilarious person, and seeing her again reminded me of a lot of wonderful qualities inherent in her that I respect and aspire to attain myself. We wandered all over the city, wove in and out of tattoo parlors and dive bars, and took a road trip up to Tahoe, all the while having fun and being ridiculous at a time when having fun and being ridiculous have been the absolute last things on my mind.

She’s the sort of person with whom I can talk about anything and everything, and every conversation with her is a chance to learn something new. Our conversations freewheeled from immigration reform and border-control policies, having children (always a topic of discussion among women, it seems), the morality of suicide, and what it means to be happy to what we think our own deepest flaws are. The ideas that we discuss and bounce off one another never hit a wall where one person simply refuses to change their point of view. Rather, the only barrier we hit is when one of us realizes that she knows too little about the subject for the conversation to be productive any longer.

As much as this has been a time for me to reflect on how I need to work on myself and the next steps I need to take in order to grow and improve, this recent visit my friend paid me also opened my eyes to just how amazing the interconnection of human relationships can be. As I reconsider all of my initial anxieties in college about finding purpose, I realize that the works of Thoreau and Camus that affected me so profoundly emphasized only a solitary search for meaning, making no mention of what reassurance an individual might find in the associations with those around her. I’m now starting to properly appreciate how many wonderful, kind, and generous people there are around me. I treasure the friendships that I have very highly, and I’m making it my mission to live a life where those I call my friends can say that I had a positive impact on them.

This has been a textbook exercise in psychosocial development for me, in which I have been trying to reconcile my own sense of self within the context of my surrounding social environment. Through it all, I think I’ve finally managed to consolidate what carries meaning for me from all of the experiences that I’ve had so far. What’s important to me is continual self-improvement, measured in part by how I foster and cultivate my relationships with those around me. I want to always be exploring more, learning more, and becoming a better person, and I want to find the people in my life that make me a priority and focus on them as well. To that end, I’m setting tentative, longer-term goals for myself and taking the smaller, more immediate steps necessary in order to reach them. For instance, I imagine that somewhere along the path of my software engineering career, I’ll be confronted with the choice of becoming either a manager or a more senior individual contributor. Regardless, I’ll need to work on my leadership skills, so for now, I’m setting immediate goals to improve my speaking, writing, empathy, self-awareness, and overall communications skills.

I am still being constantly surprised by the seemingly limitless bounds of my own arrogance and naiveté. I also know that I’ll never reach a state where I am completely secure about all the decisions I’ve made and the life I lead. However, instead of being dragged down by the weight of my own doubts and insecurities, I’m learning to find peace in the acknowledgement of the perpetuity of my imperfection. Clearly pinpointing all of my flaws will allow me to confront them and work through them, laying out the path I need to follow and allowing me to incorporate an element of deliberation in the everyday choices I make.

I know there is still a great deal of uncertainty in my future. The steps I’m taking today will likely lead me to places that look wildly different from where I expected to be in 10, 20, and 30 years. However, I believe true importance lies not in what the larger narrative of my life ultimately looks like but in being mindful and finding the joy in the instances of human connection in each of the small moments that comprises it. If I die tomorrow, I’ll be at peace believing I made something of the time that I had here.

Though I rather suspect that some thread of existential agony will weave permanently throughout my state of being, this most recent struggle has helped me to uncover which ideals truly matter to me at this moment, and I have found much solace in that.

Sidekiq-ifying Emails at Reflektive

At Reflektive, we offer the option of sending reminder emails after kicking off performance review cycles in order to notify participants of that cycle that they should fill out their reviews. We used Delayed::Job, an asynchronous priority queue system pioneered by the Shopify engineering team, to handle the delivery of those emails.

However, we noticed that there was a huge bottleneck when it came to delivering emails to cycles with large numbers of participants. For instance, one company that had approximately 30,000 employees would clog up our worker queues for hours, preventing other companies from sending out their own emails while this large job was still being processed.

One solution might have been to just throw more hardware at it, to spin up more DJ workers to hammer through the job queues. However, we decided to explore the option of using Sidekiq as an alternative asynchronous job processing system.

Some of the factors leading to choosing Sidekiq included the fact that Sidekiq has higher concurrency than Delayed::Job because it leverages threads as opposed to single processes.

In addition, Delayed::Job stores its queued jobs in a database table (in our case, Postgres) while they’re waiting for a DJ worker to process it. We opted to pair Sidekiq with Redis, an in-memory datastore. This results in a much lower I/O cost because the threads processing Sidekiq jobs don’t have to make database queries every time they fetch a new job.

Below are some of Sidekiq’s performance statistics (obtained from their Github page)


Converting a Delayed::Job to Sidekiq

The process for converting a Delayed::Job worker to Sidekiq was fairly straightforward. The general structure of our DJ email worker code looked something like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
class DelayedJobEmail
  attr_reader :employee

  def initialize(employee)
    @employee = employee
  end

  def perform
    schedule_email(employee)
  end
end

# Enqueueing this email job:
job = DelayedJobEmail.new(Employee.find(123))
Delayed::Job.enqueue(job)

The equivalent Sidekiq implementation of that email worker would look like this:

1
2
3
4
5
6
7
8
9
10
11
12
class SidekiqEmail
  include Sidekiq::Worker
  sidekiq_options queue: :process_email

  def perform(employee_id) # Note that Sidekiq does not accept Employee objects
    employee = Employee.find(employee_id)
    schedule_email(employee)
  end
end

# Enqueueing this email job:
SidekiqEmail.perform_async(123)

It’s worth noting that Sidekiq will perform a JSON dump and load of arguments passed into its workers, so it only accepts pure JSON data types. This increases the consistency of data for edge cases where data has changed from underneath in the time that a job has spent enqueued and waiting for a worker to process it, but it will increase the total number of database calls made.


Testing Sidekiq Jobs

Of course, for every piece of code, we have to write corresponding tests. Below are various methods of testing Sidekiq jobs in Minitest. They are all different ways of testing the same thing.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
require 'sidekiq/testing'

class SidekiqEmailTest
  def setup
    Sidekiq::Testing.fake! # Should be fake by default
  end

  def teardown
    Sidekiq::Worker.clear_all # Ensure jobs don't linger between tests
  end

  test '#perform delivers an email (Sidekiq fake mode)' do
    SidekiqEmail.perform_async(123)
    assert_equal(1, SidekiqEmail.jobs.size)
    SidekiqEmailWorker.drain
    # Assert that email was sent properly
  end

  test '#perform delivers an email (Sidekiq inline mode)' do
    Sidekiq::Testing.inline! do
      SidekiqEmail.perform_async(123)
      # Assert that email was sent properly
    end
  end

  test '#perform delivers an email (direct testing)' do
    SidekiqEmail.new.perform(123)
    # Assert that email was sent properly
  end
end


Configuring Sidekiq Workers

Now that we have set up enqueueing Sidekiq jobs, it is a simple matter to spin up a process to start working on those jobs. The Sidekiq rake command allows you to configure the concurrency, weight, and queues for each process as follows:

1
bundle exec sidekiq -c 20 -q process_email,2 -q deliver_email,1

The -c flag controls the concurrency of the worker, in this case, 20. The -q commands specify which queues this worker will be pulling jobs from, and the ,2 and ,1 after the queue names indicate the weighted priority given to these queues. In this case, jobs from the process_email queue will be processed twice as frequently as those from the deliver_email queue.


Monitoring Queues

We use NewRelic to monitor our application’s performance, and we wanted to be able to leverage NewRelic to provide further transparency and statistics for our new Sidekiq queues as well. To this end, we set up a Heroku dyno to continually publish our Sidekiq queue sizes as custom events to our NewRelic instance. This way, we can see whether or not a queue is backed up, investigate the causes of any bottlenecks that we observe, and determine whether or not we should allocate more workers to process jobs.

The code we used to publish custom NewRelic events looked something like as follows:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
module QueueSizeMonitor
  def self.publish_queue_information
    Sidekiq::Queue.all.each do |queue|
      publish(queue.name, queue.size)
    end

    publish('scheduled', ::Sidekiq::ScheduledSet.new.size)
    publish('retries', ::Sidekiq::RetrySet.new.size)
    publish('dead', ::Sidekiq::DeadSet.new.size)
  end

  def self.publish(queue_name, queue_size)
    ::NewRelic::Agent.record_custom_event('QueueSizeMonitor', name: queue_name, size: queue_size)
  end
end

And afterwards, we were able to query those custom events in NewRelic and compile a chart to visualize how our Sidekiq queue sizes vary over time.


Results and Next Steps

After some monitoring, we discovered that switching from Delayed::Job to Sidekiq resulted in about a 4.5x improvement in throughput! We were able to complete a 30,000 employee email campaign in under an hour. We did hit some snags where we discovered that the concurrency we had set for our Sidekiq workers was a little too high and was resulting in a lot of open database connections at once, and we ended up scaling that down.

Moving forward, we are pushing to convert more Delayed::Job work over to Sidekiq.

Rot13 in Ruby

Just for fun, I decided to try my hand at implementing a Rot13 program, which is a simple problem which takes a string as its input and encodes it by moving each letter 13 characters forward or backward. For example, since there are 26 letters in the alphabet, we can assign indices to each letter starting at 0 for 'a' and ending at 25 for 'z'. The Rot13 program would switch (or rotate) the character 'a' with 'n' since it has the index 13.

Giving this program the string input 'apple' would (hopefully) return 'nccyr'.

To that end, the following is the first pass I made at implementing the program:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
class Translator
  ALPHABET = 'abcdefghijklmnopqrstuvwxyz'

  def translate(string)
    result = ''
    string.split(//).each do |char|
      if char == ' ' #preserve spaces if string has multiple words
        result << char
        next
      end
      result << translate_char(char)
    end
    result
  end

  private

  def translate_char(char)
    lowercase = char.downcase
    is_uppercase = char != lowercase
    index = ALPHABET.index lowercase
    translated_index = index - 13
    if translated_index < 0
      translated_index += 26
    end

    #preserve case in original string
    if is_uppercase
      ALPHABET[translated_index].upcase
    else
      ALPHABET[translated_index]
    end
  end
end

In this solution, I created a class Translator that defines constant ALPHABET string representing the alphabet and an instance method #translate that takes in a string, splits it into an array of its characters, translates each character using a private method #translate_char, and then returns a string composed of those translated characters.

The private #translate_char method checks the capitalization of the character, checks the index of the character against the ALPHABET constant, and then rotates that character by returning the appropriate character in ALPHABET at that given index minus 13.

All in all, this got the job done.

However, I later discovered Ruby’s String#tr method, which allows you to replace characters in a string. It is similar to the #gsub method, but while #gsub can match complex patterns with complex results, #tr can only replace fixed characters.

In any case, you can call 'hello'.tr('el', 'ip') to get the result hippo since it will replace every 'e' with 'i' and every 'l' with 'p'.

It turns out that the above implementation can be greatly simplified by the following:

1
2
3
4
5
class Translator
  def translate(string)
    string.tr('a-zA-Z', 'n-za-mN-ZA-M')
  end
end

Spaces and punctuation are preserved since it only checks the string for those characters and replaces them accordingly, ignoring all else. I also realized at this point that my original solution would not have accounted for punctuation or special characters and would have stripped them out in the result.

Using the String#tr method, I was able to simplify my ~30 line class down to 5 lines in a more idiomatic (and more effective!) Ruby solution.

Pretty cool.