Jeff Keen

In 1997, I was one of the few people in my high school who had a TI Graph Link, which was the little doohickey that connected the calculator to a computer’s serial port, allowing you to transfer programs from the computer to the calculator. This opened up a world of possibilities, letting me download all sorts of games from the internet which I would then distribute.

My friend Matt and I somehow schemed this silly prank idea: create a game called “Fun Game” that we would transfer along with the programs the person actually wanted, and then wait.

When it was launched, the whole sequence you see in the demo version would happen (which at the time we thought was hilarious), and then finally the program would copy a single file over and over and over and over until the calculator ran out of memory.

Turns out basic math functions take several seconds to perform after fun game ate all the memory. Fun, right?

Long ago I wrote an open source ember addon for New York Public Radio called ember-hifi that powered their web audio applications, like wnyc.org, wqxr.org, newsounds.org, wnycstudios.org. It was used and maintained consistently for years and years, and as ember evolved it lagged behind a bit. So I started a branch to bring it up to modern Ember standards, and really started thinking about it from the outside in, to try and lower the learning curve.

I find it tremendously useful to start with a interactive documentation site when building something like this, as the usability problems are easy to spot early. And building this one was no different. Naming things is the hardest part, and that stuff really jumps out at you when you’re trying to explain how to use it to someone else.

After all the updates I did to the project, the changes got too big to feasibly do a mega-pull request to NYPR as their apps were still running legacy version of Ember and couldn’t benefit anyhow. So I did a hard fork, renamed it and launched it.

CSV feels like the simplest of file formats, where it might seem that there’s not much to know after mentally expanding the acronym - Comma Separated Values. But tell me this: if it’s so simple, then why are there so many CSV parsing libraries, alternative CSV parsing libraries, and CSV parsing libraries that claim to be better or smarter, and a mountain of mangled CSVs in existence?

CSV isn’t so much a file format as it is a loose set of guidelines for converting tabular data into text. The closest thing to a spec for it is this, which deals with vital and often overlooked questions such as:

1. “What happens if a value has a comma in it?” - oh, you quote it

2. “What happens if a value has a quote in it?” - oh, you put another quote before it

One question the spec definitely does not cover is one I needed answering: “What do you do with 32,000 files claiming to be valid CSVs but of the 750,000 some lines an unknown number of them have extra unquoted commas hidden in the values, basically making the data untrustworthy?”
This is not such a simple problem, but it’s an interesting problem.

Background

I’ve been building a platform for public and community radio stations the last year and have been working with the fantastic 91.7 KOOP in Austin, TX to pilot it. Part of this project involved exporting the data from their old system (some 750,000 tracks) into the system I built for them. And noticing some tracks from an artist named “10” doing a song called “000 maniacs”, I realized the old system’s crusty CSV exporter did not do the right thing when it came to commas in values, leaving me with a real mess.

How many track titles, artists, albums, or record labels have commas in their names? Many.

I initially thought this was an impossible problem to solve (at least in a way that would not drive a person insane) so I tried reaching out to the developer of the old system to fix their ancient CSV exporter, or to send me data in another format that I could sort through myself. This felt like the most straightforward option and didn’t seem like too much of a lift.
But after weeks and weeks of back and forth, this path dead ended leaving me with no other option but to fix it myself.

The Problem

Here’s an abbreviated and simplified example of a messed up CSV I was dealing with:

The way a CSV parser would parse that data currently is like this:

This is incorrect, and would only be parsed correctly if the artist value were properly quoted, like so:

1
2
3
artist,title,album,label
"Lester Sterling, Lynn Taitt & The Jets",Check Point Charlie,Merritone Rock Steady 3: Bang Bang Rock Steady 1966–1968,Dub Store,
Lester Sterling,Lester Sterling Special,Merritone Rock Steady 2: This Music Got Soul 1966–1967,Dub Store,

Which would be parsed like this:

But how do we get there without manually looking at 750,000 lines of comma separated text?

Computers, man

First, we can determine if a line is incorrect by parsing it individually as a CSV, seeing how many values we end up with, and comparing that number with the number of headers.

1
2
3
4
5
6
lines = File.read('/path/to/file.csv').lines
header = CSV.parse(lines[0], liberal_parsing: true)

incorrect_lines = lines[1..-1].select do |line|
  CSV.parse(line, liberal_parsing: true).values.size != header.values.size
end

1
2
3
4
5
6
7
8
9
10
11
12
[
  [
    "artist", "title", "album", "label"
  ],
  [
    "Lester Sterling", " Lynn Taitt & The Jets", "Check Point Charlie", "Merritone Rock Steady 3: Bang Bang Rock Steady 1966-1968" , "Dub Store"
  ],
  [
    "Lester Sterling", "Lester Sterling Special", "Merritone Rock Steady 2: This Music Got Soul 1966-1967", "Dub Store"
  ]
]

For this example, the header row has 4 values, the first line has 5 values, and the second line has 4 values. The first line is incorrect, meaning there’s an extra comma that belongs within a value.

Knowing that, we need to figure out all the different possibilities of what it could be. One of those possibilities will be the correct one, and the rest will be wrong. With five values and four fields, we need to join two values together into one (and quote it), making four values.

I think best on paper, so I started drawing out how this might work in a brute force sort of way, to try to get a handle on the algorithm I needed.

Eliminate 1 position, Fit 5 into 4

[1, 2, 3, 4, 5] -> [x, x, x, x] =>

[[1, 2], 3, 4, 5]

[1, [2, 3], 4, 5]

[1, 2, [3, 4], 5]

[1, 2, 3, [4, 5]]

What if we had two commas in there? Then we’d have six values needing to fit into four slots.

Eliminate 2 positions, Fit 6 into 4

[1, 2, 3, 4, 5, 6] -> [x, x, x, x] =>

[[1, 2, 3], 4, 5, 6]

[1, [2, 3, 4], 5, 6]

[1, 2, [3, 4, 5], 6]

[1, 2, 3, [4, 5, 6]]

[[1, 2], [3, 4], 5, 6]

[[1, 2], 3, [4, 5], 6]

[[1, 2], 3, 4, [5, 6]]

[1, [2, 3], [4, 5], 6]

[1, [2, 3], 4, [5, 6]]

[1, 2, [3, 4], [5, 6]]

This was pretty easy to visualize and draw out on paper manually, but figuring out how to do this programmatically for any situation was a challenge. This sort of combination/permutation type math felt like the type of thing I would have learned in a Probability and Statistics class that I took in college, but you know what I remember from that class?

1. Having a huge crush on a lil’ cutie and zero game to do anything about it
2. Drawing this funny comic

That’s what I remember. Certainly not this practical math.

The Actual Question

Anyway, turns out the question we’re trying to answer is: if v is the number of values we have (5), and h is the number of headers we have (4), what are all the unique permutations of h numbers that add up to v?

The following looks similar to the written out paper breakdowns from before, but now instead of each number representing a position, each number represents how many values will be joined together.

What are all the permutations of 4 numbers that add up to 5?

[2, 1, 1, 1] # join the first two values together

[1, 2, 1, 1] # join the second and third values together

[1, 1, 2, 1] # join the third and fourth values together

[1, 1, 1, 2] # join the fourth and fifth values together

To calculate this programmatically, we can first calculate the unique combinations using the method below that I nicked from a stack overflow post that has since been lost in my browser history. There were a number of them, and this one seemed most performant.

1
2
3
4
5
6
7
8
9
10
11
12
13
def combos(desired_size, count, minimum = 1)
  # determine all combinations of [count] numbers that add up to [desired_size]
  # e.g if we have an array of 6 items and want an array of 4 items
  # we need 4 numbers that add up to 6, => [[1, 1, 1, 3], [1, 1, 2, 2]]

  return [] if desired_size < count || desired_size < minimum
  return [desired_size] if count == 1

  (minimum..desired_size - 1).flat_map do |i|
    combos(desired_size - i, count - 1, i).map { |r| [i, *r] }
  end
end

Given these combinations we can now calculate all the different permutations of those sets, which is basically just getting every ordering of those numbers and then eliminating duplicates.

Here’s a class that does all this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
module CommaSplice
  class JoinCalculator
    attr_reader :from_size, :to_size

    def initialize(value_count, header_count)
      @from_size = value_count
      @to_size   = header_count
    end

    def possibilities
      @possibilities ||= permutations(combos(from_size, to_size))
    end

    private

    def permutations(combinations)
      # get all permutations of those combinations
      # to determine every possibility of join

      all_permutations = combinations.collect do |combo|
        combo.permutation(to_size).to_a
      end

      # flatten down to a list of arrays
      all_permutations.flatten(1).uniq
    end

    def combos(desired_size, count, minimum = 1)
      # determine all combinations of [count] numbers that add up to [desired_size]
      # e.g if we have an array of 6 items and want an array of 4 items
      # we need 4 numbers that add up to 6, => [[1, 1, 1, 3], [1, 1, 2, 2]]

      return [] if desired_size < count || desired_size < minimum
      return [desired_size] if count == 1

      (minimum..desired_size - 1).flat_map do |i|
        combos(desired_size - i, count - 1, i).map { |r| [i, *r] }
      end
    end
  end
end

# CommaSplice::JoinCalculator.new(6, 4).possibilities
#  #=> [[1,1,1,2], [1,1,2,1], [1,2,1,1], [2,1,1,1]]`

Now given all the permutation options, we can loop through them and generate all the value possibilities.

1
2
3
4
5
6
7
8
9
10
11
join_possibilities.collect do |joins|
  values = @values.dup
  joins.collect do |join_num|
    v = values.shift(join_num)
    if v.size > 1
      quote_values(v)
    else
      v.first
    end
  end
end

Here are the value possibilities. Five values made into four.

At this point, we could match these up with the headers and prompt the user with the option, making the task a little less tedious, like so:

And that’s a pretty good worst case scenario! But there’s still something we can do to determine which one of these is most likely correct without dying from boredom after doing this thousands of times.

The Human Element

You might have already have caught this, and if so — nice work, you’re sharp — but this crucial fact only hit me when I was a mile deep into this problem: when people type commas out they generally put a space after it. So if a parsed value starts with a space…it’s probably not the correct choice.

So in this example we can obviously see that option number 4 is the correct one, since every other one has a value starting with a space. So most of the time, we can reject any choice where any value starts with a space.

This doesn’t always work! With 750,000 manually entered tracks, you can bet people made typos, and you can bet there are tracks that have commas without spaces after them, like this beauty:

Deftones,U,U,D,D,L,R,L,R,Select,Start,Saturday Night Wrist,Maverick

(which, by the way, generates 220 possible options with no great way to determine which one is correct without a human and/or an internet connection. Oy vey.)

But this method works well enough to turn what I initially thought to be an impossible task into a very very doable task, which I’m happy to report I completed successfully.

I packaged all this code into a ruby gem (with a command line option) on the off chance that some other poor soul has found themselves in a similar CSV parsing dilemma. If that’s you: the chances of you getting out of that situation alive? Better than average.

I just updated a gem I wrote in 2011 (which the FCC actually starred and forked, lol) to use their new API, which apparently knows about caching now. It doesn’t provide all the same data as the old one did, which is kinda weird. No “signal strength”? Why? So the gem can still query the old, horrifically slow and crusty API if you want it to.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
station = FCC::Station.find(:fm, "KOOP")

if station.exists? && station.licensed?
  #Basic attributes, available quickly because the FCC actually caches these in a CDN: 
  station.id #=> 65320
  station.status #=> LICENSED
  station.rf_channel #=> 219
  station.license_expiration_date #=> "08/01/2021"
  station.facility_type #=> ED
  station.frequency #=> 91.7 
  station.contact #=> <struct FCC::Station::Contact>
  station.owner #=> <struct FCC::Station::Contact>
  station.community #=> <struct FCC::Station::Community city="HORNSBY", state="TX">

  # Extended attributes, takes several seconds to load initially because the FCC is running this endpoint on a 1960s era mainframe operated by trained hamsters. 
  station.station_class #=> A
  station.signal_strength #=> 3.0 kW
  station.antenna_type #=> ND
  station.effective_radiated_power #=> 3.0 kW
  station.haat_horizontal #=> 26.0
  station.haat_vertical #=> 26.0
  station.latitude #=> "30.266861111111112"
  station.longitude #=> "-97.67444444444445"
  station.file_number #=> BLED-19950103KA
end

Back in those quaint times when it seemed like there was no way Texas would re-elect everyone’s favorite Senator, Ted Cruz, I woke up one morning with the url “cruzclues” in my head. Remarkably nobody else had thought of this, so I snagged the domain and then created a site to fit.

Thankfully there hasn’t been a shortage of meme content since he secured that senate seat for another term.

Seems like we can’t go two minutes without our phones buzzing, trying to pull us away from whatever we were doing to instead look at a screen.
This started out innocently: as a way to let you know that someone was personally communicating with you; a figurative tap on the shoulder through the magic of the internet.

But at some point this snowballed into us defaulting into allowing the entire world to tap us on the shoulder in the middle of family dinner.

Not only is that just annoying, but it’s partially responsible for actually changing our brains, which I will demonstrate to you using this simple test:

Next time you drive somewhere… don’t look at your phone until you get to your destination. Wait at those stop lights like you used to: existing in the physical space, and thinking the thoughts that came from your own brain.

I can already tell you that it’s not going to be easy. Having a magic device in my pocket that randomly pays out a sweet dopamine rush is the addictive slot machine my brain dreams of. And the more I think about it, the more I hate it.

I didn’t want to lose my attention span to digital junk food. I did not opt-in to this. But here I am, helpless but to compulsively pull my phone out of my pocket for no important reason at all.

What’s interesting to me is that we already know all this — that we just can’t stop looking at our screens — and we just accept it. We as a society also know that we can’t stop looking at our screens while driving and our best attempts to stop people from doing that have been about as effective as shaming an alcoholic into not drinking again. 

Is this why we’re so excited for self-driving cars? So we can finally check our tweets safely on the way to work?

It seems like phone addiction (specifically, social media addiction through our phones) is what cigarette addiction was 50 years ago: Looks good, feels good, what’s the problem? A few people say it’s bad for your health, but those people are fringe weirdos. Everyone thinks they could cut back a little, but it’s so hard.

Recently after being delivered a heavy dose of life-perspective and deciding that “no, spending hours a day on my phone is not how I’d like to use my time”, I took some desperate measures. I neutered my notification settings, I deleted all apps with an infinite feed, and stopped charging my phone next to my bed. 

Moving my phone charger six feet away to my dresser was the easiest and most effective, because that small distance is enough to keep me in bed reading a book, and not fall for the classic trap of “but I’ll just look up this one thing quickly” while my book rests unread in my lap.

Deleting those apps was harder than I imagined it would be, though. After a few weeks of picking up my phone only to remember that “oh, I deleted that”, it has confirmed my initial hypothesis: most of the stuff on our phones that we think we absolutely need and can’t live without… we don’t actually need.

“Funny, how the things you have the hardest time parting with are the things you need the least”—Bob Dylan, Lonesome Day Blues

Personally, I am loving the little bit of regained space in my brain I unintentionally gave up. It made completing this art and societal commentary project possible, for one, and is finally making a dent in the stack of books next to my bed I’ve been meaning to read forever.

What will you do with your regained brain space?

Homework

Give these things a try in an attempt to reclaim your attention span:

1. Charge your phone in a place you can’t reach from your bed. 
2. Take a look through your notification settings and start turning them all off. If you really feel like you’re missing out on something later, you can always turn them back on. 
3. The hardest part: delete those apps that have no brakes. It doesn’t have to be permanent—just try it for a few weeks. (Worst case, for those one-off-cases, use the web-version. But the always there, ready to notify you native app is a big no.)

Extra Credit / Further Exploring

• The Bored and Brilliant project, by WNYC’s Note To Self (formerly New Tech City).
• Deep Work, by Cal Newport
• The Best Interface Is No Interface, by Golden Krishna

I have unsubscribed from so much junk lately. But when I came to unsubscribe from the Huckberry mailing list… I didn’t. And it was because of this delightful experience.

But I’ll probably unsubscribe next time it comes.

NYT just bought two of the most useful review sites on the internet for 30 Million dollars, and it’s amazing.

Matt Haugley said most of everything I want to say about this here, but let me tell you why I think this is really cool.

These sites solve a real problem in the best way. The problem being finding an answer to the question “What’s the best _____ to buy?”, and the best way being by just telling you which thing is best right away, backed up by an in depth writeup of how they decided that.

Most review sites will dive deep and dish out all the data of all the different choices, sandwiching advertising between sections for profit, and in the end not even give you a definitive answer leaving you to piece together all the data to form a conclusion.

The former experience is far superior.

Even more amazing, both of these sites were created by one guy in Honolulu, neither site is ad-supported (!!), and he didn’t take any VC funding!

As Matt said in his blog post:

I imagine every step of the development of the Wirecutter/Sweethome was about people laughing at Brian.

You can’t build a tech site that doesn’t publish 20 times a day. You can’t build a content site that isn’t covered with advertising. You can’t build an entire business on Amazon affiliate revenue. You can’t take on Consumer Reports and expect to get any traction. You can’t pay for this level of in-depth reporting. Ok, great, you built this, but why would anyone ever come back?

Ignore the haters and do it anyway. Amazing work, Brian Lam.

This was a really good interview with Brian about the sites, before it was purchased.

I was in Portland for Ember Conf recently, and after much research (natch), I stayed at the Hotel Eastlund. This is my new spot in Portland, and let me tell you why.

First: this rooftop bar.

The design of this hotel is amazing. Modern feel, with really nice touches that showed they really took the time to think about the experience. As someone who does UX work for a living, I noticed these touches, and I appreciated all of them. But there were two tiny little things that took my opinion of the place from being “Nice hotel”, to “Portland hotel search is over! I’m staying here every time.”

The first night I stayed there I was going to plug my phone in and go to bed, until I realized I forgot my phone charger. That’s when I noticed they already thought of that, and had charging cords for every type of modern phone on each bedside table. Nice touch.

But the following morning when getting ready, I noticed this shower detail:

Are you seeing this? A hole cut in the long glass shower wall so you can reach the knobs without getting sprayed with cold water? That is such a ridiculously thoughtful feature that I can’t stop gushing about it.

Stay here next time you’re in Portland.

From Reconsider:

Part of the problem seems to be that nobody these days is content to merely put their dent in the universe. No, they have to fucking own the universe. It’s not enough to be in the market, they have to dominate it. It’s not enough to serve customers, they have to capture them.

This. Fucking nailed it. One of the most frustrating things lately in technology is that nobody wants to play nice together. Everyone wants to have a platform, instead of providing a great service and while getting along with others. One simple example, look at the landscape of chat services, currently. There was a time, where there were different services, and different clients. Each service had a protocol that a client could conform to, and many clients supported those protocols. I used to chat to my AIM friends and my ICQ friends and my Google friends all within the same program.

Now, everything is a walled garden. Hangouts doesn’t have any third party clients, so I have to run a shitty chrome app to talk to use that. iMessage is its own deal. Facebook has its own thing.

But that’s just one example. Amazon provides great services. Apple makes great products. Apple and Amazon are both have competing products now, the Amazon Fire TV, and the Apple TV. Amazon won’t sell the Apple TV in its stores, and Apple doesn’t have an Amazon Instant Video app available with its services.

Did either of these moves increase loyalty? Improve consumer happiness? Speaking as a consumer: no. It’s only been an inconvenience.