unethical blogger - Opinion

S.A.D. - Seasonal Ada Disorder

R. Tyler Croy — Mon, 24 Jan 2011 15:00:00 +0000

Last Sunday, I announced the "0.1" release of my memcache-ada project on comp.lang.ada, thus ending a 2 month experiment with the Ada programming language.

In my previous post on the topic, I mentioned some of the things that interested me with regards to Ada and while I didn't use all the concepts that make Ada a powerful language, I can now confidentally say that I know enough to be dangerous (not much more though).

This is what my coworkers thought of me, learning Ada.

All said and done I spent less than two months off and on creating memcache-ada, mostly on my morning and evening commutes. The exercise of beginning and ending my day with a language which tends to be incredibly strict was interesting to say the least. Due to the lack of an REPL such as Python's, I found myself writing more and more unit and integration tests to get a feel for the language and the behavior of my library. Due to my "fluency" in Python, I tend to think in Python when scratching out code, similar to how a native speaker of a language will write or speak "from the hip" instead of doing large amount of mental work to construct statements. With Ada, not only am I not yet "fluent", the langauge won't let me get away with as much as Python allows me.

The overhead of writing Ada, in my opinion, is a double-edged sword, I can very quickly informally test, debug and rewrite Python but with Ada such a process is (in my opinion) onerous. My 20 minute walk to the train station would be spent contemplating how and what I wanted to write and where. By the time I sat down on the train, I had thought out and designed things internally, so I would immediately write out tests around my ideas and assumptions before writing code to pass the tests. The time spent writing code was minimal since I rarely had to rewrite code, I can think of only one function that had to be rewritten after it had passed tests (botched some socket reading) in the whole project.

I'm not yet sure what will be my next project in Ada, I am certain that I don't want to build anything of consequence in C again. Working with a language, like C, that not only gives you the rope with which to hang yourself but will often times push you off the chair is more masochism than I feel comfortable with these days. Ada on the other hand will allow you to hang yourself, but it'll make damn certain that have the perseverence to go through with it. Frankly, I don't have that kind of drive to really shoot myself in the foot anymore. I want to build software that works with a language that doesn't want to make me suffer, which means I'll be in a weird Ada + Python love triangle until future notice.

Ada? Surely you jest Mr. Pythonman

R. Tyler Croy — Mon, 06 Dec 2010 15:00:00 +0000

The past couple weeks I've been spending my BART commutes learning the Ada programming language. Prior to starting to research Ada, I sat in my office frustrated with Python for my free time hackery. Don't get me wrong, I love the Python language, I have enjoyed the ease of use, dynamic model, rapid prototyping and expressiveness of the Python language, I just fall into slumps occasionally where some of Python's "quirks" utterly infuriating. Quirks such as its loosey-goosey type system (which I admittedly take advantage of often), lack of good concurrency in the language, import subsystem which has driven lesser men mad and its difficulty in scaling organically for larger projects (I've not yet seen a large Python codebase that hasn't been borderline "clusterfuck".)

Before you whip out the COBOL and Fortran jokes, I'd like to let it known up front that Ada is a modern language (as I mentioned on reddit, the first Ada specification was in 1983, 11 years after C debuted, and almost 30 years after COBOL and Fortran were designed). It was most recently updated with the "Ada 2005" revision and supports a lot of the concepts one expects from modern programming languages. For me, Ada has two strong-points that I find attractive: extra-strong typing and built-in concurrency.

Incredibly strong typing

The typing in Ada is unlike anything I've ever worked with before, coming from a C-inspired languages background. Whereas one might use the plus sign operator in Python to add an int and a float together without an issue, in Ada there's literally zero auto-casting (as far as I've learned) between types. To the inexperienced user (read: me) this might seem annoying at first, but it's fundamental to Ada's underlying philosophy of "no assumptions." If you're passing an Integer into a procedure that expects a Float, there will be no casting, the statement will error at compile time.

Concurrency built-in

Unlike C, Java, Objective-C and Python (languages I've used before), Ada has concurrency defined as part of the language, as opposed to an abstraction on top of an OS level library (pthreads). In Ada this concept is called "tasking" which allows for building easily concurrent applications. Unlike OS level bindings built on top of pthreads (for example) Ada provides built in mechanisms for communicating between "tasks" called "rendezvous" along with scheduling primitives.

Being able to define a "task" as this concurrent execution unit that uses this rendezvous feature to provide "entries" to communicate with it is something I still haven't wrapped my head around to be honest. The idea of a language where concurrency is a core component is so new to me I'm not sure how much I can do with it.

For my first "big" project with Ada, I've been tinkering with a memcached client in Ada which will give me the opportunity to learn some Ada fundamentals before I step on to bigger projects. Disregarding the condescending jeers from other programmers who one could classify as "leet Django haxxorz", I've been enjoying the experience of learning a new vastly different language than one that I've tried before.

So stop picking on me you big meanies :(

Experimenting with reddit's self-serve ads

R. Tyler Croy — Mon, 08 Nov 2010 14:00:00 +0000

A couple weeks ago I decided to try out reddit's self-serve advertising system for one of our products at Apture: the Apture Highlights browser extension. While I am an Apture employee, I've also turned into a rabid user of our browser plugin while browsing the web, I've found it to be perfect at answering a number of quick questions like "what does this word mean?" or "who the hell is this?" In a mix of curiosity regarding reddit's advertising system and advocacy for our browser extension, I decided to run a trial campaign on reddit.

If you've not been exposed to reddit's self-serve advertising platform, here's a quick overview. The entire system is bid-based, with minimum bids starting at 20 USD a day. Ads are created by users (like me) and submitted for approval with tentative dates. Once the ad is approved by reddit, it is scheduled to run on a particular day. From my understanding of the system, the number of impressions given to your advertisement is based on your bid and the demand for ad impressions on the given day. On top of this basic structure, you can run advertisements "targeted" to a specific subreddit or reddit-wide.

For the purposes of my campaign, I wanted to try both reddit-wide and targeted ads, for my targeted portion of the campaign I ran my ad for two days on the /r/todayilearned, a subreddit with nearly 80,000 subscribers who all are looking to share an interesting nugget of information that they have learned today. In addition to targeting the ad to the specific subreddit, I tried to make the copy of the advertisement as compelling as possible for my potential clickers:

(note: The acronym "TIL" generally is used as a substitute for "today I learned" in threads on reddit)

This ad ran for two days on /r/todayilearned and for one day reddit-wide, bringing my total campaign expenditure to $60. The breakdown in numbers is as follows:

Impressions (unique -> total): 21,420 -> 141,037
Clicks (unique -> total): 146 -> 157

While the click-through rate is frustratingly low, what I found astonishing was the huge disparity between unique and non-unique impressions. What that indicates to me is that readers have a tendency to refresh a page (such as the subreddit homepage) a number of times during the day.

What you cannot tell from those numbers above is how many of the clicks came from the targeted placement (/r/todayilearned) versus the reddit-wide run. When the ad ran reddit-wide it received zero-clicks, not only did the targeting to /r/todayilearned garner more repeated (non-unique) impressions, it received all of the clicks received throughout the entire campaign.

The big take-away lesson for me from this brief trial advertising on reddit was: avoid reddit-wide advertising. Finding a subreddit with a large number of passionate users isn't that difficult, so you should be able to identify a subreddit that overlaps with your target market and advertise to them specifically. Other than that, I don't have any great "analysis" to offer, it was an interesting experiment but not a rigorously scientific one.

If you'd like to download the CSV with the data from the campaign, you can grab that here. The columns are: date, impression_unique, impression_total, click_unique, click_total, clickrate_unique, clickrate_total.

So. I'm married.

R. Tyler Croy — Thu, 04 Nov 2010 14:45:00 +0000

A few weeks ago I finally tied the knot after a rather long engagement, putting my relationship with then-fianceé into a legally binding relationship. While a wedding should hold a very special place in the bride and groom's heart forever, I feel like it is safe to say that our wedding objectively rocked.

I don't want to dive too much into the nitty-gritty details of the entire weekend which culminated in a great ceremony and reception at the phenomenal Madrona Manor Restaurant and Inn. The entire atmosphere, from both families having a great time together, to impeccable weather and the fantastically prepared dinner, was damned near perfect. Cue a brief slideshow of pictures taken by my good friends Dave Young and Annika Lindner:

Now that we're properly married, and no longer engaged, the typical annoying question has changed from "When are you getting married?" to "When are you having children?"

Considering I can barely take care of our big moron of a cat, I don't think children are in the cards anytime soon. I'm curious what milestone comes after children though, "when are you going to retire" might be next and then perhaps "when are you going to die" after that.

Either way, I think it's safe to say, it's all down hill from here.

Being a Croy

R. Tyler Croy — Fri, 03 Sep 2010 16:00:00 +0000

The name change that I mentioned in my previous post is now official. This means I now have to update everything. I'm in for a world of hurt between the DMV, banks, brothels and strip-Parcheesi clubs.

The only thing you need to do is update your address book, lucky you! I know at least one friend of mine has, who messaged me to say:

I put your old surname in the "Maiden Name" field in Address Book. Just thought you'd want to know.

I spoke to my step-dad George on the phone immediately after the hearing was over and asked if there are "any perks to being a Croy?"

Still haven't gotten a response to that one yet.

What's in a name?

R. Tyler Croy — Mon, 30 Aug 2010 15:00:00 +0000

Tomorrow morning I will be in court, hopefully finalizing a process I started earlier this year. I will be changing my name.

When I was first considering it, I found the entire idea a bit scary. I have worked tremendously hard to make a name for myself, from my work in the open source community to conferences I've spoken at and interactions with numerous companies and people who have been instrumental in my whittling out a career in software engineering. I have been very particular about being referred to as "R. Tyler Ballance," ensuring that my "self-branding" remains consistent, netting me somewhere north of 36,000 results when searching Google.

Tomorrow I intend on throwing all that out the window, there are more important things in life than Google results (as shocking as that may sound).

I'm hesitant to go too much into the motivations for the change, knowing full well that everything I publish might as well be set in stone on the internet.

Those close to me know that my parents divorced when I was young. After a particularly nasty divorce, my mother and my three sisters parted ways with my father who I have since only had sporadic contact with. After a couple dark years for my sisters and I, my mother married another Navy man, George P. Croy, III. George came into the marriage with his daughter, bringing my sister-count up to four.

Over the past fifteen years or so, I have become George's son. Successfully exploring his emotional spectrum from tears of joy to turning him a bright crimson shade of pissed-off, never once treating me as if I were anything less than his kin. I'm convinced my attitudes towards family, women and friends not to mention my strong opinions on honor and integrity have all been heavily influenced by him

Plainly put, I would not be the man I am today without his guiding hand.

Provided everything goes well at the courthouse, I enter as R. Tyler Ballance and leave as R. Tyler Croy.

Might as well update your address books.

I love Sonic.net already

R. Tyler Croy — Thu, 08 Jul 2010 03:05:06 +0000

Thanks to @pemullen, I was introduced to Sonic.net some time ago. Unfortunately I never took the time in my old apartment to switch out my AT&T DSL for Sonic.net's Fusion service; the thought of home internet downtime was just too dreadful to even contemplate changing, despite AT&T's absolutely awful service.

Now that I've left that apartment, I can finally take the dive into some delightful Sonic.net service, and while it's not even installed yet, I can tell this is going to be a wonderful relationship just by some of the support emails I've been exchanging with their folks.

From me:

Like an idiot I moved in last weekend instead of this upcoming weekend, so I'm now in the unenviable position of zero home internet service. In the interest of time, can you guys just ship the kit instead of sending some poor tech to Berkeley? :)

I understand that AT&T still needs to install a line, but after that I'm hoping to get up and running as soon as possible, I'm almost to the point of considering opening a book to read.

Oh the horror.

After only a couple hours Kelly R. got back to me:

Sorry to hear that you've been driven to such desperate measures. I know the lead time takes a while from AT&T, but we here at Sonic.net have been working on expediting our end of the install process as much as possible. I'll keep my fingers crossed that this installation process doesn't result in a library membership.

Pride

R. Tyler Croy — Sun, 04 Jul 2010 23:18:10 +0000

This fourth of July I find myself thinking a great deal about being an American in the 21st century, and pride. In the back of my head I have that hokey country song "God Bless the USA" with its chorus:

That I'm proud to be an American, where at least I know I'm free. And I wont forget the men who died, who gave that right to me.

The concept comes off so comical to me, "proud to be an American." What does that even mean? I am no more proud to be an American than I am:

Proud to have been born in California
Proud to be white
Proud to be tall
Proud to have four sisters
Proud to have a grandpa named Bob

I had no control in any of it, I won the birth lottery and just happened to be born in the United States. I just happened to have grown up to be a tall, white guy with four sisters and a grandpa named Bob, I didn't select this configuration, it just happened to me. What's to be proud of?

Taking pride in one's country however, I entirely understand. I feel that one should take pride in the positive actions that we undertake as a nation, since it's actions are theoretically comprised of our collective wills, by the same token, I think one should feel ashamed of the negative actions.

That said, I'm struggling to find things to be proud of America lately, there's certainly a good bit to be angry and ashamed of:

Our participating in the secret ACTA treaty negotiations
The tarring of the Gulf of Mexico, a body of water I've spent nearly 40% of my life around.
Guantanamo
The shrinking middle class
Shutting down Shuttle service without a viable Shuttle replacement
Four horrifically expensive failed wars:
- Drugs
- Terror
- Afghanistan
- Iraq
A bloated federal government, with representatives who've forgotten who they represent (looking at you Orrin Hatch)
Irrational fear of nuclear power

I could go on, I could even start a whole new list of all the things we've screwed up here in California too, but it just makes angry, then sad, and then sleepy.

I'm sure there are plenty of things that Americans have done lately that one could take pride in, but none are coming to mind.

We have a mighty big hole to dig ourselves out of.

Fatso Adventures: "I wonder what's down here?"

R. Tyler Croy — Wed, 30 Jun 2010 02:06:05 +0000

Quite the mixed bag today has been, I went to court (more on that later), I signed a lease (more on that later too), and I worked from home. Since ET and I are leaving this apartment soon, the management company has been showing the apartment during the day. Not a big deal, strangers walk around the apartment, all the windows are opened, all the lights are turned on, doors are opened and closed and if you're lucky enough to be around, you get to field questions.

About an hour or two after the showing of the apartment was over, ET looks up from the couch and asks "Where's Buddy" (a.k.a. Fatso). After looking in all of the usual hiding places, she grabs a can of food and taps the lid and listens. A faint meowing is heard. She opens the closet door and taps the lid again. Meow, meow, meow. I think to myself "no way in hell is that cat in the closet, so I hold the can out the window and tap, tap, tap. Meows are coming from outside of the bathroom window. Our bathroom window opens onto this tiny area between two buildings, and is rarely opened because the view sucks, and we don't stink up our bathroom too much.

Not entirely sure where the cat is, I go to the other side of this little area, in the buildings stairwell and open the window, climb out, and poke around for Fatso, a.k.a Buddy, a.k.a Missing Kitty #1. I can't see Fatso at all but I can hear him. I tapped on the hood for the ventilation shaft and I hear meowing. I tap again, meowing. Reaching my hand around under the hood, I hear more meowing but I don't feel anything.

Thanks to a flashlight and mirror loaned from a friendly neighbor, who's more earthquake prepared than ET and I, I was able to look down the ventilation shaft. and I see Fatso's stupid little face, all the way at the bottom.

Running down to the basement confirms two things, this cat is stuck, secondly, he's stuck in the ventilation inlet to the heating system for the building. Stupid cat. While I continue to investigate possible exit strategies, something Fatso clearly hadn't considered, ET is on telephone duty. First we call the management company, who are characteristically useless, then it's on to the fire department's non-emergency line.

When the calvary (see: firemen) arrive, the first thing we do is rip the hood off the ventilation shaft to determine whether we can fish the stupid cat from the depths, which after removing the hood, turns out to be about 15ft. To add insult to injury, there are a couple pieces of wood fastened into place at the top, preventing any beings larger than a 12 pound stupid cat from fitting down the shaft. Looks like we'll have to attack it from the basement, and be "we" I mean the firemen, I'm useless.

The good boys from the SFFD find a seam in the sheet metal where the shaft attaches to the furnace and using some basic tools (pick) and their hands, are able to tear back some sheet metal so I can poke my head in the bottom of the buildings furnace, only to see our stupid cat, a.k.a Fatso, a.k.a Buddy, a.k.a Missing Cat #1, as far away as possible, entirely unwilling to exit the dark bowels of the furnace he's occupied for nigh three hours now.

I explain to the firemen, that I can probably handle it from here since they likely have "real shit to do", but they are unwilling to budge, waiting on "verification" of the cat; they had not actually seen the cat at all up until this point. I shove my head back in the furnace, this time with an arm and grab Fatso by the neck and drag him, against his will, from the furnace to greet the four smiling faces of the SFFD's finest (and ET).

The firemen are kind enough to seal the now warped sheet metal enough to hold the system over until the management company can repair the damage, and after thanking them they were on their merry way, ideally to save somebody's life, but most likely to watch Real Housewives of New Jersey back at the station while they wait for something to catch on fire or some stupid cat to poke its head where it doesn't belong.

Fatso's favoring his hind-legs a little right now but is all and all in good condition. I want to say he's learned his lesson, but I'm certain he hasn't.

Silly Problems

R. Tyler Croy — Tue, 22 Jun 2010 15:45:00 +0000

In the next few weeks, ET and I will be moving out of San Francisco, perhaps for good. I am living up on the promise I made back in April and leaving. Over the past weekend I was struck by how picky I've become, particularly with where I live.

For starters, living in San Francisco, I live in a place with:

A thriving bicycle culture, which is only looking to get better
Hundreds of restaurants with all sorts of delicious food stuffs
Surprisingly few douchebags (hipsters and Mission bartenders not-withstanding)
Fantastic weather
Low violent crime

And I'm still not happy with it.

In the past, I've lived in places where enormous cars are a status symbol, giant belt buckles that double as shields are accepted; truck nuts. Moving here from Texas I left, stale, windless 100+ degree heat, random people shouting "faggot" at pedestrians from their cars, no tolerance drug policies coupled with binge drinking and drunk driving. To its credit however, Texas is cheap and areas like Austin are wonderful (not counting traffic). When I lived in eastern Germany, I was constantly confused, cold and more than once crashed a bike due to black ice on the roads. Before that, Northern Virginia, living dangerously close to the "south will rise again" group of folks, an area of the country where the Ku Klux Klan is still surprisingly strong, albeit more hidden than before.

Every place that I have lived has had its own unique set of problems, San Francisco included; the lack of progress for a progressive city still irritates the hell out of me.

There are so many parts of this country that unabashedly fucking suck compared to San Francisco, and I'm still not satisfied. What a silly problem to have.

Being a Libor, Addendum

R. Tyler Croy — Tue, 18 May 2010 16:00:00 +0000

A couple of weeks ago I wrote a post on how to "Be a Libor", trying to codify a few points I feel like I learned about building a successful engineering team at Slide. Shortly after the post went live, I discovered that Libor had been promoted to CTO at Slide.

Over coffee today Libor offered up some finer points on the post in our discussion about building teams. It is important, according to Libor, to maintain a "mental framework" within which the stack fits; guiding decisions with a consistent world-view or ethos about building on top of the foundation laid. This is not to say that you should solve all problems with the same hammer, but rather if the standard operating procedure is to build small single-purpose utilities, you should not attack a new problem with a giant monolithic uber-application that does thirty different things (hyperbole alert!).

Libor also had a fantastic quote from the conversation with regards to approaching new problems:

Just because there are multiple right answers, doesn't mean there's no wrong answers

Depending on the complexity of the problems you're facing there are likely a number of solutions but you still can get it wrong, particularly if you don't remain consistent with your underlying mental framework for the project/organization.

As usual my discussions with Libor are interesting and enjoyable, he's one of the most capable, thoughtful engineers I know, so I'm interested to see the how Slide Engineering progresses under his careful hand as the new CTO. I hope you join me in wishing him the best of luck in his role, moving from wrangling coroutines, to herding cats.

God speed mooncat

The slow death of the indie mac dev

R. Tyler Croy — Thu, 13 May 2010 16:30:00 +0000

Once upon a time I was a Mac developer. I loved Cocoa, I loved building Mac software, Mac OS X was once upon a time the greatest thing ever. I recall writing posts, and even founding a mailing list in the earlier days of Core Data, which I was using in tandem with Cocoa Bindings, which themselves were almost a black art. I was on a couple of podcasts talking about web services with Cocoa or MacWorld. I loved the Mac platform, and would have gladly rubbed Steve Jobs' feet and thanked him a thousand times for saving Apple from the despair of the late 1990's. As Apple grew, things slowly started to change, and we started to grow apart.

As I started to drift away, I gave a presentation at CocoaHeads presenting some of the changes and improvements to the Windows development stack, not supremely keen on the idea of building Windows applications, I was clearly on the market for "something else". Further and further I drifted, until I eventually traded my MacBook Pro in for a Thinkpad, foregoing any future I might have developing Mac software. My decade long journey of tinkering and learning on Macintosh computers had ended.

When Mac OS X was in it's original Rhapsody-phase, in the weird nether-world between Platinum and Aqua, Apple realized that it had been held back by not giving developers tools to build for the platform. Apple began to push Project Builder which became Xcode, which became the key to the Intel-transition and has helped transform Mac OS from a perennial loser in the third-party software world to a platform offering the absolute best in third-party software. Third-party applications of impressive quality were built and distributed by the "indie mac devs", Adium, Voodoo Pad and Acorn from Flying Meat, Nicecast and Audio Hijack Pro from Rogue Amoeba, FuzzMeasure Pro from SuperMegaUltraGroovy, Growl, NetNewsWire or MarsEdit originally from Brent Simmons (NetNewsWire is now owned by NewsGator, while MarsEdit was acquired by Daniel Jalkut of Red Sweater Software), Yojimbo and BBEdit from BareBones, even Firefox, Camino and Opera filled the gap while Apple pulled Safari out of it's craptastic version 2 series. Applications were used on Mac OS X instead of web applications because the experience was better, faster and integrated with Address Book, iPhoto, Mail.app, iMovie and all of Apple's own stack.

Then came the iPhone, with its "Web SDK" nonsense. The story, at least at the time, was clear to me. Apple didn't care about me. Apple didn't care about its developers. Build a web application using JavaScript and AJAX (a Microsoft innovation, I might add) over AT&T's EDGE network? Fuck you! A number of months later, back-tracking on the "Web SDK" concept, the iPhone SDK came out at WWDC with a ridiculous NDA, forbidding developers from talking about it publicly. Then the App Store was bundled with iTunes and iPhone OS, with Apple becoming the gatekeeper between indie developer, and Joe User. Of course, more recently in the long line of iPhone/developer related tragedies, the infamous Section 3.3.1. There's also some hub-ub about the Apple Design Awards 2010, only focusing on iPhone and iPad apps which is quite disconcerting for indie mac devs, who routinely compete and win awards for the best Mac applications.

The message is clear, Apple wants to completely own users on its platform and sit between developers and their users, dictating terms.

It's no wonder that @rentzsch, a major voice in the indie mac dev community, and organizer of the C4 conference is throwing in the towel on organizing C4 entirely (discussed in this post).

It's not entirely clear whether the "indie mac dev" community will continue to exist for too much longer, there is some speculation that a "Mac App Store" is brewing in Cupertino right now or perhaps modifications to Mac OS X similar to what is present on the iPhone. If I were still part of the "indie mac dev" tribe, I'd feel very nervous right now about what will happen at this year's WWDC, as Dan Wood from Karelia knows, Apple feels no remorse with stomping on Mac developers.

Worst comes to worst, I sincerely invite indie Mac developers to bring their user-experience talent and software-building energy to the weird but exciting world of web software, so long as Google keeps Facebook in check, the web should remain open for a good long while.

Is programming with Twisted really as awful as it sounds?

R. Tyler Croy — Wed, 12 May 2010 16:45:00 +0000

Early this week Can forwarded this post on Quora to me, which asks the question:

Is programming with Twisted really as awful as it sounds?

Yes. Yes. YES IT IS. HOLY CRAP IT'S AWFUL

Here's some good alternatives:

Eventlet, my preference
gevent, an alternative to Eventlet tied to libevent
Java. because let's face it, if you're using Twisted, you've already decided not to write Python, so use something with proper threading support.

That is all.

Be a Libor

R. Tyler Croy — Fri, 30 Apr 2010 14:45:00 +0000

I reflect occasionally on how I've gotten to where I am right now, specifically to how I made the jump from "just some kid at a Piggly Wiggly in Texas" as Dave once said, to the guy who knows stuff about things. I often think about what pieces of the Slide engineering environment were influential to my personal growth and how I can carry those forward to build as solid an engineering organization at Apture.

The two pillars of engineering at Slide, at least in my naive world-view, were Dave and Libor. I joined Dave's team when I joined Slide, and I left Libor's team when I left Slide. Dave ran the client team, and did exceptionally well at filling a void that existed at Slide bridging engineering prowess with product management. Libor often furrowed his brow and built some of the large distributed systems that gave Slide an edge when dealing with incredible growth. In my first couple years I did my best to emulate Dave, engineers would always vie for Dave's time, asking questions and working through problems until they could return to their desk with the confidence that they understood the forces involved and solve the task at hand. Now that I'm at Apture, I'm trying to emulate Libor.

(Note: I do not intend to idolize either of them, but cite important characteristics)

To understand the Libor role, the phrase "the buck stops here" is useful. A Libor is the end of the line for engineering questions, unlike some organizations the "question-chain-of-command" is not the same as the org-chart. If a problem or question progressed up the stack to a Libor, and between an engineer and a Libor the pair cannot solve the problem, you're screwed.

What does it take to be a Libor you may be thinking: * No Guessing: When acting as a Libor, knowing is crucial. That is not to say you must understand everything about all the nooks and crannies of the code-base, but when you give an answer it is crucial you actually know what the hell you are talking about. The consequences of being wrong are far worst than the consequences of not knowing, if a fellow engineer builds on your guess, when that code ships live in a few days/weeks there is a serious risk of everything falling over.

Grok the stack: A Libor is expected to hold a wealth of information internally, much like a clock maker, a Libor should understand where every single gear and spring fit together in a large complex system. It is not necessary to understand how each component individually works but instead, understand how all the pieces operate in concert. Some amount of acting as a Libor requires direct discussions with the operations team as well as the rest of engineering, when all that JavaScript and Python rolls out to 10, 20, 100, or 1,000 machines, somebody should have at least considered the ramifications of adding 3 more database calls to every request, that's the Libor.
Maintenance and accountability: Typically working at the lower ends of the stack, a Libor has to relive and tolerate last month's and last year's short-sighted decisions over and over. A Libor should not let himself nor colleagues "fire and forget" code, poor judgement will haunt a Libor for much longer than most people's New Year's resolutions. Because of this mistake-longevity, a Libor should be quite concerned with how well thought-out and tested new changes, particularly drastic ones, are.
Focus on Engineering: Code quality and extendability are Libor's primary focus, that is not to say that a Libor's role is to impede product development, but rather ensure that it is properly framed. While a product manager's primary concern may be to get a feature deployed as soon as possible, the primary concern of a Libor is to ensure that once that feature is shipped it doesn't break or otherwise degrade the quality of service of the rest of the site. When interfacing with other engineers a Libor should be asking questions about code, intentions and implementation. Code review is as important as communication with the team, flatly rejecting code is unacceptable, but discussing with engineers the potential pitfalls of certain approaches ensures that the group moves forward.

Playing the Libor character at Apture has been interesting to say the least, I've done a lot of work getting a number of systems in place to help educate my decisions, particularly in our production environment. Focusing on the entire stack as a complex system has allowed us to make some adjustments here and there that have literally started to pay dividends the day after they ship.

Non-engineering also benefits from having a Libor character in the organization, at Apture the product development narrative has changed, I find myself emphasizing:

Tell me what you want, we'll find a way to do it

That's a breakthrough.

San Francisco (Transit) Sucks

R. Tyler Croy — Wed, 28 Apr 2010 14:30:00 +0000

I was reading this article on some of the debates going on with the SF Board of Supervisors, where one of the supervisors suggested his intention on bringing a ballot initiative up this November to "force Muni operators to negotiate their salary, benefits and work rules through collective bargaining." Currently, despite the fact that Muni faces service cuts and a budget shortfall, Muni operators are guaranteed no less than the second-highest transit operator salaries in the nation.

If you ask anybody who knows anything about say, capitalism, economics or even math, they'll tell you that the arrangement is batshit insane.

So let's recap, a supervisor suggests the Muni union be forced to use collective bargaining to determine wages, which seems reasonable, and what happens in City Hall?

Daly jumped in to respond to Elsbernd's comments, saying that if the city wants to save money, it should look at agreements with all unions, beginning with the police. ''Supervisor Elsbernd targeting basically what amounts to a largely black and/or African American union not only has the issue of racial undertones''

Holy race card Batman! Bargaining, a fundamental tenet of our capitalist society gets one deemed a racist by Supervisor Daly? He's clearly preoccupied with race, which will sure help bail Muni, and the city, out of their budget deficits this year.

For those of you unfamiliar with San Francisco city budgets, this city somehow pisses away an annual budget of over 6 billion dollars a year, and our streets are terrible, homelessness continues to be a big issue, Muni is a mess and Market St still smells like urine.

When my lease is up in June, I'm leaving San Francisco.

Regarding Blippy

R. Tyler Croy — Sat, 24 Apr 2010 15:00:00 +0000

While I wish my friend Ryan all the luck in the world at Blippy, I loved this comment on Reddit regarding the startup from jacques_chester

Blippy is a company which turns the urges of widely available trendoids at the forefront of the hipster web into venture capital investments for its founders.

I still don't understand what the point is though.

A great platform does not make yours a good product

R. Tyler Croy — Sat, 03 Apr 2010 13:45:00 +0000

As I sit here writing, burnt out from the incredible hype surrounding Apple's iPad launch and I cannot help but have flashbacks. Flashbacks to 2004/2005 when "Web 2.0" was at its peak, "ajax" and "mashup" were quite trendy. Flashbacks to mid-2007 when the Facebook Platform was gaining steam, words like "FBML" and "social applications" became the new lingo. Flashbacks to early 2008 when the iPhone SDK launched and hoards of developers rushed to submit their apps to the App Store. All bore a resemblance to today, with the iPad, the newest, hottest thing in the world with everybody and their mother vying for one.

Every step of the way, beneath the almost overwhelming marketing and hype, lies actual technological innovation. With "Web 2.0", the underlying core innovation was the rise of the web as a formidable platform, browsers became capable of supporting immersive applications built in JavaScript, the web "came alive", no longer a lexicon of static pages. With the launch of the Facebook Platform, products could engage not just on a one-to-one level with a user, but with a user's social circle. The iPhone SDK made developers re-think building applications, touch, motion and then location became fundamental building blocks for products. The iPad represents a slightly different innovation, the introduction of "casual computing" to the masses. Personal computers are becoming omnipresent, the smartphone in your pocket, the laptop on your desk and now the tablet on the coffee table.

It comes as no surprise that just as with every other hyped innovation over the past few years, there's a rush of gold-diggers trying to latch onto somebody else's innovation, using it to rocket them to the top. The announcement of a $200m "iFund" for iPad apps reminds me a lot of the crazy venture capitalists establishing Facebook app funds or even the $100m "iFund" for iPhone apps (from the same group no less).

Venture Capitalists aren't alone in their attempts to ride Apple's innovating coat tails, companies building products are falling for it too, with hundreds of thousands of apps in the iPhone App Store or the hundreds of thousands of Facebook applications, one could deceive themselves into believing tens of thousands of companies are printing money on these platforms!

I would estimate that for every 1000 applications, on any of these platforms, only one is a good product. The same thing is going to happen on the iPad, for the exact same reason that it occurred with Web 2.0 sites, the Facebook Platform and the iPhone.

It's like X, but for the Web / Facebook Platform / iPhone / iPad!

Platform innovation is no substitute for product innovation. The iPad app for the New York Times comes to mind, which apparently does nothing but format the New York Times and show the user interstitial ads. There is really no added value over using something like NetNewsWire or god forbid, browsing to newyorktimes.com with Safari for consuming content.

Supporting the iPad is not a product, it's a feature, if your product isn't already compelling there is nothing intrinsic about the iPad that will make it more compelling just as it was with the launch of the other platforms.

While I won't be buying an iPad, I wish all those jumping on the bandwagon luck. With a word of caution, bring features because your presence on the iPad alone will not be enough.

Sometimes Software as a Service Sucks

R. Tyler Croy — Tue, 23 Mar 2010 14:00:00 +0000

Being a big fan of "continuous integration", particularly with Hudson, I've often thought about the possibilities of turning it into a business. It's no surprise really, my first commercial application as a rogue Mac software developer was a product called BuildFactory which, while fun to build, never sold all that many licenses. With the advent of Amazon's EC2 service and the transition of these cloud computing resources into a building block for many businesses, I've long thought about the idea of building "continuous integration as a service."

At face value the idea sounds incredibly fun to build, I'll build a service that integrates with GitHub, Google Code, SourceForge and private source control systems. The end (paying) user would "plug-in" to the "continuous integration grid", they'd work throughout the day, committing code and then the CI grid would pick up those changes, build releases and run tests against a number of different architecture, automatically detecting failures and reporting them back to the developers. It involves some of my favorite challenges in programming:

Scaling up
Efficiently using cycles, and only when needed
Building and testing cross-architecture and cross-platform

Unfortunately, it's a crap business idea, I now have second-hand confirmation from a group of guys who've attempted the concept. The folks behind RunCodeRun are shutting down the service. In the post outlining why they're shutting down, they've hit the nail on the head on why "continuous integration as a service" can never work:

Large scale hosted continuous integration is consumed as a commodity but built as a craft, and the rewards, both emotional and financial, are insufficient to support the effort.

Elaborating further on their point, continuous integration by itself is a relatively basic task: build, test, repeat. The biggest problem with continuous integration as a service however, is that no two projects are alike. My build targets or requirements might be vastly different from project to project, let alone customer to customer, making the amount of tweaking and customization per-job too large such that at some point the only benefit that one derives from such a service is the hosting of the machines to perform the task. If you're just taking care of that, why wouldn't your customers just use Hudson in "the cloud" themselves? The CI grid at that point offers no exceptional value.

As much as I regret letting a fun idea die, I think I'll have to file this one under "To do after becoming so rich I'll care about capital gains taxes."

Hypocrites on Bikes!

R. Tyler Croy — Fri, 05 Mar 2010 14:00:00 +0000

Typically I read at least one news story a day that irritates me, usually I either don't care enough to gripe about them further, or I forget. After griping at ET about driving in the car with a phone in her hand, I remembered an article I read the SF Streets Blog titled: "Advocates Concerned That Cyclists Are Included in Distracted Driving Bill" (link)

One of the choice quotes from the article being:

The California Bicycle Coalition (CBC), which was an early supporter of the original distracted driving legislation, was not thrilled about the inclusion of cyclists in the bill. CBC Communications Director Jim Brown said that he was confused about the motivation for extending the same level of fines to cyclists, particularly absent data showing distracted cycling as a public safety hazard.

"The consequences of a distracted driver are considerably more serious than the consequences of distracted cycling," said Brown, adding that safe riding should be encouraged at all times and that talking on a cell phone or any other practice that distracted a cyclist from riding would not be advisable.

As a member of the San Francisco Bicycle Coalition, I'm partially annoyed by Mr. Brown's comments, but I don't particularly care. Reading further through the article, I found this:

Andy Thornley, Program Director for The San Francisco Bicycle Coalition, agreed with Winter that lumping cyclists with motorists in this law was not good policy. While the SFBC "teaches and preaches safe, respectful, and mindful bicycling," said Thornley, "we're very leery of any equivalence of penalty when punishing a guilty cyclist or driver for the same offense."

"Even worse, we wonder whether bicyclists would be cited more often than motorists because it's so much easier to spot someone texting while pedaling," he added. "It's already a problem of perception that individual bicycle riders seem to be noticed being naughty more than motorists, comfortably anonymous within their glass and steel boxes."

What a hypocrite! Riding your bike while on the phone or worse, texting is just as stupid as some of the no-helmet, no-light nonsense I was incensed over a few weeks ago, but the fact that these two gentlemen from Bike Coalitions want preferential treatment for cyclists in the most idiotic way possible blows my mind. To be honest, I'm entirely in favor of bicyclists being cited more often than motorists for breaking the law (running red lights or not using signals comes to mind).

This kind of no-distractions law makes a lot of sense to me, and should be applied to just about anybody operating a moving vehicle, bikes, trikes, motorcycles, mopeds, cars, tractors, law mowers, you name it. If you are operating a vehicle distracted you raise your chances of hurting yourself or others on public roads (ever been hit while walking by a cyclist?).

Inside of the San Francisco cycling community, I think we can do our part by shunning or otherwise pushing cyclists into light posts who are on their cell phones while riding. They are clearly morons and in my opinion the CBC or the SFBC has no place defending their idiocy.

Programming as an objective art

R. Tyler Croy — Mon, 01 Mar 2010 15:30:00 +0000

Writing software is an outlet for artistic expression to many people, myself included. For me, solving problems involves a good deal of creativity not only in the actual solution but also in the manipulating several moving parts in order to fit the solution into an existing code-base. Combining this creative outlet with a beautiful language, such as Python results in some developers writing code that holds an masterpiece-level of beauty to them, to the untrained eye one might look at a class and think nothing of it, but to the author of that code, it might represent a substantial amount of work and personal investment.

Like art, sometimes the beauty is entirely subjective. there has been times where I've been immensely pleased with one of my creations, only to turn to wholly unimpressed Dave. Managing or working with any team of highly motivated, passionate and creative developers presents this problem, as a group: how can you objectively judge code while preserving the sense of ownership by the author? The first step to objectively judging code in my opinion, is to separate it from the individual who wrote it when discussing the code. For a lot of people this is easier said than done, particularly for younger engineers like myself. Younger engineers tend to have "more to prove" and are thereby far more emotionally invested in the code that they write, while older engineers whether by experience or simply by having written more code than their younger counterparts are able to distance themselves emotionally more easily from the code that they write. Not to say older engineers aren't emotionally invested in their work, in my experience they typically are, it's just a matter being better at picking battles.

Code review is a common sticking point for a lot of engineers, it's incredibly important for both parties in a code review to judge the code objectively, if you are not, a code review can result in hurt feelings and resentment, personal differences bubbling up to the surface in a venue they don't belong in. I think it's immensely important to refer to code as an entity unto itself once a code review starts, phrases like "your code" are a major taboo. Separating the person who wrote the code from the code itself can help both the reviewer but also the original author of the code look at the changes in an objective light. "The code is overly complicated when all it should be doing is X." "The patch doesn't appropriately account for condition Y, which can happen if Z." With a change in semantics, the conversation changes from one developer judging another's work, to two developers objectively discussing whether or not the desired goal has been acheived with minimal downside. (Note: I'm presuming "proper code review" is being performed, devoid of nitpicking on minor style differences) You will find behavior like this in many successful open source projects that make heavy use of code review, the Git project comes to mind. When patches are posted to the mailing list, their merits are discussed as a separate entity, separated from the original author.

This same strategy of separating the individual from the code should also be applied to bugs in the code. When using git-blame(1) for example, there is a tendency to look at who authored the change, seek them out and pummel them with a herring. In a smaller team dynamic, as well as an open source environment, pinning "ownership" of a bug to a particular person is entirely non-constructive. Publicly citing and referencing somebody else's mistake does nothing other than hurt that individual's ego. The important part to refer to with git-blame(1) is the commit hash, and nothing else. With the conversation changed from "Jacob introduced a bug that causes X" into "Commit ff612a introduces a bug that causes X" those involved can then look at the code, and determine what about that code causes the issue. For simpler bugs the original author will typically pipe up with "Whoops, forgot about X, here's a fix" but there are also cases where the original author didn't know about the implications of the change, had no means of testing for X, or the bug was caused by another change the original author wasn't privvy to. If the code is not separate from the individual, those latter cases can be tension points between developers that need not exist, making it all the more important (especially in small teams) to discuss changes openly and objectively.

With code decoupled from the author himself, how does the author maintain that same sense of pride and ownership? The original author should be charge with making any changes that arise out of a code review (naturally) but also should maintain responsibility for that portion of code moving forward; this added responsibility ensures less "fire and forget" changes and adds more pressure on the code reviews to yield improvements to the stability and readability of new code.

As soon as more than one developer is working on a project, it becomes increasingly important to recognize the difference between the "works of art" and the artist himself. The ceilings of the Sistine Chapel are an incredible piece of art, not because they were painted by Michelangelo. Writing code should be no different, the art is not the artist and vice versa.

Pyrage: Static isn't just something on the radio

R. Tyler Croy — Fri, 26 Feb 2010 13:45:00 +0000

Dealing with statics in Python is something that has bitten me enough times that I have become quite pedantic about them when I see them. I'm sure you're thinking "But Dr. Tyler, Python is a dynamic language!", it is indeed, but that does not mean there aren't static variables.

The funny thing about static variables in Python, in my opinion, once you understand a bit about scoping and what you're dealing with, it makes far more sense. Let's take this static class variable for example:

>>> class Foo(object):
...   my_list = []
... 
>>> f = Foo()
>>> b = Foo()

You're trying to be clever, defining your class variables with their default variables outside of your __init__ function, understandable, unless you ever intend on mutating that variable.

>>> f.my_list.append('O HAI')
>>> print b.my_list
['O HAI']
>>>

Still feeling clever? If that's what you wanted, I bet you do, but if you wanted each class to have its own internal list you've inadvertantly introduced a bug where any and every time something mutates my_list, it will change for every single instance of Foo. The reason that this occurs is because my_list is tied to the class object Foo and not the instance of the Foo object (f or b). In effect f.__class__.my_list and b.__class__.my_list are the same object, in fact, the __class__ objects of both those instances is the same as well.

>>> id(f.__class__)
7680112
>>> id(b.__class__)
7680112

When using default/optional parameters for methods you can also run afoul of statics in Python, for example:

>>> def somefunc(data=[]):
...    data.append(1)
...    print ('data', data)
... 
>>> somefunc()
('data', [1])
>>> somefunc()
('data', [1, 1])
>>> somefunc()
('data', [1, 1, 1])
>>>

This comes down to a scoping issue as well, functions and methods in Python are first-class objects. In this case, you're adding the variable data to the somefunc.func_defaults tuple, which is being mutated when the function is being called. Bad programmer!

It all seems simple enough, but I still consistently see these mistakes in plenty of different Python projects (both pony-affiliated, and not). When these bugs strike they're difficult to spot, frustrating to deal with ("who the hell is changing my variable!") and most importantly, easily prevented with a little understanding of how Python scoping works.

PYRAGE!

If you want a viral license, use the GPL

R. Tyler Croy — Tue, 23 Feb 2010 14:45:00 +0000

My "roots" in the open source community come from the BSD side of the open source spectrum, my first major introduction being involvement with FreeBSD and OpenBSD. It is not surprising that my licensing preferences fall on the BSD (2 or 3 clause) or MIT licenses, the MIT license reading as follows:

Copyright (c) [year] [copyright holders]
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

I bring the subject up because I wanted to address a brief "kerfuffle" that occurred recently on the Eventlet mailing list with the maintainer of gevent, a fork/rewrite of Eventlet. Both projects are MIT licensed which gives anybody that would like to fork the source code of either project a great deal of leeway to hack about with the code, commercialize it, etc. Disclaimer: I personally am a fan of Eventlet, use it quite often, and have recently taking up maintaining Spawning, a WSGI server that supports multiple processes/threads, non-blocking I/O and graceful code reloading, built on top of Eventlet.

The "kerfuffle" occurred after Ryan, the maintainer of Eventlet, took a few good modules from gevent; shocking as it may seem, a developer working with liberally licensed code took liberally licensed code from a similar project. The issue that the maintainer of gevent took with the incorporation of his code was all about attribution:

I don't mind you borrowing code from gevent, the license permits it. However, please make it clear where are you getting the code from.

Upon first reading the email, I doubled over to the eventlet source on Bitbucket, checked the files that were incorporated into the codebase (timeout.py and queue.py)and sure enough the copyright attributing the original author were still in tact, surely this is a non-issue?

Unfortunately not, license pedantry is an open source community past-time, right up their with drinking and shouting. When I replied mentioning that the copyrights were correctly in place, mentioning that both projects were MIT licensed so both constraints of the license were met, that is, the MIT license notice was included with the code. In essence the disagreement revolves around what the phrase "this permission notice shall be included" entail, my interpretation of the license is such that the MIT license itself shall be included, not the specific file with additions from one project to another; after sending off my mail, I received the following reply:

Ok, it's acceptable to use one LICENSE file but only if the copyright notice from gevent is present unchanged.

That is, take the notice from here http://bitbucket.org/denis/gevent/src/tip/LICENSE (the line with the url), and put it into eventlet's LICENSE, on a separate line. (It's OK to add "Copyright (c) 2009-2010" before it to make it in line with others).

That would settle it.

Slightly pedantic in my opinion, the MIT license enumerates a line for copyright holders which has been hijacked for other information that the maintainer of gevent would like to propagate, I don't necessarily agree, but this is a mailing list not a court of law, so I'll allow it. The thread continues:

The license did not change. I've only updated the copyright notice to include the url of the project to protect against abusive borrowers, that's it.

This is where I draw the line, go all in, plant my flag in the sand and other unnecessary metaphors. Abusive borrowers? Analyzing the semantics of the phrase alone makes my head hurt, I have a mental image of two old ladies wherein one says to the other: "may I borrow a cup of sugar, you horse-faced hunch-backed bucket of moron?" The rest of the email is full of similarly head-hurting quotes, for brevity I won't include them here (you can read the thread in the archives).

I'm simply dumbfounded by the ignorance of what the MIT license actually means, unlike the LGPL or the GPL license which were specifically drafted to protect against "abusive borrowers", such as Cisco, the MIT license is so open it's almost public domain.

To a certain extent I can understand the emotions behind the thread on the mailing list, I don't agree with them. If you're seeking attribution past the copyright line in a header, than perhaps the "original" 4 clause BSD license is for you, or perhaps the LGPL or GPL which give you more control over what happens to the source code you originate? If this is what you're after, the MIT license is the wrong license.

I hope you bump your head

R. Tyler Croy — Wed, 10 Feb 2010 07:28:56 +0000

There are few things I truly enjoy in life, things that warm my heart and make me smile from ear to ear, things like hot oatmeal and coffee, cell coverage in San Francisco, back scratches and not dying. If I didn't commute every single day on my bicycle, I'd likely put "bike riding" on the list too but it's hard to be ecstatic about something you start and finish every workday with (except for maybe some recreational yelling). For about 25 minutes every morning and 25 minutes every night, I'm a cyclist in San Francisco, nothing terribly unique, I am one of many cyclists in the city during rush hour and I have been noticing some things lately.

First of all, let me address my fellow cyclists. Some of you (you hopefully know who you are) are unadulterated, complete fucking-assholes. I understand that you don't need a license to ride a bicycle but you still have to obey the rules of the road. Among other things this means you should be courteous to those you're sharing the road with which includes:

Using hand signals
Yielding (you can use the dictionary on the iPhone that you're pulling out at intersections to look up what that word means). Despite the ironic t-shirt you're wearing underneath your hoodie, you are not "King of the World".
Riding on streets safe for bicycle use. I know you love to ride down Oak St. and Van Ness in rush hour but it does nothing but piss drivers off. There are typically less trafficked streets adjacent to busy streets which are preferred for cyclist use (such as Page and Market St.).
Stopping at stop lights. I'm sure Gavin and the SFPD have given you an exemption from basic traffic laws but the drivers coming into the intersection don't know that. You are making things dangerous for everybody around you on the road; stop it.

Given the hours I work, I'm typically riding home in the dark, where I've noticed a group of people who don't share the same love of "not dying" as I do. I'm going to call them the "Suicide Cyclists." This group of morons never cease to disappoint, without fail you'll see one or two of them riding down a busy street (like Market St.) with no lights and no helmet. Some vindictive part of me really would like to see a car accidentally hit one of them. I don't want them to seriously injure or kill anybody, just a light enough tap to send them to the ground and bump their head just hard enough to knock some god-damn sense into it.

There are parts of my bike ride where I might even crash into one of these Suicide Cyclists, places where I'm riding on a two-way bike path (through a poorly lit park) or where I'm turning at an intersection. If through some unfortunate turn of events we crash into each other, the cyclist wearing a dark sweatshirt with no helmet or lights on their bicycle, and me, with my plethora of lights and hand signals, you can be certain that the next thing that's going to happen is assault with a U-Lock against some poor schmuck who's too stupid to ride on the right side of the bike path in the dark.

To be honest, I can deal with the immense number of drivers not paying enough attention to the bike lanes, red lights or speed limits, the majority of the time cars do not cut through intersections, sidewalks, parks and everywhere else. Their trajectories are far more predictable than some jerk riding his fixie with his corduroy bike cap where ever he damn well pleases.

Grumble.

Mourning Sun

R. Tyler Croy — Sun, 31 Jan 2010 03:51:52 +0000

Some users of Hudson have already started to notice a subtle addition to the latest release, 1.343, a new background watermark image.

The commit message (r26728) from Kohsuke, the incredibly talented founder and maintainer of the Hudson project, adds a bit of sadness to the whole affair:

In tribute to Sun Microsystems and all my colleagues who had to go today. I hope the community would forgive me for doing this.

Given the incredible speed at which the tech industry grows and moves, it's easy to forget that there are a number of talented engineers that have spent their careers at Sun building technologies that have helped change the face of modern computing, regardless of whether or not Sun could figure out how to sell them: SunOS/Solaris, Java, DTrace, SPARC 64-bit chips, Sun Grid Engine, JRuby, the W3C XML specification, ZFS, OpenOffice (acquisition), MySQL (acquisition), and VirtualBox (acquisition).

As a corporation, I personally think Sun was a failure, as a foundation of engineering in Silicon Valley, I think Sun has been quite successful.

To those that are being pushed out as part of the merger with Oracle, I want to sincerely thank you for your contributions to computing and wish you the best of luck. Here's the "full" version of the image, which I found via @jtnl's TwitPic stream:

Unsubstantiated Rumors about Apture and Facebook

R. Tyler Croy — Wed, 20 Jan 2010 22:19:19 +0000

Yesterday I was pointed to this post on ReadWriteWeb, suggesting that Facebook should acquire Apture next. Being an Apture employee, I would like to take some time to fuel the rumormill with these COMPLETELY TRUE (read: false) rumors:

Unsubstantiated Rumor #1:

A shadowy figure in a black Northface jacket has been spotted lingering around 539 Bryant St in San Francisco

Unsubstantiated Rumor #2:

Apture's CTO is seen regularly in Palo Alto

Unsubstantiated Rumor #3:

Apture employees were issued company Adidas sandals last December

Unsubstantiated Rumor #4:

Mark Zuckerberg and Tristan Harris talk regularly at meetings of the Bay Area Strip-Parcheesi Club in Redwood City

Unsubstantiated Rumor #5:

Apture is hiring aggressively in order to bump up their acquisition price

Thread-safety assumptions in Django

R. Tyler Croy — Tue, 19 Jan 2010 05:23:26 +0000

These days, the majority of my day job revolves around working with Apture's Django-based code which, depending on the situation, can be a blessing or a curse. In some of my recent work to help improve our ability to scale effectively, I started swapping out Apache for Spawning web servers which can more efficiently handle large numbers of concurrent requests. One of the mechanisms by which Spawning accomplishes this task, is by using eventlet's tpool (thread pool) module in addition to some other clever tricks. With Apache, we used pre-forked workers to accomplish the work needed to be done and while still using forked child processes with Spawning, threading was also thrown into the mix, that's when "shit got real" (so to speak).

We started seeing sporadic, difficult to reproduce errors. Not a lot, a trickle of exception emails throughout the day. Digging deeper into some of the exceptions, careful stepping through Apture code, into Django code and back again, I started to realize I had thread-safety problems. Shock! Panic! Despair! Lunch! Disappointment! Shock! I felt all these things and more. I've long lamented the number of globals used in Django's code base but this is the icing on the cake.

Apparently Django's threading problems are sufficiently documented in a few places. Using a slightly older version of the Django framework certainly doesn't help but it doesn't appear that recent releases (1.1.1) can guarantee thread-safety anyways. I think it's safe to assume the majority of Django framework users are not using threaded web servers in any capacity, else this would have become a far larger issue (and hopefully of been fixed) by now. From NoReverseMatch exceptions, to curious middleware problems to thread-safety issues in the WSGI support layer, Django has potholes lying all along the road to multithreadedness.

Beware.

Book of Elies

R. Tyler Croy — Fri, 15 Jan 2010 08:08:07 +0000

There's quite a big advertising blitz in San Francisco for the "Book of Eli" movie, which as far as I can tell is another in a line of quasi-religious films (fortunately Tom Hanks doesn't star in this one). I now see a billboard that's a derivation of this on my ride home from work:

To be honest, I was already not going to see this movie but their marketing campaign has hammered the final nail in the cross (er, coffin). Perhaps I'm far more woeful of religion after seeing too many documentaries like "Jesus Camp" and watching clips of Pat Robertson.

In a world of spiteful neo-conservatives hijacking public discourse on important issues with nonsense about the gays, abortion and anything else that can be misconstrued as "christian values." A world where radical sects of Islam kidnap, murder and terrorize; a world where no religion is without blood on their hands, the billboard is technically right.

I think that's why it bothers the hell out of me.

The one where I gush about Contegix.

R. Tyler Croy — Tue, 05 Jan 2010 07:29:40 +0000

Since joining Apture, I've primarily concerned myself with lower-level backend code and services, including the machines that our site runs on. While not a drastic departure from my role on the server team at Slide, there are a few notable changes, the largest of which being root. Given the size of Slide's operations team, a team separate from the "server team" (the latter being developers), my role did not necessitate server management only occasional monitoring. Apture is a different can of beans, we're simply too small for an operations team, so we work with Contegix to maintain a constant watchful eye on our production environment. Self-assigning myself the "backend guy" hat means server maintenance and operations are part of my concern (but not my focus) since the "goings on" of the physical machines will have a direct impact on the performance and level of service my work can ultimately provide to end users.

Last week while planning some changes we can make to our Django-based production environment that will help us grow more effectively, Steven pointed out that we were going to see an influx of usage today (Jan. 4th) given the large number of users returning to the internet after their holiday vacation. Over the weekend I dreaded what Monday would bring, unable to enact any changes to stave off the inevitable in time.

This morning, waking up uncharacteristically early at 7 a.m. PST, bells were already ringing. A 9 a.m. EST spike angered one of our database machines, by the time I got in the office around 8:10 a.m. PST more bells were ringing as the second wave of users once again angered the MySQL Dolphin Gods. With my morning interview candidate already on site, I furiously typed off a few emails to Contegix sounding the alarm, pleading for help, load balancer tweaks, configuration reviews, anything to help squeeze extra juice from the abnormally overloaded machines to keep our desired level of service up. Working with a few of the talented Contegix admins we quickly fixed some issues with the load balancer under utilizing certain machines in favor of others, isolated a few sources of leaked CPU cycles and discovered a few key places to add more caching with memcached(8).

As our normal peak (~9 a.m. PST to around lunchtime) passed, I started to breathe easier when alarms went of again. Once again, Contegix admins were right there to help us through one of our longest peak I've seen since joining Apture, 5:30 a.m. until around 4 p.m.

Survival was my primary objective waking up today but thanks to some initiative and good footwork by the folks at Contegix we not only survived but identified and corrected a number of issues detrimental to performance and discovered on of the key catalysts of cascading load: I/O strapped database servers (as MySQL servers starve for disk I/O, waiting requests in Apache drive the load on a machine through the roof).

I am admittedly quite smitten with Contegix's work today, I became quite accustomed to KB and his ops team at Slide fixing whatever issues would arise in our production environment and it's comforting to know that we have that level of sysadmin talent at the ready.

A picture is worth a thousand words; here's a cumulative load graph from our production Ganglia instance:

Angry Tweeter

R. Tyler Croy — Tue, 29 Dec 2009 03:05:16 +0000

This year my family celebrated the holidays in north Florida at my older sister's house, fortunately for the location is just as difficult to get to by plane as my parent's house, so I didn't have to miss out on any air travel frustration. My trip to north Florida was very boring, my flight out of San Francisco left at 6 a.m. and I arrived in Jacksonville around dinner time (having slept the majority of the flight). The return trip was far more eventful, I left my sister's house around noon to drive to Jacksonville (roughly an hour and a half trip), waited at the airport for my flight at 4 p.m., arrived in Miami at 5:30, waited for hours on a delayed flight, left Miami around 10 p.m., landed in San Francisco a hours later than anticipated, paid my exorbitant parking fee and sped home.

When I woke up the next day, I looked over at how bitchy and whiney my posts to Twitter from the previous day were. I don't think I'm normally that big of a jerk but traveling alone, I needed to vent, often. (note: times listed are PST, for the majority of the trip I was in EST)

11:10 AM Still kind of amazes me how many young women in the south are running around with babies in tow.

11:34 AM How the news should cover this incident: Guy tries something on plane, passengers take him out. Post-9/11, TSA is pointless

12:57 PM Shit. My first flight today has propellers. Fucking propellers.

1:22 PM Propeller death trap. http://flic.kr/p/7qzKVn

3:20 PM Step 1 complete; in Miami. Now to SFO.

3:29 PM How does restricting carry-ons make up for the TSA's incompetence? Predictable kneejerkery.

3:43 PM Waiting on a fucking bus to go back to the terminal. My connection is boarding right the hell now :/

3:56 PM "Fortunately" my flight was delayed an hour, so I didn't nearly catch it. Fuckin AA.

4:24 PM Let's just stop flying airplanes, that's the only real way to stay safe from the terrurists.

4:47 PM It's okay American Airlines, I really didn't want to go home anyways.

5:05 PM Outlined a couple posts I want to write on the plane; our original delayed departure time has past, I might never get to writing

6:03 PM More delayment. 2nd potential departure time passed

6:08 PM What kind of toolbox gets luggage embroidered with their initials?

6:11 PM Apparently somebody on the previous flight puked, so we're delayed while they scrub vomit off the seats. Worth it.

6:26 PM The snacks in Miami are a bit dry http://flic.kr/p/7qHbBN

6:40 PM This plane better make it to SFO, I'll be pretty pissed to have waited 3 hours to end up in a field.

7:23 PM OMFG I AM TOTALLY ON THE PLANE. 3.5 hours late.

1:45 AM Made it to SFO, only a few hours late. Now to drive home, find a parking spot and cry in the shower a bit

3:05 AM Finally home. Showered, clipped fingernails, q-tip'd ears. Feeling better.

Suffice to say, I don't think I'm flying American Airlines again for some time (or at all for that matter). The whole experienc to Florida and back was grueling to say the least; with the parking fees, baggage fees, meal fees, delays and endless hours breathing recycled air riddled with H1N1 and sneezes, I think I'm going to keep my feet on the ground for a while.

On being a good house guest

R. Tyler Croy — Mon, 28 Dec 2009 05:02:44 +0000

In a past life I traveled quite frequently, being categorically poor as I often was, I tended to rely on the kindness of friends, family and occasionally total strangers. After breaking the standing record for longest-time-spent-on-Dave's couch, I came to consider myself a pretty decent house guest. More recently I spent this past week at my older sister's house with a swarm of other family members, as I cooked breakfast for the family Saturday morning, I decided that I'm not only a pretty decent house guest, I'm a pretty stinkin' awesome guest who you should invite over if you:

Feel like cooking a big dinner but don't want to do the dishes
Need an expert Rock Band guitarist
Have children that need entertaining
Are just sick and tired of cooking and really would like somebody else to make you something delicious
Feel the need to have a wide-ranging discussion regarding national and international politics with a mildly intelligent person

While I know that you're not supposed to stay too long as a house guest, I think the social rule comes from a long line of either unwelcome guests or guests that just aren't doing it right. Here are the rules I try to follow whenever I find myself crashing on some kind person's couch, floor or air mattress.

Keep your things tidy

Very important, yes you're likely traveling, the folks you're staying with understand that you don't have a closet or dresser you can throw your clothes in, it is very important however that you keep as much of your belongings tidily stashed away in your suitcase. The extra effort goes a long way in making your presence far less impactful on those hosting you. Nobody likes a dirty home.

Offer to cook

Unless your hosts have more money than they should, chances are that they have jobs and when they come home from those jobs they have to cook themselves and their family dinner. Offering to cook goes a long way with a lot of people, especially if you can actually cook. (note: cooking delicious food is not difficult, but really just a test of your ability to read a recipe). Not everybody will take you up on your offer, some people (myself included) find cooking a good way to unwind after a day's work. If you find yourself in this situation, linger around the kitchen, socialize and try to be as helpful as possible; an extra set of hands and eyes to watch a pot, or peel potatoes is almost always appreciated.

Hang out, don't cling

Most people enjoy having company, humans are inherently social animals and having a house guest can be an nice change of pace for a lot of people. If you're traveling through, you'll have to walk a fine line of hanging out with the hosting party long enough to have fun together but not long enough to make them feel smothered. The system I've always followed is to be occupied during the day and social with my hosts in the evening. This gives them a chance to have a normal workday or weekend, and gives me the chance to explore my current location on my own and have an adventure. This set up works quite well when traveling abroad since you get the opportunity to regale your hosts with tales of your adventure in their region over dinner (note; do not trash their city, people tend to have some amount of pride for their city/region/state).

Do chores

If you're staying with any host for any time longer than a few days, it is highly likely that some cleaning, vacuuming, laundry or dishes will need to be done. A good rule of thumb with chores is not to offer to help but just to help out where you can, a quick "let me give you a hand with that.." will do.

Nobody will turn down a helping hand when it comes to cleaning.

There are occasions when I've preferred hotels to crashing with friends or family, when I really need a good night's rest and some quiet, but if you're up for a good sociable experience, you really cannot beat crashing on a couch.

Just don't borrow any money, they really hate that.