Sean Schulte

Twitter: @sirsean

Github: github.com/sirsean

Mailgun and Go, go-mailgun 24 Sep 2013

Lately, I've been playing with Go. The language itself is small enough that I quickly got to the point that I needed to write an app in order to feel like I was learning anything more. As it happens, earlier this year, I wrote an app called "MLB Notifier" that monitors the MLB datafeed for changes and emails me* about interesting changes. That was written in Java, and runs on Google App Engine. It seemed like a good opportunity to rewrite an app in Go; I'd get to deal with network communication, JSON parsing, sending email, and a not-insubstantial amount of logic.

* Note to MLB's lawyers: it only sends emails to me, and nobody else. This is an individual, non-bulk, non-commercial use.

This isn't about MLB Notifier. It's about email. From App Engine, I was able to just send messages from my Gmail account, because it's running on my Google account in Google infrastructure. For some reason, I assumed it wouldn't be an issue to just keep doing that from a VM running somewhere on the internet. But when I tried sending emails via SMTP using my Gmail credentials*, I immediately received an email from Google saying "Suspicious sign in prevented". Seems like someone trying to log into my Gmail account from a server in Indonesia seems weird to Google? Also, it turns out my VM lives in Indonesia.

* Since this would require me to leave those credentials sitting on that VM, I already didn't like this solution.

Enter Mailgun. My team uses it at work (in Ruby), and it seemed simple and effective. Googling around for how to use it from Go didn't surface any useful results (come on, you guys, I can't be the first person to do this), so I had to figure it out on my own. Mailgun provides solid API documentation, and I was able to convert their Ruby/PHP/Curl examples fairly easily.

They have a very easy endpoint that you hit over HTTPS, which I personally think is nicer than having to actually deal with SMTP myself. The basic process for sending an email is:

  • Send POST variables designating the sender, recipient, subject, and message body
  • Set the Content-Type to application/x-www-form-urlencoded
  • Set HTTP Basic Authentication to log in with api:[whatever Mailgun says is your API key]
  • Send this request to the special endpoint that Mailgun gives you, which has your username in it

You can see all the code on GitHub, in my new go-mailgun project.

I wanted to pass a struct representing my message, which is really easy to define in Go:

type Message struct {
    FromName string
    FromAddress string
    ToAddress string
    Subject string
    Body string
}

Along with a method to format the from name/address:

func (m Message) From() string {
    return fmt.Sprintf("%s <%s>", m.FromName, m.FromAddress)
}

So then I just define my Send method:

func Send(message Message) error {
    client := &http.Client{}

Here we set up the POST variables based on the Message provided:

    values := make(url.Values)
    values.Set("from", message.From())
    values.Set("to", message.ToAddress)
    values.Set("subject", message.Subject)
    values.Set("text", message.Body)

And then construct an HTTP request with the Content-Type header and Basic Auth:

    request, _ := http.NewRequest("POST", ApiEndpoint, strings.NewReader(values.Encode()))
    request.Header.Set("content-type", "application/x-www-form-urlencoded")
    request.SetBasicAuth("api", ApiKey)

Then we send the request (note that Go lets us "defer" closing the body until we return from this method, which is very convenient, in case any of you have experience writing deeply nested try/catch/finally blocks in Java to do the same thing ... actually, I'm sorry I mentioned it, let's just move on):

    response, e1 := client.Do(request)
    if e1 != nil {
        fmt.Println("Failed to send request")
        fmt.Println(e1)
        return e1
    }
    defer response.Body.Close()

And I read the response here, even though I really don't need it. Although printing it in the logs was useful, at the beginning when I hadn't set the Content-Type and Mailgun's error message told me it couldn't send a message without a "from" parameter. I took that to mean that it wasn't receiving the parameters, and it wasn't long before I figured out it needed the Content-Type.

    body, e2 := ioutil.ReadAll(response.Body)
    if e2 != nil {
        fmt.Println("Failed to read response")
        fmt.Println(e2)
        return e2
    }

    fmt.Println(string(body))
    return nil
}

So that's easy! But that's if you want to write the method yourself, which I don't intend to do again. Using it is even easier, as demonstrated in this degenerate example app:

import (
    "github.com/sirsean/go-mailgun/mailgun"
)

func main() {
    mailgun.ApiEndpoint = "https://api.mailgun.net/v2/YOURNAME.mailgun.org/messages"
    mailgun.ApiKey = "YOURKEY"

    go func() {
        err := mailgun.Send(mailgun.Message{
            FromName: "Foo Bar",
            FromAddress: "foo@bar.test",
            ToAddress: "recipient@bar.test",
            Subject: "This is an example message",
            Body: "It's pretty easy to send messages via Mailgun!",
        })
        if err != nil {
            // you can handle sending errors here
        }
    }()
}

I set my endpoint/key once (in my app I do it in main.main, having read the values from a YAML file), and then send the email within a goroutine. In this case it's an anonymous function, but obviously it doesn't have to be; in my MLB Notifier application I name a separate method that calls mailgun.Send, because in that case it makes the code clearer.

So, I was pleased by how simple it was, and I have been pleased with Mailgun's performance delivering the emails. Notably, except for one evening where I was seeing weird behavior:

  • (21:42) SD tied it up in the 9th, 2-2
  • (21:42) SD broke the tie in the 9th, 3-2
  • (22:42) SD tied it up in the 9th, 2-2
  • (22:59) SD broke the tie in the 9th, 3-2

That 9th inning took over an hour, and the same team tied it up and took the lead multiple times? I investigated my logic repeatedly, and for the life of me I could not figure out how this could be happening.

And then, I received an email from Mailgun with a link to a post explaining what happened. I was satisfied with the explanation and was impressed and pleased that they got right out in front of it without any denials or excuses. So I'll continue to happily use Mailgun to deliver interesting events in baseball games to myself ... unfortunately for them I don't really have to think about it any more, now that it's running smoothly.


Rails has_secure_password, a reminder 20 Feb 2013

I'm using the new has_secure_password feature in Rails (I believe it was introduced in Rails 3.1) for user authentication. It uses bcrypt under the hood, which I'd used directly in the past. But has_secure_password seems to be the cool new way to do this, and it does seem pretty easy.

has_secure_password gives you a bunch of things for free: password hashing and salting, authenticating against the hashed password, and password confirmation validation.

But all the tutorials and explanations I've seen omit a few key details in the name of simplicity. Namely, they don't show you what you need to do for the password confirmation to work, and they don't show you how to validate the password correctly when the User object is updated (ie, they only show you what to do when it's created).

The RailsCast on the topic makes it seem sooooo easy, which is I guess the point, but if you just do what they do you'll end up confused and with something that doesn't work.

Their User model looks like this:

class User < ActiveRecord::Base
  attr_accessible :email, :password, :password_confirmation
  has_secure_password
  validates_presence_of :password, :on => :create
end

Note that it only validates the password when the User is created; that's because they want you to be able to change the username later, without having to re-enter the password. But if you add a length restriction to your password (which you should), then later on when the user changes it, the minimum length won't be validated.

Additionally, and I think this is key, the RailsCast shows you the User model, and how to authenticate, but never actually shows how to create a user and have the password_confirmation validated.

user = User.new(:email => "my@email.com")
user.password = "password"
user.valid? => true

I would've thought that would fail validation, because there's no password_confirmation set. But this is the key, the thing that apparently isn't mentioned anywhere: the password confirmation is only validated if you attempt to set it.

user = User.new(:email => "my@email.com")
user.password = "password"
user.password_confirmation = "otherpass"
user.valid? => false

user.password_confirmation = "password"
user.valid? => true

Okay, so you have to actually set the password_confirmation or else it won't do anything. But how do you make your validation work when you're updating the user?

Here's a very basic User model that will do that:

class User < ActiveRecord::Base
  attr_accessible :id, :username, :password, :password_confirmation

  has_secure_password

  validates :username, 
    :presence => true, 
    :length => { :minimum => 3 }, 
    :uniqueness => true

  validates :password,
    :length => { :minimum => 8, :if => :validate_password? },
    :confirmation => { :if => :validate_password? }

  private

  def validate_password?
    password.present? || password_confirmation.present?
  end
end

Here, it will validate the password and its confirmation if either the password or the confirmation are set, or if it's a new User with no password set. This will allow you to update the username without setting the password, but will actually validate the password if you attempt to set it. (And still has the restriction that it only attempts to validate the confirmation if you set password_confirmation.)


AppEngine HTTP Headers 29 Jan 2013

So, I'm working on a new Android app* which requires user authentication and storing data to a backend service. Since pretty much everyone with an Android phone has a Google account, and Google is doing a good job of implementing OAuth 2 for me, I figured I'd just use their authentication. Tim Bray alerted me to the cool new way of authenticating which doesn't require the user to enter any passwords, and may not even require the user to even click anything.

* It's a game, with asynchronous multiplayer functionality.

After being confused for a while about how complicated it was supposed to be -- I thought you have to pass in the token from GoogleAuthUtil.getToken into something else to complete the authentication and pass a cookie back to AppEngine to maintain an HTTP session, when in fact you can just pass the token directly to your service to verify identity -- I was ready to start moving forward.

I decided that I'd pass the token as a custom HTTP header, so I don't have to either put it in the query parameters (which is dangerous even on HTTPS) or pollute the JSON I'm passing to the service. I figured X-GoogleAuthToken would be a good option for a header name. And this is where our story really begins.

On the local dev service, X-GoogleAuthToken worked fine, and I was able to authenticate using the token. But when I deployed to production, the value for the header always came through as null.

So I wrote an extremely simple servlet to test out the headers.

public void doGet(HttpServletRequest req, HttpServletResponse resp)
        throws IOException {
    resp.setContentType("text/plain");
    Enumeration headers = req.getHeaderNames();
    while (headers.hasMoreElements()) {
        String name = (String)headers.nextElement();
        String value = req.getHeader(name);
        resp.getWriter().println(String.format("%s: %s", name, value));
    }
}

I passed in the following headers*:

GET /lettergarden HTTP/1.1
X-Googleauthtoken: blah blah
Googleauthtoken: test
Fakeheader: here
X-Gauthtoken: howdy
X-Google-Auth-Token: 123
X-G-Auth-Token: boosh

* It was supposed to be "X-GoogleAuthToken" and "fakeheader" and "X-Gauthtoken", but I'm using an app called "HTTP Client" that apparently, and I just learned this, title cases the headers before sending them. Seems wrong?

And on the local server, here's what I got:

Host: localhost:8888
User-Agent: HTTP%20Client/0.9.1 CFNetwork/454.12.4 Darwin/10.8.0 (i386) (MacBookPro5%2C4)
X-Googleauthtoken: blah blah
Googleauthtoken: test
Fakeheader: here
X-Gauthtoken: howdy
X-Google-Auth-Token: 123
X-G-Auth-Token: boosh

Okay, that's everything I passed in, plus the Host and User-Agent. Nothing to suggest there are dragons approaching.

Here's what I get after deploying to production:

Host: some-project-this-isn't-really-the-name.appspot.com
Fakeheader: here
Googleauthtoken: test
X-Gauthtoken: howdy
X-G-Auth-Token: boosh
User-Agent: HTTP%20Client/0.9.1 CFNetwork/454.12.4 Darwin/10.8.0 (i386) (MacBookPro5%2C4)
X-AppEngine-Country: US
X-AppEngine-Region: il
X-AppEngine-City: chicago
X-AppEngine-CityLatLong: 41.878114,-87.629798

The AppEngine documentation says they'll add X-AppEngine-Country, X-AppEngine-Region, X-AppEngine-City, and X-AppEngine-CityLatLong as a service to you so you can do some location stuff easily. They also say they will remove some headers from requests: Accept-Encoding, Connection, Keep-Alive, Proxy-Authorization, TE, Trailer, Transfer-Encoding.

You may have noticed, however, that it also stripped X-Googleauthtoken and X-Google-Auth-Token from the request. The documentation does not indicate anywhere that AppEngine will strip any HTTP header that starts with "X-Google". And it does not strip headers that start with "Google" (which obviously you shouldn't do).

I'd rather Google didn't bogart the entire X-Google* namespace if they're not even using any part of it. But, frankly, it wouldn't be a big deal at all if they'd just document that they're doing it and make the dev environment mirror the actual production environment.

I spent a few hours combing Google search results, AppEngine documentation, and StackOverflow for why this was happening. Maybe, having written this, the next poor sap to be bitten by this won't have to waste so much time.


Dependo Pull Request 29 Jan 2013

Last night, I got my first pull request on GitHub from someone I don't know. Pretty cool.

As you may know, Dependo is a very simple dependency injection library for Ruby, which just overrides the #method_missing method and lets you inject methods onto an object just by mixing in the Dependo module.

Well, apparently someone wanted to be able to see if methods have been injected onto the object. I'd been doing that by calling Dependo::Registry#has_key?, but this new way is better.

module Mixin
    def respond_to?(key, include_private=false)
        if Dependo::Registry.has_key?(key)
            true
        else
            super(key, include_private)
        end
    end
end

So I quickly accepted the pull request. I probably wouldn't have thought of this without some guy finding my library, using it, and contributing the idea and making my project better.

In case you needed another reason to open source your stuff, this is a great example of why it's a good thing.

Thanks, Tomas!


Whither Scoreboard? 27 Dec 2012

My most popular apps have been MLB Scoreboard (121,664 users) and NFL Scoreboard (61,994 users). In both cases, the apps scratched an itch, started small, and grew from there.

The birth of MLB Scoreboard

I started MLB Scoreboard because MLB At Bat for Android was awful two years ago.* I needed an app that would tell me the scores of baseball games, with as few clicks as possible, with as many games on the screen as possible, and without taking several extra seconds to load for no reason.

* Now, it's merely inadequate. Big step up?

So, MLB Scoreboard was very, very simple when it first started. At the beginning, it was just a scoreboard: one screen, which showed all the games for the day in the order they were played, with the linescore like you'd see at the stadium. There was also the ability to go forward and backward by a day at a time, to see the previous days' results or tomorrow's schedule. That was it.

The app grew slowly. It took weeks to get to 100 users, and I was really excited. I kept adding new features: your favorite team would always appear at the top of the screen; I showed the current outs and runners on base; I let you click into a game to view the play-by-play and the hitting/pitching boxscore; I added highlight videos for each game; soon, there was a widget that showed today's game for your favorite team, so you didn't even have to wait for the app to open to get the score.

By the middle of the summer in 2011, it had somehow shot past 1000 users. That was the number I'd barely dared to hope it'd get to. When people had asked when I was going to put ads in it, I'd been saying that I'd bother with that when it got to 1000; but by the time it happened, I was too concerned by the fact that MLB's license for their data says it's for non-commercial uses only. If I tried to make any money, it wouldn't be non-commercial any more. So I left the ads out.

And it kept getting more popular. Near playoff time, people seemed to discover it. By the end of October, 7000 people were using it multiple times per day. On the modern internet, where ten million is the new one million, a few thousand is nothing. But for me, it was exciting.

And so, with the baseball season winding down, it was time for football season to start. And it just so happens that the NFL's Android app is much, much worse than MLB's. Like, unforgiveably awful. I needed a better app to see the scores, if I was going to follow football.

The birth of NFL Scoreboard

NFL Scoreboard started the same way as MLB Scoreboard, and perhaps even more barebones: just a list of the games and their scores.

And it similarly grew slowly, but not quite as slowly. Maybe the kind-of-popularity of MLB Scoreboard helped? It quickly shot past a few hundred users. By the end of the (much shorter) season, it had 1000 users.

Despite being monumentally ugly, it had a few advantages over NFL's official app. The official app took 30 seconds to start, and after a couple of clicks you actually got to the scores, and you could see two games on a screen. My app started in under one second, and went straight to the list of scores, and you could see seven games on a screen.

A new baseball season

In March 2012, I was contacted by AT&T, who wanted to promote MLB Scoreboard as a featured part of their new Android app store. Since baseball season was about to start, they wanted an entire baseball section. So they had me fill out some informational forms about the app, with logos and a description, etc. Pretty much all the same stuff you'd get from the actual Google Play (nee Android Market). I knew it wasn't going to be a big deal, but I told people that I was letting myself become foolishly excited for all the downloads I was going to get from AT&T users.

I should have known it wouldn't go that way. AT&T, being AT&T, never got their act together with the store. There was no baseball section. There was a sports section, but MLB Scoreboard was never in it. The only apps in it were those ultra-lame cookie-cutter ringtone/wallpaper apps. And every image on their app store site was broken. It was almost as poorly executed as AT&T's actual phone service.

But the 2012 baseball season rolled on anyway, and even without AT&T's help, MLB Scoreboard had somehow gotten popular.

At the start of spring training, there were 14000 installs (5000 active users). The rate of installation increased throughout March, and at the start of the season there were 26000 installs (14000 active users). Throughout the entire month, I avidly checked my download stats every morning, shocked that another few hundred people had downloaded it the previous day.

Because there was suddenly so much interest, I felt I needed to improve the app. It was pretty ugly. So, I spent a few weeks rebuilding the entire UI, making it more modern and fitting better into the rest of Android (ie, ActionBar), and adding animations to make things look cooler and smoother. At first, people very vocally hated what I was doing, and I got a bunch of angry emails.

And then, on Opening Day, the actual baseball-loving public started caring about baseball. And I got 1000 downloads that day. And the next. And more the next. For the first couple of weeks in April, I was getting 3000-4000 downloads every day. It slowed back down in May, to the point where I was only getting about 400-500 per day* for the rest of the season.

* It's amazing how quickly I got used to that. A year earlier, I would have been amazing by having a total of 400 users.

The growth rate stopped at the end of the playoffs, once again, peaking around 120K (63K active). Turns out people don't really care about the score of today's baseball game when there are no games.

A new football season

Right before the start of the 2012 football season, I received an email from a shady Russian company, offering to buy NFL Scoreboard from me for $1000. It was ugly, and only had 1000 users. I briefly considered it, but decided I'd rather make the app better, than sell out for what is basically a pittance.

I modernized the app's interface, again with ActionBar. I added features, like full game stats, full play by play, and all the passing/rushing/receiving stats for the players. People seemed to love these new features. Every Sunday morning before the games, I got several thousand new downloads.

After week two, NFL Scoreboard had shot up to 60000 users, and my growth rate was increasing.

I was loving it.

First tangle with a legal department

On September 14, 2012, I received an email from Google. They informed me that NFL Scoreboard had been suspended, and removed from Google Play, for "alleged trademark infringement".

They informed me that repeated violations of this nature would result in my entire developer account being suspended; this made me immediately fearful about MLB Scoreboard. I did not want to lose my developer account. Especially since my experience with Google is that they don't communicate with you when you try to appeal these decisions; you're just screwed.

I contacted the NFL legal department. The email from Google had included a list of a few hundred other apps that had been caught for infringement; mine stood out like a sore thumb in that list, since every single other app that was caught was a lame, obviously infringing app called something like "New England Patriots Ringtone" or "Super Bowl Wallpaper" that had no features, and was probably just malware.

In my email to the NFL, I asked if there were any changes they'd like to see to the app in order to get it reinstated. I told them I was willing to work together to find common ground; I was willing to remove features, to include a banner ad advertising their official app, et cetera.

I pointed out that there was a small group of devoted users who loved the app, and that that implied there was room in the market for a lighter-weight option than the official NFL '12 app, and that was the role I was trying to fill.

The NFL never responded. They had gotten what they wanted -- NFL Scoreboard was dead.

It happens again

I should have known I was flying too close to the sun, when I saw that Google Play was running its algorithmic app-finding helper thing, with the line "Users who downloaded MLB At Bat also downloaded MLB Scoreboard!" MLB would have to notice that, right?

And a little bit after the end of the baseball season, I received an email from MLB.

Please read the attached letter from the legal department at MLB Advanced Media regarding your MLB Scoreboard mobile application.

Uh oh.

We write with respect to your unauthorized commercial use of MLB Material in your MLB Scoreboard application available in the Google Play market

Unauthorized commercial use, did you say? I asked for an explanation, since I very explicitly had not attempted to gain any commercial benefit from this free app with no ads in it.

Even though the application is being offered for free and doesn’t contain advertisements, it’s still being made available in commerce, which in this case constitutes commercial use in violation of the copyright notice and license described in our letter. Since you do not have a license from MLBAM, you are not authorized to use our Materials in this way and we must again insist that you immediately remove this application from the Google Play Market, sign the letter in the spaces provided and return it to us

So, I wondered, just being in Google Play constitutes "in commerce"? I asked that.

Yes it does, as it would if the application were available in the iTunes App Store, the Windows Marketplace, or any other similar application distribution platform.

Since it was apparent that if any third party might potentially derive some indirect benefit from the existence of the app meant that the app was "commercial", I asked for an example of a non-commercial use. You know, since it didn't seem like there was much possibility of avoiding that.

If, moving forward, you have ideas for ways in which to use the Materials that you believe fall under the category of “individual, non-commercial, non-bulk use,” you can send them along to me and I will be able to confirm whether or not they do.

Seriously. That was MLB's response to "can you give me an example of a non-commercial use". I don't consider that an example.

In the end, MLB forced me to remove MLB Scoreboard from the market. By my own hand. I didn't want to press them too hard, because they could just go to Google alleging copyright infringement, and I didn't want that to happen. Frankly, I'm glad MLB dealt directly with me. It was just painful, that I was forced to murder my own app.

End of an era?

So, my two best and most popular apps have been killed.

Some people surely feel this is a good thing. Before they were removed, I was talking to someone from Google and when he heard what the apps were, he responded "so, they're just pure copyright infringement." Those poor little professional sports leagues need to be protected from thieving bullies like me, standing on their backs to hog all the money and glory for myself. Removing any and all competition from the market is how American capitalism is supposed to work, right?

Others have pointed out that their vociferousness doesn't really make any sense. There's no conceivable way in which someone would download my apps so they could stop being a fan of the MLB or the NFL. I'm providing free labor to multiple-billion-dollar organizations in service of furthering the fanaticism of the people whose fanaticism makes them all their money.*

* I wish I were slightly less of a baseball fan, or that I lived in the same market as my favorite baseball team. Then, I'd be more likely to cancel my MLB.tv package in response to MLB's hit job on me. It'd actually cost them money, for no benefit to them. But my money will continue to flow to them, because I lack the courage of my convictions. And I love baseball.

Over the past two years, I've received requests from hundreds of people -- friends and strangers alike -- to make a version of my Scoreboard apps for their favorite sports. NBA, NHL, soccer. That will not be happening now. The poor little sports leagues can keep all the control they so crave, and they can continue to utterly fail to serve their customers.

Unless one of the leagues comes to me and wants me to help them make an app that doesn't suck, and that people actually want, I'm done with sports apps.