Rails has_secure_password, a reminder 20 Feb 2013
I'm using the new has_secure_password feature in Rails (I believe it was introduced in Rails 3.1) for user authentication. It uses bcrypt under the hood, which I'd used directly in the past. But has_secure_password seems to be the cool new way to do this, and it does seem pretty easy.
has_secure_password gives you a bunch of things for free: password hashing and salting, authenticating against the hashed password, and password confirmation validation.
But all the tutorials and explanations I've seen omit a few key details in the name of simplicity. Namely, they don't show you what you need to do for the password confirmation to work, and they don't show you how to validate the password correctly when the User object is updated (ie, they only show you what to do when it's created).
The RailsCast on the topic makes it seem sooooo easy, which is I guess the point, but if you just do what they do you'll end up confused and with something that doesn't work.
Their User model looks like this:
class User < ActiveRecord::Base
attr_accessible :email, :password, :password_confirmation
has_secure_password
validates_presence_of :password, :on => :create
end
Note that it only validates the password when the User is created; that's because they want you to be able to change the username later, without having to re-enter the password. But if you add a length restriction to your password (which you should), then later on when the user changes it, the minimum length won't be validated.
Additionally, and I think this is key, the RailsCast shows you the User model, and how to authenticate, but never actually shows how to create a user and have the password_confirmation validated.
user = User.new(:email => "my@email.com")
user.password = "password"
user.valid? => true
I would've thought that would fail validation, because there's no password_confirmation set. But this is the key, the thing that apparently isn't mentioned anywhere: the password confirmation is only validated if you attempt to set it.
user = User.new(:email => "my@email.com")
user.password = "password"
user.password_confirmation = "otherpass"
user.valid? => false
user.password_confirmation = "password"
user.valid? => true
Okay, so you have to actually set the password_confirmation or else it won't do anything. But how do you make your validation work when you're updating the user?
Here's a very basic User model that will do that:
class User < ActiveRecord::Base
attr_accessible :id, :username, :password, :password_confirmation
has_secure_password
validates :username,
:presence => true,
:length => { :minimum => 3 },
:uniqueness => true
validates :password,
:length => { :minimum => 8, :if => :validate_password? },
:confirmation => { :if => :validate_password? }
private
def validate_password?
password.present? || password_confirmation.present?
end
end
Here, it will validate the password and its confirmation if either the password or the confirmation are set, or if it's a new User with no password set. This will allow you to update the username without setting the password, but will actually validate the password if you attempt to set it. (And still has the restriction that it only attempts to validate the confirmation if you set password_confirmation.)
AppEngine HTTP Headers 29 Jan 2013
So, I'm working on a new Android app* which requires user authentication and storing data to a backend service. Since pretty much everyone with an Android phone has a Google account, and Google is doing a good job of implementing OAuth 2 for me, I figured I'd just use their authentication. Tim Bray alerted me to the cool new way of authenticating which doesn't require the user to enter any passwords, and may not even require the user to even click anything.
* It's a game, with asynchronous multiplayer functionality.
After being confused for a while about how complicated it was supposed to be -- I thought you have to pass in the token from GoogleAuthUtil.getToken into something else to complete the authentication and pass a cookie back to AppEngine to maintain an HTTP session, when in fact you can just pass the token directly to your service to verify identity -- I was ready to start moving forward.
I decided that I'd pass the token as a custom HTTP header, so I don't have to either put it in the query parameters (which is dangerous even on HTTPS) or pollute the JSON I'm passing to the service. I figured X-GoogleAuthToken would be a good option for a header name. And this is where our story really begins.
On the local dev service, X-GoogleAuthToken worked fine, and I was able to authenticate using the token. But when I deployed to production, the value for the header always came through as null.
So I wrote an extremely simple servlet to test out the headers.
public void doGet(HttpServletRequest req, HttpServletResponse resp)
throws IOException {
resp.setContentType("text/plain");
Enumeration headers = req.getHeaderNames();
while (headers.hasMoreElements()) {
String name = (String)headers.nextElement();
String value = req.getHeader(name);
resp.getWriter().println(String.format("%s: %s", name, value));
}
}
I passed in the following headers*:
GET /lettergarden HTTP/1.1
X-Googleauthtoken: blah blah
Googleauthtoken: test
Fakeheader: here
X-Gauthtoken: howdy
X-Google-Auth-Token: 123
X-G-Auth-Token: boosh
* It was supposed to be "X-GoogleAuthToken" and "fakeheader" and "X-Gauthtoken", but I'm using an app called "HTTP Client" that apparently, and I just learned this, title cases the headers before sending them. Seems wrong?
And on the local server, here's what I got:
Host: localhost:8888
User-Agent: HTTP%20Client/0.9.1 CFNetwork/454.12.4 Darwin/10.8.0 (i386) (MacBookPro5%2C4)
X-Googleauthtoken: blah blah
Googleauthtoken: test
Fakeheader: here
X-Gauthtoken: howdy
X-Google-Auth-Token: 123
X-G-Auth-Token: boosh
Okay, that's everything I passed in, plus the Host and User-Agent. Nothing to suggest there are dragons approaching.
Here's what I get after deploying to production:
Host: some-project-this-isn't-really-the-name.appspot.com
Fakeheader: here
Googleauthtoken: test
X-Gauthtoken: howdy
X-G-Auth-Token: boosh
User-Agent: HTTP%20Client/0.9.1 CFNetwork/454.12.4 Darwin/10.8.0 (i386) (MacBookPro5%2C4)
X-AppEngine-Country: US
X-AppEngine-Region: il
X-AppEngine-City: chicago
X-AppEngine-CityLatLong: 41.878114,-87.629798
The AppEngine documentation says they'll add X-AppEngine-Country, X-AppEngine-Region, X-AppEngine-City, and X-AppEngine-CityLatLong as a service to you so you can do some location stuff easily. They also say they will remove some headers from requests: Accept-Encoding, Connection, Keep-Alive, Proxy-Authorization, TE, Trailer, Transfer-Encoding.
You may have noticed, however, that it also stripped X-Googleauthtoken and X-Google-Auth-Token from the request. The documentation does not indicate anywhere that AppEngine will strip any HTTP header that starts with "X-Google". And it does not strip headers that start with "Google" (which obviously you shouldn't do).
I'd rather Google didn't bogart the entire X-Google* namespace if they're not even using any part of it. But, frankly, it wouldn't be a big deal at all if they'd just document that they're doing it and make the dev environment mirror the actual production environment.
I spent a few hours combing Google search results, AppEngine documentation, and StackOverflow for why this was happening. Maybe, having written this, the next poor sap to be bitten by this won't have to waste so much time.
Dependo Pull Request 29 Jan 2013
Last night, I got my first pull request on GitHub from someone I don't know. Pretty cool.
As you may know, Dependo is a very simple dependency injection library for Ruby, which just overrides the #method_missing method and lets you inject methods onto an object just by mixing in the Dependo module.
Well, apparently someone wanted to be able to see if methods have been injected onto the object. I'd been doing that by calling Dependo::Registry#has_key?, but this new way is better.
module Mixin
def respond_to?(key, include_private=false)
if Dependo::Registry.has_key?(key)
true
else
super(key, include_private)
end
end
end
So I quickly accepted the pull request. I probably wouldn't have thought of this without some guy finding my library, using it, and contributing the idea and making my project better.
In case you needed another reason to open source your stuff, this is a great example of why it's a good thing.
Thanks, Tomas!
Whither Scoreboard? 27 Dec 2012
My most popular apps have been MLB Scoreboard (121,664 users) and NFL Scoreboard (61,994 users). In both cases, the apps scratched an itch, started small, and grew from there.
The birth of MLB Scoreboard
I started MLB Scoreboard because MLB At Bat for Android was awful two years ago.* I needed an app that would tell me the scores of baseball games, with as few clicks as possible, with as many games on the screen as possible, and without taking several extra seconds to load for no reason.
* Now, it's merely inadequate. Big step up?
So, MLB Scoreboard was very, very simple when it first started. At the beginning, it was just a scoreboard: one screen, which showed all the games for the day in the order they were played, with the linescore like you'd see at the stadium. There was also the ability to go forward and backward by a day at a time, to see the previous days' results or tomorrow's schedule. That was it.
The app grew slowly. It took weeks to get to 100 users, and I was really excited. I kept adding new features: your favorite team would always appear at the top of the screen; I showed the current outs and runners on base; I let you click into a game to view the play-by-play and the hitting/pitching boxscore; I added highlight videos for each game; soon, there was a widget that showed today's game for your favorite team, so you didn't even have to wait for the app to open to get the score.
By the middle of the summer in 2011, it had somehow shot past 1000 users. That was the number I'd barely dared to hope it'd get to. When people had asked when I was going to put ads in it, I'd been saying that I'd bother with that when it got to 1000; but by the time it happened, I was too concerned by the fact that MLB's license for their data says it's for non-commercial uses only. If I tried to make any money, it wouldn't be non-commercial any more. So I left the ads out.
And it kept getting more popular. Near playoff time, people seemed to discover it. By the end of October, 7000 people were using it multiple times per day. On the modern internet, where ten million is the new one million, a few thousand is nothing. But for me, it was exciting.
And so, with the baseball season winding down, it was time for football season to start. And it just so happens that the NFL's Android app is much, much worse than MLB's. Like, unforgiveably awful. I needed a better app to see the scores, if I was going to follow football.
The birth of NFL Scoreboard
NFL Scoreboard started the same way as MLB Scoreboard, and perhaps even more barebones: just a list of the games and their scores.
And it similarly grew slowly, but not quite as slowly. Maybe the kind-of-popularity of MLB Scoreboard helped? It quickly shot past a few hundred users. By the end of the (much shorter) season, it had 1000 users.
Despite being monumentally ugly, it had a few advantages over NFL's official app. The official app took 30 seconds to start, and after a couple of clicks you actually got to the scores, and you could see two games on a screen. My app started in under one second, and went straight to the list of scores, and you could see seven games on a screen.
A new baseball season
In March 2012, I was contacted by AT&T, who wanted to promote MLB Scoreboard as a featured part of their new Android app store. Since baseball season was about to start, they wanted an entire baseball section. So they had me fill out some informational forms about the app, with logos and a description, etc. Pretty much all the same stuff you'd get from the actual Google Play (nee Android Market). I knew it wasn't going to be a big deal, but I told people that I was letting myself become foolishly excited for all the downloads I was going to get from AT&T users.
I should have known it wouldn't go that way. AT&T, being AT&T, never got their act together with the store. There was no baseball section. There was a sports section, but MLB Scoreboard was never in it. The only apps in it were those ultra-lame cookie-cutter ringtone/wallpaper apps. And every image on their app store site was broken. It was almost as poorly executed as AT&T's actual phone service.
But the 2012 baseball season rolled on anyway, and even without AT&T's help, MLB Scoreboard had somehow gotten popular.
At the start of spring training, there were 14000 installs (5000 active users). The rate of installation increased throughout March, and at the start of the season there were 26000 installs (14000 active users). Throughout the entire month, I avidly checked my download stats every morning, shocked that another few hundred people had downloaded it the previous day.
Because there was suddenly so much interest, I felt I needed to improve the app. It was pretty ugly. So, I spent a few weeks rebuilding the entire UI, making it more modern and fitting better into the rest of Android (ie, ActionBar), and adding animations to make things look cooler and smoother. At first, people very vocally hated what I was doing, and I got a bunch of angry emails.
And then, on Opening Day, the actual baseball-loving public started caring about baseball. And I got 1000 downloads that day. And the next. And more the next. For the first couple of weeks in April, I was getting 3000-4000 downloads every day. It slowed back down in May, to the point where I was only getting about 400-500 per day* for the rest of the season.
* It's amazing how quickly I got used to that. A year earlier, I would have been amazing by having a total of 400 users.
The growth rate stopped at the end of the playoffs, once again, peaking around 120K (63K active). Turns out people don't really care about the score of today's baseball game when there are no games.
A new football season
Right before the start of the 2012 football season, I received an email from a shady Russian company, offering to buy NFL Scoreboard from me for $1000. It was ugly, and only had 1000 users. I briefly considered it, but decided I'd rather make the app better, than sell out for what is basically a pittance.
I modernized the app's interface, again with ActionBar. I added features, like full game stats, full play by play, and all the passing/rushing/receiving stats for the players. People seemed to love these new features. Every Sunday morning before the games, I got several thousand new downloads.
After week two, NFL Scoreboard had shot up to 60000 users, and my growth rate was increasing.
I was loving it.
First tangle with a legal department
On September 14, 2012, I received an email from Google. They informed me that NFL Scoreboard had been suspended, and removed from Google Play, for "alleged trademark infringement".
They informed me that repeated violations of this nature would result in my entire developer account being suspended; this made me immediately fearful about MLB Scoreboard. I did not want to lose my developer account. Especially since my experience with Google is that they don't communicate with you when you try to appeal these decisions; you're just screwed.
I contacted the NFL legal department. The email from Google had included a list of a few hundred other apps that had been caught for infringement; mine stood out like a sore thumb in that list, since every single other app that was caught was a lame, obviously infringing app called something like "New England Patriots Ringtone" or "Super Bowl Wallpaper" that had no features, and was probably just malware.
In my email to the NFL, I asked if there were any changes they'd like to see to the app in order to get it reinstated. I told them I was willing to work together to find common ground; I was willing to remove features, to include a banner ad advertising their official app, et cetera.
I pointed out that there was a small group of devoted users who loved the app, and that that implied there was room in the market for a lighter-weight option than the official NFL '12 app, and that was the role I was trying to fill.
The NFL never responded. They had gotten what they wanted -- NFL Scoreboard was dead.
It happens again
I should have known I was flying too close to the sun, when I saw that Google Play was running its algorithmic app-finding helper thing, with the line "Users who downloaded MLB At Bat also downloaded MLB Scoreboard!" MLB would have to notice that, right?
And a little bit after the end of the baseball season, I received an email from MLB.
Please read the attached letter from the legal department at MLB Advanced Media regarding your MLB Scoreboard mobile application.
Uh oh.
We write with respect to your unauthorized commercial use of MLB Material in your MLB Scoreboard application available in the Google Play market
Unauthorized commercial use, did you say? I asked for an explanation, since I very explicitly had not attempted to gain any commercial benefit from this free app with no ads in it.
Even though the application is being offered for free and doesn’t contain advertisements, it’s still being made available in commerce, which in this case constitutes commercial use in violation of the copyright notice and license described in our letter. Since you do not have a license from MLBAM, you are not authorized to use our Materials in this way and we must again insist that you immediately remove this application from the Google Play Market, sign the letter in the spaces provided and return it to us
So, I wondered, just being in Google Play constitutes "in commerce"? I asked that.
Yes it does, as it would if the application were available in the iTunes App Store, the Windows Marketplace, or any other similar application distribution platform.
Since it was apparent that if any third party might potentially derive some indirect benefit from the existence of the app meant that the app was "commercial", I asked for an example of a non-commercial use. You know, since it didn't seem like there was much possibility of avoiding that.
If, moving forward, you have ideas for ways in which to use the Materials that you believe fall under the category of “individual, non-commercial, non-bulk use,” you can send them along to me and I will be able to confirm whether or not they do.
Seriously. That was MLB's response to "can you give me an example of a non-commercial use". I don't consider that an example.
In the end, MLB forced me to remove MLB Scoreboard from the market. By my own hand. I didn't want to press them too hard, because they could just go to Google alleging copyright infringement, and I didn't want that to happen. Frankly, I'm glad MLB dealt directly with me. It was just painful, that I was forced to murder my own app.
End of an era?
So, my two best and most popular apps have been killed.
Some people surely feel this is a good thing. Before they were removed, I was talking to someone from Google and when he heard what the apps were, he responded "so, they're just pure copyright infringement." Those poor little professional sports leagues need to be protected from thieving bullies like me, standing on their backs to hog all the money and glory for myself. Removing any and all competition from the market is how American capitalism is supposed to work, right?
Others have pointed out that their vociferousness doesn't really make any sense. There's no conceivable way in which someone would download my apps so they could stop being a fan of the MLB or the NFL. I'm providing free labor to multiple-billion-dollar organizations in service of furthering the fanaticism of the people whose fanaticism makes them all their money.*
* I wish I were slightly less of a baseball fan, or that I lived in the same market as my favorite baseball team. Then, I'd be more likely to cancel my MLB.tv package in response to MLB's hit job on me. It'd actually cost them money, for no benefit to them. But my money will continue to flow to them, because I lack the courage of my convictions. And I love baseball.
Over the past two years, I've received requests from hundreds of people -- friends and strangers alike -- to make a version of my Scoreboard apps for their favorite sports. NBA, NHL, soccer. That will not be happening now. The poor little sports leagues can keep all the control they so crave, and they can continue to utterly fail to serve their customers.
Unless one of the leagues comes to me and wants me to help them make an app that doesn't suck, and that people actually want, I'm done with sports apps.
Running two pieces of middleware with Dependo 31 Jul 2012
Middleware is an awesome feature of Rack -- I can add functionality that wraps around an HTTP call without having to change the actual server.
We use it with some success in our r509-ca-http project. r509-ca-http is a Certificate Authority based on r509 that serves over an HTTP REST API. Its functionality is intentionally as simple as possible -- the r509-ca-http project is intentionally not responsible for storing information about certificates' validity or a record of which certificates have been issued. Its only responsibility is issuing and revoking certificates.
But if we're going to actually run a CA, we need it to store validity information, among other things. Enter middleware.
We created a new project, r509-middleware-validity, responsible for storing validity information (issuance and revocation) about every certificate into a Redis database. Originally, we structured it such that it relied on the server it was wrapping around to have a #log method on it:
module R509
module Middleware
class Validity
def initialize(app)
@app = app
end
def call(env)
status, headers, response = @app.call(env)
# this will just intercept the calls to /1/certificate/issue and ignore anything else
if not (env["PATH_INFO"] =~ /^\/1\/certificate\/issue\/?$/).nil? and status == 200
@app.log.info "I intercepted /1/certificate/issue"
end
[status, headers, response]
end
end
end
end
In your config.ru, you activate the middleware like so:
use R509::Middleware::Validity
server = R509::CertificateAuthority::Http::Server
run server
And on each call to the server, the middleware will be executed first; you need to send the call along to the server with that @app.call(env) line, and return the results after you're done. But because we're using that @app.log.info method call, we're relying on our Sinatra server having a #log method; since it uses the Dependo::Mixin, and we added a Logger to the Dependo::Registry in our config.ru, that method will be available.
Which is pretty cool, and was working -- until we needed to add another piece of middleware. In addition to storing validity information in a database, we also need to store every issued certificate on disk. This is another thing we don't want to add to the HTTP service, so middleware is the perfect place for it. So we created r509-middleware-certwriter and got it saving every certificate that we issued. It was working great in testing ... and then we tried running the server with both middlewares active at once.
use R509::Middleware::Certwriter
use R509::Middleware::Validity
server = R509::CertificateAuthority::Http::Server
run server
And as soon as we tried to issue a certificate, we got this:
I, [2012-07-31T14:42:15.260855 #9007] INFO -- : Writing serial: 1233561620675808731887525784788727751733782628003, Issuer: /C=US/ST=Illinois/L=Chicago/O=Ruby CA Project/CN=Test CA
NoMethodError: undefined method `log' for #<R509::Middleware::Validity:0x00000100c0dab8>
/Users/sschulte/code/r509-middleware-certwriter/lib/r509/middleware/certwriter.rb:35:in `rescue in call'
/Users/sschulte/code/r509-middleware-certwriter/lib/r509/middleware/certwriter.rb:28:in `call'
/Users/sschulte/.rvm/gems/ruby-1.9.3-p0/gems/rack-1.4.1/lib/rack/lint.rb:48:in `_call'
/Users/sschulte/.rvm/gems/ruby-1.9.3-p0/gems/rack-1.4.1/lib/rack/lint.rb:36:in `call'
/Users/sschulte/.rvm/gems/ruby-1.9.3-p0/gems/rack-1.4.1/lib/rack/showexceptions.rb:24:in `call'
/Users/sschulte/.rvm/gems/ruby-1.9.3-p0/gems/rack-1.4.1/lib/rack/commonlogger.rb:20:in `call'
/Users/sschulte/.rvm/gems/ruby-1.9.3-p0/gems/rack-1.4.1/lib/rack/chunked.rb:43:in `call'
/Users/sschulte/.rvm/gems/ruby-1.9.3-p0/gems/rack-1.4.1/lib/rack/content_length.rb:14:in `call'
/Users/sschulte/.rvm/gems/ruby-1.9.3-p0/gems/rack-1.4.1/lib/rack/handler/webrick.rb:59:in `service'
Turns out that when you're using two middlewares, the second one wraps around the server, and the first one wraps around the second one -- ie, they don't both somehow wrap directly around the server. So in our case, the #log method exists on the HTTP server so the validity middleware can use it, but it doesn't exist on the validity middleware so the certwriter middleware can't use it and dies.
Here comes dependo to the rescue!
module R509
module Middleware
class Validity
include Dependo::Mixin
def initialize(app)
@app = app
end
def call(env)
status, headers, response = @app.call(env)
log.info "Now I can just use the log method directly"
[status, headers, response]
end
end
end
end
Since the server is using Dependo::Mixin, it has a #log method. If we add Dependo::Mixin to both our middlewares, they each have a #log method and can wrap around each other in either direction.
Maybe this is obvious, maybe pointless ... but I hadn't thought of it until I ran into it. This is how I got past it.