Category Archives: Software Development

ottoneu, Software Development, Statistics

Lineup records, innings caps, and scaling

29 Comments

I haven’t rolled out any major new changes to the ottoneu platform yet this year, because I am working on a doozy. If you go back to my post about the 2014 plan, you’ll see the first bullet point is a new stats backend. The original goal was to get this done pre-season, but this problem ended up being more complex than I expected. In this post, I am going to walk through the infrastructure as it exists today, the proposed changes, and why it ended up taking a lot longer than expected.

Today

Currently, the system is fairly simplistic. Every 5 minutes, a script runs that checks if games have started and thus if players should be ‘locked in’ to their fantasy team’s lineup. This creates a record in a table, and this table is now the script by how stats are compiled – it contains every team’s locked-in lineup for every day in the season. As long as nothing happens to this table, stats can be re-run at any time for any reason.

Every 20 minutes, a script is run against this table to update stats for the day. This script writes a record in a stats table for every single lineup record.

Let me write that again in a bit bigger font:

This script writes a record in a stats table for every single lineup record.

This is both completely unscalable and also required if a game is going to have a hard innings cap. Consider the following scenario: it is the last day of the baseball season, and in my league I am starting Clayton Kershaw and in your league you are starting Clayton Kershaw. However, I have 1497 IP this year, and you have 1490. Kershaw spins a complete game masterpiece, and I get the first 3 innings of it because I run directly into my 1500IP cap, and you get all 9 innings. This necessitates having two records for the two Clayton Kershaws we have created – one who pitched 3 innings and then the team couldn’t take anymore innings, and the other that pitched 9. (Aside: if only it was this easy to clone Clayton Kershaw)

The solution: moving from a hard inning cap to a soft inning cap.

About innings caps

Inning caps simply define how many innings your fantasy team can collectively throw in a season. Similar to game limits for positional players, the idea is to keep all the teams in a league around the same number of innings, so that teams are not incentivized to start every single pitcher they can in order to win counting stats.

There are two ways of implementing an innings cap: hard and soft. Every major site uses soft inning caps, meaning on the day a team goes over the cap, that team will get all the innings pitched on that day but none going forward. For example, if there is a 1500 IP cap and your fantasy team has thrown 1499 innings in the season, you can start 5 starting pitchers and 5 relievers and get all their innings on that day, but once you are over the 1500 IP cap you will no longer get credit for any innings your team throws going forward. A hard cap is more restrictive – as soon as your team hits the 1500 IP cap, that is it, no more innings for you. The hard cap was originally implemented to move fantasy owners away from ‘streaming’ pitchers on the last day of the season, or playing as many pitchers as possible in order to get as many counting stats as possible. Streaming felt like gaming the system, and ottoneu doesn’t care much for gaming the system.

However, it became clear that there are existing incentives in ottoneu for not streaming: keepers and cap penalties. Cutting players in order to pick up potential starters late in the season is expensive due to cap penalties, and towards the end of the season a lot of players on an ottoneu team are either keepable or expensive but productive players that simply should not be cut. If an owner chooses to cut productive and keepable players in order to stream pitchers at the end of the season, they probably have a good reason and the system of the game should accept that.

In leagues without keepers and without cap penalties, streaming remains game-y, which is all the more reason to play ottoneu.

Tomorrow (or, soon…)

The switch to soft caps will happen over the all-star break, because there are significant database operations required to make this switch. However, I can advocate for this change with two numbers:

6,613,323

As of writing this post, this is how many records are in the stats_batting table.

159,935

After making this change, this is how many records are left in the stats_batting table. That’s a lot fewer records.

How does this affect you? Speed. After this change, getting updated stats will become significantly faster, meaning your standings pages will be updated faster. It will also allow ottoneu to grow its userbase considerably, which would not be possible without this change.

What’s next?

Once I get this change ready to deploy, I am going to turn entirely to the lineup page experience. I believe it can be better, and I’d love your thoughts on what a great lineup page for ottoneu looks like. Or, let me know what you think could be a huge improvement to ottoneu – I’m always open to new ideas!

Please let me know your feedback on these changes and if posts like this are interesting!

Meta, ottoneu, Software Development

Auction Drafts, ottoneu, and you

1 Comment

Tonight, two leagues had long delays with their auction drafts. I want to write about what happened, the full context around the circumstances, and how it will be fixed going forward. I think it is very important to be transparent about these issues to show how ottoneu is going to improve to address these concerns.

At 3:07pm ET, I received this tweet:

I am currently in Copenhagen, Denmark on vacation from my day job, and my data plan is wonky to say the least. I received tweets regarding this issue for the next hour:

I also received a number of emails during this time period.

I happened to see this last tweet when opening my Twitter client on my phone. Up to this point in the trip, I received push notifications upon new tweets, but this time I didn’t see anything until I happened to open my phone’s client. As soon as I saw these tweets, I rushed back to the apartment I am staying in and spent about 2 minutes debugging the issue and resolving it.

There are a number of questions about this scenario:

  1. Why was the issue not acknowledged earlier?
  2. How was it fixed so quickly when it was open for so long?
  3. Why are there so many issues with auction drafts on ottoneu?

I will address each of these issues in order. Of course, if you have other concerns, I’m more than happy to address them over email or in the comments.

1. Why was the issue not acknowledged earlier?

I am in Denmark right now, for my first vacation since July of last year. Unlike last July or the previous vacation in November 2012, my laptop was not with me the entire trip and I did not put someone else in charge of any issues in my absence. This is also my first trip during the peak ottoneu months, which are February and March, when all the auction drafts occur. Finally, while I expected Twitter push notifications to my phone, I did not receive any this evening.

Solution:

While ottoneu does not make very much money at all, what little I do make this year will go towards a small laptop that I can keep on my person throughout February and March. I will also be more conscientious of vacations during this time, and much closer to this or some computer during this time, until ottoneu makes enough money to warrant a second employee. There is no excuse for this not being addressed faster during such a sensitive time.

2. How was it fixed so quickly when it was open for so long?

Plenty of ottoneu issues are actually quite simple, and only come up when some code I hastily write is pushed into production. In this case, I did a big rewrite of the auction draft in the offseason to try and improve performance. Part of this was to introduce redis to the ottoneu technology stack. Redis comes highly recommended from the aforementioned day job, and I have some experience with it but I made a couple of fairly simple mistakes. These cropped up quickly when faced with production load, and I was able to sort them out and resolve them quickly.

Solution:

See the above – faster response time will almost always mean a faster resolution. A longer-term outlook has a better test environment and more robust testing, but honestly that is a luxury right now.

3. Why are there so many issues with auction drafts on ottoneu?

While no one has straight-up asked me this question, this is a question I ask myself often. There are basically two competing interests:

1) auctions are hard to schedule and when they are scheduled, everyone wants to run their auction.
2) auctions are computationally difficult to keep real-time, and they are also very sensitive to errors, so there should be 100% confidence in live auction drafts when they are run.

There are two solutions that I am capable of: the first is rewriting large portions of the auction draft code to use more redis and less database. Database bad, cache good. The second is to invest in more servers. I plan on doing the former extensively, as talking to a few colleagues indicates that this will increase capacity considerably. I’ve already done this a bit, and I hope to do this more.

I’ve already increased the number of auction drafts that can run at a single time by 50% over last year. I’m hoping to pop it up to a full 100% and then start exploring more server capacity. So this is a “stay-tuned”, but is also a catch-22, because like I said earlier, ottoneu really doesn’t make much money at all (it was a net-loss the last two years even without any full-time employees). So until ottoneu truly has enough users to afford more server capacity, more efficient code is the best way forward. Like I said, I will continue to work towards this end to make this a reality.

That is the full situation around the issue tonight and the overall auction draft issue. I’m back home in 2 days and will be vigilant on any draft issues through April, when capacity drops considerably and we return to the predictable, boring, wonderful grind of a new baseball season. I hope this has been helpful, and please let me know if you have any further questions or concerns.

ottoneu, Software Development

Going in to 2014

6 Comments

Man, I forgot about this place. No time for apologies, let’s get into it.

It is February 19, 2014. There is a high likelihood of a lot of new users coming to ottoneu in the next month before the season starts, but let’s look at where things are now, shall we?

  1. 1917 active teams.
  2. 25 active prize leagues and 123 active non-prize leagues.
  3. 401,293 player transactions, spending $538,395 in ottoneu cap.

I’m going to throw a parade when we get over $1mil in cap spent. There are a few other important things to note, especially in regards to the now woefully outdated first 30 days post. My users remain incredibly patient and smart and understanding. The number of “shit is broken” emails have decreased to a trickle – ottoneu has become a relatively stable (KNOCK ON WOOD) platform! Even the auction drafts are performing smoothly, to the point where most of the emails about them are for really interesting feature enhancements, not “man this was slow, give me some money back.”

So, that is the basic state of things today. Now, what are the big things on the horizon? Well, there are 3 big initiatives this year, each worthy of its own series of posts:

  1. Rewriting the stats backend – after determining that a soft innings cap (and thus allowing pitcher streaming on the last day of the season) was a double-edged sword for owners and self-balancing, there is a massive opportunity for performance improvements and general efficiency with ottoneu’s stats backend. This project is ongoing and should be wrapping up in the next couple of weeks. It should be completely transparent to users, and it has to get done before the season starts. So, that’s priority #1.
  2. A better lineup page experience – the lineups page is fine, it gets the job done. But during the long baseball season, this is where a lot of time is spent. It also is the basis for the off-season roster organizer. More stats, splits, and a smoother experience swapping players in and out will go a long way towards making ottoneu feel world-class.
  3. A full redesign – while I may have sentimental attachment to the Waste Management colors and my crazy-good (read: not) logo, I think ottoneu can look significantly better. This project is actually really close to being done, but probably won’t be released until the All-Star break, just to tighten everything up and not get in the way of the top two priorities. The new logo looks boss.

That is where things stand going into this season. I plan on keeping a loose development blog here, so feel free to comment here, tweet at me, hit me up on Facebook, or just plain email me (help at ottoneu dot com). I cannot wait for baseball. Good luck this season!

ottoneu, Software Development

The Solution? More Tables

Leave a comment

I don’t know very much about database design.  I tried taking a course once in college but the professor scared me on the first class and I ended up dropping it, which I guess now that I write it out I regret.  What little I do know I have taught myself from both production examples from my previous jobs and just general experience.  That being said, there is one  rule that I’m slowly learning that I feel the need to record.  Whenever there is some sort of complication with storing or retrieving data in a relational database, the solution is always more tables.

Yesterday I was struggling with a problem.  If two teams agree to a trade involving a given set of players, all other trade proposals including those players should no longer be active.  For example, if Albert Pujols is involved in a trade that has been accepted, he shouldn’t be still out there in other trade proposals – we don’t want two or more teams concurrently accepting trades with Albert Pujols!

On its face, this seems like a basic enough problem, but the twist is my existing database schema.  It looks like this:
[cc lang=”mysql”](
ID int(11) NOT NULL auto_increment,
ProposalDate date NOT NULL,
ProposingTeam tinyint(4) NOT NULL,
TargetTeam tinyint(4) NOT NULL,
ProposingTeamPlayers varchar(30) NOT NULL,
TargetTeamPlayers varchar(30) NOT NULL,
ProposingTeamLoan int(11) NOT NULL,
TargetTeamLoan int(11) NOT NULL,
Accepted tinyint(4) NOT NULL default ‘0’,
Rejected tinyint(4) NOT NULL default ‘0’,
PRIMARY KEY (ID)
)[/cc]The problem here is that I’m storing “ProposingTeamPlayers” and “TargetTeamPlayers” as strings. So for each trade, I implode the array of players involved into a comma-delimited list. This is all well and good until you realize you want to search for all trade proposals involving a given player. The process then becomes:

  1. Find all trade proposals involving either team
  2. Get all players involved in said trade proposals
  3. Check if any of the players are involved in the just accepted trade
  4. If yes, then mark that trade proposal as rejected (which, implicitly, it is)

The solution to getting rid of this awful, slow code? Add a new table:[cc lang=”mysql”]TradeProposalID int(11) NOT NULL,
PlayerID int(11) NOT NULL,
TeamID int(11) NOT NULL,
PRIMARY KEY (TradeProposalID,PlayerID)[/cc]Then this query gives us what we want:[cc lang=”php”]$sql = “SELECT ID FROM TradeProposalsIndexTable JOIN TradeProposalsPlayersTable ON ID=TradeProposalID WHERE Accepted=’0′ AND Rejected=’0′ AND PlayerID=$playerID”;[/cc]Mark all those trades as rejected, and all done.

Without adding new tables, I had a mess trying to store multi-player trades in my database, and I had a hacky solution that clearly was not well thought out.  Add a table, remove those ProposingTeamPlayers and TargetTeamPlayers columns, and a multi-line complete mess of a solution turns into 2 elegant lines of SQL.

So we’re left with a simple rule to follow at all times: if you are getting confused by how to get or store data with your relational database, add some more tables.

Software Development

Frameworks

1 Comment

As the other half of this little venture, I have not been pulling my weight on this blog.  But no more!  It is my turn to chime in with a few lines of thought that might only be interesting to me.

My free time the last few days has been spent developing my side of the project, a new fantasy baseball game.  Developing is not the right word – more like refactoring for scale.  Right now the game is written to handle the one league and 12 teams that have been using it for the last 5 years.  However, if I want it to go anywhere, people should be able to sign up, create teams, join leagues, and so on and so forth.  So after multiple false starts, I’ve decided to take the working code I have now, refactor it for legibility and to fix the bugs that I have found, and push it out to the world next spring.

There turned out to be two clear options for me in considering scale for ottoneu.  One option is the one I took – take my manually written, inefficient, and generally goofy PHP and turn it into somewhat legible, slightly less inefficient, still remarkably goofy PHP that allows more than 12 people to play this fantasy game.  The other option would be to start over from scratch and develop a scalable, efficient piece of software against a proven MVC framework.  I tried the second path two or three times, and I’ve learned, basically, that I hate development frameworks.

I decided to work with Zend, a PHP framework with which I had heard good things.  I started from knowing nothing, and only towards the very end of my experience did I start feeling any bit of understanding towards the framework.  Basically, learning Zend was like learning a new language, but with this whole level of indirection on top of it – it looks like something I knew, but it wasn’t anything that I knew, and hiding everywhere in the framework was some call or some method that was the “Right” way to solve a problem.  Some people have this problem with base PHP – lots of silly built-in methods/functions, lots of ways of solving a problem, but the difference is that when using a framework, there really was the “Right” way and the dangerous way, whereas base PHP has multiple right ways, depending on what you are trying to do.

The second attempt at going down the Zend path had me hire a developer who had serious production experience with the framework and, I hoped, would be able to guide me and help me develop the new version of ottoneu.  However, the problems here were fairly straightforward – I still wasn’t learning anything about Zend, I barely had an idea of what was going on, and the developer was a freelancer, not invested or interested in being invested in the project.  So fine, this isn’t a beef against the framework, but… Goddamnit, I’m a reasonably smart person and a decent developer.  If your framework is too obtuse for me to pick up and build a reasonably complex project with, it isn’t doing its job.  All the “Hello Worlds” and simplistic authentication examples in the world won’t convince me otherwise.

So I decided, with a little help from this post from Joel Spolsky, to run with what I had.  Yes, it was ugly code.  Yes, I’m embarrassed to share it with anyone with anything more than self-taught PHP experience.  Yes, it had bugs galore, didn’t interact consistently with MySQL, and was way slower than it should be.  But look – it works.  The damn code works, and has worked for 5 years.  Best of all, I understand what I am working with – I wrote all this code, so I know what it is trying to do, how to fix various issues with it, etc.  And for just that reason, I know it will be faster for me to get to launch day, Spring 2011 than it would have been using Zend.

My conclusion is this – frameworks are good for someone starting out from scratch, but only if 1) the project is going to use a ton of the built-in help the framework gets you and 2) won’t need a lot of stuff that is outside the scope of the framework.  There are ways for me to eventually get my code to something much cleaner and more legible and even faster, but starting over from scratch was not one of those ways.

Next up, I’ll discuss a more interesting, more fantasy baseball-oriented topic: how do game and inning limits work?