Gaming the Issue Credit System: Is the Problem Real or Perceived?
This story follows a few Drupal developers’ tweets and has added insights since.
Contrary to popular misconception, Ostriches do not stick their heads in the sand. But gamers do. Every time there happens to be somebody who would try to game the system, and like a hot iron, the system would take the beatings for a while. But it always turns out better.
SEO optimization hacks are everywhere, so there is no genuine ranking possible unless the system finds a way around it. And that is why search engines, not just Google, change their ranking algorithms from time to time.
The time has come for Drupal dot org to find a way to keep the gamers at bay. Wait. But why bring up this subject now?
Twitter is abuzz with Drupalers venting it out in the open. For example, check out this tweet from Matt Glaman and the responses it gathered.
This discussion did not suddenly sprout out from a tweet. The 16 August podcast from Talking Drupal’s episode #361 focused on the Drupal credit system. John Picozzi and Nic Laflin were hosting the show with Cathy Theys as guest host. Matthew Tift, the lead engineer at Lullabot, was on the show.
“Ultimately, the Drupal credit system is expanding what the git logs offer: to give people the opportunity to credit not just a developer but also the developer’s employer or the client for whom the developer is working, and going beyond that, to give credit to people that aren’t doing any sort of coding work at all and to give them credit for their work in the community such as attending a meeting or hosting a podcast about Drupal: those are all the kinds of things you could use and those are just a few of the kinds of things that you could use the Drupal credit system for,”
Matthew Tift starts the talking point.
It is pertinent to note that gaming and gamifying are two things. Nic Laflin argues that
“gamifying credit and recognition in the best sense to incentivize contributions and even to credit non-code contributions is the key. When you have something you can now measure, that gives people more incentive to do the thing being measured, but it also gives people insight into what they are actually doing.”
But Tift has a counter view to share. He had beautifully written about it in Lullabot’s blog last September: How We Compare: Leaderboards and Related Comparison Metrics in the Drupal Community. An interesting quote I found in that article attributed to Vietnamese monk and zen master Thich Nhat Hanh is also relevant to this discussion: “Where there is perception, there is deception!”
“Having a system like this invites gamification, it encourages people to gamify things, it changes how people behave. Just like any leaderboard in any community, it can be beneficial in that people are motivated to work because they want to be on that whereas people feel like I will never have the oppurtunity to be at the top, so that my contributions don’t matter. So there’s all kinds of implications for having a system like that,”
Matthew Tift adds. It also talks about people who are negatively affected by the credit system. The podcast simultaneously webcasted on youtube is a must-watch.
We might go back to the tweets. The accusation Matt is raising is serious. Without naming a specific entity, he has squarely raised suspicion over the activities followed by an organization that would reportedly hack the issue credit system so that the employees and the organization move higher up the credit ladder.
According to Matt, the organization has users “verify” Drupal 10 patches by uploading screenshots of the command line interface of applying patch and module list before and after the patch is applied. As three files are attached, they will automatically accrue issue credit. But the patches aren’t ready or tested, says Matt.
Being modest, he says he does not doubt they’re gaming the system. He had ‘heard of organizations holding issue credits as a merit system to keep employment.’ Employees might take this easy path to fulfill the demand and save jobs, it is implied.
And he has sane advice for them:
“Don’t throw the juniors at this without being paired with a senior and educating them. Otherwise, it’s blindly wasted work.”
But everybody is not as nice. Twitter user René Bakx|raw adds up on where Matt let off.
“mostly from countries where labor is relatively cheap. It’s a quick and somewhat proven method for a “company” to rise in the top contributor rankings by just throwing a lot of hands that run the tasks, hoping it will bring more paid work their way.”
Whether or not to get offended or deal with the problem head-on is a choice to make. But the sentiment is deafening.
“That’s not true for all companies, though. I work for a top-10ish Drupal contributor and our HQ is in a country like you describe. We maintain several modules, have contributed to Drupal core. Our devs are asked to give 4 hours a week to OSS projects, of which Drupal is one,”
another Twitter handle by the screen name Pixelnaut moves his horse. Checkmate.
Mike Herchel, a board member of the Drupal Association, suggests implementing a penalty system for organizations that do this (similar to how Google will penalize people who get caught doing black-hat SEO).
“We’d need to set up a review and appeals system though. The system and rules would need to be transparent, too,”
“This has been gamed, fixed, discussed, and re-gamed since the rankings were implemented.”
He is determined to bring it up at the board for internal discussion.
“It's not an easy problem to properly solve. I'm willing to bet that if we create, and publicize some official rules on what is "gaming" and what the consequences can be, it'd do a lot to prevent it. Right now, that's missing.”
Sam Mortenson revisited the coding standards issue from 2016-17 that turned super annoying. Many stories are unraveled. A Drupal developer with PreviousNext who goes by the handle khromium adds:
It’s fine if organizations want to create automation that posts these patches, but organizations should have almost zero credit if others made the tooling. Think rector/ca. These patches should be posted by special “bot”-tagged accounts. Projects should be able to opt-out of individual bots (i.e., ban a user from posting in the project issue queue) or entirely opt-out of all bots. The credit abuse has got to stop.
Responding to this series of tweets, Rachel Lawson rows the discussion boat in a positive direction. She says that nobody automatically gets issue credit. They may be selected by default, but it is always the discretion of the maintainer to choose (whether to grand it). Though Matt ayes to the ‘getting automatically checked, not automatically given’ argument, he is doubtful, given the number of issue credits showing up in the end.
“Maybe the other maintainers are happy to accept and credit them,”
quips Rachel, to which Matt Glaman responds:
‘It doesn’t make the situation any less frustrating.’
As a true conflict counsel who is the senior manager (community strategy) at the United Nations Digital Impact Alliance, Rachel Lawson tries to resolve the issue. Her point:
“there is value in reminding maintainers that they do “hold the keys” to choosing what is credited and giving good examples. That is worth following up.”
In a way, Lawson is right. There is two way to deal with it. You can either penalize the ‘violaters’ or incentivize the ‘good behavior.’ Or do both.
Here lies the problem. We are doomed when we approach this problem from a rigid moral compass and compartmentalize developers and companies as good and evil. Humans are not binary objects. It is not always black and white.
Glaman’s argument that companies incentivize contributions and credits and tie those with the employee’s annual performance appraisal might be true in many cases. Open-source companies are expected to do that. Often the first credit, the first recognition, and the first listing of their name give the energy to provide more. Think of it like an initiation ritual. It is how new volunteers are enticed into contributing to a just world. A positive affirmation of their actions will let them go a long way.
Gamers are part and parcel of any system, immemorial. But not everyone we assume a gamer is a gamer. New contributors typically make mistakes. They should be handheld and taught the philosophy behind the Free Libre Open Source Software movement.
Amanda Luker, a senior frontend engineer at the Four Kitchens, wrote a blog post last January that describes her organization’s work life: How Our Workplace Mentorships Set You on a Path to Professional Satisfaction. The importance of establishing a mentoring relationship between a junior and senior developer is stressed in that article. Matt Glaman, who opened up this can of worms himself, said:
“don’t throw the juniors at this without being paired with a senior and educating them.”
So there lies the solution.
Instead of policing such behavior, give more credits to actual contributions while not entirely throwing out the not-so-critical benefactions. Let the first-time offenders off the hook, but let them know about the expected behavior. If the same people repeat it, we might check the activities.
Jess, the Drupal Core Release Manager at Acquia who goes by the handle xjm opined that
‘this is another place where automation can reduce gaming by removing the incentive to game. Automated coding standards testing on DrupalCI has virtually eliminated this noise from queues that have it. We didn’t have said automated testing in 2016.’
She shares a detailed issue crediting guidelines in her following tweets. She says:
For core we manage this with detailed issue crediting guidelines. We have response templates to take the emotional labor out of trying to teach people what’s helpful and what’s not. Sometimes, if there is a pattern of unhelpful/noisy non-contributions from a particular org, I will reach out to a regular contributor or senior person via contact form or DM. Usually works!
She suggests a solution to the problem at hand:
“I now think we should un-credit non-patch attachments by default. For core it’s just about how many boxes we have to check or uncheck and minimizing that work. Bad attachments are starting to win out, but we can reduce gaming by changing the default. Needs issue?
Comment template 1/2: “Thank you for looking into this issue. Posting screenshots of your codebase or command-line interface does not advance the issue, since the automated testing infrastructure tells us whether the change set still applies correctly. [...]”
Comment template 2/2: “So, I’ve removed the issue credit for that screenshot. In the future, you can get credit for issues by reading the issue to understand its purpose, and posting your review or testing of that purpose. Thank you!”
And then all the comment templates close with: “See the issue credit guidelines for more information.”
Eternal damnation and purging are too moralistic. It might seem to be the ideal workaround. But when you start strict gatekeeping, the organic nature of the evolution of an open source contributor gets hedged. The entry for a new volunteer becomes too complex, as their welcome turns almost always rude. What happens to overt regulations is evident from the examples of both Wikipedia and Stack Overflow. Should Drupal.org be such a place?
I am not a tech guy and have zero direct contributions to Drupal. But I was accustomed to Drupal from D4, and still am I. Over these years, I have helped the companies I worked with to make this platform better by reporting the issues in their respective issue queues.
I could have done that upstream, but I left that to the participating organizations and developers. It was despite my thriving community experience in one of India’s most active language computing communities. This exactly is called an entry barrier.
There is always a first-time contribution, and it should be more open. If we harden the entry barrier, people will desist from reporting bugs and raising feature requests upstream. They will stay away from packaging a new module for others to use.
The problem senior Drupalers put forward is real. The quality of contributions from the beginners are not that great. But what is more incredible than helping them mold (mould) into a better contributor?
In developing economies with a vast population and a steady supply of developers, survival is the most critical aspect. So even if the companies put up guidelines, some bad apples will always search for a shortcut. Most times, they do it unaware of the operational decency. It is something the upstream developers must learn to cope with.
Drupal is a very welcoming community. The passion and helping mentality we witness here is hard to find elsewhere.
The Arch Linux Community I am familiar with assists noobies better than here. Maybe, we have something to learn from them, especially about how you would build a wiki, a manual, to the degree that one could confidently say, without offending someone to RTFM. Read the effing manual!
We can improve our wiki; for that, new volunteers who squander their time and effort for an issue-credit would be the suitable candidates. We can point them to study modules and improve the wiki. Companies can channel their energy into perfecting the help pages.
All of the key contributors to Drupal were once noobies. They shouldn’t forget their baby steps. The organizations should take a leadership role in finding the right mentor for each new intake. Maybe even better—wishful thinking on my part—participating organizations can help each other by mentoring someone from another company and helping navigate organizational cultures within different organizations.
The second Drupal 10 global porting day is around the corner. It is a virtual event held online so that anybody across the globe can participate. The 24-hour event starts on 26 August 2022 at 03:00 UTC and ends on 27 August 2022 at 03:00 UTC. The sprint is led by SystemSeed’s head of engineering Evgeniy Maslovskiy, Full stack tech lead Kate a.k.a kalabro, and head of growth Tamsin Fox-Davies.
Cut some slack for the slipping of the past and urge those junior developers to help around a bit. Equip them gain good values, and Drupal, new and sustained contributors. Show them a little love and make this community more inclusive. Ask them to read the Contributor Guide and prepare themselves.
The time has come for Drupal dot org to find a way to keep the gamers at bay. But not at the expense of new contributors.
Note: The vision of this web portal is to help promote news and stories around the Drupal community and promote and celebrate the people and organizations in the community. We strive to create and distribute our content based on these content policy. If you see any omission/variation on this please let us know in the comments below and we will try to address the issue as best we can.