mether's Fedora Blog

Random thoughts, usually on Fedora

On upstreaming code

Fedora or more specifically Fedora Engineering Steering Committee which itself is a elected body of Fedora contributors voted 8-1 on a proposal to remove separate kernel module packages and patch the main kernel package in some exceptional cases with the general goal of not deviating from the upstream kernel. The kernel modules interface is fluid and considered internal to the kernel and generally unsuitable for maintaining modules separately despite some clever hacks like DKMS.

In the recent 2007 Linux kernel summit, Linus has claimed that anytime a distribution ships a out of tree driver, the process has failed and he is in favor of making it easy on developers to merge their patches upstream. From my understanding, this was also a key motivation for Fedora’s decision.

Apparently people aren’t generally very happy with that decision however. Quite a few people repeatedly spread misinformation that the Fedora kernels are heavily patched while in reality other distribution kernels like Ubuntu or whatever tend to go along that path much more than Fedora does. Go check for yourself.

I do understand that not everyone might view what is called as a upstream bias as a benefit for end users. So I write up a general set of guidelines explaining why upstreaming code is a good idea and make a case for valid exceptions too.

I am still puzzled on why people consider Slackware not patching the kernel to be a good thing and Fedora not doing it to be a bad thing however. Either you want Red Hat to patch their kernels heavily or you don’t. Which side is your bread buttered today?

Written by mether

September 24, 2007 at 6:40 pm

Posted in Uncategorized

24 Responses

Subscribe to comments with RSS.

  1. What SOME People Want

    I am still puzzled on why people consider Slackware not patching the kernel to be a good thing and Fedora not doing it to be a bad thing however. Either you want Red Hat to patch their kernels heavily or you don’t. Which side is your bread buttered today?

    I think what some people want is for them to be able to plug a kernel in which is not directly from Fedora into a Fedora system (the same way they can do with any other application package). The problem with Fedora patching their kernels so much was that people who occasionally roll out kernels usually do it via the old extract-make *config-make bzImage-make *install route find themselves with a system that doesn’t work as well *because* of the need for those heavy patches. Event the “make rpm” option that is/was present in the kernel source tree doesn’t work as well. Personally, I’ve just learned to suck it in and create custom-kernels based off of Fedora kernel SRPMS, but that method isn’t as popular and documentation for going down that route is sketchy and hard to find.

    The current decision by FESCo gives the impression that out-of-kernel-tree kernel modules will no longer be included. End-users obviously won’t be happy about this because there’s going to be a time when a kernel module they’d want might be unavailable for an indefinite time. (That’s the theory. Whether it will happen in practice is something the end-users would rather not have to find out the hard way.)

    Some kernel drivers and module developers would prefer not to have their code included in the source tree. Particularly since there’s a very partisan behavior by the kernel devs, some people would just rather avoid all the hassle and do things independently. It was alright before because users would be able to get at their work through kmods, but now they are being excluded and must wait for third party repos to pick up their stuff. End-users wouldn’t be happy with that either.

    The reason that people aren’t happy with Fedora is that they’re noticing that most decisions made by Fedora are very Fedora maintainer-oriented. From outside, it seems like all the decisions were made to make lives easier for the Fedora maintainers and developers. While people can make the argument that if the maintainer is happy, the end-user will be happy, why would you expect them to wait for that trickle-down effect when there are other distros that are end-user-centric?

    Personally, I’m glad that Fedora is aggressively pushing to have patches committed upstream and rely on less patching, but I’m not too sure about removing out-of-tree kernel mods from the main repo. It’s understandable, however, if there’s not enough resource to make sure that these kernel modules are in-sync and always available whenever new kernel packages come out. But don’t expect people to be happy with the cop out.

    richip

    September 24, 2007 at 5:25 pm

    • Re: What SOME People Want


      I think what some people want is for them to be able to plug a kernel in which is not directly from Fedora into a Fedora system (the same way they can do with any other application package). The problem with Fedora patching their kernels so much”

      Hold on right there. You are repeating a myth. Read the links in my posts for getting a dose of reality.

      “The current decision by FESCo gives the impression that out-of-kernel-tree kernel modules will no longer be include”

      Hold again. I already explained how this is not true. So you might lose this impression and concentrate on the facts.

      “But don’t expect people to be happy with the cop out.”

      I see. So expecting patches to be pushed upstream instead of being distribution specific is described as a cop out these days. I see. I got that. So again, do you want the kernels to be heavily patched or do you not want that. You can’t have it both ways.

      mether

      September 24, 2007 at 5:33 pm

      • Once you have the kernel unpacked, if you have any patches you wish to apply, now is a good time. Let’s say you don’t wish to run kernel 2.

        yvettemanik

        July 16, 2008 at 6:03 am

      • Again, if there are not enough others to matter, then you _should_ make the workflow be around your own personal sandbox.

        zdenajuteq

        July 16, 2008 at 7:49 pm

    • A: There is a bug in the kernel which causes the media check to say some CDs are bad when they are not, on some systems.

      esterylevu

      July 17, 2008 at 8:24 am

  2. Hold on right there. You are repeating a myth. Read the links in my posts for getting a dose of reality.

    First of all, I’ve no idea in what context you’re using the phrase “getting a dose of reality”, but to me, it sounds like you’re insulting my intelligence. If I understand you correctly, then that’s not the best way to get to a rational conversation.

    Second, it’s not exactly a myth since I myself encountered that exact same problem when I tried to use a custom kernel with some patches I made and borrowed from a site (profiling patches that weren’t available in the Fedora kernel) and ended up with an partially working desktop because of some missing udev patches that Fedora had.

    That was about two years ago. I’m actually glad to see that the current kernel src.rpm has the option to compile a vanilla kernel. I’m assuming that all the non-upstream patches are non-essential now, but that wasn’t the case then.

    Hold again. I already explained how this is not true. So you might lose this impression and concentrate on the facts.

    I don’t doubt that you’ve explained this when I ask: Where do you say that existing out-of-tree kernel modules that currently exist in the Fedora repositories will continue to be available? I’ve read your Why Upstream post and it doesn’t mention anything about retaining availability of out-of-kernel modules in the Fedora repo. If anything, there’s only a link to a post containing the ff.:

    “Existing kernel module packages must be removed (or merged into the main kernel package) before Fedora 9.”

    Fortunately, though, there only seems to be two packages (kmod-em8300 and kmod-sysprof) that are out-of-tree and in the Fedora repo, so that argument might actually be moot.

    I see. So expecting patches to be pushed upstream instead of being distribution specific is described as a cop out these days. I see. I got that. So again, do you want the kernels to be heavily patched or do you not want that. You can’t have it both ways.

    The cop out I as referring to wasn’t in pushing patches upstream. I already mentioned that I think that’s a good idea. What I referred to as a cop out was excluding packages for simply being too hard to maintain.

    Again, it might very well be a moot point since there the decision will apparently only affect 2 packages. The only people who would really care (judging from what I’m seeing in Fedora 7) are people who would use kmod-em8300 and kmod-sysprof. If those actually get pushed to upstream before Fedora 9 (when this policy will take effect), then there’s really no problem.

    The thing is, _I_ understand. In fact, I’m actually pro-Fedora (that’s why I’m on the devel) list. I’m trying to explain why people are feeling that way because you said you were puzzled (were you talking figuratively?). If you knew the reasons that they are feeling that way and find them to be in error, please be more specific.

    What I don’t understand, though, is why resentment from readers wasn’t cut off at the pass with additional statements like: 1) listing which kernel module packages are in danger of being removed if not pushed upstream, or 2) which third-party repos have made the reassurances that out-of-tree kernel modules will be maintained and dropped-ones picked up?

    richip

    September 24, 2007 at 6:18 pm

  3. Reality


    First of all, I’ve no idea in what context you’re using the phrase “getting a dose of reality”, “

    See Dave Jones blog post I linked to. Did you read that? Are you still claiming that the Fedora kernels are heavily patched? If so you need to do a comparative study in between other mainstream distribution kernels and Fedora and then come back. It was always considered a bug if the vanilla kernel didn’t work on Fedora boxes and that has nothing to do with the amount of patches but WHAT those patches are even if they are a small amount.

    “Where do you say that existing out-of-tree kernel modules that currently exist in the Fedora repositories will continue to be available”

    What I said that out of tree kernel modules as separate may not be going away at all. Instead they may get into the main kernel package instead of being separate packages which is already explained in the announcement if you read it completely.

    “What I referred to as a cop out was excluding packages for simply being too hard to maintain.”

    If you think that pushing patches upstream is a good idea, the additional maintenance is simply not required at all and is a unnecessary waste of time which can be spend in fixing bugs or adding features to a common code base. This is a matter of principles and priorities and one that yields long term benefits over short term convenience the details of which are explained in my guidelines. You are free to debate that if you want to but do it in fedora-devel list which is the appropriate place where I asked for comments on it. My blog isn’t appropriate for that.

    “I’m trying to explain why people are feeling that way because you said you were puzzled”

    You explanation doesn’t actually answer what I asked for which is why is there a difference in viewpoints when slackware does it vs when Fedora does it nor does it actually answer the other question which is do you want the kernels to be more patches or less patched?

    In the start, you claim that deviations in Fedora kernel which you incorrectly claim as heavy caused problems. Now I don’t doubt that deviations might cause problems but there is no heavy patching required for the problems to occur. Even a small amount of patches might cause problems. It all depends on what those patches are.

    Having said that, you also argue that not taking over additional maintenance of out of tree kernel modules kernel modules is cop out which appears to argue for patching the kernel more. I see a self contradiction there which is what I get puzzled about. So people involved need to make up their minds on whether they align themselves with either one of those viewpoints and make their case clear instead of trying to argue it both ways simultaneously.

    “1) listing which kernel module packages are in danger of being removed if not pushed upstream”

    Impossible to predict whether these patches will be upstreamed before Fedora 9 or not.

    “2) which third-party repos have made the reassurances that out-of-tree kernel modules will be maintained and dropped-ones picked up?”

    They will pick whatever the maintainers are ready to put an effort on which may or may not happen.

    mether

    September 24, 2007 at 6:52 pm

    • Re: Reality

      For Pete’s sake, Rahul … What is wrong with you?? Why do you insist on shooting the messenger?! I’ve already explicitly mentioned that what I’m stating are other people’s perception of things surrounding Fedora and it’s policies with regards to kernel packaging, and you keep addressing things as if _I_ was the one saying it.

      I’ve already said that I did have a problem about 2 years ago where the desktop wouldn’t work with a custom kernel I compiled from a vanilla source tree because it was missing udev patches. I then congratulated Fedora for its current kernel src.rpm and how it can be built to make a vanilla kernel that made ME assume Fedora will run with vanilla kernels. It’s not biggie to me, but I’m glad that it’s gotten better. You don’t have to explain it to me. You don’t even have to explain it to them unless you want to. I’m just bringing up the reason that people had problems with Fedora and their patched kernels.

      You keep telling me to wade through stuff you’ve posted about in other pages and even linked to, but you can’t even read a simple comment I wrote on THIS page thoroughly and understand it. I said that pushing patches upstream instead of maintaining them separately is good. I also said that unavailability of kernel modules at a given point in time is bad. I NEVER said that that was what was going to happen. IF it happened, then it would be bad.

      English may not be my first language, but it certainly isn’t bad that my remarks could be taken so out-of-context.

      Let me simplify this:

      1) I am NOT for keeping a separate, patched kernel, but agree that it must be done if necessary (and you’ve given some good reasons to).

      2) I am all for pushing things upstream, though of course that has the caveat that it may be hard to negotiate with upstream, particularly with the kernel devs since they’re swamped with things and there are times when you might have to wage an uphill battle just to get a patch in.

      3) IF there are out-of-tree kernel modules that aren’t in the kernel tree by the time Fedora 9 comes around and NO ONE decides to release packages for those, then that would be a bad thing for end-users. Is that going to happen? I don’t know! I’m in no position to say.

      4) When I asked to list packages that are “in danger of being removed”, I asked for a list of modules that could POSSIBLY be removed. Not that are. Let’s see if I can simplify: IF things remain as they are and kernel upstream doesn’t accept any new patches, which kernel module packages TODAY would be removed if the policy were to come into effect this very moment.

      5) I guess I was a little vague with the third-party repos thing. Let me rephrase, then: It would go a long way to reassuring people (not necessarily me) to just mention that some third-party repos are going to pick up the missing kernel modules, but only IF some of them have indeed made clear their intention to do so.

      Seriously, I don’t know why you keep using “you” so much to refer to arguments that I’m not now nor have I ever made.

      richip

      September 24, 2007 at 9:07 pm

      • Re: Reality

        “I’ve already said that I did have a problem about 2 years ago where the desktop wouldn’t work with a custom kernel”

        You stated that this was because Fedora kernel was heavily patched which was not the case and I don’t think you understood or accepted that point.
        “For Pete’s sake, Rahul … What is wrong with you?? Why do you insist on shooting the messenger?! I’ve already explicitly mentioned that what I’m stating are other people’s perception of things surrounding Fedora and it’s policies with regards to kernel packaging, and you keep addressing things as if _I_ was the one saying it.”

        It doesn’t matter who says it. My arguments are going to remain the same. If you are the messenger, feel free to convey things both ways.

        “You keep telling me to wade through stuff you’ve posted about in other pages and even linked to, but you can’t even read a simple comment I wrote on THIS page thoroughly and understand it.

        First of all, I only ask you to understand the links referenced before responding because your comments show that you have not read them or understood them. If you don’t want to read those, then it would be better to not respond since a incomplete understanding is not very useful in this conversation.

        “I asked for a list of modules that could POSSIBLY be removed”

        Everything that is not upstream can be possibly removed. The announcement was clear on that.

        “It would go a long way to reassuring people (not necessarily me) to just mention that some third-party repos are going to pick up the missing kernel modules, but only IF some of them have indeed made clear their intention to do so.”

        Read the RPMFusion announcement. It has already been made clear.

        “Seriously, I don’t know why you keep using “you” so much to refer to arguments that I’m not now nor have I ever made.”

        You here refers to anyone who makes an argument along the lines and not necessarily you literally. It is a pretty common way of phrasing things.

        Anonymous

        September 24, 2007 at 9:22 pm

      • Re: Reality

        Well, thanks for clarifying things.

        Good luck on future attempts at educating people.

        richip

        September 24, 2007 at 9:38 pm

  4. open-vm-tools

    The new policy has essentially doomed the open-vm-tools effort from the start, and Fedora is likely to become the only distribution that will not support VMware out of the box, that won’t install cleanly into a VM.

    The “push upstream” argument is just too simplistic. Not all third-party kernel code authors have the time or option to do the (considerable) work required to make that happen. And not all third-party kernel code are “device drivers”. In the case of open-vm-tools, some of the modules are very ad-hoc stuff and closely tied to the way VMware virtualization technology works. Will upstream even accept them ? Add to that an apparent feud between the vmware and LKML folks, competition with Xen and disagreement over ABIs, APIs ,etc.., and you have a complicated situation on your hands.

    None of this, however, means that the open-vm-tools code is “bad”, or bad quality, or unstable, or generally not good enough for Fedora…

    denisleroy

    September 24, 2007 at 7:01 pm

    • Re: open-vm-tools

      “he “push upstream” argument is just too simplistic. Not all third-party kernel code authors have the time or option to do the (considerable) work required to make that happen”

      So instead of them pushing their patches upstream, every distribution should patch their kernels? How is that going to scale?

      “Will upstream even accept them ?”

      They will if you make it generally useful. That is the better choice compared to everybody doing busy work.

      “None of this, however, means that the open-vm-tools code is “bad”, or bad quality, or unstable, or generally not good enough for Fedora…”

      With the current policy, that’s exactly what it means.

      mether

      September 24, 2007 at 7:07 pm

      • Re: open-vm-tools

        > So instead of them pushing their patches upstream, every distribution should > patch their kernels? How is that going to scale?

        open-vm-tools is not a kernel patch. It’s a user-space project, helped by a set of fairly specialized kernel modules, and as such I don’t think patching the kernel is the solution here. Just a separate kmod or dkms package is really the ideal solution. And support in the installer to detect the VM hardware. That is what the other distros are aiming for. Does that seem unreasonable ?

        Also note that I don’t recommend accepting kmod or dkms packages into Fedora lightly. I would have liked a policy where it’s a case-by-case discussion, based on technical merits rather than bureaucratic red tape (no offense). By technical merits, I mean things such as code quality, maintenance activity, user community.

        > “Will upstream even accept them ?”
        >
        > They will if you make it generally useful. That is the better choice
        > compared to everybody doing busy work.

        If “you make it” ? And who will do that ? Who will sign your paychecks while you do that work ? What if it can’t be made general enough ? It’s a vague technical concept that may or may not apply. It does apply, to a certain extent, to the open-vm-tools modules, but it’s a long process that provides little benefits to VMware (that may sound cruelly pragmatic, but that’s a reality). I’m told that process is under way too, but not sufficiently far along to sway the fedora kernel maintainers (who have my respect).

        > “None of this, however, means that the open-vm-tools code is “bad”,
        > or bad quality, or unstable, or generally not good enough for Fedora…”
        >
        > With the current policy, that’s exactly what it means.

        You just proved my point about the policy being just plain shortsighted and arbitrary, because technically it can proven that the open-vm-tools code is perfectly fine.

        Anonymous

        September 24, 2007 at 8:13 pm

      • Re: open-vm-tools

        (sorry I did not mean to reply anonymously)

        denisleroy

        September 24, 2007 at 8:14 pm

      • Re: open-vm-tools

        “open-vm-tools is not a kernel patch. It’s a user-space project, helped by a set of fairly specialized kernel modules, and as such I don’t think patching the kernel is the solution here. Just a separate kmod or dkms package is really the ideal solution”

        Nope. Kmod or DKMS packages are NEVER the ideal solution. Even folks who created DKMS would agree with me on that. The kernel doesn’t have any stable interface for maintaining modules as separate packages. Realistically you need code merged upstream and that is going to be the working solution.

        “If “you make it” ? And who will do that ? Who will sign your paychecks while you do that work ?”

        If VMWare wants to do it, they will fund that work and make it usable generically as they have already done so with the VMI interface.

        “You just proved my point about the policy being just plain shortsighted and arbitrary, because technically it can proven that the open-vm-tools code is perfectly fine.”

        It really need a code review in LKML to determine whether it is technically fine or not. We merely don’t know yet. There is no proof being offered here about the nature of the code base and without knowing that, putting in kernel modules into the repository would be indeed just shortsighted. There is both sides of this argument and you just cannot dismiss the other side as arbitrary.

        Anonymous

        September 24, 2007 at 8:56 pm

    • Re: open-vm-tools

      > The “push upstream” argument is just too simplistic. Not all third-party
      > kernel code authors have the time or option to do the (considerable) work
      > required to make that happen.

      So the same work should be done many time separately by distribution maintainers who don’t know all the kernel code areas as well as the people on LKML? How sane is that?

      Either this considerable work is worth is and it should be done upstream or it’s not and Fedora has no business inflicting half-baked code it didn’t even create on its users.

      Third-party kernel code authors just have to understand that if they want to offload integration to someone, this someone must work through LKML, not distributions. Distributions have no business vetting code in the stead of LKML

      Anonymous

      September 25, 2007 at 7:56 am

      • Re: open-vm-tools

        > Third-party kernel code authors just have to understand that if they want to > offload integration to someone, this someone must work through LKML, not
        > distributions. Distributions have no business vetting code in the stead of
        > LKML

        Absolutely. But who said they wish to offload the work ? VMWare is maintaining the open-vm-tools code, they never said they wanted that to change. And carrying open-vm-tools does not carry any work burden onto the distribution.

        denisleroy

        September 25, 2007 at 8:03 am

  5. The deeper question….

    What can we do as a distribution to help make it easier to merge code upstream? Is there a place to leverage Fedora’s structures to ease the path of integration for code? Do we start building a Fedora community kernel module review team under the Fedora umbrella that interacts with the kernel upstream? Where do we as a distribution project fit into Linus’s ideas on how to make it easier?

    carrots and sticks…
    If our new policy is the stick, what process do we use as the corresponding carrot? I don’t have a ready answer, but I’d really like one.

    -jef

    jspaleta

    September 24, 2007 at 7:04 pm

  6. Misrepresentations for everyone!

    Dave jone’s post is meaningless without reference to either the size of -mm or Ubuntu or any other kernel that isn’t kernel.org. I haven’t checked recently, but Ubuntu does maintain several out-of-tree drivers, and likely does patch their kernel heavily. But I have no idea how fedora sits in relation.

    But the bigger problem is this: I read the kmod discussion you failed to link to, and came to a different conclusion. They basically said, no more kmods packages, but if whoever’s in charge of the rawhide kernel wants to pull it into his source tree, the module will be shipped. So for Fedora it seems like upstream is Dave Jones’ tree. To say that Fedora will no longer ship drivers unless they’re in mainline seems a bit of a misrepresentation.

    jldugger

    September 25, 2007 at 7:00 pm

    • Re: Misrepresentations for everyone!

      “Dave jone’s post is meaningless without reference to either the size of -mm or Ubuntu or any other kernel that isn’t kernel.org.”

      Dave Jones post is useful to show that Fedora kernels don’t carry many patches and overall the patches have declined over a period of time. If you want to do a comparative look with other distributions, do that yourself like I mentioned and post the results. I agree that would be a interesting metric.

      “To say that Fedora will no longer ship drivers unless they’re in mainline seems a bit of a misrepresentation.”

      Umm, I explicitly mentioned that separate kernel module *packages* were being removed and the kernel can be patched directly in some cases. Read it again.

      mether

      September 25, 2007 at 9:17 pm

      • Re: Misrepresentations for everyone!


        Umm, I explicitly mentioned that separate kernel module *packages* were being removed and the kernel can be patched directly in some cases. Read it again.

        Sorry. My Debian based background assumed that source packages were different than binary packages. For example, a kernel package can split out into a base kernel, and modules for wacom-tools, etc. I take it you don’t mean that optional kernel driver binary packages are going away, but that source packages outside the main kernel won’t be supported.

        jldugger

        September 25, 2007 at 9:23 pm

      • Re: Misrepresentations for everyone!

        We currently include some out of tree GPL’ed kernel module (kmod) packages which are going to be phased out by Fedora 9. These kmod packages already had a requirement that it requires FESCO’s approval before being accepted.

        After Fedora 9, either the kernel modules would have to merged upstream or the current Fedora kernel maintainers (ie) Dave Jones and Chuck should be convinced to merge those modules (which they are very unlikely to do if the code isn’t headed upstream fairly quickly).

        Fedora has never included proprietary modules in it’s repository just as an additional clarification.

        mether

        September 25, 2007 at 9:31 pm

      • Re: Misrepresentations for everyone!

        RPM _does_ support building separate subpackages from the same source RPM (SRPM), but this isn’t done for kernel modules.

        kevin_kofler

        September 25, 2007 at 9:40 pm

      • Re: Misrepresentations for everyone!

        Interesting. I should pay more attention to Ubuntu’s kernel some day. It’s a nightmare to rebuild, let alone replace with a stock kernel. Especially with those pesky closed drivers.

        jldugger

        September 25, 2007 at 9:44 pm


Comments are closed.

%d bloggers like this: