14:47 <rocket> busy morning .. heh
14:48 <rocket> anyway I am trying to work out how genkernel should work in the future .. and trying to come up with ideas that make
          it less difficult to manage in catalyst
14:48 <rocket> right now its a bit of a pain ..
14:49 <johnm> Can you paste to me how it runs at the moment, via catalyst?
14:49 <rocket> heh its a huge script
14:49 <rocket> I can post it to my dev space
14:49 <johnm> sure
14:51 <rocket> d.g.o/~rocket/kmerge.sh
14:52 <rocket> I can try and explain what its doing and why
14:53  * johnm is just reading through iut now.. I recall seeing this before actually
14:53 <rocket> we are trying to cache as much info as possible between runs of catalyst since kernel building takes a long time
14:53 <rocket> so when kerncache is enabled I actually make a directory outside the chroot for the kernel sources to go into
14:53 <rocket> and then bind mount that directory to /usr/src inside the chroot
14:53 <johnm> how many does it build normally?
14:54 <rocket> some arches build up to 6 on a livecd
14:54 <rocket> and they are slow arches so it could take a couple hours to build them all
14:56 <rocket> anyway I am trying to make a newer version of genkernel and catalyst that is simpler to maintain and understand
14:56 <rocket> but the issues I see so far include .. the external ebuild modules need the kernel sources
14:56 <rocket> so I need ways of keeping a kernel source tree with a specific binary kernel build
14:57 <johnm> Gotcha
14:57 <johnm> ok
14:57 <johnm> before we go any further..
14:57 <rocket> and then I need to kind of keep track which binary ebuilds go with a specific kernel binary
14:57 <johnm> were not planning on supporting 2.4 anymore with this new release?
14:58 <johnm> or do we sill have to?
14:58 <johnm> still*
14:58 <johnm> if we dont.. there is a very convenient way of doing this
14:58 <rocket> not with genkernel 4 .. the plan is to drop it out of genkernel 4
14:58 <johnm> ok.
14:58 <johnm> In that case then..
14:58 <johnm> we can utilise KBUIL
14:58 <rocket> I was going to ask about that :)
14:58 <johnm> check out the linux-mod stuff and you will see it supports KBUILD_OUTPUT and KERNEL_DIR var's
14:58 <johnm> basically...
14:59 <johnm> something like this will do just fine:
14:59 <johnm> mykernel=image1
14:59 <johnm> kernelkv=2.6.14
14:59 <johnm> if [[ -d ${KERNEL_ROOT}/${kernelkv} ]]; then
14:59 <rocket> and before we go too far I do have some issues with kernel versions but we can talk about that in a bit
15:00 <johnm> KERNEL_DIR=${KERNEL_ROOT}/${kernelkv}
15:00 <johnm> KBUILD_OUTPUT=/var/cache/kernel/${kernelkv}-${mykernel}/
15:01 <rocket> ok following that so far .. :)
15:01 <johnm> then you can also simply emerge <modules>, passing KERNEL_DIR and KBUILD_OUTPUT
15:01 <johnm> localversions support is both srctree and objtree
15:01 <johnm> so you can alter localversions if you wish by dropping files into $KBUILD_OUTPUT
15:01 <rocket> ok so the infrastructure is already in place to support that for the modules ebuilds ..
15:02 <johnm> and of course... the caching is still done by make/kbuild
15:02 <johnm> yes
15:02 <johnm> with most mdules
15:02 <johnm> any which dont, need fixing
15:02 <rocket> I assume file bug requests on them ..
15:02 <johnm> yep
15:02 <johnm> there is a tracker open
15:02 <johnm> I can't think of anything off hand which isn't updated at the moment
15:03 <rocket> ok about localversion .. genkernel has some nasty hacks to determine what it should be
15:03 <rocket> because it can change after a user configs their kernel
15:04 <johnm> correct.
15:04 <johnm> but that does depend
15:04 <johnm> there are 3 methods for LOCALVERSION
15:04 <johnm> a CONFIG option
15:04 <johnm> srctree localver*
15:04 <johnm> objtree localver*
15:04 <johnm> the src and objtree should be OK
15:05 <johnm> (all this can run as a user just fine.. and probably should)
15:05 <johnm> the CONFIG is of course when they configure the stuff
15:05 <johnm> all three options are glued together
15:05 <rocket> yup .. I have read the makefile on the matter
15:05 <johnm> catalyst can simply drop the files into objtree
15:06 <rocket> yeah .. thats what I was starting to think about .. maybe it wont matter as long as I am using objtree
15:06 <johnm> it wont :)
15:06 <johnm> remember this...
15:06 <johnm> srctree is your src tree
15:06 <rocket> I honestly dont remember exactly where we are using the exact name of things
15:06 <johnm> obj tree is where everything you change needs to be
15:06 <johnm> which immediately means every different image you choose, can have totally different .configs etc in objtree
15:07 <johnm> all you alter each time is KBUILD_OUTPUT
15:07 <rocket> oh I remember why we needed the localversion
15:07 <rocket> its because of the modules
15:07 <johnm> the modules work that out anyways as mentioned
15:07 <johnm> assuming the above system is used
15:07 <rocket> when we did a make modules_install we needed to find where it would put them on the filesystem
15:08 <johnm> or something to the same degree
15:08 <johnm> ah I see
15:08 <johnm> well.. you can see how the kbuild output stuff effects that in the makefile anyways
15:08 <rocket> I think with 2.6 I can just do a MODULE_INSTALL_DIR=/tmpdir/foo
15:09 <rocket> and just follow the tree and figure it out that way if I had to
15:09 <johnm> there are a lot of ways to do it :)
15:09 <rocket> ok does KBUILD_OUTPUT modify the original source tree at all?
15:10 <johnm> Makefile.modinst
15:10 <johnm> not at all.
15:10 <johnm> if you're doing all this as root
15:11 <johnm> something you might find useful is to build a basic .cnfig in srctree first
15:11 <rocket> yeah it will be as root .. at least from catalyst
15:11 <johnm> that way when you rebuild every image, you have a mostly compiled tree already
15:11 <johnm> I recall if the objects exist, and dont need rebuilding, it just uses them
15:11 <rocket> oh .. thats cool .. thats a good idea ..
15:11 <johnm> worth a try and checking, I forget some of this
15:12 <johnm> I'm fairly sure it works :)
15:12 <rocket> but basically I could make an ebuild of the kernel source tree if I wanted ...
15:12 <rocket> actually thats already done never mind
15:12 <johnm> yeah
15:12 <johnm> lol
15:13 <rocket> heh my brain has been working to hard .. 2.4 support has really messed me up :p
15:13 <johnm> THe idea here is once oyu have all the objtree built, and it's sane, you can tar it up or somethign and use an ebuild
          to install correctly
15:13 <rocket> yeah .. thats what I am thinking now ..
15:13 <johnm> something worth noting...
15:13 <johnm> this should all be modular!
15:13 <johnm> but not just in stages
15:14 <johnm> by kernel version also
15:14 <rocket> I can make an obj_tree_ebuild .. and use that as a caching mechanism ..
15:14 <rocket> brb
15:14 <johnm> 2.6_gkcongiure.sh or some such
15:14 <johnm> yes you could if you wanted to :)
15:14 <johnm> or simply build the first then cp the first to all the others in the queue.
15:15 <johnm> no extra fuss
15:20 <rocket> yeah I could do something along those lines if they use the same source tree ..
15:20 <johnm> yeah
15:21 <rocket> now what are you saying about stages .. and kernel version and modular I didnt quite follow what you were saying by
          stages ..
15:21 <johnm> ah
15:21 <johnm> bascially...
15:21 <johnm> the parts of genkernel which does this work, needs to be modularised
15:21 <johnm> as I know you know anyways
15:22 <johnm> similar to portage phases I guess really
15:22 <rocket> yeah .. thats the plan anyway :p
15:22 <johnm> an unpack, a configure, a compile, a packaging stage
15:22 <johnm> right..
15:22 <johnm> well those also need to be version dependant
15:22 <johnm> ie: 2.6
15:22 <johnm> 2.7
15:22 <johnm> 2.8
15:22 <johnm> that way you can share a common API between kernel support
15:22 <johnm> if might even be worth looking to push it even mre in depth
15:23 <rocket> hrmm .. yea thats a good idea ..
15:23 <johnm> 2.6.14 (strip last part off if file doesn't exist)
15:23 <johnm> that way you can properly support kernel releases
15:23 <johnm> KBUILD during 2.5->2.6.6 ish was still changing
15:23 <rocket> yea .. thats a good idea
15:23 <johnm> and things were always going a different direction
15:23 <johnm> 2.8 will be no different
15:23 <rocket> basically do some initial tests based on source tree
15:24 <rocket> then run the appropriate scripts
15:24 <johnm> the framework is much mre important than some of the actual work :)
15:24 <johnm> right
15:24 <johnm> something like this would do:
15:24 <rocket> <root>/2.6/config.sh
15:24 <johnm> what is KV? ok KV = 2.6.14
15:26 <rocket> ok I think I am following that concept
15:26 <johnm> gimme a sec
15:27 <rocket> ok ... and then I have some kernel source based ebuild questions
15:49 <johnm> big things happening here so i might be quite some time.
15:49 -|- johnm [n=johnm@gentoo/developer/johnm]
15:49 -|-  ircname  : John Mylchreest
15:49 -|-  channels : #asterisk @#gentoo-kernel #gentoo-commits @#gentoo-dev
15:49 -|-  server   : irc.freenode.net [http://freenode.net/]
15:49 -|-           : is identified to services
15:49 -|- End of WHOIS
15:53 <rocket> ok thats fine ..
15:54 <rocket> no rush ..
18:53 <rocket> ok :)
18:53 <rocket> where were we?
18:53 <johnm> 15:30 <johnm> right
18:53 <johnm> 15:31 <johnm> something like this would do:
18:53 <johnm> 15:31 <rocket> <root>/2.6/config.sh
18:53 <johnm> 15:31 <johnm> what is KV? ok KV = 2.6.14
18:53 <johnm> 15:33 <rocket> ok I think I am following that concept
18:53 -|- johnm [n=johnm@gentoo/developer/johnm]
18:53 -|-  ircname  : John Mylchreest
18:53 -|-  channels : #asterisk @#gentoo-kernel #gentoo-commits @#gentoo-dev
18:53 -|-  server   : irc.freenode.net [http://freenode.net/]
18:53 -|-           : is identified to services
18:53 -|- End of WHOIS
18:53 <johnm> 15:33 <johnm> gimme a sec
18:53 <johnm> 15:34 <rocket> ok ... and then I have some kernel source based ebuild questions
18:53 <johnm> so yeah...
18:53 <johnm> something like
18:54 <johnm> <root>/2.6/14/.config
18:54 <johnm> <root>/2.6/14/gk_comfigure.sh
18:54 <johnm> <root>/2.6/gk_configure.sh
18:54 <johnm> <root>/2.6/gk_compile.sh
18:54 <johnm> etc.
18:55 <rocket> yeah .. something like that
18:55 <johnm> will be as simple as KV=${KV//-*}; KV=${KV/./\/}
18:55 <johnm> well
18:55 <johnm> almost
18:56 <rocket> yeah ..
18:57 <johnm> # cat /etc/portage/package.provided
18:57 <johnm> net-www/apache-2.0.54-r31
18:57 <johnm> [ebuild  N    ] net-www/apache-2.0.54-r31  -apache2 -debug -doc +ldap -mpm-leader -mpm-peruser -mpm-prefork
          -mpm-threadpool -mpm-worker -no-suexec (-selinux) +ssl -static-modules -threads 5,488 kB
18:57 <johnm> whats going on there!?
18:57 <johnm> [ebuild  N    ] net-analyzer/cacti-0.8.6g  +snmp -vhosts 1,051 kB
18:57 <rocket> huh?
18:57 <johnm> [ebuild  N    ] net-analyzer/cacti-cactid-0.8.6e  128 kB
18:58 <johnm> just getting annoyed at portage/www
18:58 <rocket> so then I guess I have questions on how to handle the ebuilds for external modules or ebuilds that require kernel
          source trees
18:59 <johnm> sure
18:59 <johnm> ask away
18:59 <rocket> I am trying to cache pkgs
18:59 <rocket> take the ivtv ebuild as an example
18:59 <rocket> say I have 2 kernels that I need the modules in .. is there are way to have 2 ivtv pkgs?
18:59 <rocket> does that make sense?
19:00 <johnm> KERNEL_DIR="/mt/kernel/sources/1/" emerge ivtv
19:00 <johnm> KERNEL_DIR="/mt/kernel/sources/2/" emerge ivtv
19:00 <johnm> you might need to set KBUILD_OUTPUT as well of course
19:01 <rocket> but how do I cache that?
19:01 <johnm> export KBUILD_OUTPUT to the environment which portage runs in, and it should be fine
19:01 <johnm> you dont/cant.
19:01 <rocket> wont that make just ivtv.tar.bz2
19:01 <johnm> of cource you will get some ccache benefits
19:01 <johnm> the actual production of the package is up to the script right?
19:01 <johnm> well hold on
19:01 <johnm> this is the question
19:01 <rocket> ok so I guess I will have to use my trick then
19:02 <johnm> yor buiding a module for a specific set of sources
19:02 <johnm> you can either build it, and roll it into the final kernel tar
19:02 <rocket> PACKAGE_DIR="/mt/kernel/packages/1/" emerge -kb ivtv
19:02 <johnm> or you can tar every single one individually
19:02 <rocket> PACKAGE_DIR="/mt/kernel/packages/2/" KERNEL_DIR="/mt/kernel/sources/2/" emerge -kb ivtv
19:02 <johnm> why not just put it into the kernel tar?
19:02 <johnm> afterall, it likely wont run on the others anyways
19:03 <rocket> well I guess ideally .. and I dont know if its possible .. it would be nice to have a ivtv-core.tar and an
          ivtv-modules.tar for each kernel module
19:03 <johnm> were planning on doing that in the tree
19:04 <johnm> you will see increasingly, kernel modules with userland portions split into two or three ebuilds
19:04 <johnm> driver-tools/userland
19:04 <johnm> driver-modules
19:04 <rocket> that would be good to see
19:04 <johnm> driver-firmware
19:04 <johnm> it is happening :)
19:04 <johnm> lots have gone that way already
19:04 <johnm> but there are more to do
19:04 <rocket> I know you guys are working on stuff like that .. its just a matter of time ..
19:05 <johnm> yeah, well any time you find a module which installs a userland, which can be split out please let us know
19:05 <rocket> will it be possible to link an ebuild to a certain kernel ebuild?
19:05 <johnm> or help :)
19:05 <johnm> you can do that with DEPEND already in theory
19:05 <rocket> eg ivtv-modules-gentoo-sources-2.4.12.tar.bz2 ..
19:05 <johnm> ah I see
19:06 <johnm> well, not really. portage doesn't give you the ability
19:06 <rocket> ie the one it compiled against ..
19:06 <johnm> although, thats niot to say since you wrap it all you can't do it
19:06 <johnm> you will know the full KV and you know the output file from the merge
19:07 <rocket> how am I getting the kv .. is emerge exporting that as it runs?
19:09 <johnm> it will do yes, actually.
19:10 <johnm> check out linux-info get_version
19:10 <rocket> ok
19:11 <rocket> I think I have what I need then to get started .. lots of good ideas .. I appreciate them :)
19:11 <rocket> oh one other thing to run by you .. if you have a sec
19:12 <johnm> yeah any time.
19:12 <johnm> I'd be interested in helping too if you're up for it
19:13 <johnm> the actual core framework (the main module loader and stuff I guess is the best way to phrase it) is probably the
          most important thing
19:13 <rocket> ok right now we compile everything for the initramfs manually .. ie we dont rely on portage at all
19:13 <johnm> ok.
19:14 <rocket> yeah that stuff I might be able to lift from the eselect tool I havent fully digested it yet .. or I might use it as
          a reference .. but yeah that will be a big improvement
19:14 <johnm> Well, I would look at eselect, and ujse it as a base
19:14 <johnm> basically.. eselect wont do quite what you need
19:14 <rocket> anyway I am thinking of making a module that compiles manually like the old genkernel .. so that it will still run
          on non-gentoo machines
19:14 <johnm> ie: it loads individual scripts
19:15 <rocket> yea but there are good ideas in its code
19:15 <johnm> what you want is something which sources everything you need, and executes the designated function at the right time
          etc.
19:15 <johnm> yeah exactly
19:15 <johnm> it does similar to what you want
19:15 <johnm> and yeah..
19:15 <johnm> I agree with you
19:15 <johnm> that manual part can actually wrap the other stuff
19:16 <johnm> or as you said.. simply replace the compile script from any given kernel version
19:16 <rocket> well we are thinking of adding a new target for catalyst that builds the initramfs portion
19:16 <johnm> i wouldn't bother
19:16 <johnm> I would put it all into the genkernel stuff, right at the latest point
19:17 <johnm> that way you can include userland tools and such for third party modules and the likes
19:17 <rocket> well we are thinking of the people who want funky initramfs for things like netboot .. etc
19:17 <rocket> see the initramfs part of genkernel is the part that is extremely difficult to maintain ..
19:17 <johnm> moreso.. if a user wants nvidia-kernel say, they can pass it to genkernel and it's automatically included in the
          initramfs or whatever
19:17 <johnm> yeah
19:18 <rocket> brb
19:19 <rocket> ok I am back
19:19 <rocket> I think this area of genkernel is going to be the most difficult and complex
19:19 <johnm> depends where you do it
19:20 <rocket> the other parts are relatively straightforward
19:20 <johnm> in that modular design you can totally ignore it for the time being
19:20 <rocket> yes .. thats my plan  for now :p
19:20 <johnm> then you can actually just implement it wherever is best
19:21 <rocket> yeah ..
19:21 <rocket> I just hope when I come to that part I can make it flexible enough to be easier to add or upgrade software
19:22 <johnm> well as mentioned, im happy to help out
19:22 <johnm> (assuming you can do the bulk of it)
19:22 <rocket> like right now we have a custom busybox because plasmaroo needed to write a custom ioctl script to make the md
          devices
19:23 <rocket> yeah i will be doing the bulk of it I am sure .. I probably will be the only one really working on it at all
19:26 <rocket> well I will definately have to think on this some more but I appreciate the ideas
19:27 <johnm> as mentioned, I'd like to be a little involved, so it's all more than welcome.
19:27 <johnm> and im happy to help out
19:27 <rocket> well I probably wont have much written for a while as I am pretty busy with work lately .. and its my itch to make
          genkernel better
19:29 <rocket> as far as genkernel is concerned .. right now it points to a config option to know where the kernel sources are ..
          how would you have that setup?
19:29 <johnm> imo, the exact same as how linux-mod does it
19:29 <johnm> assuming /usr/src/linux if KERNEL_DIR is not set in the env.
19:29 <johnm> of course.. that can be set via an option
19:30 <rocket> ok that makes sense
19:30 <johnm> I qwould recommend env. though
19:30 <johnm> that way it works across linux-*.eclass too
19:30 <rocket> ok