rusty: has the tun gso patch been merged yet? herbert: no :( herbert: ok, was there any negative feedback, or no feedback at all :) rutsy: hmm s/herbert/rusty/ :) herbert: Positive, just too late. We also discussed a ringbuffer interface. rusty: too late for 2.6.25 or 2.6.26? 25 rusty: ok perhaps it's time to do another push for 26 then herbert: yeah, thanks for the push... I'll also try to get a virtio ring implementation for tap... tomorrow is good for that. (Grr cut my finger making dinner, hard to type...) rusty: cool, i'll let you to enjoy your dinner then :) Thanks :) rusty: when you get around to it, the negotiation between tun/virtio-back/virtio-net needs to include the actual protocols supported rather than just GSO e.g., we only support TSO right now and not UFO or anything else in software herbert: ah... I hadn't realised. I might have to fix that somehow. otherwise the frontend may generate packets that the backend can only drop if the hardware also doesn't support the required GSO alternatively you can write the code to do UFO in software :) We currently have one feature option we tell guest: that we support GSO (ie. everything) that's fine if all backends always support all protocols which isn't currently the case Yes, I didn't know that... ok, so do we do TCPv6? hmm, even if we did implement UFO we may one day have a new protocol that we don't support yes we do TCPv4/TCPv6 are the only two protocols supported in software right now herbert: ECN ok? Well I can implement UFO, but I'll still need a feature bit to say TSO only for older kernels, or just not use GSO at all for them. rusty: see my comment above re new protocols, i think even if we had UFO it would be best if the protocols were included in the negotiation otherwise we'd be in a sticky situation should some new protocol crop up e.g., DCCP or SCTP herbert: yeah... the current GSO means UFO, TSO, ECN and TSO6. We can't change that, but I'll introduce them as individual features as well. (We don't actually use this in the kernel yet, since tap stuff not in) rusty: TSO/ECN/TSO6 is all OK, UFO isn't actually supported by the soft GSO stack e.g., if you hardware doesn't support it the packets go to the bit bucket herbert: yeah, that sucks. Think it'd be a pain to implement? rusty: should be pretty straightforward herbert: it'll be fun to learn more about that code anyway, so I might try to patch it. rusty: sure, all you need to do is copy the code in net/ipv4/tcp.c to udp.c and modify accordingly should be self-explanatory rusty: that'll handle both IPv4/IPv6 rusty: by code i mean tcp_tso_segment etc. herbert: OK, I'll do that tomorrow. To test, I'll want to sendfile onto a UDP socket, right? rusty: yeah, you can confirm with tcpdump to see whether it's actually generating large packets netperf should do the trick too i think, with the right parameters rusty: oh and of course a tcpdump on the remote end to make sure that the packets did make it out :) herbert: I'll have debug comments in my code, but I have a sendfile test already for the previous stuff so easy to change it to UDP herbert: yeah :) rusty: great herbert: my plan for tap is to make it understand the virtio rings... they're designed to be an efficient inter-guest ABI for ringbuffers, makes sense for tap too. rusty: cool rusty: btw did tell you that my benchmark shows kvm with virtio is already faster than xen? herbert: wow, no! Cool, did you tell aliguori? that's with your tun patches and a user-space frontend i told cdub, i'm not sure whether aliguori knows herbert: hey, are you going to be at the virt minisummit BTW? rusty: i was surprised that the user-space backend is able to match xen with its zero-copy on send rusty: yeah i'll be there herbert: awesome, think you can throw together some results to present? Or are you already ahead of me? rusty: i will once i do the backend properly, my earlier results were done with a hacked frontend that pretended the backend did tso :) you know it actually doesn't matter whether the backend does tso for a guest-to-host test Yeah, * Jes_wk spots volunteers for speaking at the mini summit herbert: tag! we'll need a small presentation on that! :) * jsgf has quit (Ping timeout: 480 seconds) herbert, wow! does it beat xen for both tx and rx? or just for tx? aliguori: kvm is just slightly ahead on guest => host but significantly better on host => guest fwiw the two directions are about the same under kvm while with xen guest => host is significantly faster compared to host => guest in either case tso with a 1500 mtu going from guest to host gives between 50%-60% of the native lo performance s/performance/throughput/ this is with one single cpu maxed out with netperf wow, that's impressive I'm honestly quite surprised the most surprising fact for me is that copying with kvm doesn't seem that bad but one thing to keep in mind is that with xen's model the backend is really in user-space anyway herbert, why's that? well it's running in dom0 and it has to perform mmu ops to actually map/unmap guest memory as opposed to direct access which would be available if we had a kvm kernel backend ah, I see what you mean :-) so what i'm saying is that given the current model kvm and xen are really operating on a level playing field except for one extra copy in kvm's case on the rx path but in theory kvm could be even better so the ring queue should eliminate that copy nicely in the guest => host direction, because xen uses zero-copy there aliguori: that's what i thought too, but given the current numbers i'm no longer sure :) because xen already does zero-copy yet it isn't faster than kvm i'm beginning to think that the mmu overhead associated with user-space is the real killer i suppose we'll have to wait for a kernel backend to prove/disprove this hrm, i thought rusty's tun/tap patches gave zero-copy on guest->host i thought when gso was in use, it did get_user_pages() to setup the skb aliguori: i haven't turned that on yet :) ah :-) so my tests aren't with that i used a hacked gso version which basically is equivalent to just setting a large mtu but there is definitely copying aliguori, herbert: can't you just accept that my sheer brilliance bleeds into benchmarks?