Kernel Planet

James Morris: OLS Slides + Macbook fail

2008-07-25T09:59:24+00:00

Here are the slides from my OLS talk, although the paper is way more useful.

For those who attended Dan Walsh's talk on confining the user, you can find the slides here. Quite a few people expressed interest in them during the talk, and Dan sent them to me for some reason, so there you go.

***

Btw, my MacBook seems to never work with projectors (thanks to Paul Moore for lending me his laptop for the talk). It's relatively recent and has the following graphics stuff:
00:02.0 VGA compatible controller: Intel Corporation Mobile 945GM/GMS, 943/940GML Express Integrated Graphics Controller (rev 03) 00:02.1 Display controller: Intel Corporation Mobile 945GM/GMS/GME, 943/940GML Express Integrated Graphics Controller (rev 03)
Nothing seems to work: rebooting with the cable attached (VGA text mode works, but not X), exhaustive messing about with xrandr, hitting the laptop with a hammer etc. Does anyone know something else I can try?

Dave Airlie: gdbing the X server with xkb ... finally..

2008-07-25T03:42:33+00:00

So for a long time if you ran X inside gdb, gdb would die badly when X forked xkbcomp. X hackers have worked around this by either running -kb or attaching gdb after X is started for many years.

I finally found it by accident this morning, I was working on some modesetting things, and had been gdb'ing the X server fine, when suddenly it started doing the wierd hang, I realised I'd just enabled direct rendering code and initialised the lock.

Now the kernel has this really really really really dirty hack for the DRM that I'm not 100% sure we really need anymore. The original drm designers decided that if a DRI client held the lock then if it died it might crash the GPU, so they got this special signal blocking code into the Linux kernel which we use to block certains signals across lock hold. When the lock drops, the process gets the signal. Of course one of these signals was one gdb was using to stop the X server and the X server always holds the lock when running so it got confused when an blockable signals somehow managed to block itself :)

Now the fixes for this are

a) kill the notifier stuff, I'm not sure it ever mattered, I at least could kill it for modesetting drivers as we have a lot more
control over the GPU from the kernel.
b) Have the X server drop the DRI lock around certain operations like xkb execing xkbcomp etc.. this seems worse.

What I've done for now in drm git is just not set the notifier up for the "master" process, i.e. the X server. This will at least allow people to debug their X servers with xkb.

Rik van Riel: Offshore drilling? Alternative energy? Why choose?

2008-07-24T21:59:17+00:00

Politics here in the US seems divided. Some politicians want to allow offshore drilling, while other politicians want to invest taxpayer money in alternative energy research. Personally, I believe alternative energy could be a while off from being cost effective for everything and I sure don't trust politicians to spend the money well.

On the other hand, offshore drilling will need to happen under some strict conditions to make sure the environment is well protected. Since conditions already need to be put in place, and considered by the oil companies when deciding whether or drilling for offshore oil is worth it, why not add in one more condition: for every barrel of oil recovered, invest $1 (or 1% of the price of a barrel of oil, whichever is less) in alternative energy research and production.

Harald Welte: Receiving the 2008 Open Source Award

2008-07-24T02:00:00+00:00

According to reports here and here I had the honor of being the recipient of one of the the 2008 Google+O'Reilly Open Source Awards entitled Defender of Rights", presented by Google and O'Reilly.

I'm obviously very happy to see that my work has been recognized this way. Following the FSF Award in March, this is definitely a big honor. Did anyone else receive both awards in the same year so far? ;)

Thanks to the committee for the trust they put in my work. I'd also like to use this opportunity to thank again my lawyer Dr. Till Jaeger and his law firm JBB, as well as Armijn Hemel, who has been running the day-to-day gpl-violations.org operations for quite some time now.

Harald Welte: Becoming VIA Open Source Liaison

2008-07-24T02:00:00+00:00

Today, VIA made public what I've already been doing behind the scenes for some time: I've been contracted and appointed to be VIA's Open Source Liaison. As first part of the process, they've released the Padlock programming guide and the CX700/VX700 integrated north+southbridge manuals on linux.via.com.tw.

This basically means that I'll be helping VIA with improving their strategy for Open Source support, such as Open Source driver support, merging those drivers into the respective mainline projects as well as working on publicly available reference documentation for their hardware.

This is an incredible chance to contribute my part to help a major manufacturer of CPU, Chipset, Ethernet, WiFi, Card Reader and PC Graphics components understand what it takes to interact properly with the Free Software community. This is a big learning experience for VIA, and a teaching experience on my part, of course. I feel very happy to be able to work in such a key position, and to be able to put all my knowledge about Linux driver development, the development process, the FOSS community values/ethics/practises as well as licensing related knowledge at work.

VIA is truly interested to learn, and they're already doing a lot internally which you might not have been aware about. I am well aware of many of the historic problems between VIA and the community, related to binary only drivers, not cooperating with mainline development, suboptimal press announcements with little action, etc.

I'm very confident that together we can move beyond this and take a fresh start for much better FOSS support of VIA products. Of course the change will not come overnight. It's a process, and it involves many groups in a large company, each group with their own management, R&D and so on. So please bear with us, and don't expect all drivers to be finished in mainline quality tomorrow.

If you are a Free Software developer and you have some comment/feedback/demand to via, please feel free to contact me (preferably at HaraldWelte@viatech.com. I will try my best to follow-up with all those comments. If you are missing some piece of documentation for hardware or have some other issue, please let me know. I do care, and I will take up the issue with VIA's management.

Evgeniy Polyakov: Manager's thoughts: unused extensibility and used de-facto standards.

2008-07-23T21:00:20+00:00

After some before-sleep-reading (this time DNS RFC specifications) I found, that DNS protocol is so much extensible, that is can perfectly cover not only its area, but also help in really lots of close problems. It already has (though completely unused) many interesting RRs and types, which have nothing to deal with DNS (like NULL RR, which allows to transmit binary data or TXT RR, which also is not related to DNS area). And the most popular RRs are A, PTR, SOA CNAME and MX. That's all from about 20 others. The same applies to (q)type and class (I first time read about Hesiod class for example). And DNS allows to introduce own classes, types and resource records.
It is just not used, but we could create distributed DNS system with new types. It would be really simple (and actually it can be done even without new DNS extensions).
But it is not actually needed, since people are used to have DNS just like it is.

Another example is internet video. There is de-facto Adobe standard, no matter what W3C will put into its new standard, everyone will continue to use existing one. Just because it works ok. Not excellent or perfect or whatever, it just works how we used to know.

And there are lots and lots similar examples.

People are so much intert in this questions (although I think in most areas, just because it is convenient not to do something better, when existing solution just works, even if not perfectly and even if not good), that no one will ever bother to change something dramatically, because it will not only require huge amount of money, but also changes in the way people used to think about given area, which is likely even more complex (and money-hungry) problem.

All this talk is about simple thing, I just opened for myself: when you created something completely new, even if it is not the best solution for given problem, if you will start pushing it to wide audience to be used, then you are able to get all 'the market'. That's why when you have something new on the market, where most of the users already used to work with one or another solution, (and even if your project is potentially very good and definitely much better than existing solutions) then there will not be any major gain, only single links to the completely new users.
This is probably told to the first year MBA students, but I was quite excited and dissapointed by this issue: the first new idea, when properly presented even if not the best solution for given problem, can get all the users, after which they will not switch to the new one just because they used to have it this way. Comments (1)

Jaya Kumar: Manmohan's Mischief

2008-07-23T19:19:35+00:00

Its sometimes quite sad to watch the comedy that is our politics. The following quip though, from the Prime Minister had me in stitches:

"he has made at least three attempts to topple our government. But on each occasion his astrologers have misled him. This pattern, I am sure, will be repeated today. At his ripe old age, I do not expect Shri Advani to change his thinking. But for his sake and India's sake, I urge him at least to change his astrologers so that he gets more accurate predictions of things to come"

via Newstrack.

James Morris: Notes from the SELinux Developer Summit 2008

2008-07-23T13:13:50+00:00

The SELinux Developer Summit went pretty well yesterday. It was a long day: 10 hours of talks and discussions with about forty developers attending.

I've just uploaded slides from the talks, which may be found next to their respective entries in the schedule.

Some of the talks I found particularly useful/interesting:

Josh Brindle on SELinux in Ubuntu. They're making good progress, although the idea of SELinux is to introduce ubiquitous, generalized MAC security, so he is advocating they enable SELinux by default as is done in Fedora, and as you typically do with other OS security layers.

John Weeks from Sun talking about OpenSolaris FMAC (introducing Flask/TE to their OS). It was interesting to see a dtrace graph of the AVC operating—a kernel mechanism for which I've developed an abstract mental model but never "seen".

Dan Walsh Talking about his ongoing work in utilizing SELinux to create practical security features for everyday users.

The above is from a demonstration where nsplugin (the framework for Firefox plugins, i.e. where flash etc. is run) is being sandboxed by SELinux, so that a flawed or malicious plugin cannot be used to snoop your keystrokes. In this case, a simulated (and trivial) exploit was blocked from capturing internet banking passwords by SELinux.

Btw, Dan will be demonstrating this today during his OLS talk on Confining the User. There's a lot of really cool stuff coming in this area & the talk should be well worth attending.

Karl MacMillan on alternatives to comprehensive least-privilege, where he described some ideas and plans for simplifying the way SELinux policy is deployed for general purpose use. He has some really promising ideas on reducing the granularity of the policy while still maintaining strong security. This can lead to simpler and smaller policy, which is important for all kinds of users.

Peter White talked about two higher-level languages being developed to express SELinux policy, Lobster and Shrimp, which will introduce features such as type checking and object orientation to the policy language area. Peter is a Haskell guy, and it all looks very promising.

***

Yuichi Nakamura talking about embedded systems and SELinux.
The format worked reasonably well—a series of short talks and discussions—although it would have been nicer to have a more relaxed schedule and more time for deep discussions on specific issues. There's already been discussion of what to do next year, and we may move it to a two-day event. Certainly, I think we'll want to have it again in conjunction with a major developer conference, which makes it a good environment for collaboration with the wider FOSS community.

For those that couldn't make it this year, I believe notes were taken and will be sent out to the mailing list. There are more photos here.

Evgeniy Polyakov: POHMELFS distributed facilities design notes.

2008-07-22T22:00:22+00:00

Since I'm quite busy with VISA/hotel/tickets and overall preparations for Kernel Summit, there is no development progress, but it should be completed very soon I think, and so I will write here some design notes I have in mind about how POHMELFS server will be designed. It is not a finished draft, but somewhat a rough direction paint.

POHMELFS will utilize distributed hash table approach, i.e. storage will support ability to get an obect based on some key attached to it. In a local filessytem we already work with hash table: directory lookup is no more than lookup for inode object based on its name, i.e. lookup for the value based on attached key. And although key in this case is not created based on object itself (like hash of the content or some other function), it still is a (turn on your imagination here) table lookup.

Cloud of POHMELFS servers will utilize similar approach. Consider a single server in the system. When it joins the cloud (I ommit this proccess for now, and will describe it below) first time, it is empty, so it gets some unique id, either via administrator steps or randomly, or it just waits in the queue to be filled with new data, so it will get id at that time, it does not matter for now how it gets its id, but this id is propagated to some cloud of its neighbours (or if it would be a bittorrent or napster to the main server).
There are two ideas on how to treat this ID: either as a part of the filename, or as a nameless pointer in the abstract namespace, I will show below that actually it does not matter.

Now, let's check what will happen when user wants to perform some IO on given file.
Every file access actually happen to inode, stored on disk. In our case it can be stored somewhere we do not know yet where, so we need to perform a lookup to get address of the node in cluster which contains our data. In existing schemas like bittorrent or Lustre there is a server (or small cloud of servers) which contain mapping information about where this or that object is placed in data cloud, so simple lookup to this server(s) return needed info. This approach does not scale to really lots of nodes and is failure-prone.
Instead I consider completely distributed metadata storage. Let's check how system will lookup the whole path in our case.

Each path starts from the root directory, which is '/', which in turn is a id in the global namespace (or hash from this string or whatever else mapping), so we first need to lookup a node, which is responsible to content of this directory. Each node contains routes only to the very limited set neighbour nodes (in various designs this number varys, but idea lays in the fact, that node, performing lookup, does not know which node contains needed info). Gnutella system just broadcasted this lookup request to all of its neighbours, so each one broadcasted it to its neighbours and so on until one of the system replied, that it contains needed info. Amount of unneded broadcasts killed Gnutella next day after Napster was closed.
So, this approach does not scale, and instead we need to map needed directory into node address in a more intelligent way. There are at least two the most appealing design choices: ring-based structure implemted in CHORD and multidimensional torus implemented in CAN.
Right now it does not matter, let's assume that we found a node, which has information about content of the needed directory. When we have that data, we can find next node (or this info can be cached on 'parent' directory node) and so on until get node, which is resposible for storing content of the needed object.

When new node joins the cloud it connects to one or another known node (provided either in public service or by administrator) and sends there information about its available space, gets ID and just waits until some client connects to it and start writing a data.
When node joins with some content, which was written to it by the system before, or written by local users bypassing distributed mechanism, node has to tell this information to the node, which holds parent directory. This information should be stored in each directory it exports, or it can be provided by administrator, for example this node exports dir '/zbr' which is actually a subdir of '/home', so node will lookup '/home' directory content owner and update its records, that now it contains new dir. There is a problem here: what if there is already another node, which also claims to have dir '/zbr' in '/home'? This can be handled via attached to each object extended attribute, which will tell us the last modification date, so system can select either the last modified '/zbr' dir or that node, which contains dir with the biggest number of the same replicas. It can be setup by administrator.

Main advantage of this joining scheme is the fact, that we actually do not need to know content of any object in the exported directory, we publish only high-level object, which may or may not contain some inner file or dir. Thus we do not need to hash millions of files in the exported directory and publish them one by one, we do not need to store information about each inner object, no need attach full path to each object and so on.

When we will decide to split the same object between multiple node, we will need to introduce not only name based lookup, but also extend it to the offset inside the object. This can be done by introducing ssytem wide 'block size', so each file is actually set of blocks of given size, so when we found a node, resposible for storing information about directory, where it is located, this node can also contain information where each part of the object was stored.

Looks quite simple, but... Devil is in the details.
I obviously missed some bits in the design (and I created it in mind during talk being under 'impression' of the greece spirit while talking with asm@, who suggested to look at Kademlia project), like redundancy management of the nodes, splitting of the node content between multiple nodes and other bits, but it is one of the first drafts, so things can be changed if needed.

Stay tuned, I will be very soon back to development process (DST first :), since paper work for kernel summit travel seems to reach its end. Comments (0)

Val Henson: Working for Red Hat

2008-07-22T19:56:21+00:00

As of August 18, I will be Red Hat's newest file systems developer! I'm working part-time for Ric Wheeler (if you don't know Ric yet, you're missing out). Possible projects, depending on what is most useful to Red Hat's customers:

* Merge e2fsck parallelization work. I wrote this on contract for Ric Wheeler when he was at EMC, but larger economic realities forced the end of this contract before it was quite done. Ted T'so is the ultimate arbiter, of course, but I think it just needs a little touching-up before it's mergable as a non-default option. The technique I use, which can be described as active buffer cache management, could replace DIO/AIO for several applications with a little OS support.

* General ext4 work. ext4 is very close to shipping, but I hope to be at Red Hat for the final frenzy of touch-ups before Red Hat puts it in a release.

* Improve btrfs check and repair. Btrfs already has a rudimentary file system checker, but I'd like to implement incremental online check and repair, my personal holy grail.

Things I won't be working on:

* Chunkfs. Chunkfs is a particular instance of the general repair-driven file system design principles I've been advocating. I wrote a prototype, both to convince myself that it would work and to give a concrete example for other file systems developers, but the experience convinced me that you really need to build these features in from the very beginning. If btrfs wasn't on the horizon, the layered file system approach might be worth implementing. I do need to write up a summary of what I learned working on chunkfs and in particular collect in one place the various measurements I made showing that the idea will work.

* GFS2, cachefs, NFS, etc. I'm just not that into network file systems.

And I have to admit, for all that I thought I was a blasé and worldly kernel developer, I am a little excited about the idea of working for Red Hat in and of itself. Back when I was making 6 floppies' worth of drivers so I could install Red Hat, I would have laughed if you'd told me I would actually work for Red Hat someday.

FAQ:

Q: I thought you hated having a real job. Why stop consulting?

A: Honestly? The number one reason is that health insurance is impossible to get in the U.S. if you're an independent consultant. Europeans, Australians, and other residents of sane countries are permitted to feel smug at this point.

Q: Why part-time?

A: I want time to work on my writing - science writing, most likely. Expect more regular articles in LWN.

Q: But distro work sucks, you're going to hate it. Packaging, bug fixes, support, etc...

A: Actually, I was a one-woman distro back in 2001-2002, doing a Yellow Dog-derived distro for a small embedded company. I liked it, and I did everything - customer support, building RPMs, burning CDs, and fixing installer bugs, in addition to actually writing kernel code. I only left because no one there knew more than me about operating systems (and I didn't know very much about operating systems).

Q: So, would you recommend consulting?

A: I'd recommend it for paranoid, ambitious, meticulous, compulsive people with a wide variety of interests and the ability to put with a lot of annoying extraneous crap. It can be pretty awesome - I write this overlooking the ocean in Hawaii - but it's a hell of a lot of work, at least in the beginning.

Jesse Barnes: Waiting for vblank...

2008-07-22T19:29:11+00:00

Will the vblank-rework branch ever be ready to merge upstream? With some recent work by Michel and myself, I really hope so.

It’s a little distressing that such a simple thing could cause so much trouble. The motivation for reworking vblank in the DRM branch was easy enough: save CPU power by turning off interrupts when possible. Not interrupting the CPU lets it sleep deeper and longer, potentially saving quite a bit of power. Getting rid of the 60 or so vertical blank event interrupts per second when they weren’t needed seemed like a logical first step, and so vblank-rework was born.

Being a good citizen in Linux land often means improving whole subsystems rather than stuffing a bunch of fancy features into individual drivers. Working that way can be harder, but it spreads the benefits wider, and improves Linux as a whole. So my efforts in the vblank-rework branch targetting the generic DRM vblank code, improving the driver APIs and making sure that all drivers could benefit from the new infrastructure, allowing them to disable interrupts when not needed. However, that’s where the “harder” part comes in: every driver needed an update. At first I thought, “Hey this is a neat, new set of APIs, surely everyone will want to use them, I’ll just convert the i915 driver (after some initial work on a radeon based laptop).". Unfortunately, the DRM drivers are in dire need of attention, so after several months of waiting I ended up converting all the drivers myself, though only a few like radeon and i915 actually implement the API fully enough to disable interrupts.

So far, so good. I figured everything was fine and the shiny new branch was ready, so I merged it into the master DRM branch in preparation for an upstream push. Then tragedy struck: Michel’s sharp eyes and testing discovered a potentially fatal bug. While many GPUs provide a frame count register we can use to keep the vblank count accurate (necessary since OpenGL extensions expose an absolute count to applications for some reason), they’re typically only updated at the leading edge of vactive. This means that your application may wakeup at vactive time instead of vblank time, causing ugly tearing; exactly what you’d like to avoid! After a bit of back and forth and a couple of false starts (trying to work around the problem with solutions that turned out to be racy), we decided to go back to using the atomic counter (which is only updated at interrupt time) for wakeups, rather than the hw register, using the latter for keeping the counter accurate across interrupt disable periods.

Which brings us to last week. I hacked up the scheme described above and started testing. As I found and fixed bugs (well actually Michel probably found most of them), I discovered that the API could actually be simplified a bit, and some of the code to compensate for corner cases was no longer necessary, so both the wraparound compensation logic in the pre/post modeset ioctl and the funky accounting we tried to do there could be removed. The result, I hope, is ready for upstream finally.

So where does that leave us, API-wise? Well, on the userland front we have a new ioctl, DRM_IOCTL_MODESET_CTL, with _DRM_PRE_MODESET and _DRM_POST_MODESET arguments. It should be called with _DRM_PRE_MODESET in userland drivers prior to any activity that resets the hw frame counter (typically mode setting). When the mode set completes, it should be called again with _DRM_POST_MODESET. These calls tell the kernel to account for any lost events so that the vblank count exposed to applications can stay accurate.

On the driver front, there are a few different calls and callbacks:

drm_vblank_get - increase the refcount on the vblank counter

This call just tells the core code that the caller is actively using the vblank counter for something, e.g. scheduled buffer swaps or a blocking vblank wait call.
drm_vblank_put - decrease the refcount

Tell the core you’re done with the vblank counter. When the refcount reaches 0, the kernel knows it can disable the interrupt at some point in the future.
drm_vblank_init - initialize the core vblank code

Should be called at driver load time or IRQ init ioctl time to init the core.
driver.get_vblank_counter - return the current hw frame count

Used by the core code to keep the count accurate across interrupt enable/disable periods.
driver.enable_vblank - enable vblank interrupts on a given CRTC

Used by the core to enable interrupts when the refcount increases.
driver.disable_vblank - disable vblank interrupts on a given CRTC

Used by the core to disable interrupts after a timeout period if the refcount is 0.

With a few simple changes, a given DRM driver can support the new scheme to save power. If you find bugs or have issues with the new APIs, let me know and/or file a bug at bugs.freedesktop.org.

Rusty Russell: WTF? Wikipedia deletion gone mad...

2008-07-22T13:06:00+00:00

OK, so Dave Miller's pending deletion I can understand; if you didn't know how key he was, the article itself lacks references and is lacks detail (compare it with Andrew Tridgell's page. (At least he noticed; when I was deleted last time I didn't know).

But then I find out that the article on OLS was deleted back in February. Huh? This is the major Linux conference in the world. Some would argue that it's a bit faded at the edged these days, but none of the crop of contenders can genuinely claim that crown. I know conferences don't generally get pages as sexy as humans do, but still...

Harald Welte: Arrived in Canada for OLS again

2008-07-22T02:00:00+00:00

I've just arrived in Canada for Ottawa Linux Symposium 2008. After my last visit to OLS in 2005, there were two years of intensive work that prevented me from attending the event. Last year I actually had to cancel an already accepted paper submission :(

In Year 01 post OpenMoko, I have time to visit OLS again. Unfortunately no company to pay for my travel expenses this time, but well, what can you do. Due to scheduling issues with a family celebration, I didn't know until very recently that I would be able to make it this year. Thus I happily forwarded the invitation to talk about OpenMoko to Werner. I was surprised that it's now actually one of the keynotes. Looking forward to it :)

There have been many rumors that OLS is not like what it used to be. Maybe I'm now in a good position to make up my mind about it, since I've missed two years and will be able to directly compare my memories from before with the current event.

Andy Grover: MinGW cross-compilation adventure.

2008-07-22T01:07:55+00:00

I just finished porting a Windows driver (actually 3) to use the MinGW build environment.

Due to the Windows Device Driver Kit (DDK) being under a somewhat opaque license, we could not get legal clearance to release binaries of our GPLed code compiled with the DDK. This was bad because I, being the open-source kinda guy I am, had based my work on a pretty substantial GPLed codebase. *Surely* the DDK license couldn't be so restrictive to prevent all binaries bsed on GPL source! Well maybe it is and maybe it isn't. When it's not clear, legal says no.

Not good. It looked like we were going to have to somehow pull out the work I had done and rewrite the rest of the code from scratch to be GPL-free so we could release binaries legally following the DDK's EULA.

It was in the process of estimating the work involved that I happened to talk to Jamey Sharp at a local Linux beering. We were each talking about our work and it turned out that he was in the same boat as me, a Linux dev who through some strange quirk was working on Windows drivers. But he wasn't using the DDK, he was using MinGW, which is a port of GCC to compile Windows binaries.

Sometime later he was kind enough to come over and show me his setup and makefile, and sure enough it worked. So, we *could* use the GPLed code if we just distributed DDK-free binaries.

Once on that path, it still did take me a while to get the driver building. One nice thing about trying to get a different build env going is that you have something that works (although you can't distribute it :)

I got it compiling by modifying the driver to have function definitions etc. that MinGW lacked. But it wouldn't link. Sigh. I was using functions that were not in MinGW's import libraries, and could not get around this by fudging it in my driver; I had to fix MinGW. This is a big fear of mine every time I use a piece of open-source software -- if your code has a Foo dependency and Foo is broken, you pretty much have to be willing to start hacking Foo if you want it fixed. (Sure you can file a bug, but obviously nobody cares about your bug except you or it would be fixed, right?)

Once I dived into MinGW (actually w32api) it was straightforward to fix. Many of my changes to get the driver compiling actually belonged in the MinGW header. There were also functions omitted from the import libraries. (Import libs tell the linker how to link to the actual shared libraries that will be present at runtime.) These were defined in .DEF files, and all it needed to know was for each function, how many bytes were its arguments? Easy, once I knew what to do. (Nothing hard, just if one doesn't do this thing regularly, you forget how it all fits together.)

So for anyone who wants to write Windows kernel code without the DDK, it can be done. Some tips:

Debian/Ubuntu includes mingw cross-compiler
Don't use KMDF/WDF, MinGW doesn't support it
Sprinkle all callback functions with DDKAPI, so they use stdcall calling method (the compiler will complain, makes it easy.) including DriverEntry().
Add -DDBG to enable KdPrint()s
Inline asm will need to be rewritten
Recent GCC doesn't have __FUNCTION__, and DDK doesn't have __func__, so plan on wrapping with your own macros for tracing and handling this difference based on #ifdef __MINGW32__. I'm a big fan of gcc's preprocessor, the variable-argument macros are pretty nice.
WinDBG needs .pdb files to set breakpoints and step through code, MinGW doesn't generate these, so KdPrint() is your friend
You can't link with libc yet sometimes you need snprintf. I pulled in Linux's snprintf code to fill this gap, yay GPL!
MinGW is 32-bit only right now. For 64-bit binaries, plan on becoming a serious MinGW contributor for a while!
Here's how my makefile build cmd ended up: i586-mingw32msvc-gcc $(CFLAGS) -o $@ -shared -Wl,--entry,_DriverEntry@8 -nostartfiles -nostdlib $^ -lntoskrnl -lhal -lndis
For reference, see the Xen GPLPV repo. More help is always welcome.

Hopefully this blog post will contain enough keywords to be helpful to someone in the future :-)

Evgeniy Polyakov: Foot, fingers, shins, knees, thigs, shoulder, back.

2008-07-21T22:00:19+00:00

No, it is not parts of the body I know (half of it I looked in the dictionary), it is what is being aching right now.

Its called football.

Yes, sounds a bit scary, but that was hell the super game today. We were much stronger, but I have to admit, that mostly because we get a right transfer decision and selected right players at the beginning, so our previous team was strengthenen. I managed to make a goal, couple of nice saves and even make quite technical outplay sometimes, which was quite surprisingly, since I did not play football for 5 years.
I would not say I'm getting into the shape, but have a progress. Comments (0)

Matthew Garrett: The fallacy of the completely inclusive community

2008-07-21T12:35:30+00:00

Emma Jane Hogbin gave a presentation on the gender gap in free software at Lugradio Live this weekend. One of the central messages was that a great deal of how to avoid putting women off computing can be distilled down to "Don't be a dick". This ties in well with Mako's restatement of the Ubuntu code of conduct as "Be excellent to each other" (a wonderful phrasing which demonstrates that Bill and Ted's Excellent Adventure is the most philosophically worthwhile film that Keanu Reeves has ever appeared in, and certainly not The fucking Matrix) and led to my 5 minute rant on why I hate the Linux community slightly later in the day, but does leave a certain problem. What standards are used to define whether given behaviour is dickish or not? The comments here show that there's disagreement even within a single sub-community of the larger free software world. Ben suggests that the reaction to perceived inappropriate behaviour is perhaps even more discouraging than the original behaviour, suggesting that bitching about things that offend you is dickish behaviour in and of itself. How do we decide whether someone is being a dick or helping the community? Is lack of tolerance a form of exclusionary behaviour?

This topic is actually one of the issues discussed in the Geek Social Fallacies, but here's a nice easy example. Would tolerating planet posts encouraging the eradication of the Jewish population be inclusive or exclusive? I suspect that most people would agree that it wouldn't be acceptable behaviour, which leads us to the next question. Why? There's two obvious arguments here. The first is that at a community level we have some form of rough moral consensus that advocating genocide is Just Wrong, and so criticising Nazis is obviously the right thing to do. The second argument is more pragmatic than philosophical - alienating millions of people in order to avoid alienating hate groups could be considered to reduce our potential contributor base in an unfortunate way.

I'm a fan of the second argument. The example I gave is emotive and relatively recent history has resulted in people tending to be pretty uniform in considering genocide to be a bad thing, but many other cases aren't clear cut. What I consider to be objectification of women is seen by others as appreciation of natural beauty. What I think of as sexist jokes are perceived by others as acceptable humour. When I advocate intolerance of certain behaviour, people are going to see me not having enough tolerance. So we end up in a situation where people make the "We should all just get along and be tolerant of each other" argument, which sounds fine but is fundamentally flawed in one significant way.

Advocating tolerance excludes the intolerant.

The reason people fail to see this is that it doesn't sound like a flaw. If you ask people whether we need to support intolerance, the immediate answer is probably no. But by advocating exclusion of intolerance, you're excluding all those who have good reasons to be intolerant. You're excluding the women who don't want to feel that the community sees them as a pair of breasts attached to some legs. You're excluding the ethnic groups who would prefer to avoid racist slurs or ethnic stereotyping. Telling anti-fascism protestors that they're being intolerant isn't likely to endear you to them. Advocating tolerance is telling the intolerant that they're wrong and should just deal with whatever it is that makes them unhappy.

So, ironically, tolerating certain types of intolerance is probably required in order to avoid alienating many potential contributors. That means some way of deciding what kinds of behaviour are acceptable and which are unacceptable. In the absence of either pre-existing community consensus or some philosophical breakthrough that allows unambiguous determination of the "rightness" of a given action, I'm going to suggest that we look at it from a pragmatic viewpoint. How many potential contributors do we discourage by criticising a certain type of behaviour? How many do we discourage by tolerating it?

The fallacy of the completely inclusive community is the idea that it includes everyone. The reality is that a certain level of social exclusion is required in order to include a wider range of people. So don't criticise people purely for criticising someone else's behaviour - make an argument for why that behaviour benefits the community. And when you see behaviour that you think discourages others, call people on it. Even if nobody's behaviour changes as a result, you're sending a signal that not everyone in the community agrees. Sometimes all people want is to know that there'll be some people on their side.

But, above all, try not to be a dick.

Dave Miller: Please help!

2008-07-21T03:33:00+00:00

Aparently there is a pending deletion tag on the wikipedia article about me, which can be found right here

It states that they will delete it later today due to lack of "reliable secondary sources attesting to notability."

So if you can help with this, I would appreciate it.

Thanks :-)

Dave Miller: Netconf 2008

2008-07-21T01:15:00+00:00

YOSHIFUJI Hideaki visited to give a presentation on IPV6 happenings. Stephen Hemminger came as well and after an excellent meal at Ototo Sushi in Queen Anne, we listened to Hideaki-san's presentation as well as one I gave on the TX multiqueue work I've been doing.

It went so well, that we decided to declare that this was Netconf for 2008 :-)

My slides are here.

Enjoy.

Dave Airlie: technology fail weekender...

2008-07-20T23:14:08+00:00

So this weekend was a total technology fail..

so on Sat night Gia rented a DVD to watch on a laptop, I was all should be no problem with this, armed with my livna (and livna-development) repositories, I started off on laptop 1, Thinkpad T60P. totem pulled an immeditate choke with some useless error message. mplayer choked similiarly with a nice bonghits in the libdvdread error message. This I tracked down to the region not being set correctly, so I set the drive region to region 4, and then mplayer failed even harder in a tight loop with kill -9 the only way out. It looked like bad sectors or something in the kernel.

So I went and tried it on laptop number 2, my Dell Insprion 6000, this machine is the oldest piece of kit I own and has been steadily on it way out the door (backlight flickers on/off due to dodgy connections internally). However it is also the only machine I have at home with a non-Linux OS on it, it has Windows XP. So I booted XP on it, stuck the disk in and it played fine, I then booted Linux on it and had the same problem as on the the other laptop. So I decided to let Gia use Windows XP to watch the DVD, I then unplugged the laptop before realising the battery was sitting on the bench (as it overheats a lot instead of charging nowadays.. and it isn't one of the goes on fire batteries either..). So it appears cutting the power wasn't such a good idea, as it appears the hard disk heads crashed or some such fail, as attempting to boot XP on the laptop after that gave a bad boot bluescreen and mounting the XP ntfs partition from Linux went into an ATA reset cycle.

So I had to watch half the rugby (it wasn't a great game in the end), and then watch some movie with Matthew McConaughty in it, at least there was scuba diving in it, so it could have been worse.

Then on Sunday on our way to the cinema, I drops my iphones on the grounds. Oh noes. My iphone volume buttons have been busted for a month or two in any case, but now the lock/power button decided to stop being useful. Of course my iphone is a US phone where I'm sure the warranty has been invalidated six ways from Tuesday at this point. So I'll probably have to go opens it ups and play with its insides. I've damaged a lot of phones in my life but the iphone has actually been the quickest to get broken, and considering I don't drink nearly as much as I used to this isn't a good showing for it.

Jaya Kumar: Cool Paper

2008-07-20T23:39:57+00:00

Just got pointed to this NY Times article, "The electronic cover will be used in only 100,000 copies that go to newsstands". I think I'd buy one just to get at the E-Ink device! Too bad they don't sell these magazines over here.

Rusty Russell: The Joy of linux-next

2008-07-20T19:03:00+00:00

Sure, linux-next is a useful way of early-detecting patch conflicts with random developers. But the second order effect has been more useful to me: forcing me to get my shit together. Now I regularly publish my patchqueue in a form which applies and compiles, and has clear "production" vs "alpha" demarcation.

Obviously, this is good for people trying to follow various patches (and there are quite a few independent efforts at the moment, including typesafe patches, virtio, lguest, module, tun/tap, stop_machine, kmod-removal and down_trylock removal), but it also makes the arrival of the merge window far less stressful.

In theory, I could have been this organized before. But just like the concept of doing homework long before the deadline, it was never going to happen. So thanks Stephen!

James Morris: Have You Driven an SELinux Lately?

2008-07-20T15:44:33+00:00

My OLS paper,

Have You Driven an SELinux Lately?

may now be downloaded as a single document, or as part of the conference proceedings.

The paper is a detailed update on the SELinux project, covering important changes to SELinux in the past few years. After the initial upstream kernel merge—which took three years and required LSM to be developed—the project proceeded rapidly in terms of integration into mainstream Linux distributions, as well as having its internal infrastructure overhauled to allow major improvements to both function and usability. A great deal has changed since many people first saw SELinux.

I'd recommend reading the paper if you want to come up to speed on where things are at in the project, and where things are headed.

I'll be giving a talk on the paper at OLS this Thursday. It's certainly a challenge trying to keep the talk length below 45 minutes without leaving something significant out. For some reason, my talks tend to self-adjust to about 90 minutes, and I always need to work to shorten them.

As a reminder, the SELinux Developer Summit is on Tuesday, and it will be held at the Ottawa Novotel from 8:30am.

Btw, I noticed Linux being used at Sydney Airport on the way over:

Ubuntu ~~6.02, I believe~~ 6.06.2.

Evgeniy Polyakov: Crazy security idea.

2008-07-19T21:00:37+00:00

I've just thought, that I do not know a way to make some (running) application to encrypt all its data, which hits the disk (either via swap or usual way, like editor writing the file and all its temporary files).
I actually consider this as a very useful feature for the editors, browsers, instant messengers and mail clients, downloading applications and musical players and so on. This is especially valid for temporary files, when one expects editor to be highly secure (or even working on encrypted partition), while its temprary files are stored somewhere in /tmp which is not encrypted.

It could be started via some wrapper, which will tell the kernel encryption algorithm, key, iv and all needed info, it will attach a crypto processing callback to the process, so when disk activity is started by given pid (swap or data writing or reading), it is encrypted/decrypted in flight.
Kernel should check all file descriptors opened by the given process and appropriately process them. There may be some problems with communication with unprotected applications, which should be thought out, but overall I like the idea...

Has put it into todo list. Comments (0)

Evgeniy Polyakov: Project presentation.

2008-07-19T21:00:37+00:00

I've just realized, that lots of my blog posts are valid enough presentation abstracts, at least they contain enough words describing the problem, possible solution and overall interested for given area topics. But I never presented such projects in english before, although quite frankly I'm not that bad speaker in russian, at least I am not afraid to talk and probably like a contact with interesting auditory. After all there is this blog :) and even had number of similar kind of presentations from 15 minutes to couple of hours including question/answer part.
My english used in blog is rather ugly, but I rarely (if at all) fix errors which I detect after subsequent reading of the text in the browser (and I detect lots of them) as long as in mails and other posts.
So probably eventually we will have interesting talks about diferent areas, but expect to 'listen' a world-wide language of the gestures :) Comments (0)

Evgeniy Polyakov: Disributed storage is dead, long live the Distributed storage!

2008-07-19T17:00:38+00:00

As you may know, DST project was an attempt to implement redundant, failover resistant, flexible block level storage subsytem. Among other features it supported ability to map multiple remote nodes via linear or mirroring algorithms to single node, reconnect to failed node, reading balancing and parallel writing to multiple nodes (in case of mirroring) and so on.

Now it has gone. There is no more distributed storage you knew before, instead there is completely new project being developed, which main goal is to provide a transport layer for the block requests only. Consider it as Network Block Device on huge steroids. Consider it as iSCSI on huge steroids. Consider it as ATA-over-Ethernet on even more huge steroids.
It is just an example of what all those protocols should have. And only that.
An it does not sound very ambitious, previous DST versions already supported lots of features, which never existed (and in some cases were impossible to be added) in another block level network storages.
DST moves further.

There will be no mirroring and overall ability to map multiple devices into single one, instead one should use Device Mapper for this goal, since its features were simply mirrored (although I tried to optimize them sometimes) in DST, and amount of targets was noticebly smaller.

Now DST is just a simple block device which operates on top of network connection. With just a single exception: its done right.

Features planned for the new Distributed Storage:

kernelspace client and server
initial autoconfiguration between client and server nodes
automatic reconnect to failed target
transaction model: resending, timeout error completion, full rollback of the failed transaction
wire speed performance
data channel encryption, strong checksumming
cryptographical authentification
ability to work on top of any network protocol
barriers support (when, if any, Device Mapper will start support them, DST will not need to be changed)
flexible protocol with simple ability to extend it to needed functionality
trivial configuration

Project is being written from scratch, but it is actually very simple, and should be quite small, so expect its first release quite soon.
It will be pushed upstream when ready. Comments (8)

Andy Grover: CFP deadline approaching

2008-07-18T18:39:35+00:00

Hey it's Friday! Chillin' at work? This afternoon might be a good time to write up that idea you had for a talk at the Linux Plumbers Conference. We do have a bunch of microconfs, but we're interested in all interesting talks.

Evgeniy Polyakov: Completed distributed storage redesign.

2008-07-18T18:00:19+00:00

I also managed to play second octave F# and sometimes the whole chromatic scale down to small (minor?) octave F on my trumpet, and I belive I started to understand overall trumpet kung-fu, but expect it is not what you wanted to read under DST tag.

So, DST becomes smaller, cleaner and simpler. Notably, I decided to drop userspace target completely for now.
Kernel part now operates on transaction entity, which holds a reference to the node, where data should be sent/received. There can be at most two such nodes if block IO request spans the boundary. In case of mirroring (which will be dropped for the first release) list of nodes to mirror this data to will be maintained by the first node, so transaction will not need to know about them.
In theory block request can be as much as BIO_MAX_PAGES pages, which is 256 for now, but I decided to limit minimum node size to be not smaller than above bio limit, so there will be always at most two nodes per request.
Each node has either block device behind it (so it will just call generic_make_request() with different block device for given bio), or network state machine.
Network state will have two threads: RX and TX. Receive one is used to get replies for the read/write messages, search appropriate transaction and complete it. In case of DST server it will also handle read/write requests and generate replies, but the whole processing will be exactly the same, client node will have a switch to process read/write requests from the network, but they should be only received by server.
Sending thread is tricky. It is used as fallback for non-blocking sockets, which are used first at generic_make_request() time, i.e. when higher level user performed read or write, if block was not fully sent, then it is queued to this thread and it will try to send the rest of the data when polling allows. ->make_request_fn() function returns in this case and higher layer can proceed with own operations.
Transaction is not freed until reply is received from the remote side or resending retry count fires.
Transaction is always allocated (from the appropriate memory pool) and that is actually all allocations in DST itself. In case it works with block devices, it is possible to clone a bio, when it crosses the boundaries (or even always, I have to check it, but it is essentially what device mapper with lots of own additional allocations), but it should be very rare condition.
Network stack will allocate data itself too.

That was a theory. Practice tells me, that essentially 90% of the code should be rewritten from scratch, so I recloned the tree and so far implemented generic bits of registering block device, creating various sysfs files and directories and other similar trivial bits. I still plan to finish it this weekend (without mirroring), but things may turn to me a different side though... Comments (0)

Evgeniy Polyakov: Have sent all documents for US visa.

2008-07-18T10:00:32+00:00

Checked my passports and decided that if other countries allowed to let me in with that photos, then US custom officers should not frown too much upon current ones.

So, waiting for the results. I almost sure that I will get visa and will met with interesting people at kernel summit and Plumbers conference, but anyway would like to draw the line.

For instance, Zach Brown will talk about CRFS (as long as show some chocolate and coctail bars around, imho the only good coctail is rum with cola (smaller colla) and ice), so there will be something to listen. Comments (0)

Dave Miller: TX multiqueue mostly done...

2008-07-18T08:13:00+00:00

Ok, it's in and commited to net-2.6 which will be pushed to Linus likely on Sunday. Relief is in sight.

Aurora Bridge in "photographer's light"

It seems the approach finally taken is for the most part, sound.

There are some synchronization issues Patrick McHardy noted but those all seem solvable at the moment.

What is really cool is that packet schedulers can be shared amongst TX queues of a device. So you can have things happen now such as starting with an enqueue of a packet for queue 0 and then when you run the qdisc what pops out is a packet for queue 4 :-)

This also means that non-default qdiscs can in fact change the SKB queue mapping if they like, and it would just work.

The sun is starting to rise, so, it's time for bed...

Harald Welte: Judge determines NXP has no right to prevent researchers from publicizing about MIFARE insecurity

2008-07-18T02:00:00+00:00

As reported here and here, NXP has apparently not been able to convince a Judge that the researchers at the University of Nijmegen should be restrained from publicizing about weaknesses in their MIFARE RFID products.

This is really good news. And it came so quick! Sometimes, I can still believe that there is some good left in this world. Almost too good to be true.

Evgeniy Polyakov: "The Gun Seller" by Hugh Laurie.

2008-07-17T18:00:39+00:00

Just finished to read this excellent detective novell (at Amazon, electronic version in russian).
People call it the best english humor novell for reason: it indeed is fun and interesting, although I suspect lots of its witty satire was a bit lost in translation, but nevertheless I do recommend it for easy reading.

And of course if you like House M.D., you have to read this novel, and you will not waste your time for sure. Comments (0)

Evgeniy Polyakov: Morning trumpet exercises.

2008-07-17T07:00:32+00:00

Today's morning I raped ears almost two hours, and at the end managed to play a chromatic scale (glide?) from second octave D (E trumpet) down to minor octave F (E trumpet), at least that is what my Korg tuner showed. Much more frequently I was able to play single first octave (only via descended direction though, I did not yet try to rise tones).
Korg AW1 tuner does not show octaves, but I really do not think, that it is possible to play one octave lower than what my the lowest sound was, but pretty sure it is possible to have at least one octave higher than my the highest tone, so I decided that I play several tones around first octave.

Ugh, it was supposed to move earlier to the office (before heat and traffic jams), but instead I fucked my brain via ears (and probably neighbours were not happy either, although I did not play on the full volume). Comments (0)

Zach Brown: propose a talk for LPC, come enjoy Portland

2008-07-16T21:49:29+00:00

This weekend marks the deadline for submitting speaker proposals for the Linux Plumbers Conf. I figure that CRFS falls under the category of future Linux storage so I submitted a proposal to talk about it.

If you have something that you’d like to discuss with your peers, and which falls under their broad list of categories, you should send in a submission.

If it’s accepted then you’ll get to come enjoy Portland with us! I have a secret plan that will make this sound like a lot more fun than it seems like it should at first glance. I want to try and provide some kind of list of the more interesting places to dine in Portland. I don’t know about you guys, but I’m pretty tired of the beer-and-mediocre-pizza meal that our community so often gravitates to.

Doesn’t that sound great? I sure hope so. Here, as a pair of teasers, are two places for after dinner treats: Cacao and Teardrop.

Evgeniy Polyakov: New toys: Korg AW1 tuner.

2008-07-16T17:00:37+00:00

I believe I can produce enough sounds out of my trumpet, so I need to have tuner, which will tell me how bad that sounds are.

So, now I'm starting to seriously tune my sounds.
So far I ~~hope~~ think that I can play at least two octaves, actually I mean not to play, but to produce a sound, it is still not that simple and not always very clean. But since I 'play' (or better say ~~rape ears~~ study) my trumpet only couple of months and never played any instrument (not counting couple guitar riffs in the university) before and do not play with a teacher, I think this tuner will be very good addition. Comments (4)

Pete Zaitcev: Livejournal spam

2008-07-16T02:02:13+00:00

There's so much LJ spam recently that I'm thinking about disabling comments. Comments are useless anyway. Anyone who has anything meaningful to tell me should be able to send an e-mail. Case in point, Ani-nouto never had comments, and it's doing great.

James Morris: FOSS Conference Observations

2008-07-16T01:17:42+00:00

Andrew Morton's slides from the LF Japan Symposium are now here (PDF), along with the rest of the presentations (scroll down). No video as yet, it seems.

I microblogged this at the time, but it's worth mentioning here that 15% of kernel contributions are now coming from Japan, as also noted at the LF blog. (I'm not sure if the media has picked up on this yet, but it was also announced at the Symposium that the 2009 Kernel Summit will be held in Tokyo).

Recently, Michael Chen from Red Hat was quoted in an interview that India is the third largest contributor to Fedora. That's "following North America and Europe", so I don't know what the per-country rankings are, but India may be doing even better in that respect.

It's probably impossible to say precisely what's driving these increasing global levels of FOSS contributions, but my impressions are that in these cases, at least, that conferences such as FOSS.IN and the LF symposia are having very significant effects.

It was also most likely similarly the case in Australia in 1999 with the introduction of CALU (which evolved into LCA), that local FOSS efforts were greatly stimulated by holding a technical conference which attracted a combination of leading international and local contributors. It was certainly a major factor in my own subsequent involvement in kernel hacking (which was at the time very sporadic, but became vastly more focused after meeting & seeing the likes of Dave Miller, Rusty and Tridge).

It's interesting to compare how this works differently in different regions. India and Australia developed their own grassroots conferences, while Japan (and recently China) have developed conferences based around industry consortia. Whether this is cultural or a sign of the times is unclear, but it seems the ultimate effect is basically the same. More people contributing—not just code—to the community process.

A comment made by Toshiharu Harada during his genuinely entertaining TOMOYO talk was particularly interesting:

Merging TOMOYO Linux started as our mission, but now they are our personal goals.

While FOSS has become increasingly mainstream, it seems that the underlying dynamics of the community also remain the same, in terms of people fulfilling personal goals (technical, social, economic), as well as those of their employers.

Evgeniy Polyakov: Distributed storage development roadmap.

2008-07-15T17:00:37+00:00

Yes, DST project is alive and will beat out the crap very soon, since I decided to change its underlying architecture, and switch to transaction model just like POHMELFS. This basically means that as long as system has enough RAM writing operations will be extremely fast, reading can be balanced between multiple nodes (in mirror), transactions can be resent, failover mechanism becomes much simpler, and system overall will be much more robust to failures.

Transaction model also means that system requires explicit acknowlege from remote side, and there are two possibilities here: two handle implicit ack which comes with TCP ack packets like I experimented before, and send explicit ack from server for each client's request.
\ The former approach although has smaller performance overhead, still suffers from the fact, that pages sent via DST are always stateless, i.e. at this layer there is no knowledge about who sends this page. We can determine inode page belongs to, can even get a socket when page is about to be released when ack has been received, but we can not know from exactly which PIPE it was submitted into given socket, so when multiple threads send the same page via miltiple sendfile() calls we do not know when and how page will be released. We can put pipes this page belong to into single-linked list (since page has only two unused at this point pointers: LRU list head, and one of them is used to determine that this page belongs to sendfile()/splice codepath), and likely traversing this list will not hurt usual users, but malicios one can create a local DoS with this approach. After some experiments with the splice code today I decided to drop this idea implementation for now.
There is a strong argument in favour of explicit acks from the server: this allows to make asynchronous transaction processing (with implicit acks we can not hook into processing path, since we do not know where exactly skb with our pages is chained), and this does not hurt perfromance (which was proven by POHMELFS benchmarks).

So, overall plan to develop DST is to switch to transaction model and perform async processing of all events (there are only two actually: reading and writing of the given pages to given locations).
This task is not that complex, so I expect some new results later this week. Stay tuned! Comments (5)

James Morris: SELinux and Security in the 2.6.26 Kernel

2008-07-15T11:39:35+00:00

What's new and exciting with SELinux and security in the new 2.6.26 kernel? Expanding some of the items from the excellent Kernel Newbies 2.6.26 page:

security= boot parameter

This patch by Ahmed Darwish allows a particular security module to be selected at kernel boot time, so that distributions can ship multiple security modules and allow the user to decide which one (if any) to enable. For example: security=selinux selects SELinux, while security=smack selects SMACK. (In Fedora, you don't need to do anything: SELinux is the default).

New SELinux open permission

Until now, opening a file under SELinux invoked the same permission checks as the intended operation on the file, such as read, write, execute and append. There was no separate "open" check: opening a file for write, for example, was considered by SELinux policy as equivalent to actually writing to the file. Experience has shown that this approach is not ideal for handling cases such as IO redirection via the shell, because policy writers cannot usefully guess where users will send redirected output. This is a very common use-case for Linux, so a solution is most definitely necessary, while also preserving strong security. Can it be done? Yes!

Implemented by Eric Paris, the new open permission provides a way to address the issue by providing applications with liberal access to read/write/execute/append permissions but tightly locking down the ability to open a file. In the case of redirecting output via the shell:

bash# /sbin/do-stuff > /tmp/output

the shell forks and creates /tmp/output, calls dup2(2) to replace stdin with the newly created file descriptor, then execs do-stuff. With the old permissions, do-stuff would have required an SELinux write permission on the new file, which it very likely would not have had. By providing do-stuff with liberal file access permissions, but not the new open permission, its output may now be redirected to the file without needing to give it the ability to directly open the file. The invoking shell of course needs the open permission, which it effectively delegates to do-stuff via the open file descriptor.

Updated security policy which utilizes this technique should be available soon in rawhide, and integrated into Fedora 10, providing significant usability improvements for sysadmins and power users.

Permissive Types

Also implemented by Eric, permissive types (aka permissive domains) allows permissive mode to be selected on the fly on a per-domain basis. Permissive mode is where security policy is being checked and logged, but not actually enforced, and was previously only possible on a system-wide basis. By making this per-domain, applications which are experiencing SELinux policy issues may be flipped into permissive mode, allowing them to do what they need until a proper fix is available, without disabling policy enforcement for the rest of the system.
Network Port SID Cache

Paul Moore implemented a cache to improve the performance of the SELinux networking code, so that network port labels are no longer looked up in the (typically large) kernel policy database on a per-packet basis, and is instead retrieved from an RCU-based cache. This addresses a long standing network performance issue which has been observed with very high loads on network servers.

There's quite a lot happening in security for 2.6.27, some of which has already been merged into Linus' tree. Due to the pervasive nature of some of the patches (including David Howells' credentials rework), I'm feeding all of the SELinux stuff via my security-testing tree. The "devel" branch is where bleeding edge changes are initially stabilized before being applied to the "next" branch, which is in turn fed into to linux-next.

Jesse Barnes: Prepping the PCI tree

2008-07-15T02:03:30+00:00

Figured I’d give an overview of the latest PCI stuff for those of you that don’t drink from the lkml firehose.

The PCI linux-next branch was a bit more exciting than I expected it to be. We’ve got lots of good changes queued up. Some of the highlights:

PCI slot detection driver (from Alex Chiang)
This driver exposes additional per-slot information that can help users identify where slots are physically located, making hotplug easier to deal with.
ROM allocation avoidance (Gary Hade)
With the “don’t allocate space for ROMs” patch reverted, lots of address space can be gobbled up by unnecessary expansion ROMs. To prevent this on large systems, Gary added an option eschew ROM allocation, so that machines not needing access to the ROMs can use more of their address space for important MMIO and I/O regions instead.
PCIe hotplug cleanups/fixes (Kenji Kaneshige)
Kenji spent a lot of time working on improving the PCIe and other hotplug drivers. Things should be more reliable and the code should be easier to follow now thanks to his efforts.
suspend/resume & wakeup enhancements (Rafael J. Wysocki)
Rafael coded up quite a few improvements to our suspend/resume infrastructure, and fixed up PCI/ACPI wakeup while he was at it. The improved wakeup code should work on more platforms and in more situations than the old code, but we still expect additional platform specific quirks and workarounds will be necessary, so testing in this area is welcome. But everyone’s already setting their systems to go to sleep automatically though, for power savings & general “green” goodness, right? These improvements should make things like wake-on-lan a bit more reliable, so if you’re not already in the green camp, please give these bits a try.

There’s also an assortment of fixes here; hopefully we haven’t broken anything too badly…

The bottom line is this though: if you’ve been hesitant to try suspend/resume with Linux, or have had bad wake-on-lan or other wakeup event experiences, or you use PCI hotplug at all, this is a good release for you to try. You can report bugs to linux-kernel@vger.kernel.org, linux-pci@vger.kernel.org and/or http://bugzilla.kernel.org and we’ll take a look!

Matthew Garrett

2008-07-15T01:15:57+00:00

Plans for the month:

18th-20th: Wolverhampton for Lugradio Live where I'll be doing some sort of presentation on some sort of power management stuff.
22nd-27th: Portland for OSCON and the Oregon Brewers Festival (which I guess settles any arguments about how to spell plumbers)
28th-30th: San Francisco for a meeting that's been moved to the week before I'm there
31st-4th: Boston in order to meet some of the people that I actually work with

Things not planned for the month:

Genocide
Puritanism
upskirt photos
Telling women who fail to cover their legs that they're asking for it

Rusty Russell: UNSW CS: Employment @ IBM OzLabs Talk: 1pm Tuesday September 2nd

2008-07-14T21:05:00+00:00

UNSW School of Computer Science and Engineering are having "Employer of the Week" experiment: September 1st is IBM's week. I'll be spruking for OzLabs, so if you know anyone at UNSW who worth talking to, drag them there (I don't know which room, I'm guessing the signs in CS will be pretty clear).

I'm going to try to talk about the stuff people in the office are hacking on, to give an idea what it's like being in what AFAICT is Australia's largest bunch of Free and Open Source Software hackers.

Evgeniy Polyakov: Football match has made the day!

2008-07-14T21:00:21+00:00

That was exceptionally bloody cool evening!
We had three teams of 6 playerrs in each and played on a small mini-football field about 2.5 hours, each match took either 7 minutes or 2 goals into single gates. It sucked power so much cool, that even exceptional tireness right now brings kind of masochistic pleasure.
My breathing system really sucks, and actually it is not a surprise, I did not play football more than 5 years already, but nevertheless shoes and ball are in a good shape.

I managed to damage knees, shoulder and fingers on the leg in various 'contacts' during the game, but that's not a problem.
Our team was not the best one really, but we strongly hold second place, and actually can fight for the first one, since all our players had long enough pauses in own games, while first team players regulary train in its own teams (including youth football champion).

That was the super time! Comments (0)

Dave Jones: Upstream participation. (or a failing thereof)

2008-07-14T15:17:30+00:00

"Look for example at the fact that Ubuntu has usually better hardware support, if we all were on the same kernel the others could take the drivers we put in there and have hardware support that is just as good as Ubuntu."

Does no-one else see the hypocrisy in this statement ? Here's how it reads to me... "It would be great if everyone just shipped the Ubuntu kernel and debugged the random crap we merge that we don't have the resources to do ourselves".

If only there were some kind of process of getting drivers merged upstream to kernel.org. Perhaps then we COULD be on the same kernel. Oh wait, there is a process. Ubuntu just chooses to ignore it.

This idea makes absolutely no sense whatsoever when a distro is patching the kernel to hell.

Having distros ship the same version of major components is utterly pointless unless everyone is on the same page, and stops making moronic decisions like "lets replace a major piece of security functionality with something else because the one upstream is complicated".

Evgeniy Polyakov: ParaLLels concert.

2008-07-14T11:00:15+00:00

Visited ParaLLels concert this weekend...
Mixed feelings, but saw lots of old friends (musicians by a coincidence), which made the day. Comments (0)

Pavel Machek: 82kg

2008-07-14T08:23:46+00:00

...that's me with all the stuff neccessary for 3 week trip into Altai mountains. 0.5kg of that are NiMH batteries.... So don't try to contact me in next 3 weeks, I'll be very unavailable.

Matthew Garrett: My trip to Istanbul by Matthew Garrett aged 28 and 4/365ths

2008-07-13T23:06:46+00:00

Saturday

Arrive at Heathrow. Richard gets to be my special lounge buddy - we later discover Bastien sitting outside looking like a lost puppy. Flight is delayed by a mere hour, so we get to the hotel with little trouble and check in. Dinner with Luis and co on pillows on the street. This becomes a recurring theme. Drinking is involved. This also becomes a recurring theme. People complain about my shortcut finding skills.

Sunday

See beautiful things. Thankful to finally get to see buildings older than my old college. Dinner under some bridge with Red Hat folks where we discover the joys of the "special price". People once more complain about my shortcut finding skills. Suspect pillows are involved, but hazy recollections.

Monday

Arrive at the university by taxi. More to the point, arrive at the university by taxi without taking a 30km detour. Dinner with Canonical people, then a party in some overpriced bar on the top of a very big hill. Discover that taxi drivers in Istanbul either have no fucking clue where anything in the entire world is, or that driving tourists to bizarre locations is a national pastime. Avoid this fate and arrive home at something like 4AM.

Tuesday

Corporate whoring continues with dinner with Collabora people (delayed by two and a half hours because Christian values getting sweaty with other men over eating, or something), followed by more drinks on the pillows. Am vindicated when it turns out that my route back to the hotel would in fact be the shortest possible route back if people didn't keep insisting on making random left turns when we're almost there.

Wednesday

Fascinating (and increasingly drunken) discussion of the neurobiological aspects of colour spaces with Pippin precedes drinks on the roof of the university. Narrowly prevent Luis' attempt to leave the foundation open to flagrant copyright violation suits. More pillows.

Thursday

Wake up with a twisted ankle and a broken laptop. I had a good birthday, it seems. Collabora host a party on a boat. Miraculously, nobody falls off the boat. We head back to the pillows while others trudge up the hill to Taksim. Aaron wins the inaugural Aaron Bockover award for services to the GNOME community.

Friday

Dinner with Novell, demonstrating my even-handedness when it comes to accepting corporate favours. Miguel reveals his true nature as a Microsoft shill by admitting to owning an XBox. I advise him to raise his free software credentials by getting a Freerunner - as a bonus it'll crash whenever anyone tries to call him, preventing Microsoft from whispering more sweet nothings in his ear. A foolproof plan. Party paid for by Google. Ear bleedingly loud. Some drinking involved. Return to bizarro deviant pillows where we are treated to the spectacle of child labour. Spend some time wondering why Planet Fedora is discussing upskirt photos and decide that hating the entire human race is probably the best option.

Saturday

Turns out that while the Grand Bazaar is effectively just a shopping centre, it's a fucking giant shopping centre that sells nothing I can find any excuse to want. Wander over to Asia for dinner. Conversation with Lennart about using cgroups as a mechanism for providing application latency requirements and how the current lack of standardisation of mountpoints and exposure of irritating implementation details makes it almost impossible to use them for anything. Fascinating diversion into how to deal with getting a SIGBUS when your mmap()ed soundcard gets hotswapped, including deciding that the easiest way of getting a reliable indication of your current process maps is by, erm, parsing /proc/self/maps. Final pillows trip. Emotional farewells (by which I mean more drinking)

Sunday

Airport. Plane. Train. Home. Transfer hard drive into laptop with working screen. Discover that productivity not actually enhanced by having more than a 300x200 pixel area of functionality. Decide to write libelous article about previous week instead.

Evgeniy Polyakov: Hermite interpolation.

2008-07-13T21:00:26+00:00

This interpolation uses cardinal splines approach, and namely Catmull-Rom splines. Next task is to test how the Kochanek-Bartels splines (also called TCB-splines) behave. The latter are used in all popular 3d modelling engines. Since math behind them is very non-trivial, I will try just to use existing formulas for hermite tangents, which are quite simple.

Now its time to think, how to use this knowledge and how to apply given approach to detect and decode letters on the image... Comments (0)

Evgeniy Polyakov: Monday evening prognosis.

2008-07-12T20:00:32+00:00

It promises to be just bloody excellent!

I did not play football several years already, but I've found people, who do like it, so our games promse to be really fun and interesing! There are already three commands (4+1 players in the team). This will be my first game after about 5 years of football silence, likely there is nothing in the legs which can help playing football, well climbing likely does not correlate with it, as long as so my experience in other physical trainings, but nevertheless I'm looking forward this ~~promised to be excelptionally cool~~ game! Comments (0)

Harald Welte: NXP sues security researchers studying Mifare classic

2008-07-12T02:00:00+00:00

In the last couple of days multiple reports stated that NXP has filed a lawsuit against security researchers from a Dutch university who were looking at security flaws of their proprietary MIFARE Classic products.

This is so ridiculous. I'm surprised that this still happens! We live in the 21st century, and IT security has become a well-established field within computer science. Furthermore, systems based on security by obscurity should be long gone.

So we have a company that in 1994 first ships a allegedly secure RFID technology. They developed a proprietary algorithm that did not receive public peer review in the cryptographic community, and used weak random number generation as well as made some mistakes in the protocol/system design. They ship this even back then questionable product without any fix/update for 14 years, irrespective the advances in technology and cryptographic research. During all that time, NXP marketing material claimed the product was fit for 'high security applications'.

Any reasonably skilled person in IT security could determine that the public statements "proprietary cipher" and "48 bit key length" did certainly not sound like high security at all. Thus, it's not surprising that in the last two years, some people, mostly friends of mine, started to look closer at what MIFARE classic is and what it does.

They should be honored and rewarded for their public service in demonstrating the irresponsible behavior of mostly NXP's customers (system integrators) and NXP itself. And exactly those companies are the ones that should be sued for continuing to milk a known-insecure cash cow for more than a decade.

I'd be more than happy to see somebody actually standing on their feet and demanding damages from those vendors. Imagine a small system integrator for a vertical market who wants to look for a secure/safe electronic wallet system and believes in the vendor promises. Now he gets defrauded because some criminal energy - not the ethical researchers at universities - exploit some weakness.

The only reason why large technology companies rarely get sued over the massive security problems they cause in their proprietary products is the fact that almost nobody (even the system integrators and developers) really understand that very technology enough. I sincerely hope that this changes at some point, and we see all those lame promises about alleged (but completely unverified) security go away.

If people would just use publicly disclosed, well-known, well-studied and well-analyzed cryptographic algorithms and implementations thereof, this world would be a much more secure place.

Evgeniy Polyakov: Spline graphical interpolation fun.

2008-07-11T19:00:24+00:00

Playing with different spline interpolation methods. So far they seems to be quite simple when written in matrix form, so I cooked up simple GTK application to test various methods.
There is no interpolation implementation yet, since I devoted last two days to read lots of materials about Bezier and Hermite interpolation techniques (as long as lots of papers about distributed hash tables, which I will use as a filesystem storage base for POHMELFS). Comments (6)

Dave Miller: TX multiqueue part two...

2008-07-10T07:21:00+00:00

Ok, progress.

I respun the remaining of the patches that aren't in net-next-2.6 yet and I even believe I sorted all of the wireless stack issues out.

So what's left?

Downtown St. Kilda, in Melbourne, Victoria, Australia

I have to finally sort out the qdisc configuration semantics. And this requires partly that I learn a thing or two about what was allowed in the uni-TX-queue state of affairs.

Right now one idea I have is to truly replicate. This means that each root qdisc on each TX queue is a perfect clone of the one on queue zero. Same ID, same everything.

And the same thing will be done when people graft new qdisc trees and classifiers onto the device.

I have a pretty solid idea how to implement this, but I am so lazy that I do not want to tackle all of the error unwinding. Specifically, doing an unwind when some queues perform the config change just fine but a subsequent TX queue fails. It certainly won't be fun to make that work.

Luna Park, also in Melbourne

Later on we can add a netlink attribute or similar to let the user operate on a specific TX queue in a qdisc config change. But I have a suspicion that we'll need to be careful about that.

It could get tricky if the user makes a specific TX queue config change, and then later we get an "anonymous" one that should be replicated.

I suppose the answer to that is that we merely follow the user's orders even if it might not make sense, and the user simply has to properly specify TX queues when appropriate.

Jesse Barnes: Fun in Folsom

2008-07-10T04:44:20+00:00

Just got back from a nice dinner & “gelato” (it was really ice cream) with Keith & Nanhai. We went to the Cliff House here in Folsom; we all ended up getting tasty steaks.

But back to the matter at hand. We’re all here at GfxCon 2008, an Intel event that brings together graphics architects and developers from across the company. There’s a lot of fun stuff going on, with some good sessions today and interesting demos (though of course I can’t talk about any of them, you’ll just have to wait for the products!).

The best part though has been the networking. It’s great to meet the developers I’ve only worked with over email, and have discussions face to face. Keith and I had some good brainstorming sessions about how to handle hardware contexts, and how we want to deal with vblank syncing in the GEM-enabled world. Fortunately, our discussions ended up sounding very similar to the discussions I had with krh back at XDC about vblank syncing in DRI2. Now I just have to code it up…

Hm, and let’s see how I’ve done keeping my promises from last week:

SSC detection/usage: still need review before pushing
TV detection: still need review before pushing
fbc fixes: pushed
memory arbitration: test patch attached to 16169
page flipping/buffer swap fixes: still working on these
PCI “what’s next” msg: still working on it

Hopefully I’ll be able to find time tomorrow with Nanhai or Keith to review and push some of the stuff above… Then on to hardware contexts.

Pete Zaitcev: Follow focus nazi

2008-07-09T23:23:04+00:00

I have a laptop with Synaptics touchpad, and one of the biggest annoyances in its implementation is how the "wheel" strips act upon the GUI element where the mouse is pointing regardless of the current focus. It took years to beat focus-follow-mouse people into a marginal submission, yet they still find every crack to seep through, like kerosene. Naturally, this behaviour is not tunable.

Another retardity in this area is the behaviour of the panel. Some genious came up with an awesome idea to flip windows if panel receives a wheel click. The problem with that is, a strip on the laptop is difficult to make to deliver just one click, so the function is useless to me. Also, it is too easy to hit the strip accidentially (which would not be a big deal if not for the enforced focus-follow). So, for one reason or the other, I would like to disable this behaviour. I don't even want to seek excuses for this wish. Just do it. But it's impossible.

So far the only group that displayed a clue in this was, strangely enough, Mozilla. They fixed bugs I filed and in general had an understanding of various input and display technologies, despite only being browser people. You'd think desktop developers would know how a touchpad differs from a wheel mouse.

UPDATE: Chris pointed out in comments that a fundamental reason exists why wheel behaves this way: the extra button events are mouse buttons, and those go where pointer points. I knew it, but it didn't click. I still think that applications should be able to work around this, although perhaps at the cost of some extra events being delivered.

Evgeniy Polyakov: Captcha transformation algorithms.

2008-07-09T13:00:32+00:00

Couple of first ideas. Pretty trivial.

Next step is to squize images, so that all bold lines moved to single-pixel ones. In theory it should not be very complex (I have an algorithm in mind), but in practice it will - starting to recall why in the hell I learnt LISP.
Basic idea is to transform above BW pictures into simple binary format, which will be read by LISP application, since I do not know and do not want to devote much time to learn how to parse/process various image formats, instead it is done by GTK application written in C. I belive LISP was called the best language for artificial intellengence development for reason, so will try to find why.

Slacking - rox :) Comments (4)

James Morris: Linux Foundation Japan Symposium Notes

2008-07-09T12:39:32+00:00

I presented on the SELinux project today at the Linux Foundation Japan Symposium in Tokyo. The slides from my talk may be downloaded here.

It's been an interesting conference, with some smaller BoF sessions planned for tomorrow. I live micro-blogged the conference via my identi.ca account, which I guess turned out as a kind of public note-taking.

Andrew Morton covered quite a lot of interesting kernel process material, highlighting some areas which we need to address (such as whether we're ready at all to support solid state disks), and explaining his view of the linux-next tree, one unpublished purpose of which was to get kernel hackers to test each others code before upstream merge. He also said that around 15% of kernel contributions are now coming from Japan.

Greg DeKoenigsberg kindly shipped a pile of Fedora DVDs and Live CDs across to give to the attendees. The CDs & DVDs proved very popular and were all distributed.

More photos here.