android upstreaming: Lowmem

Meta:
Headline: Upstream Android low memory killer functionality
Acceptance: Android applications can run on a kernel.org kernel with fully functioning low memory killer APIs. Changes to Android libraries to use new kernel interface are accepted into AOSP.
Roadmap id: KWG2011-ANDROID-OOM-KILLLER-UPSTREAMING

[jakub-pavelek 2012-09-27] Acceptance criteria - can I see to which git tree this shall be merged into when done? We do not have control over Android libs to make them use this - should we move that criterion to separate blueprint/card?

Anton's patch has been merged upstream w/ 3.10-rc1:
http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=70ddf637eebe47e61fb2be08a59315581b6d2f38

Work items for 11.12:
Deep dive study of the Android lowmem code to fully understand requirements: DONE
Address 'easy to do' comments on the previous lowmem driver submission: DONE

Work items for 12.01:
Send a new driver w/ a cover letter explaining alternatives. This is mostly to start a new discussion, we don't expect the driver to be merged as is: DONE
Get feedback on LMK driver, discuss - (Personally, I've made my mind regarding lowmemory killer, so unless there are new arguments for it, there is no point in discussing it any further. We will improve current lowmemorykiller.c driver as we can, but long-term plan is the memory notifiers approach): DONE
Synchronize Android's and upstream LMK drivers - (sent an update to Greg KH): DONE
Get rid of tasklist lock in the current LMK driver, post a patch for review: DONE

Work items for 12.02:
Prepare proof-of-concept code for Android that would use one of memory notifications mechanism, post for review: DONE
Discuss various methods of memory pressure notifications: DONE

Work items for 12.03:
Make ulmkd buildable on ARM using Android NDK: DONE
Run and test userspace uLMKd on an Android image: DONE

Work items for 12.04:
Add mlockall: DONE
Write up initial mail to send to the Google Android developers: DONE
Develop patch to add two-way notification vmevents (cgroups already does two-way notifications): DONE
Discuss userspace LMK code with Android folks: DONE
Rework makefile so it builds on both Android NDK and normal userland: DONE
Look at adding a kernel interface to decide if we want file-cache data counted when we do cgroup and vmevent based low-memory notifications: DONE

Work items for 12.05:
Address easy to do (non-fundamental) comments on vmevents, based on the latest community discussion: DONE

Work items for 12.08:
Check/evaluate current state of cgroups slab memory accounting fixes, helps task selection (the item was stalled, no one cares in the community?): DONE
Announce ulmkd in LKML (Cc Samsung/Tizen, Nokia people; The item is doable but unsure about if it is appropriate nowadays, since the whole LMK idea is in flux): DONE

Work items for 12.09:
Add experimental support for deferrable timers, send RFC: DONE

Work items for 12.10:
Add shrinker notifier to vmevent (this includes Mel's idea): DONE
Submit and discuss the new API: DONE
Add shrinker notifier support for ulmkd: DONE
Submit v2 of the vmevent pressure factor API: DONE
Modify vmevent to forcibly update global vmstats (there seems to be a way to make it w/o IPI storm, although stats may be somewhat inaccurate): DONE
Convert vmevents API to use the new vmstat interface: DONE
Consider ways to add memcg/numa/zones support for vmevent (seems like mm folks insist on adding it before merging vmevent?) (shrinkers seem to be per-cgroup -- this might be a solution): DONE
Add a complete man-page documentation for vmevent_fd: DONE

Work items for 12.11:
Slim down the vmevent API, get rid of raw nr_pages attributes (Nov 26): DONE
Discuss all the cgroups-specific work (Nov 26): DONE
Validate/fix stock OOM killer for Android usage (according to Android folks patch exists and posted somewhere) (Nov 26): DONE
vmeventfd/vmstat accuracy improvements (fixes KOSAKI's comments) (Nov 26): DONE

Work items for 12.12:
Create mempressure cgroup per David Rientjes suggestions (this also involves making vmpressure calculations per-thread): DONE
Develop feedback cycle interface per akpm's & sent to lkml: DONE
Got some interest in using the new interface from kvm developer for automatic memory ballooning: DONE

Work items for 13.01:
Anton sent out non RFC version of mempressure cgroup patchset, got ack from Kirill A. Shutemov (Jan 4): DONE
Anton's work was written up in an lwn article https://lwn.net/Articles/531077/ (Jan3): DONE

Work items for 13.02:
Anton sent out new version of mempressure cgroup patchset, got further feedback from Glauber: DONE

Work items for 13.03:
Anton sent out another version (v3) of the mempressure cgroup patchset. Akpm reviewed and gave feedback on needed modifications, but it looks like its progressing well. Also KAMEZAWA Hiroyuki reviewed and acked the patch (March 26): DONE

Work items for 13.04:
Anton sent out v4 of his patch, addressing issues from akpm and others (April 2nd): DONE
Anton's v4 patch was added to akpm's -mm tree!!!! (April 11): DONE
Wait and see if akpm pushes the patch when the 3.10 merge window opens (akpm submitted the patch to linus as of 4/29): DONE

Work items for 13.05:
Review backlog items & sort out which are actually still needed: DONE
Sent mail to Colin and Android devs to see what they are thinking about this functionality (June 4): DONE
Adjust ulmkd to use new interface: TODO

Work items for backlog:
Switch vmeventfd API to eventfd interface, make the API very similar to cgroups, but available w/o actual CGROUPS=y: TODO
Develop and integrage unit test (once API is settled): TODO
Integrate ulmkd into Android and get ulmkd service accepted into AOSP (same as above -- might be not needed): TODO
Modify Android Activity Manager (Java part) to use early low memory notifications, effectively putting low memory killer into Java part of Android (should be started once we have some API accepted in mainline, ulmkd can be used as proof-of-concept meanwhile): TODO