From: Paul E. M. <pa...@li...> - 2008-05-16 19:10:35
|
On Fri, May 09, 2008 at 09:32:30PM +0200, Andrea Arcangeli wrote: > From: Andrea Arcangeli <an...@qu...> The hlist_del_init_rcu() primitive looks good. The rest of the RCU code looks fine assuming that "mn->ops->release()" either does call_rcu() to defer actual removal, or that the actual removal is deferred until after mmu_notifier_release() returns. Acked-by: Paul E. McKenney <pa...@li...> > With KVM/GFP/XPMEM there isn't just the primary CPU MMU pointing to > pages. There are secondary MMUs (with secondary sptes and secondary > tlbs) too. sptes in the kvm case are shadow pagetables, but when I say > spte in mmu-notifier context, I mean "secondary pte". In GRU case > there's no actual secondary pte and there's only a secondary tlb > because the GRU secondary MMU has no knowledge about sptes and every > secondary tlb miss event in the MMU always generates a page fault that > has to be resolved by the CPU (this is not the case of KVM where the a > secondary tlb miss will walk sptes in hardware and it will refill the > secondary tlb transparently to software if the corresponding spte is > present). The same way zap_page_range has to invalidate the pte before > freeing the page, the spte (and secondary tlb) must also be > invalidated before any page is freed and reused. > > Currently we take a page_count pin on every page mapped by sptes, but > that means the pages can't be swapped whenever they're mapped by any > spte because they're part of the guest working set. Furthermore a spte > unmap event can immediately lead to a page to be freed when the pin is > released (so requiring the same complex and relatively slow tlb_gather > smp safe logic we have in zap_page_range and that can be avoided > completely if the spte unmap event doesn't require an unpin of the > page previously mapped in the secondary MMU). > > The mmu notifiers allow kvm/GRU/XPMEM to attach to the tsk->mm and > know when the VM is swapping or freeing or doing anything on the > primary MMU so that the secondary MMU code can drop sptes before the > pages are freed, avoiding all page pinning and allowing 100% reliable > swapping of guest physical address space. Furthermore it avoids the > code that teardown the mappings of the secondary MMU, to implement a > logic like tlb_gather in zap_page_range that would require many IPI to > flush other cpu tlbs, for each fixed number of spte unmapped. > > To make an example: if what happens on the primary MMU is a protection > downgrade (from writeable to wrprotect) the secondary MMU mappings > will be invalidated, and the next secondary-mmu-page-fault will call > get_user_pages and trigger a do_wp_page through get_user_pages if it > called get_user_pages with write=1, and it'll re-establishing an > updated spte or secondary-tlb-mapping on the copied page. Or it will > setup a readonly spte or readonly tlb mapping if it's a guest-read, if > it calls get_user_pages with write=0. This is just an example. > > This allows to map any page pointed by any pte (and in turn visible in > the primary CPU MMU), into a secondary MMU (be it a pure tlb like GRU, > or an full MMU with both sptes and secondary-tlb like the > shadow-pagetable layer with kvm), or a remote DMA in software like > XPMEM (hence needing of schedule in XPMEM code to send the invalidate > to the remote node, while no need to schedule in kvm/gru as it's an > immediate event like invalidating primary-mmu pte). > > At least for KVM without this patch it's impossible to swap guests > reliably. And having this feature and removing the page pin allows > several other optimizations that simplify life considerably. > > Dependencies: > > 1) Introduces list_del_init_rcu and documents it (fixes a comment for > list_del_rcu too) > > 2) mm_take_all_locks() to register the mmu notifier when the whole VM > isn't doing anything with "mm". This allows mmu notifier users to > keep track if the VM is in the middle of the > invalidate_range_begin/end critical section with an atomic counter > incraese in range_begin and decreased in range_end. No secondary > MMU page fault is allowed to map any spte or secondary tlb > reference, while the VM is in the middle of range_begin/end as any > page returned by get_user_pages in that critical section could > later immediately be freed without any further ->invalidate_page > notification (invalidate_range_begin/end works on ranges and > ->invalidate_page isn't called immediately before freeing the > page). To stop all page freeing and pagetable overwrites the > mmap_sem must be taken in write mode and all other anon_vma/i_mmap > locks must be taken too. > > 3) It'd be a waste to add branches in the VM if nobody could possibly > run KVM/GRU/XPMEM on the kernel, so mmu notifiers will only enabled > if CONFIG_KVM=m/y. In the current kernel kvm won't yet take > advantage of mmu notifiers, but this already allows to compile a > KVM external module against a kernel with mmu notifiers enabled and > from the next pull from kvm.git we'll start using them. And > GRU/XPMEM will also be able to continue the development by enabling > KVM=m in their config, until they submit all GRU/XPMEM GPLv2 code > to the mainline kernel. Then they can also enable MMU_NOTIFIERS in > the same way KVM does it (even if KVM=n). This guarantees nobody > selects MMU_NOTIFIER=y if KVM and GRU and XPMEM are all =n. > > The mmu_notifier_register call can fail because mm_take_all_locks may > be interrupted by a signal and return -EINTR. Because > mmu_notifier_reigster is used when a driver startup, a failure can be > gracefully handled. Here an example of the change applied to kvm to > register the mmu notifiers. Usually when a driver startups other > allocations are required anyway and -ENOMEM failure paths exists > already. > > struct kvm *kvm_arch_create_vm(void) > { > struct kvm *kvm = kzalloc(sizeof(struct kvm), GFP_KERNEL); > + int err; > > if (!kvm) > return ERR_PTR(-ENOMEM); > > INIT_LIST_HEAD(&kvm->arch.active_mmu_pages); > > + kvm->arch.mmu_notifier.ops = &kvm_mmu_notifier_ops; > + err = mmu_notifier_register(&kvm->arch.mmu_notifier, current->mm); > + if (err) { > + kfree(kvm); > + return ERR_PTR(err); > + } > + > return kvm; > } > > mmu_notifier_unregister returns void and it's reliable. > > Signed-off-by: Andrea Arcangeli <an...@qu...> > Signed-off-by: Nick Piggin <np...@su...> > Signed-off-by: Christoph Lameter <cla...@sg...> > --- > > Full patchset is here: > > http://www.kernel.org/pub/linux/kernel/people/andrea/patches/v2.6/2.6.26-rc1/mmu-notifier-v17 > > Thanks! > > diff --git a/arch/x86/kvm/Kconfig b/arch/x86/kvm/Kconfig > --- a/arch/x86/kvm/Kconfig > +++ b/arch/x86/kvm/Kconfig > @@ -21,6 +21,7 @@ config KVM > tristate "Kernel-based Virtual Machine (KVM) support" > depends on HAVE_KVM > select PREEMPT_NOTIFIERS > + select MMU_NOTIFIER > select ANON_INODES > ---help--- > Support hosting fully virtualized guest machines using hardware > diff --git a/include/linux/list.h b/include/linux/list.h > --- a/include/linux/list.h > +++ b/include/linux/list.h > @@ -747,7 +747,7 @@ static inline void hlist_del(struct hlis > * or hlist_del_rcu(), running on this same list. > * However, it is perfectly legal to run concurrently with > * the _rcu list-traversal primitives, such as > - * hlist_for_each_entry(). > + * hlist_for_each_entry_rcu(). > */ > static inline void hlist_del_rcu(struct hlist_node *n) > { > @@ -760,6 +760,34 @@ static inline void hlist_del_init(struct > if (!hlist_unhashed(n)) { > __hlist_del(n); > INIT_HLIST_NODE(n); > + } > +} > + > +/** > + * hlist_del_init_rcu - deletes entry from hash list with re-initialization > + * @n: the element to delete from the hash list. > + * > + * Note: list_unhashed() on the node return true after this. It is > + * useful for RCU based read lockfree traversal if the writer side > + * must know if the list entry is still hashed or already unhashed. > + * > + * In particular, it means that we can not poison the forward pointers > + * that may still be used for walking the hash list and we can only > + * zero the pprev pointer so list_unhashed() will return true after > + * this. > + * > + * The caller must take whatever precautions are necessary (such as > + * holding appropriate locks) to avoid racing with another > + * list-mutation primitive, such as hlist_add_head_rcu() or > + * hlist_del_rcu(), running on this same list. However, it is > + * perfectly legal to run concurrently with the _rcu list-traversal > + * primitives, such as hlist_for_each_entry_rcu(). > + */ > +static inline void hlist_del_init_rcu(struct hlist_node *n) > +{ > + if (!hlist_unhashed(n)) { > + __hlist_del(n); > + n->pprev = NULL; > } > } > > diff --git a/include/linux/mm.h b/include/linux/mm.h > --- a/include/linux/mm.h > +++ b/include/linux/mm.h > @@ -1067,6 +1067,9 @@ extern struct vm_area_struct *copy_vma(s > unsigned long addr, unsigned long len, pgoff_t pgoff); > extern void exit_mmap(struct mm_struct *); > > +extern int mm_take_all_locks(struct mm_struct *mm); > +extern void mm_drop_all_locks(struct mm_struct *mm); > + > #ifdef CONFIG_PROC_FS > /* From fs/proc/base.c. callers must _not_ hold the mm's exe_file_lock */ > extern void added_exe_file_vma(struct mm_struct *mm); > diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h > --- a/include/linux/mm_types.h > +++ b/include/linux/mm_types.h > @@ -10,6 +10,7 @@ > #include <linux/rbtree.h> > #include <linux/rwsem.h> > #include <linux/completion.h> > +#include <linux/cpumask.h> > #include <asm/page.h> > #include <asm/mmu.h> > > @@ -19,6 +20,7 @@ > #define AT_VECTOR_SIZE (2*(AT_VECTOR_SIZE_ARCH + AT_VECTOR_SIZE_BASE + 1)) > > struct address_space; > +struct mmu_notifier_mm; > > #if NR_CPUS >= CONFIG_SPLIT_PTLOCK_CPUS > typedef atomic_long_t mm_counter_t; > @@ -235,6 +237,9 @@ struct mm_struct { > struct file *exe_file; > unsigned long num_exe_file_vmas; > #endif > +#ifdef CONFIG_MMU_NOTIFIER > + struct mmu_notifier_mm *mmu_notifier_mm; > +#endif > }; > > #endif /* _LINUX_MM_TYPES_H */ > diff --git a/include/linux/mmu_notifier.h b/include/linux/mmu_notifier.h > new file mode 100644 > --- /dev/null > +++ b/include/linux/mmu_notifier.h > @@ -0,0 +1,279 @@ > +#ifndef _LINUX_MMU_NOTIFIER_H > +#define _LINUX_MMU_NOTIFIER_H > + > +#include <linux/list.h> > +#include <linux/spinlock.h> > +#include <linux/mm_types.h> > + > +struct mmu_notifier; > +struct mmu_notifier_ops; > + > +#ifdef CONFIG_MMU_NOTIFIER > + > +/* > + * The mmu notifier_mm structure is allocated and installed in > + * mm->mmu_notifier_mm inside the mm_take_all_locks() protected > + * critical section and it's released only when mm_count reaches zero > + * in mmdrop(). > + */ > +struct mmu_notifier_mm { > + /* all mmu notifiers registerd in this mm are queued in this list */ > + struct hlist_head list; > + /* to serialize the list modifications and hlist_unhashed */ > + spinlock_t lock; > +}; > + > +struct mmu_notifier_ops { > + /* > + * Called either by mmu_notifier_unregister or when the mm is > + * being destroyed by exit_mmap, always before all pages are > + * freed. This can run concurrently with other mmu notifier > + * methods (the ones invoked outside the mm context) and it > + * should tear down all secondary mmu mappings and freeze the > + * secondary mmu. If this method isn't implemented you've to > + * be sure that nothing could possibly write to the pages > + * through the secondary mmu by the time the last thread with > + * tsk->mm == mm exits. > + * > + * As side note: the pages freed after ->release returns could > + * be immediately reallocated by the gart at an alias physical > + * address with a different cache model, so if ->release isn't > + * implemented because all _software_ driven memory accesses > + * through the secondary mmu are terminated by the time the > + * last thread of this mm quits, you've also to be sure that > + * speculative _hardware_ operations can't allocate dirty > + * cachelines in the cpu that could not be snooped and made > + * coherent with the other read and write operations happening > + * through the gart alias address, so leading to memory > + * corruption. > + */ > + void (*release)(struct mmu_notifier *mn, > + struct mm_struct *mm); > + > + /* > + * clear_flush_young is called after the VM is > + * test-and-clearing the young/accessed bitflag in the > + * pte. This way the VM will provide proper aging to the > + * accesses to the page through the secondary MMUs and not > + * only to the ones through the Linux pte. > + */ > + int (*clear_flush_young)(struct mmu_notifier *mn, > + struct mm_struct *mm, > + unsigned long address); > + > + /* > + * Before this is invoked any secondary MMU is still ok to > + * read/write to the page previously pointed to by the Linux > + * pte because the page hasn't been freed yet and it won't be > + * freed until this returns. If required set_page_dirty has to > + * be called internally to this method. > + */ > + void (*invalidate_page)(struct mmu_notifier *mn, > + struct mm_struct *mm, > + unsigned long address); > + > + /* > + * invalidate_range_start() and invalidate_range_end() must be > + * paired and are called only when the mmap_sem and/or the > + * locks protecting the reverse maps are held. The subsystem > + * must guarantee that no additional references are taken to > + * the pages in the range established between the call to > + * invalidate_range_start() and the matching call to > + * invalidate_range_end(). > + * > + * Invalidation of multiple concurrent ranges may be > + * optionally permitted by the driver. Either way the > + * establishment of sptes is forbidden in the range passed to > + * invalidate_range_begin/end for the whole duration of the > + * invalidate_range_begin/end critical section. > + * > + * invalidate_range_start() is called when all pages in the > + * range are still mapped and have at least a refcount of one. > + * > + * invalidate_range_end() is called when all pages in the > + * range have been unmapped and the pages have been freed by > + * the VM. > + * > + * The VM will remove the page table entries and potentially > + * the page between invalidate_range_start() and > + * invalidate_range_end(). If the page must not be freed > + * because of pending I/O or other circumstances then the > + * invalidate_range_start() callback (or the initial mapping > + * by the driver) must make sure that the refcount is kept > + * elevated. > + * > + * If the driver increases the refcount when the pages are > + * initially mapped into an address space then either > + * invalidate_range_start() or invalidate_range_end() may > + * decrease the refcount. If the refcount is decreased on > + * invalidate_range_start() then the VM can free pages as page > + * table entries are removed. If the refcount is only > + * droppped on invalidate_range_end() then the driver itself > + * will drop the last refcount but it must take care to flush > + * any secondary tlb before doing the final free on the > + * page. Pages will no longer be referenced by the linux > + * address space but may still be referenced by sptes until > + * the last refcount is dropped. > + */ > + void (*invalidate_range_start)(struct mmu_notifier *mn, > + struct mm_struct *mm, > + unsigned long start, unsigned long end); > + void (*invalidate_range_end)(struct mmu_notifier *mn, > + struct mm_struct *mm, > + unsigned long start, unsigned long end); > +}; > + > +/* > + * The notifier chains are protected by mmap_sem and/or the reverse map > + * semaphores. Notifier chains are only changed when all reverse maps and > + * the mmap_sem locks are taken. > + * > + * Therefore notifier chains can only be traversed when either > + * > + * 1. mmap_sem is held. > + * 2. One of the reverse map locks is held (i_mmap_lock or anon_vma->lock). > + * 3. No other concurrent thread can access the list (release) > + */ > +struct mmu_notifier { > + struct hlist_node hlist; > + const struct mmu_notifier_ops *ops; > +}; > + > +static inline int mm_has_notifiers(struct mm_struct *mm) > +{ > + return unlikely(mm->mmu_notifier_mm); > +} > + > +extern int mmu_notifier_register(struct mmu_notifier *mn, > + struct mm_struct *mm); > +extern int __mmu_notifier_register(struct mmu_notifier *mn, > + struct mm_struct *mm); > +extern void mmu_notifier_unregister(struct mmu_notifier *mn, > + struct mm_struct *mm); > +extern void __mmu_notifier_mm_destroy(struct mm_struct *mm); > +extern void __mmu_notifier_release(struct mm_struct *mm); > +extern int __mmu_notifier_clear_flush_young(struct mm_struct *mm, > + unsigned long address); > +extern void __mmu_notifier_invalidate_page(struct mm_struct *mm, > + unsigned long address); > +extern void __mmu_notifier_invalidate_range_start(struct mm_struct *mm, > + unsigned long start, unsigned long end); > +extern void __mmu_notifier_invalidate_range_end(struct mm_struct *mm, > + unsigned long start, unsigned long end); > + > +static inline void mmu_notifier_release(struct mm_struct *mm) > +{ > + if (mm_has_notifiers(mm)) > + __mmu_notifier_release(mm); > +} > + > +static inline int mmu_notifier_clear_flush_young(struct mm_struct *mm, > + unsigned long address) > +{ > + if (mm_has_notifiers(mm)) > + return __mmu_notifier_clear_flush_young(mm, address); > + return 0; > +} > + > +static inline void mmu_notifier_invalidate_page(struct mm_struct *mm, > + unsigned long address) > +{ > + if (mm_has_notifiers(mm)) > + __mmu_notifier_invalidate_page(mm, address); > +} > + > +static inline void mmu_notifier_invalidate_range_start(struct mm_struct *mm, > + unsigned long start, unsigned long end) > +{ > + if (mm_has_notifiers(mm)) > + __mmu_notifier_invalidate_range_start(mm, start, end); > +} > + > +static inline void mmu_notifier_invalidate_range_end(struct mm_struct *mm, > + unsigned long start, unsigned long end) > +{ > + if (mm_has_notifiers(mm)) > + __mmu_notifier_invalidate_range_end(mm, start, end); > +} > + > +static inline void mmu_notifier_mm_init(struct mm_struct *mm) > +{ > + mm->mmu_notifier_mm = NULL; > +} > + > +static inline void mmu_notifier_mm_destroy(struct mm_struct *mm) > +{ > + if (mm_has_notifiers(mm)) > + __mmu_notifier_mm_destroy(mm); > +} > + > +/* > + * These two macros will sometime replace ptep_clear_flush. > + * ptep_clear_flush is impleemnted as macro itself, so this also is > + * implemented as a macro until ptep_clear_flush will converted to an > + * inline function, to diminish the risk of compilation failure. The > + * invalidate_page method over time can be moved outside the PT lock > + * and these two macros can be later removed. > + */ > +#define ptep_clear_flush_notify(__vma, __address, __ptep) \ > +({ \ > + pte_t __pte; \ > + struct vm_area_struct *___vma = __vma; \ > + unsigned long ___address = __address; \ > + __pte = ptep_clear_flush(___vma, ___address, __ptep); \ > + mmu_notifier_invalidate_page(___vma->vm_mm, ___address); \ > + __pte; \ > +}) > + > +#define ptep_clear_flush_young_notify(__vma, __address, __ptep) \ > +({ \ > + int __young; \ > + struct vm_area_struct *___vma = __vma; \ > + unsigned long ___address = __address; \ > + __young = ptep_clear_flush_young(___vma, ___address, __ptep); \ > + __young |= mmu_notifier_clear_flush_young(___vma->vm_mm, \ > + ___address); \ > + __young; \ > +}) > + > +#else /* CONFIG_MMU_NOTIFIER */ > + > +static inline void mmu_notifier_release(struct mm_struct *mm) > +{ > +} > + > +static inline int mmu_notifier_clear_flush_young(struct mm_struct *mm, > + unsigned long address) > +{ > + return 0; > +} > + > +static inline void mmu_notifier_invalidate_page(struct mm_struct *mm, > + unsigned long address) > +{ > +} > + > +static inline void mmu_notifier_invalidate_range_start(struct mm_struct *mm, > + unsigned long start, unsigned long end) > +{ > +} > + > +static inline void mmu_notifier_invalidate_range_end(struct mm_struct *mm, > + unsigned long start, unsigned long end) > +{ > +} > + > +static inline void mmu_notifier_mm_init(struct mm_struct *mm) > +{ > +} > + > +static inline void mmu_notifier_mm_destroy(struct mm_struct *mm) > +{ > +} > + > +#define ptep_clear_flush_young_notify ptep_clear_flush_young > +#define ptep_clear_flush_notify ptep_clear_flush > + > +#endif /* CONFIG_MMU_NOTIFIER */ > + > +#endif /* _LINUX_MMU_NOTIFIER_H */ > diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h > --- a/include/linux/pagemap.h > +++ b/include/linux/pagemap.h > @@ -19,6 +19,7 @@ > */ > #define AS_EIO (__GFP_BITS_SHIFT + 0) /* IO error on async write */ > #define AS_ENOSPC (__GFP_BITS_SHIFT + 1) /* ENOSPC on async write */ > +#define AS_MM_ALL_LOCKS (__GFP_BITS_SHIFT + 2) /* under mm_take_all_locks() */ > > static inline void mapping_set_error(struct address_space *mapping, int error) > { > diff --git a/include/linux/rmap.h b/include/linux/rmap.h > --- a/include/linux/rmap.h > +++ b/include/linux/rmap.h > @@ -26,6 +26,14 @@ > */ > struct anon_vma { > spinlock_t lock; /* Serialize access to vma list */ > + /* > + * NOTE: the LSB of the head.next is set by > + * mm_take_all_locks() _after_ taking the above lock. So the > + * head must only be read/written after taking the above lock > + * to be sure to see a valid next pointer. The LSB bit itself > + * is serialized by a system wide lock only visible to > + * mm_take_all_locks() (mm_all_locks_mutex). > + */ > struct list_head head; /* List of private "related" vmas */ > }; > > diff --git a/kernel/fork.c b/kernel/fork.c > --- a/kernel/fork.c > +++ b/kernel/fork.c > @@ -54,6 +54,7 @@ > #include <linux/tty.h> > #include <linux/proc_fs.h> > #include <linux/blkdev.h> > +#include <linux/mmu_notifier.h> > > #include <asm/pgtable.h> > #include <asm/pgalloc.h> > @@ -386,6 +387,7 @@ static struct mm_struct * mm_init(struct > > if (likely(!mm_alloc_pgd(mm))) { > mm->def_flags = 0; > + mmu_notifier_mm_init(mm); > return mm; > } > > @@ -418,6 +420,7 @@ void __mmdrop(struct mm_struct *mm) > BUG_ON(mm == &init_mm); > mm_free_pgd(mm); > destroy_context(mm); > + mmu_notifier_mm_destroy(mm); > free_mm(mm); > } > EXPORT_SYMBOL_GPL(__mmdrop); > diff --git a/mm/Kconfig b/mm/Kconfig > --- a/mm/Kconfig > +++ b/mm/Kconfig > @@ -205,3 +205,6 @@ config VIRT_TO_BUS > config VIRT_TO_BUS > def_bool y > depends on !ARCH_NO_VIRT_TO_BUS > + > +config MMU_NOTIFIER > + bool > diff --git a/mm/Makefile b/mm/Makefile > --- a/mm/Makefile > +++ b/mm/Makefile > @@ -33,4 +33,5 @@ obj-$(CONFIG_SMP) += allocpercpu.o > obj-$(CONFIG_SMP) += allocpercpu.o > obj-$(CONFIG_QUICKLIST) += quicklist.o > obj-$(CONFIG_CGROUP_MEM_RES_CTLR) += memcontrol.o > +obj-$(CONFIG_MMU_NOTIFIER) += mmu_notifier.o > > diff --git a/mm/filemap_xip.c b/mm/filemap_xip.c > --- a/mm/filemap_xip.c > +++ b/mm/filemap_xip.c > @@ -188,7 +188,7 @@ __xip_unmap (struct address_space * mapp > if (pte) { > /* Nuke the page table entry. */ > flush_cache_page(vma, address, pte_pfn(*pte)); > - pteval = ptep_clear_flush(vma, address, pte); > + pteval = ptep_clear_flush_notify(vma, address, pte); > page_remove_rmap(page, vma); > dec_mm_counter(mm, file_rss); > BUG_ON(pte_dirty(pteval)); > diff --git a/mm/fremap.c b/mm/fremap.c > --- a/mm/fremap.c > +++ b/mm/fremap.c > @@ -15,6 +15,7 @@ > #include <linux/rmap.h> > #include <linux/module.h> > #include <linux/syscalls.h> > +#include <linux/mmu_notifier.h> > > #include <asm/mmu_context.h> > #include <asm/cacheflush.h> > @@ -214,7 +215,9 @@ asmlinkage long sys_remap_file_pages(uns > spin_unlock(&mapping->i_mmap_lock); > } > > + mmu_notifier_invalidate_range_start(mm, start, start + size); > err = populate_range(mm, vma, start, size, pgoff); > + mmu_notifier_invalidate_range_end(mm, start, start + size); > if (!err && !(flags & MAP_NONBLOCK)) { > if (unlikely(has_write_lock)) { > downgrade_write(&mm->mmap_sem); > diff --git a/mm/hugetlb.c b/mm/hugetlb.c > --- a/mm/hugetlb.c > +++ b/mm/hugetlb.c > @@ -14,6 +14,7 @@ > #include <linux/mempolicy.h> > #include <linux/cpuset.h> > #include <linux/mutex.h> > +#include <linux/mmu_notifier.h> > > #include <asm/page.h> > #include <asm/pgtable.h> > @@ -823,6 +824,7 @@ void __unmap_hugepage_range(struct vm_ar > BUG_ON(start & ~HPAGE_MASK); > BUG_ON(end & ~HPAGE_MASK); > > + mmu_notifier_invalidate_range_start(mm, start, end); > spin_lock(&mm->page_table_lock); > for (address = start; address < end; address += HPAGE_SIZE) { > ptep = huge_pte_offset(mm, address); > @@ -843,6 +845,7 @@ void __unmap_hugepage_range(struct vm_ar > } > spin_unlock(&mm->page_table_lock); > flush_tlb_range(vma, start, end); > + mmu_notifier_invalidate_range_end(mm, start, end); > list_for_each_entry_safe(page, tmp, &page_list, lru) { > list_del(&page->lru); > put_page(page); > diff --git a/mm/memory.c b/mm/memory.c > --- a/mm/memory.c > +++ b/mm/memory.c > @@ -51,6 +51,7 @@ > #include <linux/init.h> > #include <linux/writeback.h> > #include <linux/memcontrol.h> > +#include <linux/mmu_notifier.h> > > #include <asm/pgalloc.h> > #include <asm/uaccess.h> > @@ -632,6 +633,7 @@ int copy_page_range(struct mm_struct *ds > unsigned long next; > unsigned long addr = vma->vm_start; > unsigned long end = vma->vm_end; > + int ret; > > /* > * Don't copy ptes where a page fault will fill them correctly. > @@ -647,17 +649,33 @@ int copy_page_range(struct mm_struct *ds > if (is_vm_hugetlb_page(vma)) > return copy_hugetlb_page_range(dst_mm, src_mm, vma); > > + /* > + * We need to invalidate the secondary MMU mappings only when > + * there could be a permission downgrade on the ptes of the > + * parent mm. And a permission downgrade will only happen if > + * is_cow_mapping() returns true. > + */ > + if (is_cow_mapping(vma->vm_flags)) > + mmu_notifier_invalidate_range_start(src_mm, addr, end); > + > + ret = 0; > dst_pgd = pgd_offset(dst_mm, addr); > src_pgd = pgd_offset(src_mm, addr); > do { > next = pgd_addr_end(addr, end); > if (pgd_none_or_clear_bad(src_pgd)) > continue; > - if (copy_pud_range(dst_mm, src_mm, dst_pgd, src_pgd, > - vma, addr, next)) > - return -ENOMEM; > + if (unlikely(copy_pud_range(dst_mm, src_mm, dst_pgd, src_pgd, > + vma, addr, next))) { > + ret = -ENOMEM; > + break; > + } > } while (dst_pgd++, src_pgd++, addr = next, addr != end); > - return 0; > + > + if (is_cow_mapping(vma->vm_flags)) > + mmu_notifier_invalidate_range_end(src_mm, > + vma->vm_start, end); > + return ret; > } > > static unsigned long zap_pte_range(struct mmu_gather *tlb, > @@ -861,7 +879,9 @@ unsigned long unmap_vmas(struct mmu_gath > unsigned long start = start_addr; > spinlock_t *i_mmap_lock = details? details->i_mmap_lock: NULL; > int fullmm = (*tlbp)->fullmm; > + struct mm_struct *mm = vma->vm_mm; > > + mmu_notifier_invalidate_range_start(mm, start_addr, end_addr); > for ( ; vma && vma->vm_start < end_addr; vma = vma->vm_next) { > unsigned long end; > > @@ -912,6 +932,7 @@ unsigned long unmap_vmas(struct mmu_gath > } > } > out: > + mmu_notifier_invalidate_range_end(mm, start_addr, end_addr); > return start; /* which is now the end (or restart) address */ > } > > @@ -1544,10 +1565,11 @@ int apply_to_page_range(struct mm_struct > { > pgd_t *pgd; > unsigned long next; > - unsigned long end = addr + size; > + unsigned long start = addr, end = addr + size; > int err; > > BUG_ON(addr >= end); > + mmu_notifier_invalidate_range_start(mm, start, end); > pgd = pgd_offset(mm, addr); > do { > next = pgd_addr_end(addr, end); > @@ -1555,6 +1577,7 @@ int apply_to_page_range(struct mm_struct > if (err) > break; > } while (pgd++, addr = next, addr != end); > + mmu_notifier_invalidate_range_end(mm, start, end); > return err; > } > EXPORT_SYMBOL_GPL(apply_to_page_range); > @@ -1756,7 +1779,7 @@ gotten: > * seen in the presence of one thread doing SMC and another > * thread doing COW. > */ > - ptep_clear_flush(vma, address, page_table); > + ptep_clear_flush_notify(vma, address, page_table); > set_pte_at(mm, address, page_table, entry); > update_mmu_cache(vma, address, entry); > lru_cache_add_active(new_page); > diff --git a/mm/mmap.c b/mm/mmap.c > --- a/mm/mmap.c > +++ b/mm/mmap.c > @@ -26,6 +26,7 @@ > #include <linux/mount.h> > #include <linux/mempolicy.h> > #include <linux/rmap.h> > +#include <linux/mmu_notifier.h> > > #include <asm/uaccess.h> > #include <asm/cacheflush.h> > @@ -2048,6 +2049,7 @@ void exit_mmap(struct mm_struct *mm) > > /* mm's last user has gone, and its about to be pulled down */ > arch_exit_mmap(mm); > + mmu_notifier_release(mm); > > lru_add_drain(); > flush_cache_mm(mm); > @@ -2255,3 +2257,152 @@ int install_special_mapping(struct mm_st > > return 0; > } > + > +static DEFINE_MUTEX(mm_all_locks_mutex); > + > +/* > + * This operation locks against the VM for all pte/vma/mm related > + * operations that could ever happen on a certain mm. This includes > + * vmtruncate, try_to_unmap, and all page faults. > + * > + * The caller must take the mmap_sem in write mode before calling > + * mm_take_all_locks(). The caller isn't allowed to release the > + * mmap_sem until mm_drop_all_locks() returns. > + * > + * mmap_sem in write mode is required in order to block all operations > + * that could modify pagetables and free pages without need of > + * altering the vma layout (for example populate_range() with > + * nonlinear vmas). It's also needed in write mode to avoid new > + * anon_vmas to be associated with existing vmas. > + * > + * A single task can't take more than one mm_take_all_locks() in a row > + * or it would deadlock. > + * > + * The LSB in anon_vma->head.next and the AS_MM_ALL_LOCKS bitflag in > + * mapping->flags avoid to take the same lock twice, if more than one > + * vma in this mm is backed by the same anon_vma or address_space. > + * > + * We can take all the locks in random order because the VM code > + * taking i_mmap_lock or anon_vma->lock outside the mmap_sem never > + * takes more than one of them in a row. Secondly we're protected > + * against a concurrent mm_take_all_locks() by the mm_all_locks_mutex. > + * > + * mm_take_all_locks() and mm_drop_all_locks are expensive operations > + * that may have to take thousand of locks. > + * > + * mm_take_all_locks() can fail if it's interrupted by signals. > + */ > +int mm_take_all_locks(struct mm_struct *mm) > +{ > + struct vm_area_struct *vma; > + int ret = -EINTR; > + > + BUG_ON(down_read_trylock(&mm->mmap_sem)); > + > + mutex_lock(&mm_all_locks_mutex); > + > + for (vma = mm->mmap; vma; vma = vma->vm_next) { > + struct file *filp; > + if (signal_pending(current)) > + goto out_unlock; > + if (vma->anon_vma && !test_bit(0, (unsigned long *) > + &vma->anon_vma->head.next)) { > + /* > + * The LSB of head.next can't change from > + * under us because we hold the > + * global_mm_spinlock. > + */ > + spin_lock(&vma->anon_vma->lock); > + /* > + * We can safely modify head.next after taking > + * the anon_vma->lock. If some other vma in > + * this mm shares the same anon_vma we won't > + * take it again. > + * > + * No need of atomic instructions here, > + * head.next can't change from under us thanks > + * to the anon_vma->lock. > + */ > + if (__test_and_set_bit(0, (unsigned long *) > + &vma->anon_vma->head.next)) > + BUG(); > + } > + > + filp = vma->vm_file; > + if (filp && filp->f_mapping && > + !test_bit(AS_MM_ALL_LOCKS, &filp->f_mapping->flags)) { > + /* > + * AS_MM_ALL_LOCKS can't change from under us > + * because we hold the global_mm_spinlock. > + * > + * Operations on ->flags have to be atomic > + * because even if AS_MM_ALL_LOCKS is stable > + * thanks to the global_mm_spinlock, there may > + * be other cpus changing other bitflags in > + * parallel to us. > + */ > + if (test_and_set_bit(AS_MM_ALL_LOCKS, > + &filp->f_mapping->flags)) > + BUG(); > + spin_lock(&filp->f_mapping->i_mmap_lock); > + } > + } > + ret = 0; > + > +out_unlock: > + if (ret) > + mm_drop_all_locks(mm); > + > + return ret; > +} > + > +/* > + * The mmap_sem cannot be released by the caller until > + * mm_drop_all_locks() returns. > + */ > +void mm_drop_all_locks(struct mm_struct *mm) > +{ > + struct vm_area_struct *vma; > + > + BUG_ON(down_read_trylock(&mm->mmap_sem)); > + BUG_ON(!mutex_is_locked(&mm_all_locks_mutex)); > + > + for (vma = mm->mmap; vma; vma = vma->vm_next) { > + struct file *filp; > + if (vma->anon_vma && > + test_bit(0, (unsigned long *) > + &vma->anon_vma->head.next)) { > + /* > + * The LSB of head.next can't change to 0 from > + * under us because we hold the > + * global_mm_spinlock. > + * > + * We must however clear the bitflag before > + * unlocking the vma so the users using the > + * anon_vma->head will never see our bitflag. > + * > + * No need of atomic instructions here, > + * head.next can't change from under us until > + * we release the anon_vma->lock. > + */ > + if (!__test_and_clear_bit(0, (unsigned long *) > + &vma->anon_vma->head.next)) > + BUG(); > + spin_unlock(&vma->anon_vma->lock); > + } > + filp = vma->vm_file; > + if (filp && filp->f_mapping && > + test_bit(AS_MM_ALL_LOCKS, &filp->f_mapping->flags)) { > + /* > + * AS_MM_ALL_LOCKS can't change to 0 from under us > + * because we hold the global_mm_spinlock. > + */ > + spin_unlock(&filp->f_mapping->i_mmap_lock); > + if (!test_and_clear_bit(AS_MM_ALL_LOCKS, > + &filp->f_mapping->flags)) > + BUG(); > + } > + } > + > + mutex_unlock(&mm_all_locks_mutex); > +} > diff --git a/mm/mmu_notifier.c b/mm/mmu_notifier.c > new file mode 100644 > --- /dev/null > +++ b/mm/mmu_notifier.c > @@ -0,0 +1,276 @@ > +/* > + * linux/mm/mmu_notifier.c > + * > + * Copyright (C) 2008 Qumranet, Inc. > + * Copyright (C) 2008 SGI > + * Christoph Lameter <cla...@sg...> > + * > + * This work is licensed under the terms of the GNU GPL, version 2. See > + * the COPYING file in the top-level directory. > + */ > + > +#include <linux/mmu_notifier.h> > +#include <linux/module.h> > +#include <linux/mm.h> > +#include <linux/err.h> > +#include <linux/rcupdate.h> > +#include <linux/sched.h> > + > +/* > + * This function can't run concurrently against mmu_notifier_register > + * because mm->mm_users > 0 during mmu_notifier_register and exit_mmap > + * runs with mm_users == 0. Other tasks may still invoke mmu notifiers > + * in parallel despite there being no task using this mm any more, > + * through the vmas outside of the exit_mmap context, such as with > + * vmtruncate. This serializes against mmu_notifier_unregister with > + * the mmu_notifier_mm->lock in addition to RCU and it serializes > + * against the other mmu notifiers with RCU. struct mmu_notifier_mm > + * can't go away from under us as exit_mmap holds an mm_count pin > + * itself. > + */ > +void __mmu_notifier_release(struct mm_struct *mm) > +{ > + struct mmu_notifier *mn; > + > + spin_lock(&mm->mmu_notifier_mm->lock); > + while (unlikely(!hlist_empty(&mm->mmu_notifier_mm->list))) { > + mn = hlist_entry(mm->mmu_notifier_mm->list.first, > + struct mmu_notifier, > + hlist); > + /* > + * We arrived before mmu_notifier_unregister so > + * mmu_notifier_unregister will do nothing other than > + * to wait ->release to finish and > + * mmu_notifier_unregister to return. > + */ > + hlist_del_init_rcu(&mn->hlist); > + /* > + * RCU here will block mmu_notifier_unregister until > + * ->release returns. > + */ > + rcu_read_lock(); > + spin_unlock(&mm->mmu_notifier_mm->lock); > + /* > + * if ->release runs before mmu_notifier_unregister it > + * must be handled as it's the only way for the driver > + * to flush all existing sptes and stop the driver > + * from establishing any more sptes before all the > + * pages in the mm are freed. > + */ > + if (mn->ops->release) > + mn->ops->release(mn, mm); > + rcu_read_unlock(); > + spin_lock(&mm->mmu_notifier_mm->lock); > + } > + spin_unlock(&mm->mmu_notifier_mm->lock); > + > + /* > + * synchronize_rcu here prevents mmu_notifier_release to > + * return to exit_mmap (which would proceed freeing all pages > + * in the mm) until the ->release method returns, if it was > + * invoked by mmu_notifier_unregister. > + * > + * The mmu_notifier_mm can't go away from under us because one > + * mm_count is hold by exit_mmap. > + */ > + synchronize_rcu(); > +} > + > +/* > + * If no young bitflag is supported by the hardware, ->clear_flush_young can > + * unmap the address and return 1 or 0 depending if the mapping previously > + * existed or not. > + */ > +int __mmu_notifier_clear_flush_young(struct mm_struct *mm, > + unsigned long address) > +{ > + struct mmu_notifier *mn; > + struct hlist_node *n; > + int young = 0; > + > + rcu_read_lock(); > + hlist_for_each_entry_rcu(mn, n, &mm->mmu_notifier_mm->list, hlist) { > + if (mn->ops->clear_flush_young) > + young |= mn->ops->clear_flush_young(mn, mm, address); > + } > + rcu_read_unlock(); > + > + return young; > +} > + > +void __mmu_notifier_invalidate_page(struct mm_struct *mm, > + unsigned long address) > +{ > + struct mmu_notifier *mn; > + struct hlist_node *n; > + > + rcu_read_lock(); > + hlist_for_each_entry_rcu(mn, n, &mm->mmu_notifier_mm->list, hlist) { > + if (mn->ops->invalidate_page) > + mn->ops->invalidate_page(mn, mm, address); > + } > + rcu_read_unlock(); > +} > + > +void __mmu_notifier_invalidate_range_start(struct mm_struct *mm, > + unsigned long start, unsigned long end) > +{ > + struct mmu_notifier *mn; > + struct hlist_node *n; > + > + rcu_read_lock(); > + hlist_for_each_entry_rcu(mn, n, &mm->mmu_notifier_mm->list, hlist) { > + if (mn->ops->invalidate_range_start) > + mn->ops->invalidate_range_start(mn, mm, start, end); > + } > + rcu_read_unlock(); > +} > + > +void __mmu_notifier_invalidate_range_end(struct mm_struct *mm, > + unsigned long start, unsigned long end) > +{ > + struct mmu_notifier *mn; > + struct hlist_node *n; > + > + rcu_read_lock(); > + hlist_for_each_entry_rcu(mn, n, &mm->mmu_notifier_mm->list, hlist) { > + if (mn->ops->invalidate_range_end) > + mn->ops->invalidate_range_end(mn, mm, start, end); > + } > + rcu_read_unlock(); > +} > + > +static int do_mmu_notifier_register(struct mmu_notifier *mn, > + struct mm_struct *mm, > + int take_mmap_sem) > +{ > + struct mmu_notifier_mm * mmu_notifier_mm; > + int ret; > + > + BUG_ON(atomic_read(&mm->mm_users) <= 0); > + > + ret = -ENOMEM; > + mmu_notifier_mm = kmalloc(sizeof(struct mmu_notifier_mm), GFP_KERNEL); > + if (unlikely(!mmu_notifier_mm)) > + goto out; > + > + if (take_mmap_sem) > + down_write(&mm->mmap_sem); > + ret = mm_take_all_locks(mm); > + if (unlikely(ret)) > + goto out_cleanup; > + > + if (!mm_has_notifiers(mm)) { > + INIT_HLIST_HEAD(&mmu_notifier_mm->list); > + spin_lock_init(&mmu_notifier_mm->lock); > + mm->mmu_notifier_mm = mmu_notifier_mm; > + mmu_notifier_mm = NULL; > + } > + atomic_inc(&mm->mm_count); > + > + /* > + * Serialize the update against mmu_notifier_unregister. A > + * side note: mmu_notifier_release can't run concurrently with > + * us because we hold the mm_users pin (either implicitly as > + * current->mm or explicitly with get_task_mm() or similar). > + * We can't race against any other mmu notifier method either > + * thanks to mm_take_all_locks(). > + */ > + spin_lock(&mm->mmu_notifier_mm->lock); > + hlist_add_head(&mn->hlist, &mm->mmu_notifier_mm->list); > + spin_unlock(&mm->mmu_notifier_mm->lock); > + > + mm_drop_all_locks(mm); > +out_cleanup: > + if (take_mmap_sem) > + up_write(&mm->mmap_sem); > + /* kfree() does nothing if mmu_notifier_mm is NULL */ > + kfree(mmu_notifier_mm); > +out: > + BUG_ON(atomic_read(&mm->mm_users) <= 0); > + return ret; > +} > + > +/* > + * Must not hold mmap_sem nor any other VM related lock when calling > + * this registration function. Must also ensure mm_users can't go down > + * to zero while this runs to avoid races with mmu_notifier_release, > + * so mm has to be current->mm or the mm should be pinned safely such > + * as with get_task_mm(). If the mm is not current->mm, the mm_users > + * pin should be released by calling mmput after mmu_notifier_register > + * returns. mmu_notifier_unregister must be always called to > + * unregister the notifier. mm_count is automatically pinned to allow > + * mmu_notifier_unregister to safely run at any time later, before or > + * after exit_mmap. ->release will always be called before exit_mmap > + * frees the pages. > + */ > +int mmu_notifier_register(struct mmu_notifier *mn, struct mm_struct *mm) > +{ > + return do_mmu_notifier_register(mn, mm, 1); > +} > +EXPORT_SYMBOL_GPL(mmu_notifier_register); > + > +/* > + * Same as mmu_notifier_register but here the caller must hold the > + * mmap_sem in write mode. > + */ > +int __mmu_notifier_register(struct mmu_notifier *mn, struct mm_struct *mm) > +{ > + return do_mmu_notifier_register(mn, mm, 0); > +} > +EXPORT_SYMBOL_GPL(__mmu_notifier_register); > + > +/* this is called after the last mmu_notifier_unregister() returned */ > +void __mmu_notifier_mm_destroy(struct mm_struct *mm) > +{ > + BUG_ON(!hlist_empty(&mm->mmu_notifier_mm->list)); > + kfree(mm->mmu_notifier_mm); > + mm->mmu_notifier_mm = LIST_POISON1; /* debug */ > +} > + > +/* > + * This releases the mm_count pin automatically and frees the mm > + * structure if it was the last user of it. It serializes against > + * running mmu notifiers with RCU and against mmu_notifier_unregister > + * with the unregister lock + RCU. All sptes must be dropped before > + * calling mmu_notifier_unregister. ->release or any other notifier > + * method may be invoked concurrently with mmu_notifier_unregister, > + * and only after mmu_notifier_unregister returned we're guaranteed > + * that ->release or any other method can't run anymore. > + */ > +void mmu_notifier_unregister(struct mmu_notifier *mn, struct mm_struct *mm) > +{ > + BUG_ON(atomic_read(&mm->mm_count) <= 0); > + > + spin_lock(&mm->mmu_notifier_mm->lock); > + if (!hlist_unhashed(&mn->hlist)) { > + hlist_del_rcu(&mn->hlist); > + > + /* > + * RCU here will force exit_mmap to wait ->release to finish > + * before freeing the pages. > + */ > + rcu_read_lock(); > + spin_unlock(&mm->mmu_notifier_mm->lock); > + /* > + * exit_mmap will block in mmu_notifier_release to > + * guarantee ->release is called before freeing the > + * pages. > + */ > + if (mn->ops->release) > + mn->ops->release(mn, mm); > + rcu_read_unlock(); > + } else > + spin_unlock(&mm->mmu_notifier_mm->lock); > + > + /* > + * Wait any running method to finish, of course including > + * ->release if it was run by mmu_notifier_relase instead of us. > + */ > + synchronize_rcu(); > + > + BUG_ON(atomic_read(&mm->mm_count) <= 0); > + > + mmdrop(mm); > +} > +EXPORT_SYMBOL_GPL(mmu_notifier_unregister); > diff --git a/mm/mprotect.c b/mm/mprotect.c > --- a/mm/mprotect.c > +++ b/mm/mprotect.c > @@ -21,6 +21,7 @@ > #include <linux/syscalls.h> > #include <linux/swap.h> > #include <linux/swapops.h> > +#include <linux/mmu_notifier.h> > #include <asm/uaccess.h> > #include <asm/pgtable.h> > #include <asm/cacheflush.h> > @@ -198,10 +199,12 @@ success: > dirty_accountable = 1; > } > > + mmu_notifier_invalidate_range_start(mm, start, end); > if (is_vm_hugetlb_page(vma)) > hugetlb_change_protection(vma, start, end, vma->vm_page_prot); > else > change_protection(vma, start, end, vma->vm_page_prot, dirty_accountable); > + mmu_notifier_invalidate_range_end(mm, start, end); > vm_stat_account(mm, oldflags, vma->vm_file, -nrpages); > vm_stat_account(mm, newflags, vma->vm_file, nrpages); > return 0; > diff --git a/mm/mremap.c b/mm/mremap.c > --- a/mm/mremap.c > +++ b/mm/mremap.c > @@ -18,6 +18,7 @@ > #include <linux/highmem.h> > #include <linux/security.h> > #include <linux/syscalls.h> > +#include <linux/mmu_notifier.h> > > #include <asm/uaccess.h> > #include <asm/cacheflush.h> > @@ -74,7 +75,11 @@ static void move_ptes(struct vm_area_str > struct mm_struct *mm = vma->vm_mm; > pte_t *old_pte, *new_pte, pte; > spinlock_t *old_ptl, *new_ptl; > + unsigned long old_start; > > + old_start = old_addr; > + mmu_notifier_invalidate_range_start(vma->vm_mm, > + old_start, old_end); > if (vma->vm_file) { > /* > * Subtle point from Rajesh Venkatasubramanian: before > @@ -116,6 +121,7 @@ static void move_ptes(struct vm_area_str > pte_unmap_unlock(old_pte - 1, old_ptl); > if (mapping) > spin_unlock(&mapping->i_mmap_lock); > + mmu_notifier_invalidate_range_end(vma->vm_mm, old_start, old_end); > } > > #define LATENCY_LIMIT (64 * PAGE_SIZE) > diff --git a/mm/rmap.c b/mm/rmap.c > --- a/mm/rmap.c > +++ b/mm/rmap.c > @@ -49,6 +49,7 @@ > #include <linux/module.h> > #include <linux/kallsyms.h> > #include <linux/memcontrol.h> > +#include <linux/mmu_notifier.h> > > #include <asm/tlbflush.h> > > @@ -287,7 +288,7 @@ static int page_referenced_one(struct pa > if (vma->vm_flags & VM_LOCKED) { > referenced++; > *mapcount = 1; /* break early from loop */ > - } else if (ptep_clear_flush_young(vma, address, pte)) > + } else if (ptep_clear_flush_young_notify(vma, address, pte)) > referenced++; > > /* Pretend the page is referenced if the task has the > @@ -457,7 +458,7 @@ static int page_mkclean_one(struct page > pte_t entry; > > flush_cache_page(vma, address, pte_pfn(*pte)); > - entry = ptep_clear_flush(vma, address, pte); > + entry = ptep_clear_flush_notify(vma, address, pte); > entry = pte_wrprotect(entry); > entry = pte_mkclean(entry); > set_pte_at(mm, address, pte, entry); > @@ -717,14 +718,14 @@ static int try_to_unmap_one(struct page > * skipped over this mm) then we should reactivate it. > */ > if (!migration && ((vma->vm_flags & VM_LOCKED) || > - (ptep_clear_flush_young(vma, address, pte)))) { > + (ptep_clear_flush_young_notify(vma, address, pte)))) { > ret = SWAP_FAIL; > goto out_unmap; > } > > /* Nuke the page table entry. */ > flush_cache_page(vma, address, page_to_pfn(page)); > - pteval = ptep_clear_flush(vma, address, pte); > + pteval = ptep_clear_flush_notify(vma, address, pte); > > /* Move the dirty bit to the physical page now the pte is gone. */ > if (pte_dirty(pteval)) > @@ -849,12 +850,12 @@ static void try_to_unmap_cluster(unsigne > page = vm_normal_page(vma, address, *pte); > BUG_ON(!page || PageAnon(page)); > > - if (ptep_clear_flush_young(vma, address, pte)) > + if (ptep_clear_flush_young_notify(vma, address, pte)) > continue; > > /* Nuke the page table entry. */ > flush_cache_page(vma, address, pte_pfn(*pte)); > - pteval = ptep_clear_flush(vma, address, pte); > + pteval = ptep_clear_flush_notify(vma, address, pte); > > /* If nonlinear, store the file page offset in the pte. */ > if (page->index != linear_page_index(vma, address)) |