Rather than unconditionally reloading cr3, only do so if the pud we're
updating is within the active pgd.

This eliminates TLB flushes most of the time. The
performance-critical uses of pud_clear are during execve and exit, but
in those cases cr3 is referring to some other pagetable. The only
other use of pud_clear is during a large (1Gbyte+) munmap, and those
are sufficiently rare that a couple of cr3 reloads won't hurt.

Signed-off-by: Jeremy Fitzhardinge
include/asm-x86/pgtable-3level.h | 11 +++++++----
1 file changed, 7 insertions(+), 4 deletions(-)

diff --git a/include/asm-x86/pgtable-3level.h b/include/asm-x86/pgtable-3level.h
--- a/include/asm-x86/pgtable-3level.h
+++ b/include/asm-x86/pgtable-3level.h
@@ -93,17 +93,20 @@ static inline void native_pmd_clear(pmd_

static inline void pud_clear(pud_t *pudp)
+ unsigned long pgd;
set_pud(pudp, __pud(0));

* Pentium-II erratum A13: in PAE mode we explicitly have to flush
* the TLB via cr3 if the top-level pgd is changed...
- * XXX I don't think we need to worry about this here, since
- * when clearing the pud, the calling code needs to flush the
- * tlb anyway. But do it now for safety's sake. - jsgf
+ * Make sure the pud entry we're updating is within the
+ * current pgd to avoid unnecessary TLB flushes.
- write_cr3(read_cr3());
+ pgd = read_cr3();
+ if (__pa(pudp) >= pgd && __pa(pudp) < (pgd + sizeof(pgd_t)*PTRS_PER_PGD))
+ write_cr3(pgd);

#define pud_page(pud) \

To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/