Re: set_page_dirty vs truncate
On Fri, Dec 18, 2020 at 10:03:16PM +0000, Matthew Wilcox wrote:
On Fri, Dec 18, 2020 at 04:05:31PM +0000, Matthew Wilcox wrote:
A number of implementations of ->set_page_dirty check whether the page
has been truncated (ie page->mapping has become NULL since entering
set_page_dirty()). Several other implementations assume that they can do
page->mapping->host to get to the inode. So either some implementations
are doing unnecessary checks or others are vulnerable to a NULL pointer
dereference if truncate() races with set_page_dirty().
I'm touching ->set_page_dirty() anyway as part of the page folio
conversion. I'm thinking about passing in the mapping so there's no
need to look at page->mapping.
The comments on set_page_dirty() and set_page_dirty_lock() suggests
there's no consistency in whether truncation is blocked or not; we're
only guaranteed that the inode itself won't go away. But maybe the
comments are stale.
The comments are, I believe, not stale. Here's some syzbot
reports which indicate that ext4 is seeing races between set_page_dirty()
and truncate():
https://groups.google.com/g/syzkaller-lts-bugs/c/s9fHu162zhQ/m/Phnf6ucaAwAJ
The reproducer includes calls to ftruncate(), so that would suggest
that's what's going on.
Hmmm ... looks like __set_page_dirty_nobuffers() has a similar problem:
{
lock_page_memcg(page);
if (!TestSetPageDirty(page)) {
struct address_space *mapping = page_mapping(page);
unsigned long flags;
if (!mapping) {
unlock_page_memcg(page);
return 1;
}
xa_lock_irqsave(&mapping->i_pages, flags);
BUG_ON(page_mapping(page) != mapping);
sure, we check that the page wasn't truncated between set_page_dirty()
and the call to TestSetPageDirty(), but we can truncate dirty pages
with no problem. So between the call to TestSetPageDirty() and
the call to xa_lock_irqsave(), the page can be truncated, and the
BUG_ON should fire.
I haven't been able to find any examples of this, but maybe it's just a very
narrow race. Does anyone recognise this signature? Adding the filesystems
which use __set_page_dirty_nobuffers() directly without extra locking.
$ git grep set_page_dirty.=.__set_page_dirty_nobuffers
fs/9p/vfs_addr.c: .set_page_dirty = __set_page_dirty_nobuffers,
fs/cifs/file.c: .set_page_dirty = __set_page_dirty_nobuffers,
fs/cifs/file.c: .set_page_dirty = __set_page_dirty_nobuffers,
fs/fuse/file.c: .set_page_dirty = __set_page_dirty_nobuffers,
fs/hostfs/hostfs_kern.c: .set_page_dirty = __set_page_dirty_nobuffers,
fs/jfs/jfs_metapage.c: .set_page_dirty = __set_page_dirty_nobuffers,
fs/nfs/file.c: .set_page_dirty = __set_page_dirty_nobuffers,
fs/ntfs/aops.c: .set_page_dirty = __set_page_dirty_nobuffers, /* Set the page dirty
fs/orangefs/inode.c: .set_page_dirty = __set_page_dirty_nobuffers,
fs/vboxsf/file.c: .set_page_dirty = __set_page_dirty_nobuffers,
On 12/18/20 9:18 PM, Matthew Wilcox wrote:
On Fri, Dec 18, 2020 at 10:03:16PM +0000, Matthew Wilcox wrote:
On Fri, Dec 18, 2020 at 04:05:31PM +0000, Matthew Wilcox wrote:
A number of implementations of ->set_page_dirty check whether the page
has been truncated (ie page->mapping has become NULL since entering
set_page_dirty()). Several other implementations assume that they can do
page->mapping->host to get to the inode. So either some implementations
are doing unnecessary checks or others are vulnerable to a NULL pointer
dereference if truncate() races with set_page_dirty().
I'm touching ->set_page_dirty() anyway as part of the page folio
conversion. I'm thinking about passing in the mapping so there's no
need to look at page->mapping.
The comments on set_page_dirty() and set_page_dirty_lock() suggests
there's no consistency in whether truncation is blocked or not; we're
only guaranteed that the inode itself won't go away. But maybe the
comments are stale.
The comments are, I believe, not stale. Here's some syzbot
reports which indicate that ext4 is seeing races between set_page_dirty()
and truncate():
https://groups.google.com/g/syzkaller-lts-bugs/c/s9fHu162zhQ/m/Phnf6ucaAwAJ
The reproducer includes calls to ftruncate(), so that would suggest
that's what's going on.
Hmmm ... looks like __set_page_dirty_nobuffers() has a similar problem:
{
lock_page_memcg(page);
if (!TestSetPageDirty(page)) {
struct address_space *mapping = page_mapping(page);
unsigned long flags;
if (!mapping) {
unlock_page_memcg(page);
return 1;
}
xa_lock_irqsave(&mapping->i_pages, flags);
BUG_ON(page_mapping(page) != mapping);
sure, we check that the page wasn't truncated between set_page_dirty()
and the call to TestSetPageDirty(), but we can truncate dirty pages
with no problem. So between the call to TestSetPageDirty() and
the call to xa_lock_irqsave(), the page can be truncated, and the
BUG_ON should fire.
I haven't been able to find any examples of this, but maybe it's just a very
narrow race. Does anyone recognise this signature? Adding the filesystems
which use __set_page_dirty_nobuffers() directly without extra locking.
That sounds like the same kind of failure that Jan Kara and I were
seeing on live systems[1], that led eventually to the gup-to-pup
conversion exercise.
That crash happened due to calling set_page_dirty() on pages that had no
buffers on them [2]. And that sounds like exactly the same thing as
calling __set_page_dirty_nobuffers() without extra locking. So I'd
expect that it's Just Wrong To Do, for the same reasons as Jan spells
out very clearly in [1].
Hope that helps.
[1] https://www.spinics.net/lists/linux-mm/msg142700.html
[2] which triggered this assertion:
#define page_buffers(page)
({
BUG_ON(!PagePrivate(page));
((struct buffer_head *)page_private(page));
})
$ git grep set_page_dirty.=.__set_page_dirty_nobuffers
fs/9p/vfs_addr.c: .set_page_dirty = __set_page_dirty_nobuffers,
fs/cifs/file.c: .set_page_dirty = __set_page_dirty_nobuffers,
fs/cifs/file.c: .set_page_dirty = __set_page_dirty_nobuffers,
fs/fuse/file.c: .set_page_dirty = __set_page_dirty_nobuffers,
fs/hostfs/hostfs_kern.c: .set_page_dirty = __set_page_dirty_nobuffers,
fs/jfs/jfs_metapage.c: .set_page_dirty = __set_page_dirty_nobuffers,
fs/nfs/file.c: .set_page_dirty = __set_page_dirty_nobuffers,
fs/ntfs/aops.c: .set_page_dirty = __set_page_dirty_nobuffers, /* Set the page dirty
fs/orangefs/inode.c: .set_page_dirty = __set_page_dirty_nobuffers,
fs/vboxsf/file.c: .set_page_dirty = __set_page_dirty_nobuffers,
...wow, long list of these.
thanks,
John Hubbard
NVIDIA
On Fri, Dec 18, 2020 at 10:10:01PM -0800, John Hubbard wrote:
On 12/18/20 9:18 PM, Matthew Wilcox wrote:
On Fri, Dec 18, 2020 at 10:03:16PM +0000, Matthew Wilcox wrote:
On Fri, Dec 18, 2020 at 04:05:31PM +0000, Matthew Wilcox wrote:
A number of implementations of ->set_page_dirty check whether the page
has been truncated (ie page->mapping has become NULL since entering
set_page_dirty()). Several other implementations assume that they can do
page->mapping->host to get to the inode. So either some implementations
are doing unnecessary checks or others are vulnerable to a NULL pointer
dereference if truncate() races with set_page_dirty().
I'm touching ->set_page_dirty() anyway as part of the page folio
conversion. I'm thinking about passing in the mapping so there's no
need to look at page->mapping.
The comments on set_page_dirty() and set_page_dirty_lock() suggests
there's no consistency in whether truncation is blocked or not; we're
only guaranteed that the inode itself won't go away. But maybe the
comments are stale.
The comments are, I believe, not stale. Here's some syzbot
reports which indicate that ext4 is seeing races between set_page_dirty()
and truncate():
https://groups.google.com/g/syzkaller-lts-bugs/c/s9fHu162zhQ/m/Phnf6ucaAwAJ
The reproducer includes calls to ftruncate(), so that would suggest
that's what's going on.
Hmmm ... looks like __set_page_dirty_nobuffers() has a similar problem:
{
lock_page_memcg(page);
if (!TestSetPageDirty(page)) {
struct address_space *mapping = page_mapping(page);
unsigned long flags;
if (!mapping) {
unlock_page_memcg(page);
return 1;
}
xa_lock_irqsave(&mapping->i_pages, flags);
BUG_ON(page_mapping(page) != mapping);
sure, we check that the page wasn't truncated between set_page_dirty()
and the call to TestSetPageDirty(), but we can truncate dirty pages
with no problem. So between the call to TestSetPageDirty() and
the call to xa_lock_irqsave(), the page can be truncated, and the
BUG_ON should fire.
I haven't been able to find any examples of this, but maybe it's just a very
narrow race. Does anyone recognise this signature? Adding the filesystems
which use __set_page_dirty_nobuffers() directly without extra locking.
That sounds like the same kind of failure that Jan Kara and I were
seeing on live systems[1], that led eventually to the gup-to-pup
conversion exercise.
That crash happened due to calling set_page_dirty() on pages that had no
buffers on them [2]. And that sounds like exactly the same thing as
calling __set_page_dirty_nobuffers() without extra locking. So I'd
expect that it's Just Wrong To Do, for the same reasons as Jan spells
out very clearly in [1].
Interesting. It's a bit different, but Jan's race might be what's
causing this symptom. The reason is that the backtrace contains
set_page_dirty_lock() which holds the page lock. So there can't be
a truncation race because truncate holds the page lock when calling
->invalidatepage.
That said, the syzbot reproducer doesn't have any O_DIRECT in it
either. So maybe this is some other race?
Hope that helps.
[1] https://www.spinics.net/lists/linux-mm/msg142700.html
[2] which triggered this assertion:
#define page_buffers(page)
({
BUG_ON(!PagePrivate(page));
((struct buffer_head *)page_private(page));
})
$ git grep set_page_dirty.=.__set_page_dirty_nobuffers
fs/9p/vfs_addr.c: .set_page_dirty = __set_page_dirty_nobuffers,
fs/cifs/file.c: .set_page_dirty = __set_page_dirty_nobuffers,
fs/cifs/file.c: .set_page_dirty = __set_page_dirty_nobuffers,
fs/fuse/file.c: .set_page_dirty = __set_page_dirty_nobuffers,
fs/hostfs/hostfs_kern.c: .set_page_dirty = __set_page_dirty_nobuffers,
fs/jfs/jfs_metapage.c: .set_page_dirty = __set_page_dirty_nobuffers,
fs/nfs/file.c: .set_page_dirty = __set_page_dirty_nobuffers,
fs/ntfs/aops.c: .set_page_dirty = __set_page_dirty_nobuffers, /* Set the page dirty
fs/orangefs/inode.c: .set_page_dirty = __set_page_dirty_nobuffers,
fs/vboxsf/file.c: .set_page_dirty = __set_page_dirty_nobuffers,
...wow, long list of these.
thanks,
John Hubbard
NVIDIA
On 12/18/20 10:50 PM, Matthew Wilcox wrote:
...
Hmmm ... looks like __set_page_dirty_nobuffers() has a similar problem:
{
lock_page_memcg(page);
if (!TestSetPageDirty(page)) {
struct address_space *mapping = page_mapping(page);
unsigned long flags;
if (!mapping) {
unlock_page_memcg(page);
return 1;
}
xa_lock_irqsave(&mapping->i_pages, flags);
BUG_ON(page_mapping(page) != mapping);
sure, we check that the page wasn't truncated between set_page_dirty()
and the call to TestSetPageDirty(), but we can truncate dirty pages
with no problem. So between the call to TestSetPageDirty() and
the call to xa_lock_irqsave(), the page can be truncated, and the
BUG_ON should fire.
I haven't been able to find any examples of this, but maybe it's just a very
narrow race. Does anyone recognise this signature? Adding the filesystems
which use __set_page_dirty_nobuffers() directly without extra locking.
That sounds like the same kind of failure that Jan Kara and I were
seeing on live systems[1], that led eventually to the gup-to-pup
conversion exercise.
That crash happened due to calling set_page_dirty() on pages that had no
buffers on them [2]. And that sounds like exactly the same thing as
calling __set_page_dirty_nobuffers() without extra locking. So I'd
expect that it's Just Wrong To Do, for the same reasons as Jan spells
out very clearly in [1].
Interesting. It's a bit different, but Jan's race might be what's
causing this symptom. The reason is that the backtrace contains
set_page_dirty_lock() which holds the page lock. So there can't be
a truncation race because truncate holds the page lock when calling
->invalidatepage.
That said, the syzbot reproducer doesn't have any O_DIRECT in it
either. So maybe this is some other race?
Jan's race can be also be reproduced without O_DIRECT. I first saw
it via a program that just did these steps on a normal ext4 filesystem:
a) pin ext4 file-backed pages, via get_user_pages(). Actually the way
it got here was due to using what looked like anonymous RAM to the
program, but was really file-backed RAM, because the admin had it
set up to mount ext4 on /tmp, instead of using tmpfs, to "save RAM",
but I digress. :)
b) wait a while, optionally do some DMA on the pages from a GPU, drink
coffee...
c) call set_pages_dirty()
d) unpin the pages
e) BUG_ON() in page_buffers().
thanks,
John Hubbard
NVIDIA