aboutsummaryrefslogtreecommitdiff
path: root/fs/splice.c
AgeCommit message (Collapse)Author
2006-05-01[PATCH] pipe: enable atomic copying of pipe data to/from user spaceJens Axboe
The pipe ->map() method uses kmap() to virtually map the pages, which is both slow and has known scalability issues on SMP. This patch enables atomic copying of pipe pages, by pre-faulting data and using kmap_atomic() instead. lmbench bw_pipe and lat_pipe measurements agree this is a Good Thing. Here are results from that on a UP machine with highmem (1.5GiB of RAM), running first a UP kernel, SMP kernel, and SMP kernel patched. Vanilla-UP: Pipe bandwidth: 1622.28 MB/sec Pipe bandwidth: 1610.59 MB/sec Pipe bandwidth: 1608.30 MB/sec Pipe latency: 7.3275 microseconds Pipe latency: 7.2995 microseconds Pipe latency: 7.3097 microseconds Vanilla-SMP: Pipe bandwidth: 1382.19 MB/sec Pipe bandwidth: 1317.27 MB/sec Pipe bandwidth: 1355.61 MB/sec Pipe latency: 9.6402 microseconds Pipe latency: 9.6696 microseconds Pipe latency: 9.6153 microseconds Patched-SMP: Pipe bandwidth: 1578.70 MB/sec Pipe bandwidth: 1579.95 MB/sec Pipe bandwidth: 1578.63 MB/sec Pipe latency: 9.1654 microseconds Pipe latency: 9.2266 microseconds Pipe latency: 9.1527 microseconds Signed-off-by: Jens Axboe <axboe@suse.de>
2006-05-01[PATCH] splice: call handle_ra_miss() on failure to lookup pageJens Axboe
Notify the readahead logic of the missing page. Suggested by Oleg Nesterov. Signed-off-by: Jens Axboe <axboe@suse.de>
2006-05-01[PATCH] pipe: introduce ->pin() buffer operationJens Axboe
The ->map() function is really expensive on highmem machines right now, since it has to use the slower kmap() instead of kmap_atomic(). Splice rarely needs to access the virtual address of a page, so it's a waste of time doing it. Introduce ->pin() to take over the responsibility of making sure the page data is valid. ->map() is then reduced to just kmap(). That way we can also share a most of the pipe buffer ops between pipe.c and splice.c Signed-off-by: Jens Axboe <axboe@suse.de>
2006-05-01[PATCH] splice: fix bugs in pipe_to_file()Jens Axboe
Found by Oleg Nesterov <oleg@tv-sign.ru>, fixed by me. - Only allow full pages to go to the page cache. - Check page != buf->page instead of using PIPE_BUF_FLAG_STOLEN. - Remember to clear 'stolen' if add_to_page_cache() fails. And as a cleanup on that: - Make the bottom fall-through logic a little less convoluted. Also make the steal path hold an extra reference to the page, so we don't have to differentiate between stolen and non-stolen at the end. Signed-off-by: Jens Axboe <axboe@suse.de>
2006-04-30[PATCH] splice: fix bugs with stealing regular pipe pagesJens Axboe
- Check that page has suitable count for stealing in the regular pipes. - pipe_to_file() assumes that the page is locked on succesful steal, so do that in the pipe steal hook - Missing unlock_page() in add_to_page_cache() failure. Signed-off-by: Jens Axboe <axboe@suse.de>
2006-04-27[PATCH] splice: make the read-side do batched page lookupsJens Axboe
Use the new find_get_pages_contig() to potentially look up the entire splice range in one single call. This speeds up generic_file_splice_read() quite a bit. Signed-off-by: Jens Axboe <axboe@suse.de>
2006-04-27[PATCH] splice: switch to using page_cache_readahead()Jens Axboe
Avoids doing useless work, when the file is fully cached. Signed-off-by: Jens Axboe <axboe@suse.de>
2006-04-26[PATCH] splice: rearrange moving to/from pipe helpersJens Axboe
We need these for people writing their own ->splice_read/write hooks. Signed-off-by: Jens Axboe <axboe@suse.de>
2006-04-26[PATCH] Add support for the sys_vmsplice syscallJens Axboe
sys_splice() moves data to/from pipes with a file input/output. sys_vmsplice() moves data to a pipe, with the input being a user address range instead. This uses an approach suggested by Linus, where we can hold partial ranges inside the pages[] map. Hopefully this will be useful for network receive support as well. Signed-off-by: Jens Axboe <axboe@suse.de>
2006-04-26[PATCH] splice: fix offset problemsJens Axboe
Make the move_from_pipe() actors return number of bytes processed, then move_from_pipe() can decide more cleverly when to move on to the next buffer. This fixes problems with pipe offset and differing file offset. Signed-off-by: Jens Axboe <axboe@suse.de>
2006-04-26[PATCH] splice: fix min() warningAndrew Morton
Signed-off-by: Andrew Morton <akpm@osdl.org> Signed-off-by: Jens Axboe <axboe@suse.de>
2006-04-20[PATCH] splice: fix smaller sized splice readsJens Axboe
Signed-off-by: Jens Axboe <axboe@suse.de>
2006-04-19[PATCH] splice: fixup writeout path after ->map changesJens Axboe
Since ->map() no longer locks the page, we need to adjust the handling of those pages (and stealing) a little. This now passes full regressions again. Signed-off-by: Jens Axboe <axboe@suse.de>
2006-04-19[PATCH] splice: offset fixesJens Axboe
- We need to adjust *ppos for writes as well. - Copy back modified offset value if one was passed in, similar to what sendfile does. Signed-off-by: Jens Axboe <axboe@suse.de>
2006-04-19[PATCH] tee: link_pipe() must be careful when dropping one of the pipe locksJens Axboe
We need to ensure that we only drop a lock that is ordered last, to avoid ABBA deadlocks with competing processes. Signed-off-by: Jens Axboe <axboe@suse.de>
2006-04-19[PATCH] splice: cleanup the SPLICE_F_NONBLOCK handlingJens Axboe
- generic_file_splice_read() more readable and correct - Don't bail on page allocation with NONBLOCK set, just don't allow direct blocking on IO (eg lock_page). Signed-off-by: Jens Axboe <axboe@suse.de>
2006-04-19[PATCH] splice: close i_size truncate races on readJens Axboe
We need to check i_size after doing a blocking readpage. Signed-off-by: Jens Axboe <axboe@suse.de>
2006-04-11[PATCH] splice: add support for sys_tee()Jens Axboe
Basically an in-kernel implementation of tee, which uses splice and the pipe buffers as an intelligent way to pass data around by reference. Where the user space tee consumes the input and produces a stdout and file output, this syscall merely duplicates the data inside a pipe to another pipe. No data is copied, the output just grabs a reference to the input pipe data. Signed-off-by: Jens Axboe <axboe@suse.de>
2006-04-11[PATCH] splice: pass offset around for ->splice_read() and ->splice_write()Jens Axboe
We need not use ->f_pos as the offset for the file input/output. If the user passed an offset pointer in through sys_splice(), just use that and leave ->f_pos alone. Signed-off-by: Jens Axboe <axboe@suse.de>
2006-04-11[PATCH] splice: comment stylesIngo Molnar
- capitalize consistently - end sentences in one way or another - update comment text to match the implementation Signed-off-by: Ingo Molnar <mingo@elte.hu> Signed-off-by: Jens Axboe <axboe@suse.de>
2006-04-11[PATCH] splice: add Ingo as addition copyright holderJens Axboe
The comment is also somewhat out of date, correct that as well. Signed-off-by: Jens Axboe <axboe@suse.de>
2006-04-11[PATCH] splice: unlikely() optimizationsJens Axboe
Also corrects a few comments. Patch mainly from Ingo, changes by me. Signed-off-by: Ingo Molnar <mingo@elte.hu> Signed-off-by: Jens Axboe <axboe@suse.de>
2006-04-11[PATCH] splice: speedups and optimizationsJens Axboe
- Kill the local variables that cache ->nrbufs, they just take up space. - Only set do_wakeup for a real pipe. This is a big win for direct splicing. - Kill i_mutex lock around ->f_pos update, regular io paths don't do this either. Signed-off-by: Jens Axboe <axboe@suse.de>
2006-04-11[PATCH] splice: speedup __generic_file_splice_readJens Axboe
Using find_get_page() is a lot faster than find_or_create_page(). This gets splice a lot closer to sendfile() for fd -> socket transfers. Signed-off-by: Jens Axboe <axboe@suse.de>
2006-04-11[PATCH] splice: add direct fd <-> fd splicing supportJens Axboe
It's more efficient for sendfile() emulation. Basically we cache an internal private pipe and just use that as the intermediate area for pages. Direct splicing is not available from sys_splice(), it is only meant to be used for sendfile() emulation. Additional patch from Ingo Molnar to avoid the PIPE_BUFFERS loop at exit for the normal fast path. Signed-off-by: Jens Axboe <axboe@suse.de>
2006-04-10[PATCH] splice: add optional input and output offsetsIngo Molnar
add optional input and output offsets to sys_splice(), for seekable file descriptors: asmlinkage long sys_splice(int fd_in, loff_t __user *off_in, int fd_out, loff_t __user *off_out, size_t len, unsigned int flags); semantics are straightforward: f_pos will be updated with the offset provided by user-space, before the splice transfer is about to begin. Providing a NULL offset pointer means the existing f_pos will be used (and updated in situ). Providing an offset for a pipe results in -ESPIPE. Providing an invalid offset pointer results in -EFAULT. Signed-off-by: Ingo Molnar <mingo@elte.hu> Signed-off-by: Jens Axboe <axboe@suse.de>
2006-04-10[PATCH] introduce a "kernel-internal pipe object" abstractionIngo Molnar
separate out the 'internal pipe object' abstraction, and make it usable to splice. This cleans up and fixes several aspects of the internal splice APIs and the pipe code: - pipes: the allocation and freeing of pipe_inode_info is now more symmetric and more streamlined with existing kernel practices. - splice: small micro-optimization: less pointer dereferencing in splice methods Signed-off-by: Ingo Molnar <mingo@elte.hu> Update XFS for the ->splice_read/->splice_write changes. Signed-off-by: Jens Axboe <axboe@suse.de>
2006-04-10[PATCH] splice: be smarter about calling do_page_cache_readahead()Jens Axboe
We don't want to call into the read-ahead logic unless we are at the start of a page, _or_ we have multiple pages to read. Signed-off-by: Jens Axboe <axboe@suse.de>
2006-04-10[PATCH] splice: optimize the splice buffer mappingJens Axboe
We don't really need to lock down the pages, just make sure they are uptodate. Signed-off-by: Jens Axboe <axboe@suse.de>
2006-04-10[PATCH] splice: cleanup __generic_file_splice_read()Jens Axboe
The whole shadow/pages logic got overly complex, and this simpler approach is actually faster in testing. Signed-off-by: Jens Axboe <axboe@suse.de>
2006-04-10[PATCH] splice: only call wake_up_interruptible() when we really have toJens Axboe
__wake_up_common() is pretty heavy in the kernel profiles, this brings it down to a more acceptable level. Signed-off-by: Jens Axboe <axboe@suse.de>
2006-04-10[PATCH] splice: potential !page dereferenceDave Jones
We can get to out: with a NULL page, which we probably don't want to be calling page_cache_release() on. Signed-off-by: Dave Jones <davej@redhat.com> Signed-off-by: Jens Axboe <axboe@suse.de>
2006-04-10[PATCH] splice: mark the io page as accessedJens Axboe
We should do that, since we do the LRU manipulation ourselves now. Suggested by Nick Piggin. Signed-off-by: Jens Axboe <axboe@suse.de>
2006-04-02[PATCH] splice: fix page stealing LRU handling.Jens Axboe
Originally from Nick Piggin, just adapted to the newer branch. You can't check PageLRU without holding zone->lru_lock. The page release code can get away with it only because the page refcount is 0 at that point. Also, you can't reliably remove pages from the LRU unless the refcount is 0. Ever. Signed-off-by: Nick Piggin <nickpiggin@yahoo.com.au> Signed-off-by: Jens Axboe <axboe@suse.de>
2006-04-02[PATCH] splice: page stealing needs to wait_on_page_writeback()Jens Axboe
Thanks to Andrew for the good explanation of why this is so. akpm writes: If a page is under writeback and we remove it from pagecache, it's still going to get written to disk. But the VFS no longer knows about that page, nor that this page is about to modify disk blocks. So there might be scenarios in which those blocks-which-are-about-to-be-written-to get reused for something else. When writeback completes, it'll scribble on those blocks. This won't happen in ext2/ext3-style filesystems in normal mode because the page has buffers and try_to_release_page() will fail. But ext2 in nobh mode doesn't attach buffers at all - it just sticks the page in a BIO, finds some new blocks, points the BIO at those blocks and lets it rip. While that write IO's in flight, someone could truncate the file. Truncate won't block on the writeout because the page isn't in pagecache any more. So truncate will the free the blocks from the file under the page's feet. Then something else can reallocate those blocks. Then write data to them. Now, the original write completes, corrupting the filesystem. Signed-off-by: Jens Axboe <axboe@suse.de>
2006-04-02[PATCH] splice: export generic_splice_sendpageJens Axboe
Forgot that one, thanks Jeff. Also move the other EXPORT_SYMBOL to right below the functions. Signed-off-by: Jens Axboe <axboe@suse.de>
2006-04-02[PATCH] splice: add a SPLICE_F_MORE flagJens Axboe
This lets userspace indicate whether more data will be coming in a subsequent splice call. Signed-off-by: Jens Axboe <axboe@suse.de>
2006-04-02[PATCH] splice: add comments documenting more of the codeJens Axboe
Hopefully this will make Andrew a little more happy. Signed-off-by: Jens Axboe <axboe@suse.de>
2006-04-02[PATCH] splice: improve writeback and clean up page stealingJens Axboe
By cleaning up the writeback logic (killing write_one_page() and the manual set_page_dirty()), we can get rid of ->stolen inside the pipe_buffer and just keep it local in pipe_to_file(). This also adds dirty page balancing logic and O_SYNC handling. Signed-off-by: Jens Axboe <axboe@suse.de>
2006-04-02[PATCH] splice: fix shadow[] filling logicJens Axboe
Clear the entire range, and don't increment pidx or we keep filling the same position again and again. Thanks to KAMEZAWA Hiroyuki. Signed-off-by: Jens Axboe <axboe@suse.de>
2006-04-02splice: add SPLICE_F_NONBLOCK flagLinus Torvalds
It doesn't make the splice itself necessarily nonblocking (because the actual file descriptors that are spliced from/to may block unless they have the O_NONBLOCK flag set), but it makes the splice pipe operations nonblocking. Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-03-30[PATCH] splice exportsJeff Garzik
Woe be unto he who builds their filesystems as modules. Signed-off-by: Jeff Garzik <jeff@garzik.org> [ Obscure quote from the infamous geek bible? ] Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-03-30[PATCH] splice: add support for SPLICE_F_MOVE flagJens Axboe
This enables the caller to migrate pages from one address space page cache to another. In buzz word marketing, you can do zero-copy file copies! Signed-off-by: Jens Axboe <axboe@suse.de> Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-03-30[PATCH] Introduce sys_splice() system callJens Axboe
This adds support for the sys_splice system call. Using a pipe as a transport, it can connect to files or sockets (latter as output only). From the splice.c comments: "splice": joining two ropes together by interweaving their strands. This is the "extended pipe" functionality, where a pipe is used as an arbitrary in-memory buffer. Think of a pipe as a small kernel buffer that you can use to transfer data from one end to the other. The traditional unix read/write is extended with a "splice()" operation that transfers data buffers to or from a pipe buffer. Named by Larry McVoy, original implementation from Linus, extended by Jens to support splicing to files and fixing the initial implementation bugs. Signed-off-by: Jens Axboe <axboe@suse.de> Signed-off-by: Linus Torvalds <torvalds@osdl.org>
generated by cgit v1.2.3 (git 2.25.1) at 2024-07-02 07:38:51 +0000