CL-Mmapped-File

CL-Mmapped-File is a small wrapper around the MMAP library to read and write mmap()'ed files. It's goal is not to support everything that you can do with mmap(), but to instead provide a convenient stream-like interface for very fast reading/writing to files that are mmap()'ed.

Wanna support Remilia? Buy me a coffee on Ko-Fi, or support me through Liberapay.

How do I get set up?

Most dependencies can be installed using QuickLisp (though this hasn't been tested). The exception is the CL-SDM library, which can be obtained here.

You will need Fossil installed to clone this repo.

Place this library (and the CL-SDM dependency) somewhere where ASDF can find them, then run this in your REPL:

(asdf:load-system :cl-mmapped-file)

Usage

There are two ways to use CL-Mmapped-File: using the core interface, or the Grey Streams interface. The Grey Streams interface lets you use (most) standard stream functions, while the core interface is more strict but very slightly faster due to aggressive typing and no method calls.

;; Example using the core interface.
(use-package :cl-mmapped-file)

;; Print all the data in the file as 32-bit signed integers.
(with-mmapped-file (in #P"/path/to/file")
  (handler-case
      (loop do
        (print (mmapped-file-read-int32 in)))
    (mmapped-file-eof-error ()
      (print "All int32's read")))
      
  ;; Now get a five bytes from a random position in the file without changing
  ;; the internal read cursor's position.
  (print (mmapped-file-get-bytes in (random (- (mmapped-file-size in) 5)) 5)))

;; Example using the Grey Streams interface.
(use-package :cl-mmapped-file-grey)

;; Print a text file.
(with-mmapped-stream (in #P"/path/to/file")
  (loop for line = (read-line in nil nil)
        while line do (format t "~a~%" line)))

;; Read a file in blocks of 512 bytes.
(let ((all-data (make-array 0)))
  (with-mmapped-stream (in #P"/path/to/file")
    (loop with buf = (make-array 512 :element-type '(unsigned-byte 8)
                                     :initial-element 0)
          for pos = (read-sequence buf in)
          while (plusp pos) do
            (setf all-data (concatenate 'vector all-data (subseq buf 0 pos)))))
  (print (length all-data)))

Development

Style info

I use a slightly unorthodox style for my code. Aside from these differences, please use normal Lisp formatting.

• Keep lines 118 characters or shorter. Obviously sometimes you can't, but please try. Use 115 characters for Markdown files, though.
• I mark types using the form T/..... For example, T/SOME-NEAT-TYPE. For predicates on these, use SOME-NEAT-TYPE-P.
• No tabs. Only spaces.

How do I contribute?

1. Go to https://chiselapp.com/user/MistressRemilia/repository/cl-mmapped-file and clone the Fossil repository.
2. Create a new branch for your feature.
3. Push locally to the new branch.
4. Create a bundle with Fossil that contains your changes.
5. Get in contact with me.

Contributors

• Remilia Scarlet - creator and maintainer
  • Homepage: https://remilia.sdf.org/
  • Gemini: gemini://nanako.mooo.com/
  • Mastodon: @MistressRemilia@social.sdf.org
  • Email: zremiliaz@postzeoz.jpz My real address does not contain Z's

Links and Licenses

CL-RemiGemini is under the GNU Affero General Public License version 3.