Home » Computers & Electronics

The Magic of Emacs Comint

3 Comments

3 Like

Like 3

Pin It

Ranked #2,487 in Computers & Electronics, #43,510 overall

A Simple Emacs Comint Subscriber

Comint is one of the main reasons I moved to emacs after using vi/vim for 14 years. It is an emacs library for interacting with external processes. Many text editors have a way of calling out to a process and synchronously getting the result. Emacs goes one step beyond this with its awesome support for asynchronous processes.

If you find this lens interesting/useful, please consider logging in and rating in it. Thank you.

Emacs External Processes

Interacting with Asynchronous Processes

Emacs does have some low level commands for starting and interacting with long-running processes. However, comint makes this much easier. It was initially designed for interacting with an external interpreter. However, it is perfect for any long running process.

A Stock Price Generator

Let me demonstrate how easy it is with an example. Let's say we want to connect to a program that is constantly pumping out some data. An example might be something reporting stock prices (although admittedly just at the moment I don't know why anyone wants to know about them).

As this is just a simple example, I don't really want to connect to a real stock price feed. Instead, I'll make an example program that generates dummy prices. As most data seems to be passed around as xml these days a price for a particular stock might looks like this:

<data>
<ticker>SADCO</ticker>
<price>101.2191</price>
</data>

Perl Script - ticker.pl

The ticker indicates which company this is, for example MSFT is Microsoft. I wrote a perl script to generate a new price for a random fake ticker every second.

#!/usr/bin/perl

use strict;
use warnings;

use English qw(-no_match_vars);

$OUTPUT_AUTOFLUSH = 1;

my @tickers = qw(MYCO YRCO ANCO ROSCO SADCO);

while (1) {
    my $ticker = $tickers[int(rand(5))];
    my $price = sprintf "%.4lf", (rand(20) + 90);

    print <<"EOF"
<data>
  <ticker>$ticker</ticker>
  <price>$price</price>
</data>
EOF

    sleep 1;
}

Getting Started with Comint

The first comint call generally starts the process and redirects any output to a buffer. We do this with (apply 'make-comint <buffer> <program> nil args). As we are connecting to the stdout. Note that comint can also connect to a TCP socket.

(require 'comint)

(defconst *perl* "/usr/bin/perl")
(defconst *ticker*
  (expand-file-name "~/emacs-files/testing/ticker.pl"))

(defun my-subscribe ()
  (apply 'make-comint "subscriber" *perl* nil
         (list *ticker*)))

Extract the Ticker and the Price

We need a function that extracts the ticker and the price from the xml. The easiest thing is to extract it to an assoc array. I can't be bothered to write a proper parser and this regex will obviously work on the simple xml in the example. Apologies to Zawinski but it doesn't seem like I now have two problems.

(defconst *my-fields* '(ticker price))
(defconst *my-re-fields*
  (format "<\$%s\$>\$[^<]*\$"
          (mapconcat (lambda (e) (symbol-name e)) *my-fields* "\\|")))

(defun my-extract-fields (xml)
  (let ((pos 0)
        (ret-val nil))
    (while (string-match *my-re-fields* xml pos)
      (push (cons (match-string 1 xml) (match-string 2 xml)) ret-val)
      (setq pos (match-end 2)))
    ret-val))

Formatting The Output

We will want to transform the extracted data into something for people to read. my-data-format handles this.

(defun my-get-value (map key)
  (let ((pair (assoc key map)))
    (if pair (cdr pair) "undefined")))

(defun my-data-format (fields)
  (format "Ticker: %5s Price: %8.4f"
          (my-get-value fields "ticker")
          (string-to-number (my-get-value fields "price"))))

Planning on Displaying the Output

Now for the complicated bit. Comint provides some hooks you can add code to for when the external process has sent some data. The main ones are comint-output-filter-functions and comint-preoutput-filter-functions. *preoutput* is called as the data arrives but before it has been written into the buffer. This gives you the option of changing what is written to the buffer. *output* is called after the data has been written.

One mini-gotcha when using the output filters is that emacs doesn't read a line of input at a time. It tries and fills its input buffer as far as it is able. Because of this, we can never be sure we received a complete message so it is best to concatenate all input together until we have what we need. Once we have the complete message we can extract the fields we want and then we need to decide what to do with it.

Now, we could transform the data on the way into the buffer, but I prefer to write the data into a brand new buffer. This way, I can make a mode for the new buffer (if necessary) without deriving from comint. It also makes it easier to debug as I can see exactly what information was received from the external process.

I decided to write a line like the following:

Ticker: MYCO Price: 95.1115

Any price updates for the same ticker will overwrite the original line.

Subscriber Output Filter

The Initial Version

(defvar my-buffer "")

(defun my-output-filter (output)
  (setq my-buffer (concat my-buffer output))
  (while (string-match "" my-buffer)
    (let* ((end (match-end 0))
           (xml (substring my-buffer 0 end))
           (fields (my-extract-fields xml))
           (ticker (my-get-value fields "ticker")))
      (when (and output (get-buffer "*display*"))
        (with-current-buffer "*display*"
          (goto-char (point-min))
          (if (re-search-forward (format "Ticker: +%s" ticker) nil t)
              (progn
                (move-beginning-of-line 1)
                (kill-line 1)
                (insert (format "%s\n" (my-data-format fields))))
            (goto-char (point-max))
            (insert (format "%s\n" (my-data-format fields))))))
      (setq my-buffer (substring my-buffer end))))
  output)

(add-hook 'comint-preoutput-filter-functions 'my-output-filter)

Improving The Output

Highlighting Updates

Then I decided, wouldn't it be nice if price updates were highlighted in some way, perhaps by flashing that line for half a second in magenta. Ambitious readers might like to modify it so that it flashes green for a price increase and red for a price decrease.

The Updated Output Filter

(defface my-face-magenta
  '((((class color)) (:background "magenta"))
    (t (:bold t)))
  "my magenta face")

(defun my-add-overlay (begin end face)
  (let ((overlay (make-overlay begin end)))
    (overlay-put overlay 'face face)
    overlay))

(defun my-output-filter (output)
  (setq my-buffer (concat my-buffer output))
  (while (string-match "" my-buffer)
    (let* ((end (match-end 0))
           (xml (substring my-buffer 0 end))
           (fields (my-extract-fields xml))
           (ticker (my-get-value fields "ticker")))
      (when (and output (get-buffer "*display*"))
        (with-current-buffer "*display*"
          (goto-char (point-min))
          (if (not (re-search-forward (format "Ticker: +%s" ticker) nil t))
              (goto-char (point-max))
            (move-beginning-of-line 1)
            (kill-line 1))
          (let ((begin (point))
                (overlay nil))
            (insert (format "%s\n" (my-data-format fields)))
            (setq overlay (my-add-overlay begin (point) 'my-face-magenta))
            (run-with-timer 0.5 nil (lambda (overlay)
                                    (with-current-buffer "*display*"
                                      (delete-overlay overlay)))
                            overlay))))
      (setq my-buffer (substring my-buffer end))))
  output)

Split the Window and Subscribe

Finishing Touches

And now we just need to configure our windows nicely and we're done. If you can think of anything else I need to do to clarify the example, let me know in the guestbook at the bottom of the lens.

(defun my-unsubscribe ()
  (with-current-buffer "*subscriber*"
    (comint-kill-subjob)))

(defun my-config-display ()
  (delete-other-windows)
  (switch-to-buffer-other-window "*display*")
  (erase-buffer)
  (other-window -1))

(my-config-display)
(my-subscribe)
(my-unsubscribe)

The Emacs Blog

Check Out My Other Emacs Lenses

Comments Or Suggestions?

Please Let Me Know

Reply

Robert McIntyre Jul 14, 2010 @ 8:13 am | delete

you need to have a semicolon at the end of your HERE print.

print "EOF" --> print "EOF";

I'm trying to get this to work, so I put this all in my .emacs file, but what do I do to actually get it to do anything?

Reply

jareddavison2009 Apr 9, 2009 @ 1:19 am | delete

Hi Seth, you're welcome. If you have any questions or suggestions for improvements, let me know.

Thanks.

Reply

Seth Apr 6, 2009 @ 11:54 am | delete

This is great. Thanks for writing this.
I'm in the midst of trying to do something similar so you just saved this elisp beginner a whole bunch of time.