Perl XS tutorial - digest version

This page was created on?Fri May 20 2011?and last changed on?Wed Jun 06 2012.

This is an abridged version of the XS tutorial which is supplied with Perl.

NAME

perlXStut - how to extend Perl with C

DESCRIPTION

This teaches creating a Perl extension in C. It starts with simple examples and becomes more complex. It was written for Unix.

TUTORIAL

EXAMPLE 1 - Hello world

The first example prints "Hello world". Run?h2xs -A -n test. This creates a directory named?test?and files including?Makefile.PL,?lib/test.pm,?test.xs,and?t/test.t. Initially,?test.xs?looks like this:

#include "EXTERN.h"
#include "perl.h"
#include "XSUB.h"

#include "ppport.h"

MODULE = test               PACKAGE = test

Edit?test.xs. Add

void
hello()
CODE:
    printf("Hello,world!n");

to the end. Run?perl Makefile.PL:

$ perl Makefile.PL
Checking if your kit is complete...
Looks good
Writing Makefile for test
$

This creates a file called?Makefile. Run the command "make":

$ make
cp lib/test.pm blib/lib/test.pm
perl xsubpp  -typemap typemap  test.xs > test.xsc && mv test.xsc test.c
Please specify prototyping behavior for test.xs (see perlxs manual)
cc -c     test.c
Running Mkbootstrap for test ()
chmod 644 test.bs
rm -f blib/arch/auto/test/test.so
cc  -shared -L/usr/local/lib test.o  -o blib/arch/auto/test/test.so
chmod 755 blib/arch/auto/test/test.so
cp test.bs blib/arch/auto/test/test.bs
chmod 644 blib/arch/auto/test/test.bs
Manifying blib/man3/test.3pm
$

Create a file called?hello?containing the following:

#!/usr/bin/perl
use ExtUtils::testlib;
use test;
test::hello();

Make?hello?executable with?chmod +x hello,and run it:

$ ./hello
Hello,world!
$

EXAMPLE 2 - Odd or even

This returns 1 if a number is even,and 0 if the number is odd. Add the following to the end of?test.xs:

int
is_even(input)
    int input
CODE:
    RETVAL = (input % 2 == 0);
OUTPUT:
    RETVAL

Run "make" again. Test that the extension works. Create a test script,?t/test.t,containing the following:

use Test::More tests => 4;
BEGIN { use_ok('test') };

is(&test::is_even(0),1);
is(&test::is_even(1),0);
is(&test::is_even(2),1);

Run it by typing?make test:

$ make test
PERL_DL_NONLAZY=1 /usr/bin/perl "-MExtUtils::Command::MM" "-e" "test_harness(0,'blib/lib','blib/arch')" t/*.t
t/test....ok
All tests successful.
Files=1,Tests=4,0 wallclock secs ( 0.03 cusr +  0.00 csys =  0.03 CPU)
$

The program?h2xs?is the starting point for creating extensions. It creates files in the extension directory.?Makefile.PL?is a program which generates a?Makefile?to build the extension. The fileslib/test.pm?and?test.xs?contain the meat of the extension. The .xs file holds the C routines. The .pm file tells Perl how to load the extension.

Running?make?creates a directory called?blib?("build library"). This directory contains the compiled output. The command?make test?invokes perl with?-I?(for "include") arguments so it finds the extension files in?blib. When testing extensions,use?make test,or run the test file using

perl -I blib/lib -I blib/arch t/test.t

Without this,the test script will either fail to run,or,if there is another version of the extension installed,it will use that,instead of the version which was meant to be tested.

EXAMPLE 3 - Rounding numbers

This example takes one argument as input and sets the argument to its rounded value. Add to the end of?test.xs?the following:

void
round(arg)
    double  arg
    CODE:
    if (arg > 0.0) {
            arg = floor(arg + 0.5);
    } else if (arg < 0.0) {
            arg = ceil(arg - 0.5);
    } else {
            arg = 0.0;
    }
    OUTPUT:
    arg

Add '-lm' to the line containing 'LIBS' in?Makefile.PL:

'LIBS'      => ['-lm'],# e.g.,'-lm'

This adds a link to the C maths library which contains?floor?and?ceil. Change the number of tests intest.t?to "9",

use Test::More tests => 9;

and add the following tests:

my $i;
$i = -1.5; &test::round($i); is( $i,-2.0 );
$i = -1.1; &test::round($i); is( $i,-1.0 );
$i = 0.0; &test::round($i);  is( $i,0.0 );
$i = 0.5; &test::round($i);  is( $i,1.0 );
$i = 1.2; &test::round($i);  is( $i,1.0 );

Run?perl Makefile.PL,?make,then?make test. It should print out that nine tests have passed.

In these new test cases,the argument passed to?round?was a scalar variable,?$i. Is it possible to round a constant? To see what happens,add the following to?t/test.t:

&test::round(3);

Run?make test. Perl dies:

Modification of a read-only value attempted at t/test.t line 23.

Perl won't allow changing the value of constants.

The value of the function is not being passed back as the function's return value,but by changing the value of the variable that was passed into the function.

Input and Output Parameters

The parameters passed into the XSUB are specified after declaring the function's return value and name. The output parameters are listed at the end of the function,after the?OUTPUT:?directive. The use of?RETVAL?tells Perl to send this value back as the return value of the XSUB function. In Example 3,the return value was placed in the original variable which was passed in,so it and not?RETVAL?was listed in the?OUTPUT:?section.

The program?xsubpp?translates XS into C. Its rules to convert from Perl data types,such as "scalar" or "array",to C data types such as?int,or?char,are found in a file called a "typemap". This file has three parts. The first part maps C types to a name which corresponds to Perl types. The second part contains C code which?xsubpp?uses for input parameters. The third part contains C code which?xsubppuses for output parameters.

For example,look at a portion of the C file created for the extension,?test.c:

XS(XS_test_round); /* prototype to pass -Wmissing-prototypes */
XS_test_round)
{
#ifdef dVAR
    dVAR; dXSARGS;
#else
    dXSARGS;
#endif
    if (items != 1)
       croak_xs_usage(cv,"arg");
    {
        double  arg = (double)SvNV(ST(0));
#line 30 "test.xs"
    arg > 0.0) {
            arg = floor(arg + 0.5);
    } else arg < 0.0) {
            ceil(arg - 0.5);
    } else {
            arg = 0.0;
    }
#line 137 "test.c"
        sv_setnv(ST(0),(double)arg);
        SvSETMAGIC(ST(0));
    }
    XSRETURN_EMPTY;
}

Download it here.

In the typemap file,doubles are of type?T_DOUBLE. In the?INPUT?section of?typemap,an argument that isT_DOUBLE?is assigned to the variable?arg?by calling?SvNV,then casting its value to?double,then assigning that to?arg. In the?OUTPUT?section,?arg?is passed to?sv_setnv?to be passed back to the calling subroutine. (ST(0)?is discussed in?"The argument stack").

EXAMPLE 4 - Using a header file

This example creates an extension that interacts with a C library. h2xs writes a .pm and .xs file for an example library.

Create a new directory called?test2. In?test2,create a subdirectory called?testlib. Change into that directory.

In the testlib directory,create?testlib.h:

#define TESTVAL     4

extern double       foo(int,long,const char*);

And?testlib.c:

#include <stdlib.h>
#include "testlib.h"

double
foo(int a,long b,const char *c)
{
    return (a + b + atof(c) + TESTVAL);
}

Create?Makefile.PL:

use ExtUtils::MakeMaker;
$Verbose = 1;
WriteMakefile(
    NAME   => 'test2::testlib',SKIP   => [qw(all static static_lib dynamic dynamic_lib)],clean  => {'FILES' => 'libtestlib$(LIB_EXT)'},);


sub MY::top_targets
{
    return <<EOF;
all :: static

pure_all :: static

static ::       libtestlib$(LIB_EXT)

libtestlib$(LIB_EXT): $(O_FILES)
t$(AR) cr libtestlib$(LIB_EXT) $(O_FILES)
t$(RANLIB) libtestlib$(LIB_EXT)

EOF
}

Change to the directory above test2 and run the following command:

$ h2xs -O -n test2 ./test2/testlib/testlib.h

This prints out a warning about overwriting test2,but the files are stored in?test2/testlib,and will be untouched.

The?Makefile.PL?that h2xs generates doesn't know about the?testlib?directory. Tell h2xs that there is a subdirectory by adding the argument?MYEXTLIB?to the call to?WriteMakefile:

WriteMakefile(
    'NAME'      => 'test2','VERSION_FROM' => 'test2.pm','LIBS'      => [''],'DEFINE'    => '','INC'       => '','MYEXTLIB' => 'testlib/libtestlib$(LIB_EXT)',);

At the end add a subroutine:

sub MY::postamble {
return <<EOF;
$(MYEXTLIB): testlib/Makefile
tcd testlib && $(MAKE) $(PASSTHRU)
EOF
}

In the .xs file,edit the?#include?line to read:

#include "testlib/testlib.h"

Add the following to the end of the .xs file:

double
foo(a,b,c)
    int             a
    long            b
    const char *    c
    OUTPUT:
    RETVAL

Run Perl on?Makefile.PL?in?test2. Notice that it also created a file called?Makefile?in the directorytestlib. Run "make". Watch that it does change directory into?testlib?and run "make" in there as well.

Edit?t/test2.t. Change the number of tests to 4. Add the following lines to the end:

is( &test2::foo(1,2,"Hello,world!"),7 );
is( &test2::foo(1,"0.0"),7 );
ok( abs(&test2::foo(0,"-3.4") - 0.6) <= 0.01 );

Run?make test. There are some warnings on missing tests for the test2::testlib extension,but these may be ignored.

Unlike previous examples,this example runs h2xs on an include file. This has caused new parts to appear in the .pm and .xs files. In the .xs file,there's now an include directive with the absolute path to?testlib.h. There's now some new C code added to the .xs file. The?constant?routine makes the values defined in the header file accessible to the Perl script (by calling either?TESTVAL?or?&test2::TESTVAL). There's also some XS code to allow calls to the?constant?routine.

Perl knows about the library in the?testlib?subdirectory. That required only the addition of?MYEXTLIB?to the WriteMakefile call and the replacement of the postamble subroutine to cd into the subdirectory and run make. This replaces the postamble subroutine. This code specifies that the library to be created here is a static archive library (as opposed to a dynamically loadable library) and provides the commands to build it.

Anatomy of .xs file

The .xs file of?"EXAMPLE 4 - Using a header file"?contains new elements. Lines before the line

MODULE = test2              PACKAGE = test2

is C. This C says which headers to include,and defines functions. xsubpp performs no translations on this part except skipping over POD documentation (see?perlpod). It goes directly into the generated output C file.

However,anything after the above line is XSUB functions.?xsubpp?translates these descriptions to C code which makes these functions visible from the Perl interpreter.

The function?constant?appears twice in the generated .xs file: once in the first part,as a static C function,then another time in the second part,when an XSUB interface to this static C function is defined.

Getting the fat out of XSUBs

In?"EXAMPLE 4 - Using a header file"?the second part of .xs file contained the following description of an XSUB:

Note that in contrast with?"EXAMPLE 1 - Hello world",?"EXAMPLE 2 - Odd or even"?and?"EXAMPLE 3 - Rounding numbers". this description does not contain code for what is done during a call to?foo(). Even if a CODE section is added to this XSUB:
 
the result is almost identical generated C code:?xsubpp?compiler figures out the?CODE:?section from the first two lines of the description of XSUB. The?OUTPUT:?section is absolutely the same. The?OUTPUT:section can be removed as well,if a?CODE:?section is not specified: xsubpp can see that it needs to generate a function call section,and will autogenerate the OUTPUT section too. Thus the XSUB can be as little as:
 
This can also be done for
 
int
is_even(input)
    int     input
    CODE:
    RETVAL = (input % 2 == 0);
    OUTPUT:
    RETVAL
 
of?"EXAMPLE 2 - Odd or even",if a C function?int is_even(int input)?is supplied. As in?"Anatomy of .xs file",this may be placed in the first part of the .xs file:
 
int
is_even(int arg)
{
    return (arg % 2 == 0);
}
 
After having this in the first part of .xs file,the "Perl glue" part becomes as simple as
 
int
is_even(input)
    int     input
 
This technique of separation of the glue part from the workhorse part has tradeoffs: to change a Perl interface requires changing two places in the code. However,it removes clutter,and makes the workhorse part independent from the idiosyncratic Perl calling conventions. (In fact,there is nothing Perl-specific in the above description,a different version of?xsubpp?might have translated this to TCL glue or Python glue as well.)
 
More about XSUB arguments
 
When arguments to routines in the .xs file are specified,three things are passed for each argument listed. The first is the order of that argument relative to the others (first,second,third). The second is the type of argument (int,?char*). The third is the calling convention for the argument in the call to the library function.
 
Suppose two C functions with similar declarations,for example
 
int string_length (char *s);
int upper_case_char (char *cp);
 
operate differently on the argument:?string_length?inspects the characters pointed to by?s?without changing their values,but?upper_case_char?dereferences?cp?and manipulates what it points to. From Perl,these functions are used in a different manner.
 
Tell?xsubpp?which is which by replacing the?*?before the argument by?&. An ampersand,?&,means that the argument should be passed to a library function by its address. In the example,238); margin:20px; padding:10px">int
string_length(s)
    char * s
 
but
 
int
upper_case_char(cp)
    char & cp
 
int
foo(a,b)
    char & a
    char * b
 
The first Perl argument to this function is treated as a char and assigned to?a,and its address is passed into?foo. The second Perl argument is treated as a string pointer and assigned to?b. The value of b is passed into the function foo. The call to?foo?that xsubpp generates looks like this:
 
foo (& a,b);
 
The argument stack
 
In the C code generated by the examples,except example 1,there are a number of references toST(0),?ST(1)?and so on.?ST?is a macro that points to the?nth argument on the argument stack.?ST(0)?is thus the first argument on the stack and therefore the first argument passed to the XSUB,?ST(1)?is the second argument,and so on.
 
The list of arguments to the XSUB in the .xs file tells?xsubpp?which argument corresponds to which of the argument stack (i.e.,the first one listed is the first argument,and so on). These must be listed in the same order as the function expects them.
 
The actual values on the argument stack are pointers to the values passed in. When an argument is listed as being an OUTPUT value,its corresponding value on the stack (i.e.,?ST(0)?if it was the first argument) is changed. Verify this by looking at the C code generated for Example 3. The code for?roundcontains lines that look like this:
 
double  arg = (double)SvNV(ST(0));
/* Round the contents of the variable arg */
sv_setnv(ST(0),(double)arg);
 
The arg variable is initially set by taking the value from?ST(0),then is stored back into?ST(0)?at the end of the routine.
 
XSUBs are also allowed to return lists,not just scalars. This must be done by manipulating stack values?ST(0),?ST(1),etc. See?perlxs.
 
XSUBs are also allowed to avoid automatic conversion of Perl function arguments to C function arguments. See?perlxs. Some people prefer manual conversion by inspecting?ST(i)?even in the cases when automatic conversion will do,arguing that this makes the logic of an XSUB call clearer. Compare with?"Getting the fat out of XSUBs"?for a similar tradeoff of a complete separation of "Perl glue" and "workhorse" parts of an XSUB.
 
EXAMPLE 5 - Returning an array
 
This example illustrates working with the argument stack. The previous examples have all returned only a single value. This example shows an extension which returns an array. It uses the?statfssystem call.
 
Return to the?test?directory. Add the following to the top of?test.xs,after?#include "XSUB.h":
 
#include <sys/vfs.h>
 
Add to the end:
 
void
statfs(path)
    char *  path
    INIT:
    int i;
    struct statfs buf;

    PPCODE:
    i = statfs(path,&buf);
    if (i == 0) {
            XPUSHs(sv_2mortal(newSVnv(buf.f_bavail)));
            XPUSHs(sv_2mortal(newSVnv(buf.f_bfree)));
            XPUSHs(sv_2mortal(newSVnv(buf.f_blocks)));
            XPUSHs(sv_2mortal(newSVnv(buf.f_bsize)));
            XPUSHs(sv_2mortal(newSVnv(buf.f_ffree)));
            XPUSHs(sv_2mortal(newSVnv(buf.f_files)));
            XPUSHs(sv_2mortal(newSVnv(buf.f_type)));
    } else {
            XPUSHs(sv_2mortal(newSVnv(errno)));
    }
 
In?test.t,increment the number of tests from 9 to 11,and add
 
@a = &test::statfs("/blech");
ok( scalar(@a) == 1 && $a[0] == 2 );
@a = &test::statfs("/");
is( scalar(@a),7 );
 
The?INIT:?directive contains code that will be placed immediately after the argument stack is decoded. This routine returns a different number of arguments depending on whether the call to?statfssucceeds. If there is an error,the error number is returned as a single-element array. If the call is successful,then a 7-element array is returned. Since only one argument is passed into this function,the stack needs room to hold the seven values which may be returned.
 
The?PPCODE:?directive does this. It tells?xsubpp?that the xsub manages the return values put on the argument stack by itself.
 
To place values to be returned to the caller onto the stack,use the series of macros that begin withXPUSH. There are five different versions,for placing integers,unsigned integers,doubles,strings,and Perl scalars on the stack. In the example,a Perl scalar was placed onto the stack.
 
The values pushed onto the return stack of the XSUB are "mortal"?SVs. They are made "mortal" so that once their values are copied by the calling program,the SV's that held the returned values can be deallocated. If they were not mortal,then they would continue to exist after the XSUB routine returned,but would not be accessible,causing a memory leak.
 
EXAMPLE 6 - Arrays and hashes
 
This example takes a reference to an array as input,and returns a reference to an array of hashes. This demonstrates manipulating Perl's data types.
 
This example is based on?"EXAMPLE 5 - Returning an array". It takes a reference to an array of filenames as input,calls?statfs?for each file name,and returns a reference to an array of hashes containing the data for each of the filesystems.
 
In the?test?directory add the following code to the end of?test.xs:
 
SV *
multi_statfs(paths)
    SV * paths
INIT:
    AV * results;
    I32 numpaths = 0;
    int i,n;
    struct statfs buf;

    if ((!SvROK(paths))
    || (SvTYPE(SvRV(paths)) != SVt_PVAV)
    || ((numpaths = av_len((AV *)SvRV(paths))) < 0))
    {
    XSRETURN_UNDEF;
    }
    results = (AV *)sv_2mortal((SV *)newAV());
CODE:
    for (n = 0; n <= numpaths; n++) {
    HV * rh;
    STRLEN l;
    char * fn = SvPV(*av_fetch((AV *)SvRV(paths),n,0),l);

    i = statfs(fn,&buf);
    if (i != 0) {
        av_push(results,newSVnv(errno));
        continue;
    }

    rh = (HV *)sv_2mortal((SV *)newHV());

    hv_store(rh,"f_bavail",8,newSVnv(buf.f_bavail),0);
    hv_store(rh,"f_bfree",7,newSVnv(buf.f_bfree),"f_blocks",newSVnv(buf.f_blocks),"f_bsize",newSVnv(buf.f_bsize),"f_ffree",newSVnv(buf.f_ffree),"f_files",newSVnv(buf.f_files),"f_type",6,newSVnv(buf.f_type),0);

    av_push(results,newRV((SV *)rh));
    }
    RETVAL = newRV((SV *)results);
OUTPUT:
    RETVAL
 
Add to?test.t
 
$results = test::multi_statfs([ '/','/blech' ]);
ok( ref $results->[0] );
ok( ! ref $results->[1] );
 
This function does not use a typemap. Instead,it accepts one?SV*?(scalar) parameter,and returns anSV*. These scalars are populated within the code. Because it only returns one value,there is no need for a?PPCODE:?directive,only?CODE:?and?OUTPUT:?directives.
 
When dealing with references,it is important to handle them with caution. The?INIT:?block first checks that?SvROK?returns true,which indicates that paths is a valid reference. It then verifies that the object referenced by paths is an array,using?SvRV?to dereference paths,and?SvTYPE?to discover its type. As an added test,it checks that the array referenced by paths is non-empty,using?av_len,which returns -1 if the array is empty. The?XSRETURN_UNDEF?macro aborts the XSUB and returns the undefined value whenever all three of these conditions are not met.
 
We manipulate several arrays in this XSUB. An array is represented internally by a pointer to?AV. The functions and macros for manipulating arrays are similar to the functions in Perl:?av_len?returns the highest index in an?AV*,much like?$#array;?av_fetch?fetches a scalar value from an array,given its index;?av_push?pushes a scalar value onto the array,extending it if necessary.
 
Specifically,we read pathnames one at a time from the input array,and store the results in an output array (results) in the same order. If?statfs?fails,the element pushed onto the return array is the value of?errno?after the failure. If?statfs?succeeds,the value pushed onto the return array is a reference to a hash containing some of the information in the?statfs?structure.
 
As with the return stack,it would be possible (and a small performance win) to pre-extend the return array before pushing data into it,since we know how many elements we will return:
 
av_extend(results,numpaths);
 
We are performing only one hash operation in this function,which is storing a new scalar under a key using?hv_store. A hash is represented by an HV* pointer. Like arrays,the functions for manipulating hashes from an XSUB mirror the functionality available from Perl. See?perlguts?and perlapi for details.
 
To create a reference,use?newRV. An?AV*?or an?HV*?can be cast to type?SV*?in this case. This allows taking references to arrays,hashes and scalars with the same function. Conversely,the?SvRV?function always returns an SV*,which may need to be cast to the appropriate type if it is something other than a scalar (check with?SvTYPE).
 
EXAMPLE 7 - Passing open files
 
To create a wrapper around?fputs,238); margin:20px; padding:10px">#define PERLIO_NOT_STDIO 0
#include "EXTERN.h"
#include "perl.h"
#include "XSUB.h"

#include <stdio.h>

int
fputs(s,stream)
    char *          s
    FILE *          stream

SEE ALSO

For more information,consult

perlguts

This documents functions such as?SvNV?which convert Perl scalars into C?doubles.

perlapi (see? http://perldoc.perl.org/perlapi.html)

This lists functions such as?croak?used for error handling. Beware that the version of the file on the?http://search.cpan.org?website is not formatted correctly.

perlxs

This is the "official" documentation for Perl XS.

perlmod

This is the "official" documentation for Perl modules.

SPECIAL NOTES

make

This tutorial assumes that the "make" program that Perl is configured to use is called?make. Instead of running "make" in the examples,a substitute may be required. The command?perl -V:make?gives the name of the substitute program.

AUTHOR

Jeff Okamoto,reviewed and assisted by Dean Roehrich,Ilya Zakharevich,Andreas Koenig,and Tim Bunce. PerlIO material contributed by Lupe Christoph,with some clarification by Nick Ing-Simmons. Changes for h2xs as of Perl 5.8.x by Renee Baecker. This digest web version (http://www.lemoda.net/xs/perlxstut/) was edited from that found in the Perl distribution by Ben Bullock.

You can download?the POD (Plain Old Documentation) format of this article. This may contain one or two bits of formatting which aren't actually POD.