NAME

perlguts - Perl's Internal Functions

DESCRIPTION

This document attempts to describe some of the internal functions of the Perl executable. It is far from complete and probably contains many errors. Please refer any questions or comments to the author below.

Datatypes

Perl has three typedefs that handle Perl's three main data types:

SV Scalar Value AV Array Value HV Hash Value

Each typedef has specific routines that manipulate the various data types.

What is an "IV"?

Perl uses a special typedef IV which is large enough to hold either an integer or a pointer.

Perl also uses two special typedefs, I32 and I16, which will always be at least 32-bits and 16-bits long, respectively.

Working with SV's

An SV can be created and loaded with one command. There are four types of values that can be loaded: an integer value (IV), a double (NV), a string, (PV), and another scalar (SV).

The four routines are:

SV* newSViv(IV); SV* newSVnv(double); SV* newSVpv(char*, int); SV* newSVsv(SV*);

To change the value of an *already-existing* scalar, there are five routines:

void sv_setiv(SV*, IV); void sv_setnv(SV*, double); void sv_setpvn(SV*, char*, int) void sv_setpv(SV*, char*); void sv_setsv(SV*, SV*);

Notice that you can choose to specify the length of the string to be assigned by using sv_setpvn or newSVpv, or you may allow Perl to calculate the length by using sv_setpv or specifying 0 as the second argument to newSVpv. Be warned, though, that Perl will determine the string's length by using strlen, which depends on the string terminating with a NUL character.

To access the actual value that an SV points to, you can use the macros:

SvIV(SV*) SvNV(SV*) SvPV(SV*, STRLEN len)

which will automatically coerce the actual scalar type into an IV, double, or string.

In the SvPV macro, the length of the string returned is placed into the variable len (this is a macro, so you do not use &len). If you do not care what the length of the data is, use the global variable na. Remember, however, that Perl allows arbitrary strings of data that may both contain NUL's and not be terminated by a NUL.

If you simply want to know if the scalar value is TRUE, you can use:

SvTRUE(SV*)

Although Perl will automatically grow strings for you, if you need to force Perl to allocate more memory for your SV, you can use the macro

SvGROW(SV*, STRLEN newlen)

which will determine if more memory needs to be allocated. If so, it will call the function sv_grow. Note that SvGROW can only increase, not decrease, the allocated memory of an SV.

If you have an SV and want to know what kind of data Perl thinks is stored in it, you can use the following macros to check the type of SV you have.

SvIOK(SV*) SvNOK(SV*) SvPOK(SV*)

You can get and set the current length of the string stored in an SV with the following macros:

SvCUR(SV*) SvCUR_set(SV*, I32 val)

But note that these are valid only if SvPOK() is true.

If you want to append something to the end of string stored in an SV*, you can use the following functions:

void sv_catpv(SV*, char*); void sv_catpvn(SV*, char*, int); void sv_catsv(SV*, SV*);

The first function calculates the length of the string to be appended by using strlen. In the second, you specify the length of the string yourself. The third function extends the string stored in the first SV with the string stored in the second SV. It also forces the second SV to be interpreted as a string.

If you know the name of a scalar variable, you can get a pointer to its SV by using the following:

SV* perl_get_sv("varname", FALSE);

This returns NULL if the variable does not exist.

If you want to know if this variable (or any other SV) is actually defined , you can call:

SvOK(SV*)

The scalar undef value is stored in an SV instance called sv_undef. Its address can be used whenever an SV* is needed.

There are also the two values sv_yes and sv_no, which contain Boolean TRUE and FALSE values, respectively. Like sv_undef, their addresses can be used whenever an SV* is needed.

Do not be fooled into thinking that (SV *) 0 is the same as &sv_undef. Take this code:

SV* sv = (SV*) 0; if (I-am-to-return-a-real-value) { sv = sv_2mortal(newSViv(42)); } sv_setsv(ST(0), sv);

This code tries to return a new SV (which contains the value 42) if it should return a real value, or undef otherwise. Instead it has returned a null pointer which, somewhere down the line, will cause a segmentation violation, or just weird results. Change the zero to &sv_undef in the first line and all will be well.

To free an SV that you've created, call SvREFCNT_dec(SV*). Normally this call is not necessary. See the section on MORTALITY.

What's Really Stored in an SV?

Recall that the usual method of determining the type of scalar you have is to use Sv*OK macros. Since a scalar can be both a number and a string, usually these macros will always return TRUE and calling the Sv*V macros will do the appropriate conversion of string to integer/double or integer/double to string.

If you really need to know if you have an integer, double, or string pointer in an SV, you can use the following three macros instead:

SvIOKp(SV*) SvNOKp(SV*) SvPOKp(SV*)

These will tell you if you truly have an integer, double, or string pointer stored in your SV. The "p" stands for private.

In general, though, it's best to just use the Sv*V macros.

Working with AV's

There are two ways to create and load an AV. The first method just creates an empty AV:

AV* newAV();

The second method both creates the AV and initially populates it with SV's:

AV* av_make(I32 num, SV **ptr);

The second argument points to an array containing num SV*'s. Once the AV has been created, the SV's can be destroyed, if so desired.

Once the AV has been created, the following operations are possible on AV's:

void av_push(AV*, SV*); SV* av_pop(AV*); SV* av_shift(AV*); void av_unshift(AV*, I32 num);

These should be familiar operations, with the exception of av_unshift. This routine adds num elements at the front of the array with the undef value. You must then use av_store (described below) to assign values to these new elements.

Here are some other functions:

I32 av_len(AV*); /* Returns highest index value in array */ SV** av_fetch(AV*, I32 key, I32 lval); /* Fetches value at key offset, but it stores an undef value at the offset if lval is non-zero */ SV** av_store(AV*, I32 key, SV* val); /* Stores val at offset key */

Take note that these two functions return SV**'s, not SV*'s.

void av_clear(AV*); /* Clear out all elements, but leave the array */ void av_undef(AV*); /* Undefines the array, removing all elements */

If you know the name of an array variable, you can get a pointer to its AV by using the following:

AV* perl_get_av("varname", FALSE);

This returns NULL if the variable does not exist.

Working with HV's

To create an HV, you use the following routine:

HV* newHV();

Once the HV has been created, the following operations are possible on HV's:

SV** hv_store(HV*, char* key, U32 klen, SV* val, U32 hash); SV** hv_fetch(HV*, char* key, U32 klen, I32 lval);

The klen parameter is the length of the key being passed in. The val argument contains the SV pointer to the scalar being stored, and hash is the pre-computed hash value (zero if you want hv_store to calculate it for you). The lval parameter indicates whether this fetch is actually a part of a store operation.

Remember that hv_store and hv_fetch return SV**'s and not just SV*. In order to access the scalar value, you must first dereference the return value. However, you should check to make sure that the return value is not NULL before dereferencing it.

These two functions check if a hash table entry exists, and deletes it.

bool hv_exists(HV*, char* key, U32 klen); SV* hv_delete(HV*, char* key, U32 klen, I32 flags);

And more miscellaneous functions:

void hv_clear(HV*); /* Clears all entries in hash table */ void hv_undef(HV*); /* Undefines the hash table */

Perl keeps the actual data in linked list of structures with a typedef of HE. These contain the actual key and value pointers (plus extra administrative overhead). The key is a string pointer; the value is an SV*. However, once you have an HE*, to get the actual key and value, use the routines specified below.

I32 hv_iterinit(HV*); /* Prepares starting point to traverse hash table */ HE* hv_iternext(HV*); /* Get the next entry, and return a pointer to a structure that has both the key and value */ char* hv_iterkey(HE* entry, I32* retlen); /* Get the key from an HE structure and also return the length of the key string */ SV* hv_iterval(HV*, HE* entry); /* Return a SV pointer to the value of the HE structure */ SV* hv_iternextsv(HV*, char** key, I32* retlen); /* This convenience routine combines hv_iternext, hv_iterkey, and hv_iterval. The key and retlen arguments are return values for the key and its length. The value is returned in the SV* argument */

If you know the name of a hash variable, you can get a pointer to its HV by using the following:

HV* perl_get_hv("varname", FALSE);

This returns NULL if the variable does not exist.

The hash algorithm, for those who are interested, is:

i = klen; hash = 0; s = key; while (i--) hash = hash * 33 + *s++;

Creating New Variables

To create a new Perl variable, which can be accessed from your Perl script, use the following routines, depending on the variable type.

SV* perl_get_sv("varname", TRUE); AV* perl_get_av("varname", TRUE); HV* perl_get_hv("varname", TRUE);

Notice the use of TRUE as the second parameter. The new variable can now be set, using the routines appropriate to the data type.

There are additional bits that may be OR'ed with the TRUE argument to enable certain extra features. Those bits are:

0x02 Marks the variable as multiply defined, thus preventing the "Indentifier <varname> used only once: possible typo" warning. 0x04 Issues a "Had to create <varname> unexpectedly" warning if the variable didn't actually exist. This is useful if you expected the variable to already exist and want to propagate this warning back to the user. If the C<varname> argument does not contain a package specifier, it is created in the current package.

References

References are a special type of scalar that point to other data types (including references).

To create a reference, use the following command:

SV* newRV((SV*) thing);

The thing argument can be any of an SV*, AV*, or HV*. Once you have a reference, you can use the following macro to dereference the reference:

SvRV(SV*)

then call the appropriate routines, casting the returned SV* to either an AV* or HV*, if required.

To determine if an SV is a reference, you can use the following macro:

SvROK(SV*)

To actually discover what the reference refers to, you must use the following macro and then check the value returned.

SvTYPE(SvRV(SV*))

The most useful types that will be returned are:

SVt_IV Scalar SVt_NV Scalar SVt_PV Scalar SVt_PVAV Array SVt_PVHV Hash SVt_PVCV Code SVt_PVMG Blessed Scalar

XSUB's and the Argument Stack

The XSUB mechanism is a simple way for Perl programs to access C subroutines. An XSUB routine will have a stack that contains the arguments from the Perl program, and a way to map from the Perl data structures to a C equivalent.

The stack arguments are accessible through the ST(n) macro, which returns the n'th stack argument. Argument 0 is the first argument passed in the Perl subroutine call. These arguments are SV*, and can be used anywhere an SV* is used.

Most of the time, output from the C routine can be handled through use of the RETVAL and OUTPUT directives. However, there are some cases where the argument stack is not already long enough to handle all the return values. An example is the POSIX tzname() call, which takes no arguments, but returns two, the local timezone's standard and summer time abbreviations.

To handle this situation, the PPCODE directive is used and the stack is extended using the macro:

EXTEND(sp, num);

where sp is the stack pointer, and num is the number of elements the stack should be extended by.

Now that there is room on the stack, values can be pushed on it using the macros to push IV's, doubles, strings, and SV pointers respectively:

PUSHi(IV) PUSHn(double) PUSHp(char*, I32) PUSHs(SV*)

And now the Perl program calling tzname, the two values will be assigned as in:

($standard_abbrev, $summer_abbrev) = POSIX::tzname;

An alternate (and possibly simpler) method to pushing values on the stack is to use the macros:

XPUSHi(IV) XPUSHn(double) XPUSHp(char*, I32) XPUSHs(SV*)

These macros automatically adjust the stack for you, if needed.

For more information, consult the perlapi manpage .

Mortality

In Perl, values are normally "immortal" -- that is, they are not freed unless explicitly done so (via the Perl undef call or other routines in Perl itself).

Add cruft about reference counts.

In the above example with tzname, we needed to create two new SV's to push onto the argument stack, that being the two strings. However, we don't want these new SV's to stick around forever because they will eventually be copied into the SV's that hold the two scalar variables.

An SV (or AV or HV) that is "mortal" acts in all ways as a normal "immortal" SV, AV, or HV, but is only valid in the "current context". When the Perl interpreter leaves the current context, the mortal SV, AV, or HV is automatically freed. Generally the "current context" means a single Perl statement.

To create a mortal variable, use the functions:

SV* sv_newmortal() SV* sv_2mortal(SV*) SV* sv_mortalcopy(SV*)

The first call creates a mortal SV, the second converts an existing SV to a mortal SV, the third creates a mortal copy of an existing SV.

The mortal routines are not just for SV's -- AV's and HV's can be made mortal by passing their address (and casting them to SV*) to the sv_2mortal or sv_mortalcopy routines.

From Ilya: Beware that the sv_2mortal() call is eventually equivalent to svREFCNT_dec(). A value can happily be mortal in two different contexts, and it will be svREFCNT_dec()ed twice, once on exit from these contexts. It can also be mortal twice in the same context. This means that you should be very careful to make a value mortal exactly as many times as it is needed. The value that go to the Perl stack should be mortal.

You should be careful about creating mortal variables. It is possible for strange things to happen should you make the same value mortal within multiple contexts.

Stashes and Objects

A stash is a hash table (associative array) that contains all of the different objects that are contained within a package. Each key of the stash is a symbol name (shared by all the different types of objects that have the same name), and each value in the hash table is called a GV (for Glob Value). This GV in turn contains references to the various objects of that name, including (but not limited to) the following: Scalar Value Array Value Hash Value File Handle Directory Handle Format Subroutine

Perl stores various stashes in a separate GV structure (for global variable) but represents them with an HV structure. The keys in this larger GV are the various package names; the values are the GV*'s which are stashes. It may help to think of a stash purely as an HV, and that the term "GV" means the global variable hash.

To get the stash pointer for a particular package, use the function:

HV* gv_stashpv(char* name, I32 create) HV* gv_stashsv(SV*, I32 create)

The first function takes a literal string, the second uses the string stored in the SV. Remember that a stash is just a hash table, so you get back an HV*.

The name that gv_stash*v wants is the name of the package whose symbol table you want. The default package is called main. If you have multiply nested packages, pass their names to gv_stash*v, separated by :: as in the Perl language itself.

Alternately, if you have an SV that is a blessed reference, you can find out the stash pointer by using:

HV* SvSTASH(SvRV(SV*));

then use the following to get the package name itself:

char* HvNAME(HV* stash);

If you need to return a blessed value to your Perl script, you can use the following function:

SV* sv_bless(SV*, HV* stash)

where the first argument, an SV*, must be a reference, and the second argument is a stash. The returned SV* can now be used in the same way as any other SV.

For more information on references and blessings, consult the perlref manpage .

Magic

[This section still under construction. Ignore everything here. Post no bills. Everything not permitted is forbidden.]

# Version 6, 1995/1/27

Any SV may be magical, that is, it has special features that a normal SV does not have. These features are stored in the SV structure in a linked list of struct magic's, typedef'ed to MAGIC.

struct magic { MAGIC* mg_moremagic; MGVTBL* mg_virtual; U16 mg_private; char mg_type; U8 mg_flags; SV* mg_obj; char* mg_ptr; I32 mg_len; };

Note this is current as of patchlevel 0, and could change at any time.

Assigning Magic

Perl adds magic to an SV using the sv_magic function:

void sv_magic(SV* sv, SV* obj, int how, char* name, I32 namlen);

The sv argument is a pointer to the SV that is to acquire a new magical feature.

If sv is not already magical, Perl uses the SvUPGRADE macro to set the SVt_PVMG flag for the sv. Perl then continues by adding it to the beginning of the linked list of magical features. Any prior entry of the same type of magic is deleted. Note that this can be overriden, and multiple instances of the same type of magic can be associated with an SV.

The name and namlem arguments are used to associate a string with the magic, typically the name of a variable. namlem is stored in the mg_len field and if name is non-null and namlem >= 0 a malloc'd copy of the name is stored in mg_ptr field.

The sv_magic function uses how to determine which, if any, predefined "Magic Virtual Table" should be assigned to the mg_virtual field. See the "Magic Virtual Table" section below.

The obj argument is stored in the mg_obj field of the MAGIC structure. If it is not the same as the sv argument, the reference count of the obj object is incremented. If it is the same, or if the how argument is "#", or if it is a null pointer, then obj is merely stored, without the reference count being incremented.

Magic Virtual Tables

The mg_virtual field in the MAGIC structure is a pointer to a MGVTBL, which is a structure of function pointers and stands for "Magic Virtual Table" to handle the various operations that might be applied to that variable.

The MGVTBL has five pointers to the following routine types:

int (*svt_get)(SV* sv, MAGIC* mg); int (*svt_set)(SV* sv, MAGIC* mg); U32 (*svt_len)(SV* sv, MAGIC* mg); int (*svt_clear)(SV* sv, MAGIC* mg); int (*svt_free)(SV* sv, MAGIC* mg);

This MGVTBL structure is set at compile-time in perl.h and there are currently 19 types (or 21 with overloading turned on). These different structures contain pointers to various routines that perform additional actions depending on which function is being called.

Function pointer Action taken ---------------- ------------ svt_get Do something after the value of the SV is retrieved. svt_set Do something after the SV is assigned a value. svt_len Report on the SV's length. svt_clear Clear something the SV represents. svt_free Free any extra storage associated with the SV.

For instance, the MGVTBL structure called vtbl_sv (which corresponds to an mg_type of '\0') contains:

{ magic_get, magic_set, magic_len, 0, 0 }

Thus, when an SV is determined to be magical and of type '\0', if a get operation is being performed, the routine magic_get is called. All the various routines for the various magical types begin with magic_.

The current kinds of Magic Virtual Tables are:

mg_type MGVTBL Type of magicalness ------- ------ ------------------- \0 vtbl_sv Regexp??? A vtbl_amagic Operator Overloading a vtbl_amagicelem Operator Overloading c 0 Used in Operator Overloading B vtbl_bm Boyer-Moore??? E vtbl_env %ENV hash e vtbl_envelem %ENV hash element g vtbl_mglob Regexp /g flag??? I vtbl_isa @ISA array i vtbl_isaelem @ISA array element L 0 (but sets RMAGICAL) Perl Module/Debugger??? l vtbl_dbline Debugger? P vtbl_pack Tied Array or Hash p vtbl_packelem Tied Array or Hash element q vtbl_packelem Tied Scalar or Handle S vtbl_sig Signal Hash s vtbl_sigelem Signal Hash element t vtbl_taint Taintedness U vtbl_uvar ??? v vtbl_vec Vector x vtbl_substr Substring??? * vtbl_glob GV??? # vtbl_arylen Array Length . vtbl_pos $. scalar variable ~ Reserved for extensions, but multiple extensions may clash

When an upper-case and lower-case letter both exist in the table, then the upper-case letter is used to represent some kind of composite type (a list or a hash), and the lower-case letter is used to represent an element of that composite type.

Finding Magic

MAGIC* mg_find(SV*, int type); /* Finds the magic pointer of that type */

This routine returns a pointer to the MAGIC structure stored in the SV. If the SV does not have that magical feature, NULL is returned. Also, if the SV is not of type SVt_PVMG, Perl may core-dump.

int mg_copy(SV* sv, SV* nsv, char* key, STRLEN klen);

This routine checks to see what types of magic sv has. If the mg_type field is an upper-case letter, then the mg_obj is copied to nsv, but the mg_type field is changed to be the lower-case letter.

Double-Typed SV's

Scalar variables normally contain only one type of value, an integer, double, pointer, or reference. Perl will automatically convert the actual scalar data from the stored type into the requested type.

Some scalar variables contain more than one type of scalar data. For example, the variable $! contains either the numeric value of errno or its string equivalent from either strerror or sys_errlist[].

To force multiple data values into an SV, you must do two things: use the sv_set*v routines to add the additional scalar type, then set a flag so that Perl will believe it contains more than one type of data. The four macros to set the flags are:

SvIOK_on SvNOK_on SvPOK_on SvROK_on

The particular macro you must use depends on which sv_set*v routine you called first. This is because every sv_set*v routine turns on only the bit for the particular type of data being set, and turns off all the rest.

For example, to create a new Perl variable called "dberror" that contains both the numeric and descriptive string error values, you could use the following code:

extern int dberror; extern char *dberror_list; SV* sv = perl_get_sv("dberror", TRUE); sv_setiv(sv, (IV) dberror); sv_setpv(sv, dberror_list[dberror]); SvIOK_on(sv);

If the order of sv_setiv and sv_setpv had been reversed, then the macro SvPOK_on would need to be called instead of SvIOK_on.

Calling Perl Routines from within C Programs

There are four routines that can be used to call a Perl subroutine from within a C program. These four are:

I32 perl_call_sv(SV*, I32); I32 perl_call_pv(char*, I32); I32 perl_call_method(char*, I32); I32 perl_call_argv(char*, I32, register char**);

The routine most often used is perl_call_sv. The SV* argument contains either the name of the Perl subroutine to be called, or a reference to the subroutine. The second argument consists of flags that control the context in which the subroutine is called, whether or not the subroutine is being passed arguments, how errors should be trapped, and how to treat return values.

All four routines return the number of arguments that the subroutine returned on the Perl stack.

When using any of these routines (except perl_call_argv), the programmer must manipulate the Perl stack. These include the following macros and functions:

dSP PUSHMARK() PUTBACK SPAGAIN ENTER SAVETMPS FREETMPS LEAVE XPUSH*()

For more information, consult the perlcall manpage .

Memory Allocation

It is strongly suggested that you use the version of malloc that is distributed with Perl. It keeps pools of various sizes of unallocated memory in order to more quickly satisfy allocation requests. However, on some platforms, it may cause spurious malloc or free errors.

New(x, pointer, number, type); Newc(x, pointer, number, type, cast); Newz(x, pointer, number, type);

These three macros are used to initially allocate memory. The first argument x was a "magic cookie" that was used to keep track of who called the macro, to help when debugging memory problems. However, the current code makes no use of this feature (Larry has switched to using a run-time memory checker), so this argument can be any number.

The second argument pointer will point to the newly allocated memory. The third and fourth arguments number and type specify how many of the specified type of data structure should be allocated. The argument type is passed to sizeof. The final argument to Newc, cast, should be used if the pointer argument is different from the type argument.

Unlike the New and Newc macros, the Newz macro calls memzero to zero out all the newly allocated memory.

Renew(pointer, number, type); Renewc(pointer, number, type, cast); Safefree(pointer)

These three macros are used to change a memory buffer size or to free a piece of memory no longer needed. The arguments to Renew and Renewc match those of New and Newc with the exception of not needing the "magic cookie" argument.

Move(source, dest, number, type); Copy(source, dest, number, type); Zero(dest, number, type);

These three macros are used to move, copy, or zero out previously allocated memory. The source and dest arguments point to the source and destination starting points. Perl will move, copy, or zero out number instances of the size of the type data structure (using the sizeof function).

AUTHOR

Jeff Okamoto<okamoto@corp.hp.com>

With lots of help and suggestions from Dean Roehrich, Malcolm Beattie, Andreas Koenig, Paul Hudson, Ilya Zakharevich, Paul Marquess, Neil Bowers, Matthew Green, Tim Bunce, and Spider Boardman.

DATE

Version 19: 1995/4/26