Problem to get Parse::Yapp and Parse:Flex working together

A) I did:

# cd ~/.cpan/build/Parse-Flex-0.03
# more src/default.y
%{
#define YY_DECL char* yyylex YY_PROTO(( void ))
#undef yywrap
int yywrap(void) { return 1 ;}


char buffer[15];
%}

%%
[a-z]+ { return strcpy(buffer, "VAR"); }
[0-9]+ { return strcpy(buffer, "NUM"); }
[ \t]+ ;
.. { return strcpy(buffer, yytext); }
\n { return strcpy(buffer, yytext); }
<> return "" ;
%%


And then make and install. A quick test

$ perl -MParse::Flex -e 'print join " ", yylex(),"\n"'
abc
VAR abc

I borrowed the Calc example from Parse::Yapp, but want to use Flex
instead, thus

# cat Calc
use Calc;
use Parse::Flex;

sub yapp_error {
my $self = shift;
warn "Error: found ", $self->YYCurtok,
" and expecting one of ", join(" or ",$self->YYExpect);
}


my $calc = new Calc;
$calc->YYParse( yylex => yylex, yyerror => \&yapp_error );

This however results in:

Invalid value for parameter 'YYLEX' at Calculator line 12

I think the reason is that the yylex is also invoking the Parse::Flex
the first time.

I changed Parse::Flex, and renames yylex in flex:

$calc->YYParse( yylex => flex, yyerror => \&yapp_error );

to no avail. I can do
($token,$value) = flex;

What do I miss/oversee here ??


B) For sure I wrote my own My::FLex, with a different approach. In a
nutshell (I'm happy to append details on request)

In Flex.pm:

sub flex {
my $token = yylex();

if ($token) {
my $value = yytext();
# print $token, " = ", $value, "\n";
return ($token, $value);
}
else {
# print "Undefined token\n";
return (undef, "");
}
}


yylex()
OUTPUT:
RETVAL

char*
yytext()
CODE:
RETVAL = yytext;
OUTPUT:
RETVAL


Make and install the module and simply working.

my $calc = new Calc;
$calc->YYParse( yylex => flex, yyerror => \&yapp_error );

and that is the same invokation as above.


Best Regards,
Ton 't Lam

Getting Parse::Yapp and Parse:Flex working together

General

Parse::Yapp and flex is a powerful combination to do grammar and pattern
matching. You have the joy of the fast lexical analyzer, and within Perl
you create a program easily.

Yapp expects two variables, and per consequence the flex module should
provide these. The problem is that both Parse::Yapp, and flex use yylex.
I couldn't find Perl syntax to force the first yylex to be from Parse::Yapp.

$calc->YYParse( yylex => yylex, ...);

To go short: I changed Parse::Flex, as well created My::Flex. The fatest
method is to use Parse::Flex.

Parse::Flex

About the man page

The Parse::Flex manual says:

yylex() Get the name of next token, and indirectly sets yytext . Returns
undef for end of input.

You should read this as:

yylex() Get the name of next token, and indirectly sets yytext . Returns
undef for end of input.
Two variables are returned.

yying = 'datafile'

should be:

yyin = 'datafile'

Modifying Parse::Flex

Modify Flex.xs:

void
flex()
PPCODE:
char* id = 0;
if (id = yyylex() ) {
XPUSHs (sv_2mortal(newSVpv(id,0)));
XPUSHs (sv_2mortal(newSVpv( yytext, 0)));
XSRETURN(2);
}
XSRETURN_EMPTY;

Modify Flex.pm

our @EXPORT = qw( yyin yyout flex );

and src/default.y contains e.g.:

%{
#define YY_DECL char* yyylex YY_PROTO(( void ))
#undef yywrap
int yywrap(void) { return 1 ;}
%}

%%
[a-z]+ { return "VAR"; }
[0-9]+ { return "NUM"; }
[ \t]+ ;
.. ECHO;
\n ECHO;
<> return "" ;
%%

# make
# make install

A quick test

$ perl -MParse::Flex -e 'print join " ", flex(),"\n"'
abc
VAR abc

Yapp

Install Parse::Yapp

# cpan -i Parse::Yapp

Take the Calc.yp example from Parse::Yapp. We will use this example, but
of course use the flex lexer instead. Parse::Yapp expects a yylex
function that returns a pair: the name of the token and the matched text.

From the Calc.yp keep

#
# Calc.yp
#
# Parse::Yapp input grammar example.
#
# This file is PUBLIC DOMAIN
#
#
%right '='
%left '-' '+'
%left '*' '/'
%left NEG
%right '^'

%%
input: #empty
| input line { push(@{$_[1]},$_[2]); $_[1] }
;

line: '\n' { $_[1] }
| exp '\n' { print "$_[1]\n" }
| error '\n' { $_[0]->YYErrok }
;

exp: NUM
| VAR { $_[0]->YYData->{VARS}{$_[1]} }
| VAR '=' exp { $_[0]->YYData->{VARS}{$_[1]}=$_[3] }
| exp '+' exp { $_[1] + $_[3] }
| exp '-' exp { $_[1] - $_[3] }
| exp '*' exp { $_[1] * $_[3] }
| exp '/' exp {
$_[3]
and return($_[1] / $_[3]);
$_[0]->YYData->{ERRMSG}
= "Illegal division by zero.\n";
$_[0]->YYError;
undef
}
| '-' exp %prec NEG { -$_[2] }
| exp '^' exp { $_[1] ** $_[3] }
| '(' exp ')' { $_[2] }
;

%%

sub yapp_error {
my $self = shift;
warn "Error: found ", $self->YYCurtok,
" and expecting one of ", join(" or ",$self->YYExpect);
}

# yapp Calc.yp

The result is a Calc.pm in your current directory.

Yapp invokes flex

# cat Calc
use Calc;
use Parse::Flex;

sub yapp_error {
my $self = shift;
warn "Error: found ", $self->YYCurtok,
" and expecting one of ", join(" or ",$self->YYExpect);
}


my $calc = new Calc;
# $calc->YYParse( yylex => \&My::Flex::flex, yyerror => \&Calc::yapp_error);
$calc->YYParse( yylex => \&flex, yyerror => \&Calc::yapp_error);

# perl Calc
12+34
46
12++34
Error: found + and expecting one of - or NUM or VAR or ( at Calc.yp line 48.

My::Flex

Here are steps to create your own flex module My::Flex

Here:

* Parse::Flex supplies flex (Flex.pm)
* flex (Flex.pm) calls yylex, and yytext (Flex.xs).
* Flex.xs calls the functions from lex.yy.o, thus Flex.l.

Next step is to build an XS module, that invokes flex to do the pattern
matching.

Create the module structure:

# h2xs -O -n My::Flex

# cd My-Flex

Create a file MyFlex.h

# more MyFlex.h
extern char* yylex();
extern char* yytext;
extern FILE *yyin, *yyout;

Modify Flex.xs, so that

#include "EXTERN.h"
#include "perl.h"
#include "XSUB.h"

#include "ppport.h"

#include "const-c.inc"
#include "myFlex.h"

MODULE = My::Flex PACKAGE = My::Flex

INCLUDE: const-xs.inc

char*
yylex()
CODE:
RETVAL = yylex();
OUTPUT:
RETVAL
char*
yytext()
CODE:
RETVAL = yytext;
OUTPUT:
RETVAL

void
yyin( file)
char* file
CODE:
yyin = fopen( file, "r");

void
yyout( file)
char* file
CODE:
yyout = fopen( file, "w");

and the myFlex.l

# more myFlex.l
%{
#define YY_DECL char* yylex() void;
char buffer[15];
%}

%%
apple ECHO;
pear ECHO;
[a-z]+ { return strcpy(buffer, "VAR"); }
[0-9]+ { return strcpy(buffer, "NUM"); }
[ \t]+ ;
\n { return strcpy(buffer, yytext); }
.. { return strcpy(buffer, yytext); }

%%
int perl_yywrap(void) {
return 1;
}

We have to map the data types from C to Perl. Next map translates the
pointers to characters to the built-in T_PV Perl type.

# cat typemap
char* T_PV

In lib/My/Flex.pm enter

# Preloaded methods go here.

sub flex {
my $token = yylex();

if ($token) {
my $value = yytext();
# print $token, " = ", $value, "\n";
return ($token, $value);
}
else {
# print "Undefined token\n";
return (undef, "");
}
}

Now the lib/My/Flex.pm

# our %EXPORT_TAGS = ( 'all' => [ qw( ) ] );
# our @EXPORT_OK = ( @{ $EXPORT_TAGS{'all'} } );

our @EXPORT = qw(yyin yyout flex );

....

sub flex {
my $token = yylex();

if ($token) {
my $value = yytext();
# print $token, " = ", $value, "\n";
return ($token, $value);
}
else {
# print "Undefined token\n";
return (undef, "");
}
}

Next step is to modify the Makefile.PL

WriteMakefile(
NAME => 'My::Flex',
VERSION_FROM => 'lib/My/Flex.pm', # finds $VERSION
PREREQ_PM => {}, # e.g., Module::Name => 1.1
($] >= 5.005 ? ## Add these new keywords supported since 5.005
(ABSTRACT_FROM => 'lib/My/Flex.pm', # retrieve abstract from module
AUTHOR => 'Ton ') : ()),
LIBS => ['-lfl'], # e.g., '-lm'
DEFINE => '', # e.g., '-DHAVE_SOMETHING'
INC => '-I.', # e.g., '-I. -I/usr/include/other'
MYEXTLIB => 'myFlex.so' # our Flexer (note there is no comma)
# Un-comment this if you add C files to link with later:
# OBJECT => '$(O_FILES)', # link all the C files too
);



sub MY::postamble {
"
\$(MYEXTLIB): lex.yy.c Flex.o
\t\$(CC) -c lex.yy.c
\t\$(AR) cr \$(MYEXTLIB) lex.yy.o
\tranlib \$(MYEXTLIB)

lex.yy.c: myFlex.l
\tflex myFlex.l
";
}

It is time to compile

# perl Makefile.PL
# make
# make test # FWIW
# make install

If you don have root access, then do

$ perl Makefile.PL PREFIX=/home/tonl
$ make
$ make install

If you want to re-compile then remove the *.[co] files first, or make clean.

Test if the module is installed.

# perl -MMy::Flex -e 1

A quick test

# perl -MMy::Flex -e 'print join " ", My::Flex::flex(),"\n"'
abc
VAR abc

And finally invoke the modules, and do the calculation ...

# use lib "/home/tonl"; # if you installed locally
use Calc;
use My::Flex;

My::Flex::yyin("file");
My::Flex::yyout "res";
My::Flex::flex();

print "===\n";

my $calc = new Calc;
$calc->YYParse( yylex => \&flex, yyerror => \&Calc::yapp_error);

and file contains

apple
pear
1+2
abc=4
def=5
abc+def
apple

# perl Calc
===
3
4
5
9

and res contains

applepearapple

Note about yyout:
flex its ECHO directs output to yyout, and the other output is return to
the Perl program.

I could have done eqally the following:

void
flex()
PPCODE:
char* id = 0;
if (id = yylex() ) {
XPUSHs (sv_2mortal(newSVpv(id,0)));
XPUSHs (sv_2mortal(newSVpv( yytext, 0)));
XSRETURN(2);
}
XSRETURN_EMPTY;

This will cause an Perl array to be returned. Then you don't need to
modify lib/My/Flex.pm.

$ perl -MMy::Flex -e 'print join " ", My::Flex::flex(),"\n"'
abc
VAR abc

use Calc;
use My::Flex;

my $calc = new Calc;
$calc->YYParse( yylex => \&My::Flex::flex(), yyerror => \&Calc::yapp_error);

References

* Parse::Yapp
* Parse::Flex
* perlxstut

Author
Ton 't Lam

Getting Parse::Yapp and Parse:Flex working together (appendix)

You may come accross next error message:

Usage: Parse::Flex::flex() at
/opt/perl/lib/site_perl/5.8.2/Parse/Yapp/Driver.pm line 271.

and if you try that

Invalid value for parameter 'YYLEX' at Calc line 11

The reason is that in Driver.pm next invokation is made:

($$token,$$value)=&$lex($self);

In other words, a parameter is expected. flex() doesn't accept this. The
workaround is to have next:

use Calc;
use Parse::Flex;

sub _Flex {
flex; # or Parse::Flex::flex;
}

my $calc = new Calc;
$calc->YYParse( yylex => \&_Flex, yyerror => \&Calc::yapp_error);

Another easy solution is to have flex accept the variable.

void
flex(path)
SV * path;
PPCODE:

and then you can have:

my $calc = new Calc;
$calc->YYParse( yylex => \&flex, yyerror => \&Calc::yapp_error );

Remark: This happened with Parse::Flex, but not with My::Flex.

Linefeed not accepted

Unknown why this is. Entering a second linefeed is accepted.

# echo "1+2" | perl Calc

works as expected. Also if using

yyin "file";

Remark: This happened with Parse::Flex, but not with My::Flex.