Friday, September 29, 2017

Freeside Development Environment (day 1)

I'm on the job market again after two wonderful years at Broadbean. Sadly, budget cuts eliminated my position and several others, but enough about that. Anyway, as part of my job search, I came upon Freeside, which is a billing application for ISP's, CLEC's, and similar. They work on an open source, paid support model, so after discussion with Ivan, the CEO and "head geek", I thought I'd do some work "on spec" as a job application of sorts. But to begin with, I need a development environment.

At Broadbean I got used to every package we worked on having its own cpanfile. This meant that, using perlbrew, all I had to do was perlbrew lib create perl-<version>@<package>, and then cpanm --installdeps ., and I was set. Freeside doesn't work like that, so I'll set up a VirtualBox VM for it instead.

I'm going to use Debian 8.9.0 (Jessie), as that is the version where installation instructions are completely documented in the Freeside wiki. So I've downloaded the Debian network install CD image from https://cdimage.debian.org/cdimage/archive/8.9.0/amd64/iso-cd/ and will install it into a VM. I've got Google Fiber, so this shouldn't take long.

Since I intend to interact with this VM via SSH and a browser, I've selected the "web server", "SSH server", and "standard system utilities" collections. A few minutes later, the system is installed, I've rebooted the VM, and installed sudo (I prefer using sudo to simply doing su). I set up a network interface, put in my SSH key for passwordless login, and I'm ready to start setting up my development environment.

First thing to install is vim. This is my preferred editor for code, and I can't abide nano. ;) Next I set up the package repositories as specified in the Freeside installation instructions and install the Freeside packages. I'll note here that aptitude is recommending that I remove the packages exim4, exim4-base, exim4-config, and exim4-daemon-light, and is not installing the recommendation of the Perl EV package. I assume since Freeside is installing Mojolicious, it is going to be running under Mojo::IOLoop (which I was just working with at Broadbean. Freeside++!) No, I was mistaken. Only very minimal usage of Mojolicious in Freeside.

OK, a few minutes later all the packages are installed and I'm setting up the database. Note where the docs say "[ as postgres/pgsql user ]" they mean "user" to be the system user (from /etc/passwd) "postgres". I set up the database role with a crappy password (this box isn't exposed to the internet, after all) and as the "freeside" user, I execute freeside-setup -d example.com. Note that the domain is important - if it's not a valid TLD, freeside-setup will throw an error and you'll have to blow away the freeside database and start over. Anyway, freeside-setup barfs, so now I need to figure out what I did wrong.

Turns out the wiki has things in the wrong order. I need to set up the RT database before I run freeside-setup. With that done, I can move on, install the system users and so on, and I should be up and running. Sure enough, I am able to restart the Freeside daemon successfully.

Since I want to hack on Freeside, I think I need to blow away the Freeside packages and set up a fresh installation via the instructions on installing from source. This might allow me to use perlbrew as well, but I'm not certain of that. But that's a task for another day.




Tuesday, October 18, 2016

Genuine, Honest-to-Ada, Taleo MTOM/XOP export example

Because this has been vexing me for months.

So the key here is that you have two Document elements. One in the SOAP envelope, and one in the attachment. Your Attributes (gotta have those) go in the Document element in the attachment.

Thursday, June 11, 2015

Freelancing again

I have once again joined the ranks of the freelance developer community. It wasn't by choice, and I've been looking for full-time gigs, but nothing's panned out as of yet. So in the mean time, I'm trying to make the best of my situation, and I'm hoping this new site I've heard about, Toptal, can help.

Why Toptal? A couple of reasons. First, I need the work, and the more sites I'm on where I can get work, the better. But second, and this remains to be seen, is that I'm hoping Toptal can put me in front of clients who can really challenge me as a developer. Where I can solve exciting new problems.

Since I'm primarily a Perl guy, I'm joining Toptal's Perl Developers Network. I don't mind learning the latest shiny new technology - like single page Javascript, for example - but that's not my end goal. I'm not looking to learn the new technologies just to check off a box, or to add a line to my resume. I'm looking to create solutions. If there's a tool I need to learn in the process of creating a solution, that's great. But it's not an end in itself. So I'm joining their Perl network in the hope of finding clients who are less concerned with the latest HR buzzword and more concerned about getting things done.

Wednesday, April 17, 2013

Auto-generating setters and getters in Perl with Moose

As part of a project for work interfacing with the RDS encoder for the radio station (an Inovonics Model 730) I thought, "wouldn't it be nice to have a Perl object that provides a complete interface to this device?"  The only trouble is that the device takes a lot of different commands, and it would be really tedious to write sub set_foo { ... }; sub get_foo { ... } umpteen million times for every last command / datum the encoder supported, since every set_foo { ... } was going to boil down to:

sub set_foo {
    my ($self, $value) = @_;
    return _set('foo', $value);
}

sub get_foo {
    my $self = shift;
    return _get('foo');
}

Where _set() and _get() took care of the actual business of communicating with the encoder.

Now, I could have done this in Vim with regular expressions - just copy/paste and use the regex to change the appropriate things. Still tedious, though. I also could have done something like this:

sub set_property {
    my ($self, $property, $value) = @_;
    return _set($property, $value);
}

sub get_property {
    my ($self, $property) = @_;
    return _get($property);
}

And that would have worked fine. I could have put some code in there to throw an exception on an invalid property, maybe something to validate the values based on the property name, all that sort of thing. But Moose gives us a nifty trick to avoid having to validate the property name:

    package MooseSketch;
    use Moose;

    my $meta = __PACKAGE__->meta;
    foreach my $prop (qw/foo bar baz bak/) {
        $meta->add_method(qq/set_$prop/, sub { 
                my $self = shift;
                my $value = shift;
                return $self->_set($prop, $value);
            }
        );
        $meta->add_method(qq/get_$prop/, sub { 
                my $self = shift;
                return $self->_get($prop);
            }
        );
    }

This $meta business comes from Class::MOP::Class, which allows us to do introspection and manipulation of Perl 5 objects. So with Class::MOP::Class, I can add or remove methods programatically at runtime, or even create entire classes. Neat, huh?

Hope you find this useful. I know I sure will. Many thanks to the good people over at Stack Overflow who helped me figure this out. My original question: How to auto-generate a bunch of setters/getters tied to a network service in Moose?

Monday, February 11, 2013

Generating a Word document with Perl and Win32::OLE

I'm about to have to throw this particular bit of code away, as I'm not able to get this to work from Scheduled Tasks on Windows 7.  Before I sent it to the bit bucket, however, I thought I'd post it here with the hope that someone will find it useful.



# Expects arguments as a hashref with the keys:
# # log_date: Date of the log
# # data: an arrayref of arrayrefs.  First line is treated as column headings, following lines are treated as data.
#
# A double horizontal rule will be added between the column headings and the data.
#
# NB: The reason that everything gets its own object (e.g. "my $tables = $doc->Tables; my $table = $tables->Add(...);")
# is not (neccessarily) for "Law of Demeter" reasons, but rather MS recommended practice when
# automating Office applications from Visual Studio (and by extension, OLE): http://support.microsoft.com/kb/317109
# Experimentally, I have noticed instances of the Word executable remaining in memory after program exit;
# refactoring the code in this way is an attempt to deal with that issue.
# 11 Feb 2013 KP
sub _print_with_word {
    my $args = shift;

    if ( ref $args ne q/HASH/ ) {
        croak(
            sprintf q/Usage: %s /,
            ( caller 0 )[$FUNCTION_NAME_POSITION]
        );
    }
    foreach my $required_key (qw/log_date data/) {
        if ( !$args->{$required_key} ) {
            croak(qq/Missing required key '$required_key' in args/);
        }
    }

    my $header = _slurp_file( $CONFIG->{'_'}{'header_file'} );
    my $footer = _slurp_file( $CONFIG->{'_'}{'footer_file'} );

    my @rows = @{ $args->{'data'} };

    my $word   = Win32::OLE->new( 'Word.Application', 'Quit' );
    _debug(q/Created new Word object/);

    my $doc    = $word->Documents->Add();
    _debug(q/Added new document/);

    my $selection = $word->Selection;
    _debug(q/Got Selection instance/);    

    
    my $selection_paragraph_format = $selection->ParagraphFormat;
    _debug(q/Got ParagraphFormat instance for selection/);
    
    $selection_paragraph_format->{'SpaceAfter'} = 0;
    _debug(q/Set paragraph spacing for header/);
    
    $selection->TypeText( { 'Text' => qq/$header\n\n/, } );
    _debug(q/Typing header into selection/);
    
    $selection->BoldRun();
    _debug(q/started bold run/);
    
    $selection_paragraph_format->{'Alignment'} = wdAlignParagraphRight;
    _debug(q/Set date paragraph format to right/);
    
    $selection->TypeText(
        {
            'Text' => Time::Piece->strptime(
                $args->{'log_date'}, q|%m/%d/%Y %H:%M:%S|
              )->strftime(qq/%A %B %d %Y\n\n/)
        }
    );
    _debug(q/Typing date header into selection/);
    
    $selection->BoldRun();
    _debug(q/End bold run/);

    my $range  = $selection->Range;
    _debug(q/Got Range instance from selection/);
    
    my $tables = $doc->Tables;
    _debug(q/Got Tables collection from document/);
    
    my $table  = $tables->Add( $range, scalar @rows, scalar @{ $rows[0] } );
    _debug(q/Added new table to document/);
    
    for my $rownum ( 0 .. $#rows ) {
        my $cols = $rows[$rownum];
        for my $colnum ( 0 .. $#{ $rows[$rownum] } ) {
            my @cellpos    = ( $rownum + 1, $colnum + 1 );
            my $cell       = $table->Cell(@cellpos);
            _debug(qq/Got cell at ($cellpos[0], $cellpos[1]) /);
            
            my $cell_range = $cell->Range;
            _debug(qq/Got Range instance for cell at ($cellpos[0], $cellpos[1])/);
            
            $cell_range->{'Text'} = $cols->[$colnum];
            _debug(qq/Set text of Range for cell at ($cellpos[0], $cellpos[1]) to "$cols->[$colnum]"/);
        }
    }
    my $rows                 = $table->Rows;
    _debug(q/Got Rows collection from table/);
    
    my $first_row            = $rows->First;
    _debug(q/Got first Row object (header row) from Rows collection/);
    
    my $first_row_range      = $rows->First->Range;
    _debug(q/Got Range instance for header row/);
    
    my $first_row_range_font = $first_row_range->Font;
    _debug(q/Got Font instance for header row Range/);
    
    $first_row_range_font->{'Bold'} = 1;
    _debug(q/Set header row Range Font to bold/);
    
    my $first_row_range_paragraph_format = $first_row_range->ParagraphFormat;
    _debug(q/Got ParagraphFormat instance for header row/);
    
    $first_row_range_paragraph_format->{'Alignment'} = wdAlignParagraphCenter;
    _debug(q/Set alignment for table headers to center/);
    
    my $first_row_bottom_border = $first_row->Borders(wdBorderBottom);
    _debug(q/Got Border instance for bottom of header row/);
    
    @{$first_row_bottom_border}{qw/LineStyle LineWidth/} =
      ( wdLineStyleDouble, wdLineWidth100pt );
    _debug(q/Set bottom border of header row to 10 pt double line/);
      
    my $paragraphs            = $doc->Paragraphs;
    _debug(q/Got Paragraphs collection from document/);
    
    my $last_paragraph        = $paragraphs->Last;
    _debug(q/Got last paragraph from document/);
    
    my $last_paragraph_format = $last_paragraph->Format;
    _debug(q/Got Format instance for last paragraph/);
    
    $last_paragraph_format->{'Alignment'}  = wdAlignParagraphLeft;
    _debug(q/Set last paragraph alignment to left/);
    
    $last_paragraph_format->{'SpaceAfter'} = 0;
    _debug(q/Set spacing on last paragraph/);
    
    my $last_paragraph_range = $last_paragraph->Range;
    _debug(q/Got Range instance for last paragraph/);
    
    $last_paragraph_range->InsertAfter( { 'Text' => qq/\n$footer/ } );
    _debug(q/Inserting footer after last paragraph range/);
    
    #$doc->SaveAs( { 'Filename' => Cwd::getcwd . '/test.doc' } );
    
    $doc->PrintOut();
    _debug(q/Printed document/);
    
    $doc->Close( { 'SaveChanges' => wdDoNotSaveChanges } );
    _debug(q/Close document without saving/);
    
    $word->Quit();
    _debug(q/Quit Word/);
    
    return 1;
}

Thursday, September 13, 2012

"Wire-to-wire" emulation with Proxy ARP

In my previous post, I stated a goal of mimicing the connectivity provided by the microwave T-1 connecting the studio and TV transmitter at my work.  I'm pleased to say that I've accomplished that... mostly.  There are still a few peculiar kinks to work out.

It all works through the magic of Proxy ARP.  Basically, what Proxy ARP does is allow one machine to answer ARP queries on behalf of another machine, saying, in effect "I'll take that packet; I know how to get to that machine".  The answering machine then forwards the packet on to the appropriate machine.  This differs from packet routing in that it happens in Layer 2 of IP, rather than Layer 3 (routing).

In practice, what you have to do is set up static routes on the host (which is acting in this case as a bridge) doing Proxy ARP for all hosts the bridge is providing the connection for, then turn on IP forwarding in the kernel, and turn on Proxy ARP for the appropriate interfaces.  Since I have an IPsec / L2TP VPN set up, the interfaces are going to be ppp0 for the VPN (since it's L2TP and IPsec, PPP is involved, where with straight IPsec it wouldn't be) and eth0 for the local network. I'm also going to have to do the same thing on the firewall out at the transmitter.

So, first of all, the studio.  This is based on Using Linux as an L2TP/IPsec VPN client by Jacco de Leeuw.

The xl2tpd configuration file at the studio:

; File: /etc/xl2tpd/xl2tpd.conf at STUDIO
[lac Transmitter]
; transmitter public IP obfuscated for security reasons
  lns = X.X.X.10 
  require chap = yes
  refuse pap = yes
  require authentication = yes
  ; Name should be the same as the username in the PPP authentication!
  name = bridge
  ppp debug = yes
  pppoptfile = /etc/ppp/options.l2tpd.client
  length bit = yes

Note that I'm not using the l2tp-secrets file as it really doesn't provide any additional security, as far as I can tell.

The PPP options file at the studio:
# /etc/ppp/options.l2tpd.client at STUDIO

ipcp-accept-local
ipcp-accept-remote
refuse-eap
noccp
noauth
crtscts
mtu 1410
mru 1410
nodefaultroute
debug
lock
connect-delay 5000

PPP authentication is done in /etc/ppp/chap-secrets:

# File: /etc/ppp/chap-secrets
# Secrets for authentication using CHAP
# client        server          secret                  IP addresses
bridge          *               "supersecret"
*               bridge          "supersecret"

To set up the static routes, I have set up two scripts in /etc/ppp/ip-up.d. Scripts in this directory are executed by pppd as ip-up scripts - scripts that are executed when the PPP interface is brought up. These set up static routing and turn on Proxy ARP.

#!/bin/bash

# File: /etc/ppp/ip-up.d/0001routes

PPP_INTERFACE=$1
LOCAL_ADDR=$4
REMOTE_ADDR=$5

# These are all the hosts that should be accessible both via the VPN
HOSTS=(
10.1.1.10
10.1.1.11
10.1.1.12
10.1.1.13
)

for HOST in ${HOSTS[*]}
do
        route add -host $HOST $PPP_INTERFACE
done


#!/bin/bash

# File: /etc/ppp/ip-up.d/0002proxyarp

for INTERFACE in ppp0 eth0 ; do
        /sbin/sysctl -w net.ipv4.conf.${INTERFACE}.proxy_arp=1
done




The ipsec (openswan) configuration file at the studio:


# File: /etc/ipsec.conf
# 
config setup
        protostack=netkey
conn Transmitter
        #
        # ----------------------------------------------------------
        # Use a Preshared Key. Disable Perfect Forward Secrecy.
        # Initiate rekeying.
        # Connection type _must_ be Transport Mode.
        #
        authby=secret
        pfs=no
        rekey=yes
        keyingtries=3
        type=transport
        #
        # ----------------------------------------------------------
        # The local Linux machine that connects as a client.
        #
        # The external network interface is used to connect to the server.
        # If you want to use a different interface or if there is no
        # defaultroute, you can use:   left=your.ip.addr.ess
        left=%defaultroute
        #
        leftprotoport=17/1701
        #
        # ----------------------------------------------------------
        # The remote server.
        #
        # Connect to the server at this IP address. (obfuscated for security)
        right=X.X.X.10
        #
        rightprotoport=17/1701
        # ----------------------------------------------------------
        #
        # Change 'ignore' to 'add' to enable this configuration.
        #
        auto=add
        DPDACTion=restart_by_peer
        dpdtimeout=30
        dpddelay=3

Next, the transmitter side.  Again, starting with xl2tpd:



; File: /etc/xl2tpd/xl2tpd.conf

[global]
        ipsec saref = no
        listen-addr = X.X.X.10

[lns default]
        ip range = 10.1.1.100 - 10.1.1.255
        local ip = 10.1.1.1
        assign ip = yes
        require chap = yes
        refuse pap = yes
        require authentication = yes
        name = Transmitter
        ppp debug = no
        pppoptfile = /etc/ppp/options.xl2tpd
        length bit = yes

And now PPP:

# File: /etc/ppp/options.xl2tpd

refuse-mschap-v2
refuse-mschap
ms-dns 8.8.8.8
asyncmap 0
auth
lock
hide-password
local
#debug
name l2tpd
#proxyarp
lcp-echo-interval 30
lcp-echo-failure 4


# File: /etc/ppp/chap-secrets

# Secrets for authentication using CHAP
# client                server                  secret                  IP addresses
user1                   *                       "secret1"               10.1.1.0/24
*                       user1                   "secret1"               10.1.1.0/24
user2                   *                       "secret2"               10.1.1.0/24
*                       user2                   "secret2"               10.1.1.0/24
bridge                  *                       "supersecret"           10.1.1.2
*                       bridge                  "supersecret"           10.1.1.2

Here I want to pause and note something. I have my regular "road warrior" users set up to get any address in 10.1.1.0/24, and in my xl2tpd config, I have that further restricted to addresses in the range of .100 - .255. The "bridge" user is restricted to 10.1.1.2, which is how I allow my "road warriors" and the bridge at the studio to coexist. The appropriate scripts in /etc/ppp/ip-up.d at the transmitter:


#!/bin/bash

# File: /etc/ppp/ip-up.d/0001routes
# The hosts here are made accessible to the transmitter network via the firewall, which is acting as a bridge similar to the one at the studio

PPP_INTERFACE=$1
LOCAL_ADDR=$4
REMOTE_ADDR=$5


# only set up the routes for the VPN bridge
case $REMOTE_ADDR in
10.1.1.2)
        for HOST in     10.1.1.20 \
                        10.1.1.21 \
                        10.1.1.22 \
                        10.1.1.23 ; do
                route add -host $HOST $1
        done
esac



#!/bin/bash
PPP_INTERFACE=$1
LOCAL_ADDR=$4
REMOTE_ADDR=$5

# File: /etc/ppp/ip-up.d/0002proxyarp
# only set up Proxy ARP for the VPN bridge
case $REMOTE_ADDR in
10.1.1.2)
        for INTERFACE in $PPP_INTERFACE eth0 ; do
                /sbin/sysctl -w net.ipv4.conf.${INTERFACE}.proxy_arp=1
        done
esac

And finally the ipsec (openswan) configuration, based on Configure L2TP/IPSec VPN on Ubuntu by Riobard Zhan:

#
# File: /etc/ipsec.conf
#

# Transmitter Firewall Side

config setup
    oe=off
    protostack=netkey
    nat_traversal=yes

conn L2TP-PSK-NAT
    rightsubnet=vhost:%no
    also=L2TP-PSK-noNAT

conn L2TP-PSK-noNAT
    authby=secret
    pfs=no
    auto=add
    keyingtries=3
    rekey=no
    ikelifetime=8h
    keylife=1h
    type=transport
        left=X.X.X.10
    leftprotoport=17/1701
    right=%any
    rightprotoport=17/%any
    dpdaction=restart_by_peer
    dpdtimeout=30
    dpddelay=3

So the way to accomplish this "wire-to-wire" emulation is with two bridges, one for each network to be bridged. If you want to try something like this, I wish you the best of luck, and I hope my experiences help.

Tuesday, August 21, 2012

Setting up an IPsec VPN

Posted here mostly for my own reference, but mayhap this will prove useful for someone else.

The setup: a remote network at the TV transmitter, with Internet access both through a commercial provider and a dedicated, but increasingly unreliable, microwave T-1 link to the studio.

The goal: mimic the behavior of the microwave T-1 (though not the unreliability, of course) using the commercial provider's service.  Set up a "virtual wire" between the network at the transmitter and the transmitter at the studio.

I've chosen an IPsec VPN to do this, and I've set things up largely as in Ch. 35 of Linux Home Networking. At the present time I've managed to achieve bidirectional communication between hosts (but not the routers/firewalls) on each network, but to do that I had to set up a private test network as shown below.


So as you can see here, I have a couple of virtual machines set up on the 172.16.1.0/24 network, which is accessible only to those VM's.  The IPsec tunnel is established between Router VM and Transmitter Firewall.  Note that I have replaced Transmitter Firewall's public IP address with "X.X.X.10" for security.

With this setup, I can achieve bidrectional communication between both networks.  Host A can reach Client VM, and Client VM can reach Host A.

Here is the relevant ipsec.conf on Router VM:

#
# File: /etc/ipsec.conf
#
# VPN Test Router VM side
config setup
        protostack=netkey
        oe=off
        nat_traversal=yes

# LEFT: Studio
# RIGHT: Transmitter
conn nettonet
  left=192.168.152.188            # Public Internet IP address of the
                                  # LEFT VPN device
  leftsubnet=172.16.1.0/24        # Subnet protected by the LEFT VPN device
  leftrsasigkey=*removed*

  right=X.X.X.10                  # Public Internet IP address of
                                  # the RIGHT VPN device
  rightsubnet=10.1.1.0/24         # Subnet protected by the RIGHT VPN device
  rightrsasigkey=*removed*
  rightnexthop=%defaultroute
  auto=start


Note here that I'm using RSA keys, rather than PSK's.  The key signatures have been removed for security.

The ipsec.conf on Transmitter Firewall is slightly different:


#
# File: /etc/ipsec.conf
#

# Transmitter Firewall Side

config setup
    oe=off
    protostack=netkey
    nat_traversal=yes

# LEFT: Studio
# RIGHT: Transmitter
conn nettonet
  left=10.1.0.2                   # Public Internet IP address of the
                                  # LEFT VPN device
  leftsubnet=172.16.1.0/24        # Subnet protected by the LEFT VPN device
  leftid=192.168.152.188
  leftrsasigkey=*removed*
  right=X.X.X.10                  # Public Internet IP address of
                                  # the RIGHT VPN device
  rightsubnet=10.1.1.0/24         # Subnet protected by the RIGHT VPN device
  rightrsasigkey=*removed*
  auto=start                      # authorizes and starts this connection
                                  # on booting

Here you'll notice that the argument to left= is 10.1.0.2, which is the public (again, these IP's have been obfuscated for security's sake) address of Studio Firewall. This is because Router VM's ultimate path to the Internet is through Studio Firewall via NAT.

Speaking of NAT, I should point out that there is no NAT of any kind being performed (currently) on Router VM or Transmitter Firewall.  Instead, the entire 172.16.1.0/24 network is visible to the 10.1.1.0/24 network at the transmitter, just as any other public network would be.

I'll note that this does not accomplish the stated goal at the beginning of this post: set up a "virtual wire" between the 10.1.1.0/24 network at the studio and the 10.1.1.0/24 network at the transmitter.  It may turn out to be the case that this "virtual wire" is extremely complicated with regard to routing and suchlike (since we would be using the same IP block on both ends) .  What this does give us, however, is a place to work from.