Webpages which break in the night.

Our xPapers site seemed to be working fine last night when I left work. When I came in this morning it had broken. Loading the homepage showed the error "Oops an error has occurred". From my desktop PC (where i'd left myself logged in with an admin account) I also got a debug message:

Can't call method "threads_o" without a package or object reference at /home/xpapers/sites/fwshub/mason/news.html line 11.

... 7: 8: <%perl> 9: 10: my $f = xPapers::Forum->get(7); 11: my @threads = $f->threads_o("true","ct desc",0,2); 12: for my $p (map { $_->firstPost } @threads) { 13: my ($c1,$c2) = $rend->wordSplit($p->body,30); 14: #$c1 =~ s/<\/?a[^>]*>?//ig; 15: my $follow = $c2 ? " (read more)" : "";

My first thought was that one of the changes i'd made yesterday to a Mason page hadn't actually appeared yesterday due to the way Mason cache's things and that the cache had now expired revealing an error. So I tried removing all the changes I made yesterday and the error persisted. I then realised it only appeared on the home page and that I could access other pages. I then looked at the forums table and noticed that there was no entry number 7 which the error page indicated it was trying to access. I changed this number to one of a forum which really exists and the error went away. I've still no idea why this happened and if there was ever an entry number 7 in the database. Looking at the Apache logs nobody seems to have been on the site overnight and deleted an entry. I've now turned on query logging in MySQL and am about to setup a daily backup script so that we can at least take a diff of the database between two days and spot changes.

editing category structures and changing colours

I've spent most of the day trying to figure out how xpapers fits together and how to modify bits of it. Still got a few issues with adding content and doing forum posts, which will be the topic of a future post.

It seems categories can be created either by going to the admin menu (if you are an admin) and choosing "edit category structure". Once done the command $XPAPERS/bin/setup/mkforums.pl needs to be run from the command line. It seems that only at this point will the categories appear on the main page.

The core 4 colours of the page seem to be controlled by the C1,C2,C3 and C4 variables in /etc/xpapers.d/main.pl. This doesn't change everything on the page, the green/blue bar across the top is actually an image and the backgrounds to headings within the main page.

make sure the hostname is correct

Our server effectively has two hostnames, fws-hub and mellte (the name of a Welsh river). The hostname we want to give out to everyone is fws-hub, but everything on the server was set to call itself mellte. This was getting picked up by apache and was causing pages to end up referring to mellte not fws-hub. Unfortunately we'd registered for recaptcha (used for user signup) as fws-hub and this was getting broken because it didn't match so new users couldn't sign up.

database problems fixed!

Thanks to David Bourget, the author of the xpapers project for his help here.

By running:

perl $XPAPERS/bin/setup/apply-sql-patches.pl 1.0 1.1

The extra fields (hisotricalFacetOf,facetRoot, historicalLeaf, summary, keyWorks, introductions, summaryUpdated, summaryChecked and advertiseChecked) now exist in the cats table and the errors referring to these are now gone. A lot more of the functions in xPapers now seem to work.

David also pointed me at a command line interface for xPapers, which allows you to manipulate the database (I wanted to make a user an admin). This can be done by running bin/cmd.pl, for example:

perl bin/cmd.pl

> load user 123

user blabla bla loaded

> set admin 1

field admin set to 1

>bye

site working again, not totally sure why

The site is now working again, but I can't totally explain why.

I changed 2 permissions which may have caused problems, these are:

chmod 755 /usr/lib/perl5/site_perl/5.12.3/i686-linux/FastMmap.pm
chmod g+w /dev/shm/xpapers

It seems backing out the last set of changes I made (to lib/xpapers/cat.pm) still breaks things so I may have been right to make them. It also seems that fastcgi does a lot of caching of stuff so that changes don't always propagate for a while unless apache is restarted.

problems with databases

Our xpapers server got rebooted accidentally at somepoint over the last few days and now visiting the site just returns an error page saying the server can't be contacted. Further investigation into the apache logs reveals the following error:

[Tue Sep 20 19:02:10 2011] [warn] FastCGI: (dynamic) server "/home/xpapers/cgi/handler.fcgi" restarted (pid 15951)
DBD::mysql::st execute failed: Unknown column 'historicalFacetOf' in 'field list' at /usr/lib/perl5/site_perl/5.12.3/Rose/DB/Object.pm line 422.
load() - DBD::mysql::st execute failed: Unknown column 'historicalFacetOf' in 'field list' at /usr/lib/perl5/site_perl/5.12.3/Rose/DB/Object.pm line 422.
at /home/xpapers/lib/xPapers/Conf/Forums.pm line 10
Compilation failed in require at /home/xpapers/cgi/handler.fcgi line 47.
BEGIN failed--compilation aborted at /home/xpapers/cgi/handler.fcgi line 47.

It seems that 'historicalFacetOf' is a database field in the Cats table referenced by lib/xPapers/Cat.pm. I tried commenting out line 45 of this file which makes the reference and then got further errors about facetRoot, historicalLeaf, summary, keyWorks, introductions, summaryUpdated, summaryChecked and advertiseChecked. After commenting all of those out the error goes away when running /home/xpapers/lib/xPapers/Conf/Forums.pm from the command line, but i'm still getting errors on the website so something else is still broken. The fact that it worked last week makes me think that this is not the root cause of this problem and is just suppressing the symptoms of something else. Unfortunately xpapers is a rather large piece of code and I've only just started working on it and don't really understand how it works yet.

Setting up main.pl

When installing xPapers a lot of things need to be set to match your own config in the xpapers config file (/etc/xpapers.d/main.pl). As this file contains passwords it shouldn't be committed into publicly accessible version control repositories.

The DB_SETTINGS section, this contains the database server hostname, username and password.

The SUBJECT variable, contains the subject of the installation. By default its philosophy, we've changed it to Food and Water Security.

EDITORS_EMAILS is the email addresses of the editors. By default this is the xpapers authors, probably don't want emails claiming to be from them! A well configured MTA won't pass these on anyway.

EMAIL_SENDER is the address all email appears to come from. Again this defaults to the xpapers authors and needs to be changed.

SMTPHOST is the email address of the email server. I wonder if this means we can get away without having a local MTA running?

TEST_MODE - set to 1 to disable email, useful for testing! Don't forget to change it back.

PASSWD_SALT - used to encrypt the passwords. Change this from the default and keep it secret to maintain strong password encryption.

FACEBOOK - used for facebook API keys, still need to work out what kind of facebook integration xpapers does.

RECAP_KEY/RECAP_PUBKEY - used to set things for reCAPTCHA, which is required to sign up any users to your xpapers install. Get a key from http://www.google.com/recaptcha.

DEFAULT_SITE_NAME - set to fws-hub from philpapers.

Part of the colour scheme is controlled by the variables C1, C2, C3 and C4.

Still to investigate:

How does the Z3950 harvesting work? It seems to just specify a single URL (library of congress), how do we harvest multiple sites?

What does the sites list variable do?

What's a push server?

Installing xPapers on Gentoo

xPapers is an open source web portal framework for disciplinary virtual research environments developed through project partners at University of London and Australian National University and part funded by JISC.

xPapers installation documentation is provided for Linux on Debian and Ubuntu systems - I'm installing on Gentoo, so here goes.

Dependencies

First we need to install the dependencies on the Gentoo server - this has been adapted from the xPapers documentation for Debian/Ubuntu.

N.B. MySQL needs to be installed from source to compile it with SphinxSE (Open Source search server) - see below.

First, add portage use flags to provide package compilation with appropriate support.

Available flags can be viewed in the file /usr/portage/profiles/use.desc and added to /etc/make.conf e.g. USE=" -apache2 -perl -mysql -mysqli ..."

apache2 - Adds Apache2 support
perl - Adds support/bindings for the Perl language
mysql - Adds mySQL Database support
mysqli - Adds support for the improved mySQL libraries

Next, emerge the following dependencies

* sys-devel/libtool
Latest version available: 2.4-r1
Latest version installed: 2.4-r1
Size of files: 832 kB
Homepage: http://www.gnu.org/software/libtool/
Description: A shared library tool for developers
License: GPL-2

* sys-libs/ncurses
Latest version available: 5.7-r7
Latest version installed: 5.7-r7
Size of files: 2,387 kB
Homepage: http://www.gnu.org/software/ncurses/ http://dickey.his.com/ncurses/
Description: console display library
License: MIT

* app-admin/sysstat
Latest version available: 10.0.1
Latest version installed: [ Not Installed ]
Size of files: 259 kB
Homepage: http://pagesperso-orange.fr/sebastien.godard/
Description: System performance tools for Linux
License: GPL-2

* app-text/aspell
Latest version available: 0.60.6-r2
Latest version installed: [ Not Installed ]
Size of files: 1,736 kB
Homepage: http://aspell.net/
Description: A spell checker replacement for ispell
License: LGPL-2

* dev-perl/Text-Aspell
Latest version available: 0.09
Latest version installed: [ Not Installed ]
Size of files: 13 kB
Homepage: http://search.cpan.org/~hank/
Description: Perl interface to the GNU Aspell Library
License: || ( Artistic GPL-2 )

* app-dicts/aspell-en
Latest version available: 6.0.0
Latest version installed: 6.0.0
Size of files: 178 kB
Homepage: http://aspell.net
Description: English (US, British, Canadian) language dictionary for aspell
License: as-is public-domain

* dev-lang/perl
Latest version available: 5.12.3-r1
Latest version installed: 5.12.3-r1
Size of files: 11,767 kB
Homepage: http://www.perl.org/
Description: Larry Wall's Practical Extraction and Report Language
License: || ( Artistic GPL-1 GPL-2 GPL-3 )

* www-servers/apache
Latest version available: 2.2.17
Latest version installed: [ Not Installed ]
Size of files: 4,896 kB
Homepage: http://httpd.apache.org/
Description: The Apache Web Server.
License: Apache-2.0 Apache-1.1

* www-apache/mod_perl
Latest version available: 2.0.4-r1
Latest version installed: [ Not Installed ]
Size of files: 3,640 kB
Homepage: http://perl.apache.org/
Description: An embedded Perl interpreter for Apache2
License: GPL-2

* www-apache/mod_fastcgi
Latest version available: 2.4.7_pre0910052141
Latest version installed: [ Not Installed ]
Size of files: 100 kB
Homepage: http://fastcgi.com/
Description: FastCGI is a open extension to CGI without the limitations of server specific APIs.
License: FastCGI


N.B. Gentoo is likely to have g++ (GNU C++ compiler) pre-installed, check with: g++ --version
Also, no ebuild was found for libaspell-dev

Compiling MySQL 5.1.x with SphinxSE

First, download sources as recommended in the xPapers documentation.

MySQL Community Server 5.1.58 and Sphinx 0.9.9

Then follow the Sphinx documentation for compilation.

Configuring Apache and mod_perl

See the default config for Gentoo.