CiviCRM on Drupal 8 (Nov 2020)

Goal

 This blog is to capture a practical example of installing a real example of CiviCRM on top of a current version of Drupal.

At this time, Drupal 8 is the standard with many sites using D9.  I have had a few (not many at all) modules or tasks that have not been ready for D9 so I am using D8 to be as safe as possible.

This is a quick attempt at documentation.  It is NOT pretty.  It is free for you to use or ignore.

I welcome discussion.

Here is the video that you can follow along with this blog.  I put timestamps in the blog to align with the video.

 

Right now, it only covers intallation.

I may get around to other basics such as

• Creating memberhip types along with a user/member creation form that can be either self-service or admin only.
• Making it so that new member get a Drupal account when the memeber is created.
• Putting prices/fees and payments on memberhsips.
• Adding +1s to a membership (ie family memberships)
• Mailings to members
• Fundraiser events/campaigns.
  
• Reserve/Book some asset that the club owns.  Using fees to reserve.  Using discount codes so members can get x for free.  I have NOT nailed this one yet.  If you can help, let's get this right.
  

My Virtual Machine Environment

Most of my moderately powerful machines in my home are Windows 10 based so that is the platform I am building on.  I originally worked on Oracle VirtualBox but had a few problems with that including it not being able to run in parallel with Docker and only having hacks available to configure it to restart on boot.

I am using Hyper-V as the platform and installed an Ubuntu 20.04.1 Desktop VM.  Normally I would NOT use a desktop environment for this tasks but I think that the built-in Ubuntu Desktop Hyper-V image has some tuning to make it perform better on top of Win10.

This does have a UI for the machine but I do all of my work through ssh into the VM from a Windows Terminal using WSL mode.  

This can all be discussed in more detail separately.

Video - 00:00 to 00:32

High Level Overview of my Ubuntu 20.04.1 Environment

Video:

00:55 - LAMP Stack

Update Ubuntu

sudo apt update sudo apt upgrade -y

  

LAMP Stack with some extras

sudo apt install -y mysql-server apache2 curl libapache2-mod-php  php   php-mysql    php-cli   php-mbstring     php-xml    php-gd   php-json    php-curl    php-common    php-intl  php-zip    git unzip zip vsftpd nodejs npm

 

01:14 - Some MySQL tweaks

Edit my.cnf file and put this setting in mysqld section:

sudo vi /etc/mysql/my.cnf [mysqld] default_authentication_plugin= mysql_native_password

Let mysql listen on more than localhost

sudo vi /etc/mysql/mysql.conf.d/mysqld.cnf   #  comment out the bind address to allow remote config sudo systemctl restart mysql

To use phpMyAdmin remotely (ie since we are using server edition we are remote to it) we need to either 1) have root be authenticated by pw or 2) create a new user with all the rights.

If we create a new user, then we can afford to disallow remote root when secuting. If not, we need to keep allowing remote root.

Still need to use root by password since this is hardcoded into the Aegir installer.

Bailing on Aegir for now so choose the latter. which is the first example below.

sudo mysql     SELECT user,authentication_string,plugin,host FROM mysql.user;  # this is just a check     ALTER USER 'root'@'localhost' IDENTIFIED WITH mysql_native_password BY 'password';     FLUSH PRIVILEGES;     SELECT user,authentication_string,plugin,host FROM mysql.user;  # this is just a check AND / OR     CREATE USER 'admin'@'localhost' IDENTIFIED WITH mysql_native_password BY 'password';     GRANT ALL PRIVILEGES ON *.* TO 'admin'@'localhost' WITH GRANT OPTION;     FLUSH PRIVILEGES;     exit

Install phpMyAdmin

sudo apt-get install -y phpmyadmin

Secure phpMyAdmin (TO-DO)

Interactive Steps (not yet done in this example)

sudo mysql_secure_installation  # Aegir gave me a headache. Disable PW Validation                                 # Disallow root for now, but Aegir has it hardcoded. Next time you log into mysql you'll need to use the, actually no since I did the latter   "mysql -u root -p" format

01:25 - Setting up the MSMTP mailer which is very important in the Aegir install.  Maybe not so much in a standard install.  You can try skipping this section

01:44 - I bumped the php memory available to 2x

sudo vi /etc/php/7.4/apache2/php.ini

memory_limit = 128M  (I went 256)

sudo systemctl reload apache2

 

01:49 - Prepare Apache for the sites we are going to install.  This inluded setting directory permissions and the virtual hosts.

Some Apache utils and enabling multi-site.

apache2ctl -M  # to list installed modules sudo a2enmod rewrite sudo phpenmod mbstring # /etc/apache2/sites-available/your_domain.conf # sudo a2ensite your_domain.conf ls /etc/apache2/sites-available 000-default.conf  default-ssl.conf sudo mv/cp /etc/apache2/sites-available/000-default.conf \         /etc/apache2/sites-available/<site>.conf sudo vi /etc/apache2/sites-available/<site>.conf ServerName warrington-pa.rotaryclubs.us DocumentRoot /var/www/warrington/docroot sudo a2ensite <site> sudo vi  /etc/apache2/apache2.conf <Directory /var/www/your-site/your-docroot/>         AllowOverride All </Directory> sudo systemctl restart apache2.service sudo systemctl restart apache2 sudo systemctl reload apache2 # Misc commands... sudo systemctl status apache2 sudo systemctl stop apache2 sudo systemctl start apache2 sudo systemctl restart apache2 sudo systemctl reload apache2 sudo systemctl disable apache2 sudo systemctl enable apache2

I make /var/www writable for my own sanity.

02:03 - Install composer (v1 not v2) and Drush.  Note that the drush command is really just a dynamically determined reference to whichever project you are in.  You install the wrapper globally but every site needs its own composer require drush/drush .

Install Composer (v1)

cd ~ php -r "copy('https://getcomposer.org/installer', 'composer-setup.php');" php -r "if (hash_file('sha384', 'composer-setup.php') === 'c31c1e292ad7be5f49291169c0ac8f683499edddcfd4e42232982d0fd193004208a58ff6f353fde0012d35fdd72bc394') { echo 'Installer verified'; } else { echo 'Installer corrupt'; unlink('composer-setup.php'); } echo PHP_EOL;" #<check output> sudo php composer-setup.php --version=1.10.17 --install-dir=/usr/local/bin --filename=composer php -r "unlink('composer-setup.php');"

# rate limit problems with composer and github

composer config -g github-oauth.github.com <I forget where I got my key>

Install Drush

Do this once, but you still need to require drush/drush in each project

get the git key first…

make sure the public and private keys are set.

ssh git@github.com

wget https://github.com/drush-ops/drush-launcher/releases/latest/download/drush.phar chmod +x drush.phar sudo mv drush.phar /usr/local/bin/drush

02:13 - I do a lot with git so I need my keys on this VM

02:27 to 03:06 - I create a database for this installation of Drupal.

03:12 - 04:00 - I have done this so many times that I created some environment variables to help me cut down on the steps and scripts.

Create a drupal instance

Use Composer to create a new project using the desired starter template, with the latest version:

composer create-project drupal/recommended-project my-project

For a specific version, use drupal/recommended-project:8.8.0. See Packagist for all available versions, or use the shell with composer show drupal/recommended-project --all.

cd /var/www drupal_root="/var/www/d8civi" echo $drupal_root sudo rm -r $drupal_root composer create-project drupal/recommended-project:^8 $drupal_root --stability dev --no-interaction d_docroot=$drupal_root/web echo $d_docroot cd $drupal_root sudo apt install php7.4-bcmath #(one of my installations I needed to add the bcmath) systemctl reload apache2 #(not sure which one) cd $drupal_root chmod 777 $d_docroot/sites/default/ cp $d_docroot/sites/default/default.settings.php   $d_docroot/sites/default/settings.php chmod 666 $d_docroot/sites/default/settings.php

 

04:00 - 06:53

<------   Web Install ----->

http://wicensshootingrange.club:8088

 

06:53

<------   Post Web Install ----->

Fix up the ownerships

Via shell

vi ../set_perms.sh #!/bin/bash # Help menu print_help() { cat <<-HELP This script is used to fix permissions of a Drupal installation you need to provide the following arguments:   1) Path to your Drupal installation.   2) Username of the user that you want to give files/directories ownership.   3) HTTPD group name (defaults to www-data for Apache). Usage: (sudo) bash ${0##*/} --drupal_path=PATH --drupal_user=USER --httpd_group=GROUP Example: (sudo) bash ${0##*/} --drupal_path=/usr/local/apache2/htdocs --drupal_user=john --httpd_group=www-data HELP exit 0 } if [ $(id -u) != 0 ]; then   printf "**************************************\n"   printf "* Error: You must run this with sudo or root*\n"   printf "**************************************\n"   print_help   exit 1 fi drupal_path=${1%/} drupal_user=${2} httpd_group="${3:-www-data}" # Parse Command Line Arguments while [ "$#" -gt 0 ]; do   case "$1" in     --drupal_path=*)         drupal_path="${1#*=}"         ;;     --drupal_user=*)         drupal_user="${1#*=}"         ;;     --httpd_group=*)         httpd_group="${1#*=}"         ;;     --help) print_help;;     *)       printf "***********************************************************\n"       printf "* Error: Invalid argument, run --help for valid arguments. *\n"       printf "***********************************************************\n"       exit 1   esac   shift done if [ -z "${drupal_path}" ] || [ ! -d "${drupal_path}/sites" ] || [ ! -f "${drupal_path}/core/modules/system/system.module" ] && [ ! -f "${drupal_path}/modules/system/system.module" ]; then   printf "*********************************************\n"   printf "* Error: Please provide a valid Drupal path. *\n"   printf "*********************************************\n"   print_help   exit 1 fi if [ -z "${drupal_user}" ] || [[ $(id -un "${drupal_user}" 2> /dev/null) != "${drupal_user}" ]]; then   printf "*************************************\n"   printf "* Error: Please provide a valid user. *\n"   printf "*************************************\n"   print_help   exit 1 fi cd $drupal_path printf "Changing ownership of all contents of "${drupal_path}":\n user => "${drupal_user}" \t group => "${httpd_group}"\n" chown -R ${drupal_user}:${httpd_group} . printf "Changing permissions of all directories inside "${drupal_path}" to "rwxr-x---"...\n" find . -type d -exec chmod u=rwx,g=rx,o= '{}' \; printf "Changing permissions of all files inside "${drupal_path}" to "rw-r-----"...\n" find . -type f -exec chmod u=rw,g=r,o= '{}' \; printf "Changing permissions of "files" directories in "${drupal_path}/sites" to "rwxrwx---"...\n" cd sites find . -type d -name files -exec chmod ug=rwx,o= '{}' \; printf "Changing permissions of all files inside all "files" directories in "${drupal_path}/sites" to "rw-rw----"...\n" printf "Changing permissions of all directories inside all "files" directories in "${drupal_path}/sites" to "rwxrwx---"...\n" for x in ./*/files; do   find ${x} -type d -exec chmod ug=rwx,o= '{}' \;   find ${x} -type f -exec chmod ug=rw,o= '{}' \; done echo "Done setting proper permissions on files and directories"

sudo bash ../set_perms.sh --drupal_path=$d_docroot --drupal_user=jflynn

vi $d_docroot/sites/default/settings.php $settings['trusted_host_patterns'] = [   '^wicens\.com$',   '^.+\.wicens\.com$',   '^wicensshootingrange\.club$',   '^.+\.wicensshootingrange\.club$', ];

07:44 - I have tinkered with lots and lots of modules and distributions.  Some combinations seem to mess with CiviCRM.  In this example I use the absolute minimal for me.

composer require drupal/admin_toolbar

composer require drush/drush         

composer require drupal/swiftmailer

composer require drupal/webform:~6.0    ##### not the 6 part for d8

08:11 - 08:26 - I have tinkered a lot with themes and look and feel.  Let's leave that out for now and skip right to CiviCRM

08:26 - Install CiviCRM

Install and Configure CiviCRM

These edits to the composer.json file may be needed if your distribution does not use ./web as its docroot directory.  

"civicrm-asset": {

    "path": "docroot/libraries/civicrm",

    "url": "/libraries/civicrm"

    },

"patches-ignore": {

   "civicrm/civicrm-core": {

        "adrienrn/php-mimetyper": {

           "This patch has known ": "https://patch-diff.githubusercontent.com/raw/adrienrn/php-mimetyper/pull/15.patch"

           }

        }

    }

composer config extra.enable-patching true

composer require civicrm/civicrm-asset-plugin:'~1.1'

composer require civicrm/civicrm-{core,packages,drupal-8}:'~5.31-dev'

09:38 - Enable CiviCRM - I have found that when you enable Civi from drush there can be lots of permissions problems because you are the user owner of the files.  Even trying to manually fix the permissions it was an extreme headache to get it to work.  I have come to the conclusion that it is simply easier to enable it via the UI.  But before, we need to temporarily open up the permissions on the settings directory

chmod g+w $d_docroot/sites/default/

<enable civi via gui>

chmod g-w $d_docroot/sites/default/

 

This SHOULD enable the module and also run some specific scripts included with the civicrm/civicrm-asset-plugin module.  Most of my headaches in experimenting have been with the proper execution of these scripts.   If you don't get a clean Civi screen complete with menus then this is likely where your problem is.

 

By the way, you can force reexecution of the script with the following.

composer civicrm:publish  # FYI to force rebuild of civi libs

11:18 - Oh, man, this is the big test.  You don't know how many times this has failed me.  When we click on the CiviCRM menu button we SHOULD get a dashboard screen and all of the Civi menus should show up.  If you get this far, have a drink and celebrate minor successes.