Frontend & Hyvä 12/16/2019

How to bundle Magento 2 with Baler or M2 DevTools extension

Update from Nov 19, 2020: Please note that due to a Google PageSpeed update the guide found here still works, but no longer has the desired effect for Lighthouse. If you are looking for an alternative to improve the performance of your shop, Hyvä Themes might be of interest to you.

Table of Contents


Part One: The “ultimate guide” to Magento 2 JavaScript Bundling
  • About Lighthouse Audits
  • Benchmarks
  • How JavaScript influences the performance
Part Two: Magento 2 front end performance settings – Magento 2 bundling patches
  • The Basis — default Magento 2 setup & Patches
  • Default Magento Bundling
Part Three: How to bundle Magento 2 with Baler or M2 DevTools extension
→ You are here
  • Baler
  • Magento 2 DevTools browser extension
  • Devtools bundling
Part Four: The integer_net solution to JavaScript bundling
  • integer_net Bundling
  • A real-life example
  • Conclusions

Baler

Baler is the latest effort from Andrew Levine to write a tool that creates one effective bundle that contains all the JavaScript that is crucial for any page on your site. It then continues to write a dependency graph of all the other JavaScript files and in which cases they are needed. The files from that graph will then be injected into the head of the page to tell the browser it needs to fetch those files as soon as possible, without blocking page-rendering.

One of the things Andrew has put a lot of effort in is to make the tool fast. And it sure is. Creating the main bundle happens through an npm script that's executed through command-line at the root of your project.

As said before, we believe - and truly hope - that this tool will become the standard for Magento 2 frontend optimization. But, in its current state it only creates the main bundle and does not build the dependency graph yet. This means you only get the main bundle, which is at this point just a slight improvement to performance.


Getting started

Base requirements:

  • All requirements listed at the 'The Basis - default Magento 2 setup' section except for JavaScript minification
  • For Magento versions up until 2.3.4: apply the patches listed before:
    • Mixins (up to 2.3.4)
    • Fix missing shims and phtml files with mage-ini directives (up to 2.3.3)

Baler main NPM tool

Clone the Baler repository https://github.com/magento/baler/

 

Follow instructions from https://github.com/magento/baler/blob/master/docs/ALPHA.md

  • Make sure your node version is >= 10.12.9
  • Run npm run build from the main Baler directory you just cloned/downloaded
  • Run npm link so that it's installed globally.

 

Install the Magento Module

Baler needs a Magento module to be installed. https://github.com/magento/m2-bale

Either download/clone the repo and put it in `app/code/` or, better, install via composer.

 

Using composer:

composer config repositories.magento-baler vcs git@github.com:magento/m2-baler.git
composer require magento/module-baler:dev-master@dev
composer install

 

bin/magento module:enable Magento_Baler
bin/magento setup:upgrade

 

Set Magento configuration

Disable all core JavaScript options, and enable Baler Bundling. Again, remove --lock-config if you want it set in database instead of app/etc/config.php

 

bin/magento config:set dev/js/enable_baler_js_bundling 1 --lock-config
bin/magento config:set dev/js/enable_js_bundling 0 --lock-config
bin/magento config:set dev/js/merge_files 0 --lock-config
bin/magento config:set dev/js/minify_files 0 --lock-config

 

Compile your themes

I usually run setup:upgrade to remove all compiled assets, and then compile DI and static content:

 

bin/magento setup:upgrade
bin/magento setup:di:compile
bin/magento setup:static-content:deploy en_US

 

By the way, did you know there is an excellent feature of Symfony Console, on which the Magento console is based, that allows you to abbreviate those commands?

This one-liner performs all three commands above:

bin/magento set:up && bin/magento set:di:co && bin/magento set:sta:dep en_US

 

Run Baler

From the root of your project, run: baler build

If you get stuck here, please see the troubleshooting section below.

 

Run Terser

For best results, minify all JavaScript files Baler did not already minify.

Code snippet for terser-bundle-bash-script

That's it

Done. Go to your frontend and see if everything works. You should see a core-bundle.js file being loaded and a 'requirejs-bundle-config.js' instead of the default 'requirejs-config.js'.

If not, check if the Magento Baler module is properly activated and enabled.

 

Troubleshooting


Could not find Magento store root
If Baler won't run, because it "Could not find Magento store root from [dir]", it means it failed to find one of the following directories/files:

  • app/
  • pub/
  • lib/
  • index.php

In case you run Baler locally with Docker, it means you need to add these directories/files to the list of synced directories. Or run the tool from inside the Docker container.

 

Errors in your store's frontend
If you see JavaScript errors in your browser's console output, and you are on a Magento version lower than 2.3.4, make sure that you have applied all patches listed before. If problems persist, you might have a faulty RequireJS configuration or you try to run JavaScript code without properly requiring it.

An issue you might run into is jQuery code that is run without properly requiring jQuery first. If anyone put a direct reference to jQuery in a template file or a CMS page/block, you might get errors with bundling that you didn't get before.

For example, this will throw an error if jQuery is not yet defined.

if(jQuery.slickSlider()) {
// initialize a slider
}

 

Instead, this should be written as:

require([ "jquery", "Magento_Theme/js/slickslider" ], function () {
// initialize a slider
});

Profile

If your frontend runs without issues, start auditing and see what results Baler gives you compared to the default Magento configuration. You should see at least an improvement by a few points.

We saw a greater improvement on actual projects than in the benchmarks we did with default Magento and sample data.

 

Results

Table of results for JavaScript Bundling without and with Baler.
Result of Lighthouse Perfomance Audits with and without Baler

Homepage

Result of Lighthouse Perfomance Audits with Baler

Checkout

Result of Lighthouse Perfomance Audits with Baler for the Luma DemoStore checkout

Implementation

As you'll see on both of the Baler repositories, these tools are currently in alpha state, meaning they are not strictly considered production-ready yet.

We've found they work quite well and heard about some webshops happily running them in production.

The Baler tool itself is incredibly fast, which is a fantastic result from Andrew Levine.

Adding the manual Terser step to it unfortunately is extremely slow. Terser is the best minifier currently out there, but when running it on the highest possible compression setting means you'll be waiting many minutes for each theme/language you run in your store.

You could however play around with the settings (you can look up the settings/flags here) and speed up the process.

We have not used this in our deployment pipeline yet. But you could install Baler locally in the project during deployment. This means you clone the project, npm install it and execute node_modules/bin/baler instead.

- Without having tested it - I assume this would work.

 

Magento 2 DevTools browser extension

Before testing the DevTools Bundling or our 'integer_net Bundling', you need to get the Magento 2 DevTools browser extension.

This extension has a RequireJS Bundle Generator that creates an optimized bundle for your Magento shop by browsing through the shop and keeping track of the used JavaScript files.

The extension is a previous endeavour also from Andrew Levine but unfortunately abandoned in favor of his new tool, Baler (you should by now know what Baler is).

The generated bundle configuration from the toolbar can be used by the RequireJS Optimizer, which is a tool by RequireJS itself, to create (you guessed it) optimized bundles.

Both the DevTools Bundling and our own solution make use of the toolbar. In our own solution, we found some ways to improve the bundles and, most importantly, the way they are loaded.

 

Installing the M2 DevTools toolbar

You only need to do this once for your device. It will install the toolbar in your browser.

Clone the M2 Devtools repositoryhttps://github.com/magento/m2-devtools

 

Follow instructions from Magento 2 DevTools Readme:

  • Make sure your node node version is >= 8.x
  • Run npm install from the main M2-Devtools directory you just cloned/downloaded
  • Run npm run build so that it's installed globally.
  • Navigate to chrome://extensions
  • Enable Developer mode
  • Click Load unpacked
  • Select the extension folder in the root of this repository

You should now have a new Inspector tab called Magento 2.

 

Now open up your Magento 2 shop, using production mode and open the DevTools RequireJS tab.

Following the instructions from M2 DevTools:

  • Visit any page in the storefront in a browser with this extension installed
  • Open RequireJS >> Bundle Generator
  • Click the Record button
  • Begin navigating to critical pages of your store (Home/Catalog/Product/Cart/Checkout)
  • Click the Record button again when you are finished
  • Copy the generated bundle configuration and save it in your Magento 2 website root as requirejs-config-bundle.js

 

Good. Now you can proceed either to the DevTools Bundling section or our integer_net Bundling section.

 

DevTools bundling

You may choose to skip this section if you just want to compare our method with default Magento and Baler. We don't make use of the Magento module used in this section and our method will most certainly get better results than the default implementation described here.

Getting started

Base requirements:

 

Optionally remove Baler:

If you just tested the results with Baler, you might want to clean that up first.

bin/magento module:uninstall Magento_Baler
composer remove magento/module-baler

 

Install the Magento module

The Magento_BundleConfig module needs to be installed. It's part of the M2-DevTools repository: https://github.com/magento/m2-devtools

Either download/clone the repo and put it in app/code/ or, better, install via composer.

 

Using composer:

composer config repositories.magento/module-bundle-config vcs git@github.com:magento/m2-devtools.git
composer require magento/module-bundle-config:dev-master@dev
composer install

bin/magento module:enable Magento_BundleConfig
bin/magento setup:upgrade

 

Set Magento configuration

Disable all core JavaScript options. Remove --lock-config if you want it set in database instead of `app/etc/config.php

 

bin/magento config:set dev/js/enable_js_bundling 0 --lock-config
bin/magento config:set dev/js/merge_files 0 --lock-config
bin/magento config:set dev/js/minify_files 0 --lock-config

 

Note that the Magento_BundleConfig actually disables JavaScript minification with a plugin, but I'd consider it good practice to also set it in configuration.

The reason why minification is being disabled is because the RequireJS optimizer will run minification after generating the bundles, and the tool works better if the files haven't been minified already (it might even break)

 

Install r.js, the RequireJS optimizer

You can either download r.js from the requirejs.org website or, better, install it with npm: npm install -g requirejs

You should now be able to run r.js from command line: r.js

See https://github.com/requirejs/r.js for usage.

 

Compile your themes

I usually run setup:upgrade to remove all compiled assets, and then compile DI and static content:

 

bin/magento setup:upgrade
bin/magento setup:di:compile
bin/magento setup:static-content:deploy en_US

 

By the way, did you know there is an excellent feature of Symfony Console, on which the Magento console is based, that allows you to abbreviate those commands?

This one liner performs all three commands above:

bin/magento set:up && bin/magento set:di:co && bin/magento set:sta:dep en_US

 

Create the bundles

We can now go to the root of our project where you should have saved your requirejs-bundle-config.js

Now we are going to perform a couple of commands to first copy the generated themes from pub/static/frontend/ to a backup directory and then run the r.js optimizer which will copy the theme back into the original directory with the optimizations and generated bundles.

Code snippet for devtools-bundle-bash-script

You should see a long list of output for the files that are being minified and the bundles that are being created.

 

That's it

To be sure that your new files are being loaded, flush your cache (and maybe even force clean your Varnish).

Now visit your store's frontend and open inspector. You should see `bundles/shared.js` loaded on any page and for any page that has a custom bundle you should see the bundle with the layout handle e.q. bundles/catalog-product-view

 

Troubleshooting


Bundling fails because certain files cannot be found.

The toolbar will pick up AMD files that might be inside a minified JavaScript file that are not in your RequireJS config. I found it easiest to simply remove those from the bundle configuration and try again.

For example, we found fizzy-ui-utils/utils in one of our bundle configs, but no such file exists.

Upon inspection, we found define.amd?define("fizzy-ui-utils/utils", ...) in a library of our project. The RequireJS bundler will try to locate that module and won't be able to find it.

 

Errors in the frontend

Did you apply the patches?

Is there any JavaScript loaded directly without the needed require([], function(){}) around it?

 

Some things don't work in the checkout

Mixins might not be working, and the first place you usually find out is in the checkout process.

Again, for Magento versions up till 2.3.4: did you apply those patches?

 

Other issues

You might have errors in your requirejs-config.js files, mixins or shims or any other issue. This is unfortunately a case by case issue and you'll have to debug.

 

Profile

You can now start profiling. Results really depend on your project, but we've seen varying results but for most pages the results are lower than with Baler and even without bundling.

It depends on how JavaScript heavy the pages are really.

The issues we found with this implementation is that the bundles are loaded into the head synchronously and therefore blocking the page-rendering.

What comes next is our implementation where we solve this issue by making the bundles asynchronous.

 

Results

Table of results for JavaScript Bundling without and with Devtools.
Result of Lighthouse Perfomance Audits with and without Devtools

Homepage

Result of Lighthouse Perfomance Audits with Devtools for the Homepage of the Luma Demostore

Checkout

Result of Lighthouse Perfomance Audits with Devtools for the Luma DemoStore Checkout

Implementation

We've managed to get this into our deployment pipeline installing requirejs locally in the project during deployment. This means you run npm install requirejs and execute node_modules/bin/r.js instead. This works well.

The downside to this implementation is that any time you change your shop and JavaScript files are added or removed, you need to create a new bundle config.

In case new files are added or missing from your bundle configuration, this is not a big issue because RequireJS will always fall back to loading a single file if the can't be found in the bundles.

If files are removed, however, you won't be able to create the bundles because r.js will fail and exit with an error file not found.

So keeping your configuration up to date is an ongoing process.

 

You finished reading the third part.
Now would be a good time to skip to Part Four.

 

How do you wish to proceed?


Part One: The “ultimate guide” to Magento 2 JavaScript Bundling
  • About Lighthouse Audits
  • Benchmarks
  • How JavaScript influences the performance
Part Two: Magento 2 front end performance settings – Magento 2 bundling patches
  • The Basis — default Magento 2 setup & Patches
  • Default Magento Bundling
Part Three: How to bundle Magento 2 with Baler or M2 DevTools extension
→ You are here
  • Baler
  • Magento 2 DevTools browser extension
  • Devtools bundling
Part Four: The integer_net solution to JavaScript bundling
  • integer_net Bundling
  • A real-life example
  • Conclusions