The Easiest Way to Speed Up Your Julia Runtimes

by Abhishek Priya | May 14, 2019 | Archive, General, Tutorial

Why do Julia and remote code deployment go hand in hand?

The increasingly popular programming language Julia was built with big data projects in mind. These projects imply either very long runtimes or access to compute power that will allow you to cut that down. Luckily, Julia has certain built-in advantages for distributed computing as well as remote deployment, and there is now a native app—Galileo—that will allow you to quickly and easily drag and drop your Julia code to run elsewhere. This is the fastest, easiest, and best method we’ve found to enhance Julia’s output and speed.

While Julia is an all-purpose language, creators were especially thinking about big data analytics, simulations for scientific or engineering research, and high-performance computing. Combining the functionality of R and Python with the speed of C++, Julia is particularly well suited for this. Creators also aimed to make it easier to innovate and to solve machine learning problems that are more difficult with other languages.

Top fields for Julia’s use are:

3D Printing
Augmented Reality
Machine Learning
Precision Medicine
Self Driving Cars

Of course, these kinds of projects are fairly demanding, which can quickly become impractical, depending on your set-up. While it is increasingly practical to run big data projects on multiple nodes in a cluster so tasks can be completed in parallel, many programming languages, like Python, were never designed for this. More or less clunky workarounds have had to be found. By contrast, “Julia was designed for parallelism from the ground up.” Julia has built-in features that make it easier to spread workloads across multiple CPU cores, and it has out-of-the-box support for distributed scientific computing.

But what if I don’t have access to a machine that can take advantage of these features? Answer: remote deployment.

When Julia was initially created, most users were data scientists working on large scientific projects with access to high-capacity machines. Julia’s parallel computing features were, at that point, most useful for running on distributed machines but not necessarily used for remote deployment. As Julia continues to gain in popularity, its user demographics are shifting. It is still used by data scientists working on big data projects, but many of them do not have access to large rigs and therefore need easy access to remote compute power. This is where Galileo comes in.

Galileo makes it extremely easy to reduce runtimes by deploying Julia projects remotely. (Check out this quick “How-To.”) It is an ideal complement to Julia, which was itself designed without any compromise between ease-of-use and performance.

Galileo also makes it possible to exploit certain aspects of Julia that are ideal for remote deployment. For one, Julia’s dependency management is much better when compared to other programming languages. This is already the case, even though there’s more work to be done re package compilations that will make deployments even easier. It’s also notable that Julia plays well with Docker. Since Julia is an adopter of Docker images, it’s basically the straightest shot for containerization, code-wise. Since Galileo is built on top of Docker, it also leverages this advantage.

What about GPUs, CPUs, and other novel architectures?

As of fall 2018, there were future plans to improve Julia support for parallel processing on GPUs and TPUs, in addition to further optimizing CPU parallel processing. If you have access to a computer with an NVIDIA card, for example, the Julia interpreter should be able to use it without any fancy code contortion on your part. Or if you have a computer cluster, you should be able to run Julia jobs on the cluster without having to directly program with MPI, because new methods will have been built around CUDA and MPI.

Again, Galileo will compliment these enhancements in that it facilitates the sending of Julia jobs from your laptop to the remote machines that have these capabilities (i.e. GPU’s or HPC cluster of nodes).

Conclusion:

Julia was designed with heavy duty jobs in mind, which are much more practically run with heavy duty compute resources. Julia already includes features that predispose it towards distributed computing. Galileo is an ideal fit for Julia enthusiasts who want to optimize their projects even further through quick and easy remote deployment. It provides an obvious solution and complement to Julia’s unique combination of ease-of-use and performance.

Further resources:

What is Julia? Brief history. It is adopted by more and more coders:

Future top Julia fields: 3D Printing, Augmented Reality, Machine Learning, Precision Medicine, Self Driving Cars

Julia and parallel distributed computing: