[Updated 09 July 2014: Previous (2012) version for reference here. Script updated for Ubuntu 14.04 based systems.]
All of my research for the past 5 years was done with free software. In this post I describe the free programs that I use every day, and what I use them for. I do not use them simply to conform to stereotypes about cheap Scotsmen. As you will see, I use them because they are portable and very powerful.
Free/Libre Open Source Software (FLOSS) allows you to make as many copies the programs as you need and distribute them as you please. This makes it portable. Any workflows or methods can be taken to different computers, different institutions or sent to friends in different countries without worries about expensive licences. For example, I use GRASS GIS instead of ArcMap, Python instead of Matlab, Zotero instead of Endnote. I also use some free (gratis) proprietary software such as Google Earth. While philosophically different to FLOSS software, for practical purposes the advantages are the same.
In this post, I have divided the programs into different categories: Operating system; Maps and Geographic Information Systems; Data Processing and Plotting; Writing Journal Articles; Conference Presentations; Programming Tools; Images, Graphics and Photos; Videos and Media; Computer Administration Tools and Miscellaneous.
I have also posted a short script that to automatically install most of this software onto a Linux machine, and I invite you to suggest any software that I may have missed in the comments.
Operating System
My current system runs a FLOSS operating system, GNU/Linux (shortened to Linux here). There are many websites about the advantages of switching to Linux and the high-profile organisations that have already done so. You definitely don’t need to be a geek to run it, but it can help to have one around to set it up in the first place.
My main reason to run Linux is the command line interface (CLI), which can be used to carry out tasks very quickly and precisely. It has the HUGE advantage that once you know the commands to do what you need, you can write them in a script and repeat the task 1000 times with very little extra effort. This makes it very powerful. It feels like your computer is working for you and most of my workflows now take advantage of this.
Of all the different Linux flavours, I chose Linux Mint 14 XFCE. It is based on the popular Ubuntu distribution so it has a wide range of software available in easily-installed packages and there are lots of helpful tutorials for it online. The latest versions of Ubuntu have a tablet-style interface; I prefer the way that Mint sets things up for the desktop. You could also try Xubuntu or Linux Mint Cinnamon instead as both are the same under the hood. Each comes as a LiveCD, so you try them out without altering your system.
The names of the Ubuntu software packages for each program are given below so that you can install them easily from the Software Centre or via the command line. Windows and Mac versions exist for most and can be found with a quick Google search.
Maps and Geographic Information Systems
- GRASS (grass): Fully-featured and extremely-powerful GIS package with both GUI and command line interfaces. It handles raster and vector data in all formats and is easily scriptable to automate workflows. I use it to create new GIS datasets from raw data e.g. by processing LiDAR point clouds, digitising field maps, image analysis of multispectral remote-sensing data.
- Quantum GIS (qgis, qgis-plugin-grass): Easy-to-use GUI-based GIS package. It is ideal for making and printing maps from pre-existing datasets. It also has a nice georeferencing tool and can be used as an interface to GRASS GIS.
- GDAL (gdal-bin): A command-line swiss-army-knife for GIS files allowing you to convert formats, change projection, join, crop and alter the resolution of raster files and much more. Includes OGR, which does the same with vector files (e.g. shape files). This is what actually does a lot of work behind-the-scenes in GRASS and QGIS.
- Proj4 (proj-bin, proj-data): Command line tools for reprojecting data points in different map projections (cs2cs). This works behind-the-scenes of GDAL.
- Generic Mapping Tools (gmt, gmt-coast-low, gmt-doc): These command-line tools for plotting publication-quality maps of geophysical data are very popular among oceanographers and seismologists. You won’t see an issue of Journal of Geophysical Research that doesn’t contain at least one figure made in GMT.
- Google Earth (instructions here): A 3D globe in your computer showing everything from the submarine mountains of the Mid-Atlantic ridge to the car parked in your street. Not FLOSS.
- GPS Babel (gpsbabel): Communicate with any handheld GPS unit, and convert formats between gpx, kml, garmin and anything else that you can think of. The Windows version has a graphical user interface.
- GPS Prune (gpsprune): GUI-based tool for editing GPS point and track data. The best feature is the ability to geotag photos then view them in Google Earth (see video here).
Data Processing and Plotting
- Python (python): An open source, cross-platform programming language. It is widely-used by scientists and is extremely versatile because it can be easily extended using addon modules such as these below. Some of the other advantages are described here. Everything that I used to do in Matlab, I now do in Python, safe in the knowledge I can take the scripts with me wherever I go. The easiest way to get Python and most of the following packages onto a Windows machine is by installing Python(x,y).
- IPython (ipython): Excellent interactive interface for Python. In particular, the IPython Notebook lets you write Python in your web browser, combining it with text, LaTeX, images, hyperlinks and videos. There are great examples the people have shared on the nbviewer website. It is going to revolutionise teaching students how to code.
- Spyder (spyder): A development environment for Python, giving it a Matlab-like appearance and with features such as code-checking, command completion and automatic display of documentation for the current command / object.
- Numpy and SciPy (python-numpy, python-numpy-doc, python-scipy, python-netcdf): Scientific and numerical computing modules for Python, allowing it to handle arrays of numbers, and the NetCDF data format.
- Matplotlib (python-matplotlib, python-matplotlib-doc): Plotting modules for Python allowing you to make all kinds of publication-quality 2D and 3D figures such as these.
- Basemap (python-mpltoolkits.basemap, python-mpltoolkits.basemap-data): Add-on for Matplotlib giving Python similar map-plotting functions to those of GMT e.g. plotting in different projections, adding coastlines or the Blue Marble image). See some examples here. It also contains the pyproj module which allows easy conversion between coordinate systems. See my post for a quick intro.
- R (r-recommended): An open source, cross-platform programming environment, with a strong emphasis on statistics. Also very powerful for geospatial data. [Added in 2012 after overwhelming support in comments below. See them for useful links.]
- SQLite (sqlite, sqlite3, sqliteman): This an open source database format. It can be accessed via the same Structured Query Language used by cutting-edge data servers, but the data are stored in a single, portable file. This allows you to perform cool queries such as getting a list of photos of samples that were collected on a Tuesday, in Scotland, and had ash in them. I am switching to storing sample data here because the data can then be accessed directly by GRASS and by Python.
- SQLiteManager (Firefox plugin): A nice viewer that lets you edit and perform queries on SQLite databases.
- XYScan (xyscan): Digitise points from plot images / maps. Ideal for getting data from old papers.
- LibreOffice Calc (libreoffice-calc): An open source spreadsheet program, and a viable substitute for Excel. LibreOffice is a slightly more independent version of Open Office. I don’t use spreadsheets that much, but it seems to do everything that I need it to. Gnumeric (gnumeric) is a quicker, but less featured, spreadsheet program.
Writing Journal Articles
- Zotero (Firefox plugin): Reference manager software. It runs in Firefox and lets you add articles to the database directly from the journal website or the results page of a Web of Science query. It has a plugin that lets you put references into Word or Writer documents and can export BibTex files, too. Also, it syncs with the cloud, so your reference library is constant across different computers.
- LaTeX (texlive, texlive-latex-extra, texlive-fonts-extra, texlive-humanities + others): LaTeX is an open source typesetting program. It is used to produce beautifully laid-out pdf documents from plain text files containing the text and some simple formatting codes e.g. section{Introduction}. The best thing is that it does referencing, section numbering, figure captions and tables of contents for you automatically. If you are about to write a thesis, then learning LaTeX will be one of the best things that you ever did. For a graphical-user-interface, try Lyx or MiKTeX.
- LibreOffice Writer (libreoffice-writer): This is an open source word processing program. This is an ideal substitute for Microsoft Word on all platforms, as it can read and write .doc and .docx files. The most important features for me, comments and track changes, work perfectly. I need these to collaborate on work with my co-authors. It also prints straight to pdf, which is nice.
Conference Presentations
- Scribus (scribus): I use this professional quality desktop publishing package to make conference posters. It is very easy to create good-looking layouts, align images and set font-themes, but that just scratches the surface of what it can do. The output is a pdf file that you can print anywhere. Read my quick-start guide here.
- Beamer (latex-beamer): Make pdf-format conference slides in LaTeX. It has all the advantages of LaTeX e.g. beautiful results, no-fussing about layout, referencing and contents all taken care of. Plus the pdf files don’t get messed up between Mac/Windows/Linux versions like Powerpoint slides can.
- LibreOffice Impress (libreoffice-impress): This is an open source Powerpoint substitute. It is definitely the weakest of the LibreOffice family. It can read and write Powerpoint files but sometimes the fonts and layouts come out differently, and it is generally a lot less slick. It does a decent job, though, and I have written a couple of lecture courses with it.
- PDF Toolkit (pdftk): Command line tool to join pdfs, extract individual pages, rotate pages and generally reshuffle pdfs.
Programming Tools
- GVIM (vim-gtk): Geeky text editor. Steep learning curve, but if you love keyboard shortcuts then give it a go. Once you discover macros you’ll be flying. Learn more here. So far, I mainly use it for LaTeX, but have recently found that it can connect to iPython to become a Python IDE.
- git (git): Distributed version control for working offline and online.
- meld (meld): Compare and merge differences between two text files.
Images, Graphics and Photos
- Gimp (gimp): The Gnu Image Manipulation Programme is equivalent to Adobe Photoshop or Corel Photopaint. The interface takes some getting used-to, but it is very powerful.
- Inkscape (inkscape): Inkscape is a vector graphics package equivalent to Adobe Illustrator or Corel Draw. It’s fast, light, and a joy to use.
- Image Magick (imagemagick): Command-line tools that allow automatic or batch processing of image files: resize, rotate, label, crop, change format etc. See my post about it here.
- Shotwell (shotwell): Photo viewing programme a bit like iPhoto on a Mac, allowing you to view your images using tags, ratings and events. Ideal for organising field photos.
- Hugin (hugin): Panorama / photo stitching software. If you have to scan a map in many parts, it’s good for joining them up again.
- Dia (dia): Software for drawing flowcharts and other structured diagrams.
Videos and Media
- VideoLan Player (vlc): Play video files in almost any format that you can think of.
- Openshot (openshot, openshot-doc): Simple video editing.
- avconv (libav-tools): Command-line tool to change the size, framerate, format etc. of videos. Good for extracting the soundtrack as an mp3. Great for chopping out clips of sounds or videos. This used to be known as ffmpeg.
- Audacity (audacity): Edit mp3 and other sound files.
- Sound Juicer (sound-juicer): Rip CDs to MP3 or other formats.
- Youtube Downloader (youtube-dl): Command line tool to download youtube videos to watch offline.
- Get iPlayer (get-iplayer): Command line tool to download BBC iPlayer programmes to watch offline (only works within the UK).
Computer Administration Tools
- Ubuntu Restricted Extras (ubuntu-restricted-extras): By default, Ubuntu only ships with open source software. This package installs commonly-used the proprietary tools such as Flash video, Microsoft fonts and MP3 codecs.
- Open SSH (openssh-client, openssh-server): Connect securely to your machine across the internet without the fuss of a VPN. Log in with a terminal to see how jobs are getting on, or use a secure FTP program such as WinSCP to copy files.
- Rsync (rsync): One-way synchronisation over SSH. I use this to automatically back up my desktop to the department server. It knows which files have changed and only sends the differences, so it runs very quickly.
- Unison (unison): Two-way synchronisation between computers over SSH. I use this to sync the files on my netbook with my desktop machine each day.
- Dropbox (dropbox): File-syncing in the cloud, if you don’t mind the NSA reading your files.
- EncFS (encfs, gnome-encfs-manager): Encrypt individual folders on your hard drive to keep confidential data (client data, student marks) if you lose your laptop. Command line, but much easier with the graphical manager.
- Baobab (baobab): Nice graphical disk usage program. See which folders are taking up most space.
- WINE (wine): Lets you run Windows programs on a Linux machine. Some people use it to play games or other complicated software, but it can be a bit hit-and-miss. I use to run the simple panorama-making software, Autostitch, which works perfectly.
Miscellaneous
- Skype (skype): Free phone calls (with video) over the internet. The “Partners” repository should be enabled in the Software Centre before installation. Not FLOSS.
- Adobe Acrobat Reader (instructions below): Evince, the pdf reader that comes as standard with Ubuntu is great for reading pdfs. But to add comments, make corrections, or fill in forms you need the Adobe version. Not FLOSS.
- Stellarium (stellarium): See what’s in the night sky above. Still cool despite the invention of the Google Sky Map app.
- Hotot (hotot): Twitter client that lets you view your lists in different columns.
- Adblock Plus (Firefox plugin): The internet is a much faster and less cluttered place without adverts.
- Pocket (Firefox plugin): Save articles from the internet to read later, and have them synchronised with your phone.
Installation script
The following script will install most of the above software onto a freshly-installed Ubuntu 12.10 machine. First ensure that the ‘universe’, ‘multiverse’ and ‘partner’ repositories are enabled in the Software Centre.
# Maps and GIS software sudo apt-get install grass qgis qgis-plugin-grass gdal-bin proj-bin proj-data gmt gmt-coast-low gmt-doc gpsbabel gpsprune # Data processing sudo apt-get install spyder python-numpy python-numpy-doc python-scipy python-matplotlib python-matplotlib-doc python-mpltoolkits.basemap python-mpltoolkits.basemap-data r-recommended sqlite sqlite3 sqliteman python-netcdf # Upgrade to the latest IPython notebooks sudo apt-get install python-pip sudo pip install --upgrade ipython sudo pip install pygments # needed for qt console # Others sudo apt-get install baobab dia dropbox scribus texlive texlive-latex-extra texlive-humanities texlive-fonts-extra texlive-lang-other latex-beamer gimp git hotot inkscape imagemagick libav-tools meld pdftk shotwell vlc openshot audacity sound-juicer youtube-dl get-iplayer ubuntu-restricted-extras openssh-server unison wine stellarium skype hugin vim-gtk xyscan # Adobe Reader (uses gdebi to install dependencies) cd /tmp wget http://ardownload.adobe.com/pub/adobe/reader/unix/9.x/9.5.5/enu/AdbeRdr9.5.5-1_i386linux_enu.deb sudo apt-get install gdebi sudo gdebi AdbeRdr9.5.5-1_i386linux_enu.deb # EncFS manager (from external repository) sudo add-apt-repository ppa:gencfsm/ppa sudo apt-get update sudo apt-get install gnome-encfs-manager
What have I missed?
These are the tools that I use in my day-to-day work as an academic geologist. I’m sure that there are plenty more for things like processing seismic data that I have missed. If you know any, please add them in the comments.
Just make sure that they don’t cost anything; don’t you know how copper wire was invented?